2010-08-15

Re: [Tccc] IETF model? Re: Different community, similar problems? (Henning Schulzrinne)

On Aug 15, 2010, at 10:00 AM, Mukul Goyal wrote:

> Osama
>
>> 1- How many people in the community have the time and interest to involve
>> as voluntary reviewers? This process needs X reviewers/paper where X>3. In
>> fact, 3 expert reviewers/paper is already difficult in the current system
>
> Current system allows only the TPC members to select reviewers.

You may want to check out a broader set of conferences. The far more typical model is that papers are assigned (typically, automatically or semi-automatically) to the TPC. The TPC-picks-reviewer model seems rare these days, I suspect, partially to ensure some quality control. (Some TPC members delegate reviews to, say, their students, but that's a separate issue.)

Henning

_______________________________________________
Tccc mailing list
Tccc@lists.cs.columbia.edu
https://lists.cs.columbia.edu/cucslists/listinfo/tccc

Re: [Tccc] IETF model? Re: Different community, similar problems? (Henning Schulzrinne)

Osama

>1- How many people in the community have the time and interest to involve
>as voluntary reviewers? This process needs X reviewers/paper where X>3. In
>fact, 3 expert reviewers/paper is already difficult in the current system

Current system allows only the TPC members to select reviewers. That's why it is difficult to find 3 expert reviewers per paper. I think there would be many voluntary reviewers for "interesting" papers.

>2- What percentage of "junk" papers's authors believe their work is not good?

That is fine. If I think my paper is ready for the world, I submit it. I will know how good it really is after the review process.

>3- Does TPC mentor have enough time to moderate the debate?

Yes, the proposed model may require more work on part of mentors of "interesting" papers.

>4- Other issues will be escalated such as bias and plaigrism and fairness
among papers.

Why do you think so??

Thanks
Mukul

> Forget about the term "IETF". What's wrong with the following review
> process for a conference/journal:
>
> 1. The review process starts with the posting of the submitted paper on a
> public "wall" for a certain time window.
> 2. Any one can post their review of the paper on the "wall" before a
> certain deadline.
> 3. The paper has a TPC mentor that has power to accept/reject the paper.
> 4. TPC mentor assigns a certain number of official reviewers for the
> paper. However, their reviews do not necessarily carry more weight than
> those by voluntary reviewers.
> 5. The authors and reviewers communicate with each other, possibly using
> pseudonyms, on the "wall" during the time window for the review process.
> However, the authors are not allowed to submit a new version of the paper
> during the review process.
> 6. At the conclusion of the time window, the TPC mentor makes the
> accept/reject decision about the paper based on posted reviews and
> author/reviewer communication.
>
> Thanks
> Mukul
>
> ----- Original Message -----
> From: "L Wood" <L.Wood@surrey.ac.uk>
> To: mukul@uwm.edu, hgs@cs.columbia.edu
> Cc: tccc@lists.cs.columbia.edu
> Sent: Saturday, August 14, 2010 3:58:41 PM
> Subject: RE: [Tccc] IETF model? Re: Different community, similar problems?
> (Henning Schulzrinne)
>
> Before advocating the IETF process, I suggest looking at the IRTF and
> seeing how the IETF process fails in research.
>
> -----Original Message-----
> From: tccc-bounces@lists.cs.columbia.edu
> [mailto:tccc-bounces@lists.cs.columbia.edu] On Behalf Of Mukul Goyal
> Sent: 14 August 2010 20:36
> To: Henning Schulzrinne
> Cc: tccc
> Subject: Re: [Tccc] IETF model? Re: Different community, similar problems?
> (Henning Schulzrinne)
>
> Henning
>
>>>
>>> A related concern with IETF-style iterative reviewing is that the
>>> reviewer may end up contributing more than some of the authors. It
>>> is easy to imagine a conscientious reviewer going through many
>>> iterations with a student whose (co-author) advisor is preoccupied.
>
>>And if people are complaining about conference
>>submission-to-publication delays of 5 months today, they will be thrilled
>> when they see the 5-year delays from -00 I-D to RFC...
>
> There is no suggestion that the review process would extend for 5 years.
> It would still be same as before. Just that it would be open.
>
>>In general, without stating what you're trying to optimize and which
>> problem you are trying to solve (quality? pick future faculty?
>> timeliness? perception of fairness), the discussion of mechanisms seems a
>> bit besides the point.
>
> In my mind, the problem is fairness of the review process. Another
> problem, that such a model may possibly solve, is dearth of reviewers and
> submission of sub-quality papers.
>
>> To once again cite the IETF process: first, you need a requirements
>> draft.
>
> By IETF model, I was basically referring to its open review process.
>
> Thanks
> Mukul
> _______________________________________________
> Tccc mailing list
> Tccc@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
> _______________________________________________
> Tccc mailing list
> Tccc@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
> _______________________________________________
> Tccc mailing list
> Tccc@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
>

_______________________________________________
Tccc mailing list
Tccc@lists.cs.columbia.edu
https://lists.cs.columbia.edu/cucslists/listinfo/tccc

Re: [Tccc] IETF model? Re: Different community, similar problems? (Henning Schulzrinne)

You are right. I did not mean that authors give up IP rights. All I meant was make the paper publically available on submission as it is done after its acceptance.

Thanks
Mukul

----- Original Message -----
From: "L Wood" <L.Wood@surrey.ac.uk>
To: mukul@uwm.edu, raysaikat@gmail.com
Cc: tccc@lists.cs.columbia.edu, hgs@cs.columbia.edu
Sent: Sunday, August 15, 2010 5:19:21 AM
Subject: RE: [Tccc] IETF model? Re: Different community, similar problems? (Henning Schulzrinne)

Unfortunately, 'putting in the public domain' does not mean what you appear to think it means.
http://en.wikipedia.org/wiki/Public_domain

-----Original Message-----
From: Mukul Goyal [mailto:mukul@uwm.edu]
Sent: 15 August 2010 03:46
To: Saikat Ray
Cc: Wood L Dr (Electronic Eng); tccc@lists.cs.columbia.edu; hgs@cs.columbia.edu
Subject: Re: [Tccc] IETF model? Re: Different community, similar problems? (Henning Schulzrinne)
Importance: High

>You are not protecting the authors' idea from getting plagiarized.

Such protection probably does not exist in the current system either (?). This is an ethics issue. I am not aware how IEEE deals with complaints about plagiarism. IEEE must evolve a system to deal with such complaints if there is none today.

> For the papers that are rejected, it is a MUST that no one can copy
>the idea - otherwise no one will ever send a paper to that venue. This
>is the reason an unpublished paper is reviewed by a small number of
>people who agree not to use the idea of a paper that s/he reviews.

I think the proposed model will force a change in mindset. Under the proposed model, submission to a conference/journal will place a paper in the public domain, which is not so under the current model. Plain stealing of ideas would not be possible since the act of submission gives a paper a timestamp that can be used in plagiarism complaints. Other papers can refer to a submitted/rejected paper as a "work in progress", much like how an IETF internet draft is referenced.

A desirable side effect of putting all submitted papers in public domain would be that people will hesitate before submitting a half-baked paper.

Thanks
Mukul

> 1. The review process starts with the posting of the submitted paper on a public "wall" for a certain time window.
> 2. Any one can post their review of the paper on the "wall" before a certain deadline.
> 3. The paper has a TPC mentor that has power to accept/reject the paper.
> 4. TPC mentor assigns a certain number of official reviewers for the paper. However, their reviews do not necessarily carry more weight than those by voluntary reviewers.
> 5. The authors and reviewers communicate with each other, possibly using pseudonyms, on the "wall" during the time window for the review process. However, the authors are not allowed to submit a new version of the paper during the review process.
> 6. At the conclusion of the time window, the TPC mentor makes the accept/reject decision about the paper based on posted reviews and author/reviewer communication.
>
> Thanks
> Mukul
>
> ----- Original Message -----
> From: "L Wood" <L.Wood@surrey.ac.uk>
> To: mukul@uwm.edu, hgs@cs.columbia.edu
> Cc: tccc@lists.cs.columbia.edu
> Sent: Saturday, August 14, 2010 3:58:41 PM
> Subject: RE: [Tccc] IETF model? Re: Different community, similar
> problems? (Henning Schulzrinne)
>
> Before advocating the IETF process, I suggest looking at the IRTF and seeing how the IETF process fails in research.
>
> -----Original Message-----
> From: tccc-bounces@lists.cs.columbia.edu
> [mailto:tccc-bounces@lists.cs.columbia.edu] On Behalf Of Mukul Goyal
> Sent: 14 August 2010 20:36
> To: Henning Schulzrinne
> Cc: tccc
> Subject: Re: [Tccc] IETF model? Re: Different community, similar
> problems? (Henning Schulzrinne)
>
> Henning
>
>>>
>>> A related concern with IETF-style iterative reviewing is that the
>>> reviewer may end up contributing more than some of the authors.  It
>>> is easy to imagine a conscientious reviewer going through many
>>> iterations with a student whose (co-author) advisor is preoccupied.
>
>>And if people are complaining about conference
>>submission-to-publication delays of 5 months today, they will be thrilled when they see the 5-year delays from -00 I-D to RFC...
>
> There is no suggestion that the review process would extend for 5 years. It would still be same as before. Just that it would be open.
>
>>In general, without stating what you're trying to optimize and which problem you are trying to solve (quality? pick future faculty? timeliness? perception of fairness), the discussion of mechanisms seems a bit besides the point.
>
> In my mind, the problem is fairness of the review process. Another problem, that such a model may possibly solve, is dearth of reviewers and submission of sub-quality papers.
>
>> To once again cite the IETF process: first, you need a requirements draft.
>
> By IETF model, I was basically referring to its open review process.
>
> Thanks
> Mukul
> _______________________________________________
> Tccc mailing list
> Tccc@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
> _______________________________________________
> Tccc mailing list
> Tccc@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
> _______________________________________________
> Tccc mailing list
> Tccc@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
>

--
Saikat Ray
Web: http://raysaikat.googlepages.com

_______________________________________________
Tccc mailing list
Tccc@lists.cs.columbia.edu
https://lists.cs.columbia.edu/cucslists/listinfo/tccc