Re: [Tccc] IETF model? Re: Different community, similar problems? (Henning Schulzrinne)

Mukul
> Osama
>
>>1- How many people in the community have the time and interest to involve
>>as voluntary reviewers? This process needs X reviewers/paper where X>3.
>> In
>>fact, 3 expert reviewers/paper is already difficult in the current system
>
> Current system allows only the TPC members to select reviewers. That's why
> it is difficult to find 3 expert reviewers per paper. I think there would
> be many voluntary reviewers for "interesting" papers.
>
I believe the problem is not only selectivity but the limited pool of
available reviewers. In the proposed system, you still have official
reviewers and you need more voluntary ones for the idea to work. Who has
time to be loaded with additional reviews?

>>2- What percentage of "junk" papers's authors believe their work is not
>> good?
>
> That is fine. If I think my paper is ready for the world, I submit it. I
> will know how good it really is after the review process.
>
Many of the bad papers are resubmitted "as is" to other conferences after
rejection. Many authors just believe the reviewers are wrong/unfamiliar
with their work.
In addition, bad papers may not be interesting enough to attract voluntary
reviewers.
>>3- Does TPC mentor have enough time to moderate the debate?
>
> Yes, the proposed model may require more work on part of mentors of
> "interesting" papers.
>
>>4- Other issues will be escalated such as bias and plaigrism and fairness
> among papers.
>
> Why do you think so??
>
plaigrism: discussed before
fairness: "interesting" papers will recieve more reviews than
"non-interesting" (and not necessarily bad) papers
abuse: A malicious author may fake several identities to provide good
reviews or may ask couple of freinds to do so. The control of such abuse
will be harder.

> Thanks
> Mukul
>

Thanks.
Osama.
>> Forget about the term "IETF". What's wrong with the following review
>> process for a conference/journal:
>>
>> 1. The review process starts with the posting of the submitted paper on
>> a
>> public "wall" for a certain time window.
>> 2. Any one can post their review of the paper on the "wall" before a
>> certain deadline.
>> 3. The paper has a TPC mentor that has power to accept/reject the paper.
>> 4. TPC mentor assigns a certain number of official reviewers for the
>> paper. However, their reviews do not necessarily carry more weight than
>> those by voluntary reviewers.
>> 5. The authors and reviewers communicate with each other, possibly using
>> pseudonyms, on the "wall" during the time window for the review process.
>> However, the authors are not allowed to submit a new version of the
>> paper
>> during the review process.
>> 6. At the conclusion of the time window, the TPC mentor makes the
>> accept/reject decision about the paper based on posted reviews and
>> author/reviewer communication.
>>
>> Thanks
>> Mukul
>>
>> ----- Original Message -----
>> From: "L Wood" <L.Wood@surrey.ac.uk>
>> To: mukul@uwm.edu, hgs@cs.columbia.edu
>> Cc: tccc@lists.cs.columbia.edu
>> Sent: Saturday, August 14, 2010 3:58:41 PM
>> Subject: RE: [Tccc] IETF model? Re: Different community, similar
>> problems?
>> (Henning Schulzrinne)
>>
>> Before advocating the IETF process, I suggest looking at the IRTF and
>> seeing how the IETF process fails in research.
>>
>> -----Original Message-----
>> From: tccc-bounces@lists.cs.columbia.edu
>> [mailto:tccc-bounces@lists.cs.columbia.edu] On Behalf Of Mukul Goyal
>> Sent: 14 August 2010 20:36
>> To: Henning Schulzrinne
>> Cc: tccc
>> Subject: Re: [Tccc] IETF model? Re: Different community, similar
>> problems?
>> (Henning Schulzrinne)
>>
>> Henning
>>
>>>>
>>>> A related concern with IETF-style iterative reviewing is that the
>>>> reviewer may end up contributing more than some of the authors. It
>>>> is easy to imagine a conscientious reviewer going through many
>>>> iterations with a student whose (co-author) advisor is preoccupied.
>>
>>>And if people are complaining about conference
>>>submission-to-publication delays of 5 months today, they will be
>>> thrilled
>>> when they see the 5-year delays from -00 I-D to RFC...
>>
>> There is no suggestion that the review process would extend for 5 years.
>> It would still be same as before. Just that it would be open.
>>
>>>In general, without stating what you're trying to optimize and which
>>> problem you are trying to solve (quality? pick future faculty?
>>> timeliness? perception of fairness), the discussion of mechanisms seems
>>> a
>>> bit besides the point.
>>
>> In my mind, the problem is fairness of the review process. Another
>> problem, that such a model may possibly solve, is dearth of reviewers
>> and
>> submission of sub-quality papers.
>>
>>> To once again cite the IETF process: first, you need a requirements
>>> draft.
>>
>> By IETF model, I was basically referring to its open review process.
>>
>> Thanks
>> Mukul
>> _______________________________________________
>> Tccc mailing list
>> Tccc@lists.cs.columbia.edu
>> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
>> _______________________________________________
>> Tccc mailing list
>> Tccc@lists.cs.columbia.edu
>> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
>> _______________________________________________
>> Tccc mailing list
>> Tccc@lists.cs.columbia.edu
>> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
>>
>
>
>

-----------
Osama Bazan, PhD
Post Doctoral Fellow
Electrical and Computer Engineering
Ryerson University
Toronto, Ontario, Canada
Phone: +1 416 979 5000 Ext. 4528
Webpage: http://www.ee.ryerson.ca/~obazan

_______________________________________________
Tccc mailing list
Tccc@lists.cs.columbia.edu
https://lists.cs.columbia.edu/cucslists/listinfo/tccc