hoonzang's CFPs: Re: [Tccc] IETF model? Re: Different community, similar problems? (Henning Schulzrinne)

Hi Marco,

>May be somehow out of topic, but I don't see the equation more
>review(er)s => more fair selection process.

>My understanding is that more review(er)s => more noise, more variance,
>larger grey area, more random selection.

But the authors would be able to rebut bad reviews. The ability of refute bad reviews is a major advantage of the proposed system. The TPC mentor can ultimately judge the worth of a review.

>Add to this that the review(er)s are now "auto-assigned" (by reviewers
>that voluntarily review a paper). The noise can only increase:
>non-experts attracted by a paper will only provide modest and light
>reviews at best.

These non-expert reviews will only get as much weight as they deserve. On the other hand, any expert, high quality voluntary reviews can significantly improve the overall review process. I think it is overly pessimistic to assume that voluntary reviews would be bad quality. They could be bad quality. They could be very good quality as well.

>Yes this happens also now, but all the claiming and paper assignment
>phases should guarantee a good match between expertize and fairness etc.

In my view, there are many people (including myself) who have suffered from bad, unfair reviews over the years and feel helpless against the system. The existing system does not meet the fairness expectations of many people.

>At last, I fully agree that the extra time that can be devoted to the
>voluntary reviews is very limited...

This may be true for many people. But, still there could be many people who would cherish the opportunity to review papers interesting to them.

>Just check _when_ reviews are submited now, _how_ many reviews are still
>missing, etc.etc.

A major reason for this, in my view, is the inability of the current system to tap the reviewer pool properly.

>This system may possibly work for small numbers (<50). It will never scale for x100 submissions.

That is really an assumption that needs to be tested.

Thanks
Mukul

> Osama,
>
>
>> I believe the problem is not only selectivity but the limited pool of
>> available reviewers. In the proposed system, you still have official
>> reviewers and you need more voluntary ones for the idea to work. Who has
>> time to be loaded with additional reviews?
>>
> I think many conferences, e.g. Infocom, will be able to attract a large pool of voluntary reviewers. It may not be so for less known conferences. BTW, the proposed system does not _depend_ on voluntary reviewers. Official reviewers are there to make sure that paper does receive a certain minimum number of reviews. The fact that any one can review and that the authors have ample opportunity to rebut reviews allows for a more fair and more thorough review process than the current one.
>
>
>> Many of the bad papers are resubmitted "as is" to other conferences after
>> rejection. Many authors just believe the reviewers are wrong/unfamiliar
>> with their work.
>> In addition, bad papers may not be interesting enough to attract voluntary
>> reviewers.
>>
> As I said before, the inability of a paper to attract voluntary reviews is not a problem.
>
>
>
>> plaigrism: discussed before
>>
> If you think your idea has been stolen, all you have to do is file a complaint against the offender with the timestamp of your submitted paper.
>
>
>> fairness: "interesting" papers will recieve more reviews than
>>
> "non-interesting" (and not necessarily bad) papers
>
> There is no unfairness here. Non-interesting papers will still receive official reviews. Since there is ample opportunity for the authors to rebut the reviews, hopefully there wont be any more complaints about unfair reviews. Since the process is open, impartial observers would come to know if the review process has been unfair.
>
>
>> abuse: A malicious author may fake several identities to provide good
>> reviews or may ask couple of freinds to do so. The control of such abuse
>> will be harder.
>>
> Ethics, ethics, ethics. The TPC mentors will know the true identities of the reviewers (even if they are using pseudonyms for the review). They can authenticate the identities if there is a concern. The authors can be asked for past collaborators and these people may be prevented from reviewing (as is done is proposal reviews). Severe penalities can be associated with a malicious attempt to influence the process.
>
> Thanks,
> Mukul
>
>
>
>>> Forget about the term "IETF". What's wrong with the following review
>>> process for a conference/journal:
>>>
>>> 1. The review process starts with the posting of the submitted paper on
>>> a
>>> public "wall" for a certain time window.
>>> 2. Any one can post their review of the paper on the "wall" before a
>>> certain deadline.
>>> 3. The paper has a TPC mentor that has power to accept/reject the paper.
>>> 4. TPC mentor assigns a certain number of official reviewers for the
>>> paper. However, their reviews do not necessarily carry more weight than
>>> those by voluntary reviewers.
>>> 5. The authors and reviewers communicate with each other, possibly using
>>> pseudonyms, on the "wall" during the time window for the review process.
>>> However, the authors are not allowed to submit a new version of the
>>> paper
>>> during the review process.
>>> 6. At the conclusion of the time window, the TPC mentor makes the
>>> accept/reject decision about the paper based on posted reviews and
>>> author/reviewer communication.
>>>
>>> Thanks
>>> Mukul
>>>
>>> ----- Original Message -----
>>> From: "L Wood"<L.Wood@surrey.ac.uk>
>>> To: mukul@uwm.edu, hgs@cs.columbia.edu
>>> Cc: tccc@lists.cs.columbia.edu
>>> Sent: Saturday, August 14, 2010 3:58:41 PM
>>> Subject: RE: [Tccc] IETF model? Re: Different community, similar
>>> problems?
>>> (Henning Schulzrinne)
>>>
>>> Before advocating the IETF process, I suggest looking at the IRTF and
>>> seeing how the IETF process fails in research.
>>>
>>> -----Original Message-----
>>> From: tccc-bounces@lists.cs.columbia.edu
>>> [mailto:tccc-bounces@lists.cs.columbia.edu] On Behalf Of Mukul Goyal
>>> Sent: 14 August 2010 20:36
>>> To: Henning Schulzrinne
>>> Cc: tccc
>>> Subject: Re: [Tccc] IETF model? Re: Different community, similar
>>> problems?
>>> (Henning Schulzrinne)
>>>
>>> Henning
>>>
>>>
>>>>> A related concern with IETF-style iterative reviewing is that the
>>>>> reviewer may end up contributing more than some of the authors. It
>>>>> is easy to imagine a conscientious reviewer going through many
>>>>> iterations with a student whose (co-author) advisor is preoccupied.
>>>>>
>>>
>>>> And if people are complaining about conference
>>>> submission-to-publication delays of 5 months today, they will be
>>>> thrilled
>>>> when they see the 5-year delays from -00 I-D to RFC...
>>>>
>>> There is no suggestion that the review process would extend for 5 years.
>>> It would still be same as before. Just that it would be open.
>>>
>>>
>>>> In general, without stating what you're trying to optimize and which
>>>> problem you are trying to solve (quality? pick future faculty?
>>>> timeliness? perception of fairness), the discussion of mechanisms seems
>>>> a
>>>> bit besides the point.
>>>>
>>> In my mind, the problem is fairness of the review process. Another
>>> problem, that such a model may possibly solve, is dearth of reviewers
>>> and
>>> submission of sub-quality papers.
>>>
>>>
>>>> To once again cite the IETF process: first, you need a requirements
>>>> draft.
>>>>
>>> By IETF model, I was basically referring to its open review process.
>>>
>>> Thanks
>>> Mukul
>>> _______________________________________________
>>> Tccc mailing list
>>> Tccc@lists.cs.columbia.edu
>>> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
>>> _______________________________________________
>>> Tccc mailing list
>>> Tccc@lists.cs.columbia.edu
>>> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
>>> _______________________________________________
>>> Tccc mailing list
>>> Tccc@lists.cs.columbia.edu
>>> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
>>>
>>>
>>
>>
>>
>
> -----------
> Osama Bazan, PhD
> Post Doctoral Fellow
> Electrical and Computer Engineering
> Ryerson University
> Toronto, Ontario, Canada
> Phone: +1 416 979 5000 Ext. 4528
> Webpage: http://www.ee.ryerson.ca/~obazan
>
> _______________________________________________
> Tccc mailing list
> Tccc@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/cucslists/listinfo/tccc
>
>

--
Ciao, /\/\/\rco

_______________________________________________
Tccc mailing list
Tccc@lists.cs.columbia.edu
https://lists.cs.columbia.edu/cucslists/listinfo/tccc

hoonzang's CFPs

2010-08-16

Re: [Tccc] IETF model? Re: Different community, similar problems? (Henning Schulzrinne)

No comments:

About Me

Search This Blog

Blog Archive

Followers