Given that reviewers are not compensated and that there is no assessment
of the quality of the reviews, I am not at all surprised by the lack of
agreement. Given the lack of agreement and assessment of quality, it is
also not surprising that "best paper" awards are largely meaningless. (01)
As for the question "what does it mean?", I suppose the obvious answer is
"you get what you pay for". (02)
-- Ken (03)
On Mon, 13 Apr 2015, John F Sowa wrote: (04)
> Referees with different backgrounds often disagree about papers
> submitted to journals and conferences. See below for some excerpts
> from a discussion about the reviewing process for NIPS (Neural
> Information Processing Systems).
>
> As the NIPS web site says, they bring together "machine learning and
> computational neuroscience": https://nips.cc/Conferences/current
>
> With that combination of reviewers and participants, disagreements
> are inevitable. But other conferences with interdisciplinary
> participants (AI, for example) have similar problems.
>
> The result of accepting papers with the highest average scores is
> that the so-called "best papers" are boring. They're inevitably
> papers that nobody objects to.
>
> John
> ______________________________________________________________________
>
> http://cacm.acm.org/blogs/blog-cacm/181996-the-nips-experiment/fulltext
>
> The 26% disagreement rate presented at the NIPS conference understates
> the meaning in my opinion, given the 22% acceptance rate. The immediate
> implication is that between half and two-thirds of papers accepted at
> NIPS would have been rejected if reviewed a second time. For analysis
> details and discussion about that, see here.
>
> Let’s give P (reject in 2nd review | accept 1st review) a name:
> arbitrariness. For NIPS 2014, arbitrariness was ~60%. Given such a stark
> number, the primary question is "what does it mean?" ...
>
> _________________________________________________________________
> Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/
> Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/
> Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
> Shared Files: http://ontolog.cim3.net/file/
> Community Wiki: http://ontolog.cim3.net/wiki/
> To join: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J (05)
_________________________________________________________________
Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/
Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/
Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
Shared Files: http://ontolog.cim3.net/file/
Community Wiki: http://ontolog.cim3.net/wiki/
To join: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J (01)
|