Not sure how this relates to the original thread, but it's a topic of
importance in getting computers to understand language: (01)
[JFS]
>>> Studies of inter-annotator agreement among well-trained humans show
>>> that 95% agreement is very rarely achieved. More typically, the best
>>> computer systems achieve about 75% accuracy.
>>
>> [PC]
>> In what - er - "context" is this true? Do you have a pointer to this?
>> This kind of number must be task-dependent.
>
[JFS]
> The percentage of 95% is widely cited as a "gold standard" and the value
of 75% comes from the SENSEVAL (sense evaluation) projects.
>
> There was a recent discussion of this point on Corpora List, which is
devoted to NLP work on large volumes of NL data.
>
> I made some comments that 95% would imply 15 errors per page and that my
high-school English teacher would not consider that good.
>
[PC] Really, really *bad* is the way I would rate it. My teachers would
usualy take 5 percentage points off for each error on a set of questions. (02)
> Adam Kilgarriff, who had originally published the value 95%, admitted that
the actual values for inter-annotator agreement are actually less than that.
Following is his response.
>
> The entire thread is archived. You can excerpt a phrase from the
following note, put it in quotes, and find the complete thread with your
favorite search engine.
>
> John (03)
As the archive indicates, the rate of agreement does depend on various
factors, especially the words tested. But what I consider far more
important is that the set of "senses" used in disambiguatiopon studies is
not a firmly based criterion for disambiguation. Mostly they use WordNet,
which is known to have problems. The most commonly cited problem is that it
is too 'fine-grained', making seeming distinctions among 'senses' that are
in fact indistinguishable. This resulted from the historical path by which
WordNet was developed, not originally intended for that purpose, but
dragooned into service because it was the only (free!!) resource that sort
of, kind of, fit the need. Hovy created a reduced set of senses by
aggregating sets of indistinguishable senses and of course got better
interannotator agreement. But that left other problems in WordNet
untouched. Wordnet is, in the context of NLU, an ancient device that has
outlived its original usefulness. Aside from questionable hierarchical
assignments, many synsets can be seen to be themselves aggregates, and many
important links among meanings are missing. To use WordNet to create a
"gold standard" is like using the king's foot as a standard of length. But
NLU researchers are busy with their own favorite tasks and there seems to be
little desire to recreate the standard of "meaning", even when the existing
one is known to be flawed. This could make an interesting case study in
the psychology of research funding.
The only ones who really know the "meaning" of a word are the ones who
created the text. It would not be too difficult to have creators of text
label the senses that they intend, and through a series of iterations, find
a set of senses that text creators and text annotators can agree on with a
precision that would satisfy Mrs. Elliott. I find little interest at NLP
meetings for work of that kind.
And computers should be able to do even better. Some human error merely
results from failure of memory, some from inattention, and some from
carelessness. Computers are better in those respects, and I would expect
computers to be able, eventully, to do better than any given pair of
annotators at finding the right meaning, provided that the "meanings" are in
fact distinguishable. (04)
Pat (05)
Patrick Cassidy
MICRA Inc.
cassidy@xxxxxxxxx
1-908-561-3416 (06)
-------- Original Message --------
Subject: Re: [Corpora-List] WSD / # WordNet senses / Mechanical Turk
Date: Tue, 16 Jul 2013 07:43:58 +0100
From: Adam Kilgarriff <adam@xxxxxxxxxxxxxxxxxx>
To: John F Sowa <sowa@xxxxxxxxxxx>
CC: corpora@xxxxxx (07)
Dear Mark, John, (08)
Let me confess to a moment of embarrassment that I've been anxious about for
years: following SENSEVAL-1 I did a (tiny) experiment to establish
inter-annotator agreement, and came up with the 95% figure cited by John. (09)
On experience since, I think the findings were not sound, and it is most
unusual to get a figure that high, and I regret having published it (and,
worse, having put it in the title of a short paper from EACL-99) (010)
For either automatic WSD, or even for the gold standard, I agree entirely
with John: (011)
Miss Elliott, my high-school English teacher, wouldn't give
anyone a gold star [for work like that] (012)
Adam (013)
========================================
Adam Kilgarriff <http://www.kilgarriff.co.uk/> adam@xxxxxxxxxxxxxxxxxx
<mailto:adam@xxxxxxxxxxxxxxxxxx> Director Lexical Computing Ltd
<http://www.sketchengine.co.uk/> Visiting Research Fellow University of
Leeds <http://leeds.ac.uk> /Corpora for all/ with the Sketch Engine
<http://www.sketchengine.co.uk>
/DANTE: a lexical database for English <http://www.webdante.com>/
======================================== (014)
_________________________________________________________________
Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/
Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/
Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
Shared Files: http://ontolog.cim3.net/file/ Community Wiki:
http://ontolog.cim3.net/wiki/ To join:
http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J (015)
_________________________________________________________________
Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/
Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/
Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
Shared Files: http://ontolog.cim3.net/file/
Community Wiki: http://ontolog.cim3.net/wiki/
To join: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J (016)
|