At 4:10 PM -0400 3/29/08, Patrick Cassidy wrote:
>I agree with *almost* everything John S, said here, but need to clarify one
>point - what is language "understanding"?
>
>[JS] > 1. None of the current systems do anything that could be called
>"understanding".
>
>If JS means that none of the current systems can understand normal language
>at, say, 80% or better of the human level, I do agree (and I think that most
>are much worse). (01)
I don't think it is meaningful to use percentages in a claim like
this. The very notion of 'level' doesn't really make sense, in fact.
I think what John said is much closer to the truth. We (researchers)
really do not know what it takes to comprehend language in the way
that we (humans) do. (02)
>
>And I do want to use an ontology that can support human-level understanding.
>So that requires not only building the ontology, but building the language
>processing system that will demonstrate that it is useful for that purpose.
>I haven't figured out how to do a few person-centuries of work in my spare
>time, but language understanding is the ultimate goal that *orients* the
>work. (03)
But it seems then that you have what one might call an outline, or a
sketch, of a theory of how NL comprehension might be achieved, that
orients your thinking. Can you share it with us? (04)
> Focusing first on the linguistic and ontological "Defining
>Vocabulary" creates a bound that makes the effort somewhat more practical
>than general language understanding, or understanding of, e.g., newspaper
>text.
>
>But the way I use the word, "understanding" is a quantitative measure of the
>ability of a language processing system to answer questions about the
>content of texts. It is therefore not all-or-nothing, but has levels.
>Adult people may vary in this ability too, for any given text. One can get
>into a discussion of how best to measure this attribute of
>language-processing systems; the measures used in the " Automatic Content
>Extraction" and "Message Understanding Conference" competitions (some data
>available at http://www-nlpir.nist.gov/related_projects/muc/) are one
>example of an attempt to define and quantify "understanding". Another test
>might be to support a normal conversation with a five-year old, but that
>would be harder to quantify. (05)
I think - indeed i am sure - that these two criteria (text and
5-year-old conversation) are almost completely different in the
linguistic abilities they would be testing. In fact, no text/answer
protocol is ever going to come to grips with the ways that language
is used in natural conversation. (06)
>I expect that, given a particular text, members of this discussion group
>would vary in the list of questions and answers they would create that would
>satisfy them individually that an automatic system actually "understood"
>the text to a degree that indicates human-level understanding. It might be
>an interesting exercise for those who want to discuss "understanding" to
>first create a text and then have the discussants provide such a list of
>questions, and the answers that they think should be produced, so that the
>discussion can be concrete. (07)
Concrete but artificial and subjective. It would be far more
scientific to look at transcriptions of recordings of natural
conversations between people in various settings (including perhaps
question/answer interviews after reading a text) and see what it
would take to recreate something similar with a machine playing one
of the roles. Turing's test, in effect, though not necessarily 'live'. (08)
>There may be better ways to evaluate whether a computer has "understood" a
>text. I would be interested in alternative suggestions. For Robotic
>systems, it may be a bit easier, since commands can be very specific - but
>there may be many who would not classify interpretation of text in such
>restricted topics as "understanding". (09)
How about the use of language in teaching or training/instruction?
Then the test of comprehension is the newly learned ability or skill
by the learner, which can be measured by established techniques.
Recent work by James Allen and his team have had some success along
these lines. (010)
Pat (011)
--
---------------------------------------------------------------------
IHMC (850)434 8903 or (650)494 3973 home
40 South Alcaniz St. (850)202 4416 office
Pensacola (850)202 4440 fax
FL 32502 (850)291 0667 cell
http://www.ihmc.us/users/phayes phayesAT-SIGNihmc.us
http://www.flickr.com/pathayes/collections (012)
_________________________________________________________________
Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/
Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
Shared Files: http://ontolog.cim3.net/file/
Community Wiki: http://ontolog.cim3.net/wiki/
To Post: mailto:ontolog-forum@xxxxxxxxxxxxxxxx (013)
|