> I think - indeed i am sure - that these two criteria (text and
> 5-year-old conversation) are almost completely different in the
> linguistic abilities they would be testing. In fact, no text/answer
> protocol is ever going to come to grips with the ways that language
> is used in natural conversation.
>
Yes, I suspect that they do represent different facets of language
understanding, and so would the other task: (01)
> How about the use of language in teaching or training/instruction? (02)
But if you don't have a rather large group to work on all facets of the
problem at once, one has to start somewhere, and question answering seems to
be one task for which one can create a meaningful measure of performance.
Before that, if one assumes as I do that some form of ontology will be
needed to do the task well (in the next decade at least), one needs to build
the ontology that one expects can support the reasoning needed. That is
what I am currently exploring. (03)
> But it seems then that you have what one might call an outline, or a
> sketch, of a theory of how NL comprehension might be achieved, that
> orients your thinking. Can you share it with us?
>
Yes, but it is even less than a sketch because I am still trying to
construct an outline of individual components that need to work together.
There is nothing particularly original in the methods I would like to try.
It is basically to revisit the old notion of "word experts" but now with
much more powerful computers than were used back in the early trials, and to
integrate an ontology that has structures that come as close as possible to
at least the most common English-language structures. Think of it as
"extreme lexicalization" of the grammar. Whether that will port to other
languages I have no idea. For the language-understanding component, I am
not concerned with exploring the nature of human linguistic communication
per se, but in building a machine that will perform the task as well, by
whatever mechanism. One facet of that approach is to try to answer the
question, which we have discussed already, of whether there is a relatively
small core (thousands probably) of concept representations that can serve as
an inventory of components for representing almost all of the more
specialized concepts. What I consider the linguistic evidence (and you
don't) is nothing more than a motivation to perform the experiment with the
ontology. (04)
The virtue, in my opinion, of the "Word Expert" approach is that it can be
very modular - one verb, one program - and therefore could be built by a
very large collaborative effort, provided that there is a common ontology in
which the meanings are represented. One big difficulty is in the great
complexity. There was one comment decades ago about the Word Expert
approach, noting the proposed complexity of even a single Word Expert, and
expressing the hope that before the field had to "don that hair shirt"
something simpler could be found. Then they got enthusiastic about
statistical approaches to NLP. I think that donning that hair shirt is in
fact what is required to tackle the problem of deeper language
understanding. Adopting a common foundation ontology for an open
collaboration of multiple investigators provides a mechanism to make it more
practical for many otherwise isolated groups to build a common system as a
collection of separately developed components. Such a system can evolve by
replacement of components with improved functionality for specific words or
phrases or grammatical structures. To enable such "evolution" one does
need some quantitative measure of functionality, which is why I focus on the
more easily measured question-answering criterion. In this architecture,
the common foundation ontology serves a function in providing a
communication protocol between different language interpretation modules,
among which statistical modules may very well serve to provide a significant
part of the interpretation. Other modules such as image processing, speech
recognition, graphic-oriented reasoning, and robotic functions would be able
to communicate using the same foundation ontology. So I think that the
foundation ontology is a critical component of such a system, and deserves
serious investigation as a topic in its own right. (05)
There are plenty of problems for which I have no meaningful proposal, among
which how to handle context is a major one. The microtheories of Cyc are
one approach, and I expect that something like that will be helpful. I
think that one may also need "context experts" to explore the more complex
aspects of how context affects the meanings of a text. The issue of
"grounding" of meaning is another problem, and may require multiple
"sensory" inputs (sound, vision, internet connections, GPS, temperature
sensors, robotic effectors with feedback, etc.) together with learning
mechanisms to provide a situation awareness environment in which the
meanings of text are not arbitrary. (06)
Now, none of that should be interpreted as a theory for which I should be
presenting evidence, and I have none. It is an engineering hypothesis of
how a system that may approach human-level language understanding may be
structured, and is at this point only the *motivating hypothesis* (one of
several) that makes me believe that answering the question of whether there
is a "Conceptual Defining vocabulary" is a problem well worth working on.
And that is all that I can do in the time available. The task for which I
am currently being paid is to develop an ontology to integrate multiple
relational databases, and that task involves no language processing at all. (07)
Pat (08)
Patrick Cassidy
MICRA, Inc.
908-561-3416
cell: 908-565-4053
cassidy@xxxxxxxxx (09)
> -----Original Message-----
> From: ontolog-forum-bounces@xxxxxxxxxxxxxxxx [mailto:ontolog-forum-
> bounces@xxxxxxxxxxxxxxxx] On Behalf Of Pat Hayes
> Sent: Saturday, March 29, 2008 10:07 PM
> To: [ontolog-forum]
> Cc: '[ontolog-forum] '
> Subject: Re: [ontolog-forum] What is "understanding" - was: Building on
> common ground
>
> At 4:10 PM -0400 3/29/08, Patrick Cassidy wrote:
> >I agree with *almost* everything John S, said here, but need to
> clarify one
> >point - what is language "understanding"?
> >
> >[JS] > 1. None of the current systems do anything that could be
> called
> >"understanding".
> >
> >If JS means that none of the current systems can understand normal
> language
> >at, say, 80% or better of the human level, I do agree (and I think
> that most
> >are much worse).
>
> I don't think it is meaningful to use percentages in a claim like
> this. The very notion of 'level' doesn't really make sense, in fact.
> I think what John said is much closer to the truth. We (researchers)
> really do not know what it takes to comprehend language in the way
> that we (humans) do.
>
> >
> >And I do want to use an ontology that can support human-level
> understanding.
> >So that requires not only building the ontology, but building the
> language
> >processing system that will demonstrate that it is useful for that
> purpose.
> >I haven't figured out how to do a few person-centuries of work in my
> spare
> >time, but language understanding is the ultimate goal that *orients*
> the
> >work.
>
> But it seems then that you have what one might call an outline, or a
> sketch, of a theory of how NL comprehension might be achieved, that
> orients your thinking. Can you share it with us?
>
> > Focusing first on the linguistic and ontological "Defining
> >Vocabulary" creates a bound that makes the effort somewhat more
> practical
> >than general language understanding, or understanding of, e.g.,
> newspaper
> >text.
> >
> >But the way I use the word, "understanding" is a quantitative measure
> of the
> >ability of a language processing system to answer questions about the
> >content of texts. It is therefore not all-or-nothing, but has levels.
> >Adult people may vary in this ability too, for any given text. One
> can get
> >into a discussion of how best to measure this attribute of
> >language-processing systems; the measures used in the " Automatic
> Content
> >Extraction" and "Message Understanding Conference" competitions (some
> data
> >available at http://www-nlpir.nist.gov/related_projects/muc/) are one
> >example of an attempt to define and quantify "understanding". Another
> test
> >might be to support a normal conversation with a five-year old, but
> that
> >would be harder to quantify.
>
> I think - indeed i am sure - that these two criteria (text and
> 5-year-old conversation) are almost completely different in the
> linguistic abilities they would be testing. In fact, no text/answer
> protocol is ever going to come to grips with the ways that language
> is used in natural conversation.
>
> >I expect that, given a particular text, members of this discussion
> group
> >would vary in the list of questions and answers they would create that
> would
> >satisfy them individually that an automatic system actually
> "understood"
> >the text to a degree that indicates human-level understanding. It
> might be
> >an interesting exercise for those who want to discuss "understanding"
> to
> >first create a text and then have the discussants provide such a list
> of
> >questions, and the answers that they think should be produced, so that
> the
> >discussion can be concrete.
>
> Concrete but artificial and subjective. It would be far more
> scientific to look at transcriptions of recordings of natural
> conversations between people in various settings (including perhaps
> question/answer interviews after reading a text) and see what it
> would take to recreate something similar with a machine playing one
> of the roles. Turing's test, in effect, though not necessarily 'live'.
>
> >There may be better ways to evaluate whether a computer has
> "understood" a
> >text. I would be interested in alternative suggestions. For Robotic
> >systems, it may be a bit easier, since commands can be very specific -
> but
> >there may be many who would not classify interpretation of text in
> such
> >restricted topics as "understanding".
>
> How about the use of language in teaching or training/instruction?
> Then the test of comprehension is the newly learned ability or skill
> by the learner, which can be measured by established techniques.
> Recent work by James Allen and his team have had some success along
> these lines.
>
> Pat
>
> --
> ---------------------------------------------------------------------
> IHMC (850)434 8903 or (650)494 3973 home
> 40 South Alcaniz St. (850)202 4416 office
> Pensacola (850)202 4440 fax
> FL 32502 (850)291 0667 cell
> http://www.ihmc.us/users/phayes phayesAT-SIGNihmc.us
> http://www.flickr.com/pathayes/collections
>
>
>
> _________________________________________________________________
> Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/
> Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontolog-
> forum/
> Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
> Shared Files: http://ontolog.cim3.net/file/
> Community Wiki: http://ontolog.cim3.net/wiki/
> To Post: mailto:ontolog-forum@xxxxxxxxxxxxxxxx
> (010)
_________________________________________________________________
Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/
Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
Shared Files: http://ontolog.cim3.net/file/
Community Wiki: http://ontolog.cim3.net/wiki/
To Post: mailto:ontolog-forum@xxxxxxxxxxxxxxxx (011)
|