doug foxvog wrote:
> Watson's win on Wednesday was made very likely because enough of the
> problems could be solved by simple look-up. These could have been answered
> by definition look-ups, quotation completion, or searches based on key
> words in the question. Any question that went beyond these techniques
> was not answered, or not answered correctly by Watson.
> (01)
Excellent analysis. Little attention, it seems, has been paid in the
media to which types of questions and categories are likely to be occur
and how -- and how well --Watson might answer them as compared with how
people would answer them. (02)
Once you understand the nature of the game, there's a temptation to call
the Jeopardy!/Watson match a cheap parlor trick. But it wasn't so cheap,
was it. Still, brilliant work by the Watson team. Clever, too. (That's
*not* a criticism.) They really understood the nature of the game.
> The vaunted ability to handle humor and mental twists did not show up in
> the problems presented.
>
> For some reason, Watson lost on the buzzer on every question in the
> category "Actors who Direct", even though it got the correct response for
> each of them with 84-96% confidence. The clues were the movie names, so
> looking up the director of each movie and verifying that the director was
> also an actor in the movie, would be straight forward.
>
> Watson could not get any problem in the category, "ALSO ON YOUR COMPUTER
> KEYS". It either ignored the category, could not figure out what the
> category meant, or had no text describing the keys on computer keyboards.
> On the last puzzle in this category, it considered "delete key", so it
> was evidently learning the category.
>
> Some of the questions seemed designed to be easily answered from a database.
> For example, the question about the "Coyote State" specified the state's
> area as 75,885 square miles. No person would remember the exact area of
> a state. Why not round it off to the nearest 100 sq. mi.?
>
Exactly. But I would not accuse Jeopardy! of tailoring questions to
Watson's strengths. (03)
1. The Jeopardy clue team consciously limits the difficulty of each clue
in several ways. (04)
* Some clues are harder than others, but most rely on "general
knowledge." Using its human experience, the clue team avoids clues that
would be too difficult for the average smart person. Such constraints
actually limit Watson's advantage. Giving the value of pi to 10 places
or listing all vice presidents of the US would be child's play for
Watson. When it comes to raw memory, Watson is going to win.
* The clues rarely require analysis of complex conditions. After
all, the object of the game is for humans to come up with the right
question in a few seconds. The absence of more complex and subtle clues
is generally an advantage for Watson.
* The clues and questions fall within the cultural experience of
Americans with a typical college education. Listing great Bollywood
films would be easy for Watson but tough for most Americans. (That may
change over time.) (05)
2. The response to most clues is a question that identifies a small
set of concepts or entities -- usually only one.
* By "entity" I mean specific people, places, or things.
[Who/What is] Henry VIII, Lake Nicaragua, and The Declaration of
Independence are among the specific "questions" I have heard.
* By "concept" I mean a class of things, whether concrete or
abstract -- like dogs, weaponry, human health, or happiness. I believe
that if we took a statistical survey of Jeopardy! questions (the
responses), we would find that the clue frequently consists of lists of
things belonging to a class (definition by extension -- a subset of the
things in that class) rather than definition by intension (a set of
properties that define a class). I suspect that this also favors Watson
in a substantial way. (06)
So Ken Jennings and Brad Rutter took a thumping on national television
because categories/types of questions that might have favored humans at
this time were eliminated, and because there are other significant
constraints imposed by the "rules" of the game itself. The thumping
could have been worse. And IBM knew that. (07)
There's more to be learned (by the general public, like me) about what
actually happened by more careful analysis of the Jeopardy!Watson
challenge. But we need to let it go as a metaphor for computers
outsmarting people. (08)
> Alex also accepted answers from Watson that he probably would not have
> accepted from a person. For "It's a poor man who blames these", Watson
> responded "tools" instead of "his tools". With a person, Alex would
> probably have axed for elaboration, but accepted Watson's response
> without pause. He also accepted "Parliament" in response to "Elected
> every 5 years, it has 736 members from 7 parties" instead of requiring
> "European Parliament".
>
> (09)
They got an unexpected boost from Alex, as you noted. My wife and I are
longtime Jeopardy! watchers. It seems to us that Alex and his "clue
team" have become increasingly arbitrary in their acceptance of specific
answers, whether for the correct phrasing of the question or for error
in facts. Some of their judgments are clearly wrong. That's
understandable. It's the trend that irritates us, so we end up yelling
at Alex. I guess we need to "get a life." ;-) (010)
Phil Murray
> =============================================================
> doug foxvog doug@xxxxxxxxxx http://ProgressiveAustin.org
>
> "I speak as an American to the leaders of my own nation. The great
> initiative in this war is ours. The initiative to stop it must be ours."
> - Dr. Martin Luther King Jr.
> =============================================================
>
>
> _________________________________________________________________
> Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/
> Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/
> Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
> Shared Files: http://ontolog.cim3.net/file/
> Community Wiki: http://ontolog.cim3.net/wiki/
> To join: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J
> To Post: mailto:ontolog-forum@xxxxxxxxxxxxxxxx
>
>
> (011)
--
--------------------- (012)
The Semantic Advantage
Turning Information into Assets
phil.murray@xxxxxxxxxxxxxxxxxxxxx
401-247-7899 (013)
Blog: http://semanticadvantage.wordpress.com
Web site: http://www.semanticadvantage.com (014)
_________________________________________________________________
Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/
Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/
Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
Shared Files: http://ontolog.cim3.net/file/
Community Wiki: http://ontolog.cim3.net/wiki/
To join: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J
To Post: mailto:ontolog-forum@xxxxxxxxxxxxxxxx (015)
|