ontolog-forum
[Top] [All Lists]

Re: [ontolog-forum] IBM Watson's Final Jeopardy error "explanation"

To: "[ontolog-forum]" <ontolog-forum@xxxxxxxxxxxxxxxx>
From: Mike Bennett <mbennett@xxxxxxxxxxxxxxx>
Date: Wed, 16 Feb 2011 23:16:03 +0000
Message-id: <4D5C5AB3.6060002@xxxxxxxxxxxxxxx>
Well there were some unpredictable contagions in previous 
crashes, e.g. where one investor or group of investors is 
over-exposed in something which seems safe enough except that 
it's over-exposed to something else. Being able to apply these 
statistical measures to a set of semantically comparable concepts 
would possibly detect these sort of correlations.    (01)

As for predictive measures from tweaking macro-economic 
variables, that seems to me to be at least partly a chaotic 
problem - so presumably even a good simulation of the facts might 
end up with a wildly different end result to reality. Not to 
mention the chaotic effect of one or more Watsons forming part of 
the system itself.    (02)

Mike    (03)

On 16/02/2011 22:42, Ron Wheeler wrote:
> On 16/02/2011 2:33 PM, Mike Bennett wrote:
>> Thanks. Very interesting. There's a few of us looking at using
>> semantics to enable systemic risk management applications
>> development. The use of this kind of thing by big investors is
>> going to change the landscape considerably I think.
>>
> It is going to make it much harder to trace back the methodology used to
> develop a risk assessment.
> The algorithms and logic used to analyze the impact of a Russian
> agricultural forecast on the price of oil may be hard for traditional
> stock brokers, analysts and traders to follow.
>
> Would it have caught the fact that a lot of the incomes quoted in
> mortgage applications were too high for the job titles and historical
> incomes of the persons who were granted the mortgages?
> The rating agencies did not (or chose not to consider it as a factor in
> rating the securities) so the bar for "better" is set pretty low.
>
> At least it would have read all of the contracts!
>
> The adoption rate will be interesting to watch. I suspect that once 1
> institution buys in the rest will follow quickly so as not to be left
> behind.
> Unfortunately it could take a few years to see if Watson is really
> making better decisions and by, that time, no one will know how to go
> back to the "old" ways if it really turns out wrong.
>
>
> Ron
>
>> Mike
>>
>> On 16/02/2011 19:01, Ron Wheeler wrote:
>>> On 16/02/2011 1:30 PM, Mike Bennett wrote:
>>>> Agreed, I think there are some very interesting avenues to
>>>> explore for systemic risk analysis.
>>> IBM's video on the Financial industry's possible use of Watson
>>> http://www.youtube.com/watch?v=IV6mfWWMIVg&feature=channel
>>> A bit short on details but gives some ideas about where IBM sees this
>>> applying.
>>>
>>> The comments section under the video is fun to look at as well.
>>> Ron
>>>> Mike
>>>>
>>>> On 16/02/2011 18:25, Ron Wheeler wrote:
>>>>> On 16/02/2011 12:16 PM, Mike Bennett wrote:
>>>>>> I think the financial industry will stick to semantics rather
>>>>>> than statistics for now, at least for reporting and risk management.
>>>>>>
>>>>> Reporting at the authoring end but I suspect that at the receiving end
>>>>> the government and NGOs are going to be very interested in tools that
>>>>> help correlate reported data with other industry news and reports.
>>>>>
>>>>> Risk management is such a broad category and entails so much information
>>>>> that I suspect that a system that can identify and explain the increased
>>>>> risk to a portfolio caused by a flood in southern China and a
>>>>> demonstration in Cairo on the same day will be hard to resist.
>>>>>
>>>>>> Stock picking is another matter, as noted by Ron.
>>>>> The IBM material that was issued around the "Watson/Jeopardy event"
>>>>> specifically mentioned M&A target identification and the feeling
>>>>> expressed by one analyst in an interview, that analysts are inundated
>>>>> with news, reports and trading patterns/events.
>>>>>>        I wonder if
>>>>>> this thing has significantly increased the risk (or scale) of
>>>>>> another Flash Crash type of event?
>>>>>>
>>>>> Anything that can get everyone making the same decision at the same time
>>>>> is bound to cause some very interesting "gold rushes" where there are
>>>>> more losers than winners but lots of transactions for the
>>>>> intermediaries. The guys selling logistics in a gold rush almost always
>>>>> made more money than the miners.
>>>>>
>>>>> Ron
>>>>>
>>>>>> Mike
>>>>>>
>>>>>> On 16/02/2011 15:37, doug foxvog wrote:
>>>>>>> Question answering by a machine such as Watson is not very useful unless
>>>>>>> the system can explain its answers.  Thus i am disappointed that IBM's
>>>>>>> "explanation" of its error in Final Jeopardy does not explain why it
>>>>>>> chose its answer.
>>>>>>>
>>>>>>> On Tue, February 15, 2011 21:19, ZENG, MARCIA said:
>>>>>>>> FYI:
>>>>>>>> Watson's Final Jeopardy Blunder In Day 2 Of IBM Challenge was updated:
>>>>>>>>
>>>>>>>> UPDATE: IBM posted this explanation for the mistake on its "Smarter
>>>>>>>> Planet" blog....
>>>>>>>>
>>>>>>>> 
>http://www.huffingtonpost.com/2011/02/15/watson-final-jeopardy_n_823795.html
>>>>>>> This explanation is little more than "Jeopardy is hard":
>>>>>>> * Category names are tricky and their significance is downgraded.
>>>>>>> * There are cities named Toronto in the US
>>>>>>> * Toronto, Ontario, has an American League baseball team.
>>>>>>>
>>>>>>> This does not explain why "Toronto" was selected as a response.
>>>>>>>
>>>>>>>
>>>>>>> In the IBM ads accompanying the show, they discuss Watson being used
>>>>>>> for diagnosis -- not just for pointing out information that a doctor
>>>>>>> may want to look at for a case.  However, without explanation for
>>>>>>> a diagnosis -- or even for a recommendation for reviewing some paper,
>>>>>>> such use seems quite improper.
>>>>>>>
>>>>>>> -- doug foxvog
>>>>>>>
>>>>>>>> On 2/15/11 8:18 PM, "John F. Sowa"<sowa@xxxxxxxxxxx>        wrote:
>>>>>>>>
>>>>>>>> On 2/15/2011 4:24 PM, Jerry Hobbs wrote:
>>>>>>>>> Actually I think the question asked for an "anatomical anomaly",
>>>>>>>>> which could be a body part, like a sixth toe, rather than a 
>disability.
>>>>>>>>> The text Watson used referred to the athlete's "wooden leg".
>>>>>>>> I agree.  It found the anomaly, but it made an error in not explaining
>>>>>>>> that it was an anomaly.  So it should at least get partial credit.
>>>>>>>>
>>>>>>>> On tonight's round, Watson did very well.  But it made a category
>>>>>>>> error for which there was no excuse.
>>>>>>>>
>>>>>>>> The category was US Cities, and all three contestants were given
>>>>>>>> 30 seconds to write their answers and the amount they were willing
>>>>>>>> to bet.
>>>>>>>>
>>>>>>>> Question:  A city whose largest airport is the name of a war hero
>>>>>>>> and whose second largest airport is the name of a WW II battle.
>>>>>>>>
>>>>>>>> Both humans got the answer right -- Chicago -- and they bet
>>>>>>>> the maximum or almost the maximum.
>>>>>>>>
>>>>>>>> That was tricky because Midway Airport happens to be the
>>>>>>>> name of a battle, but it was not named for the battle.
>>>>>>>>
>>>>>>>> But Watson wrote Toronto and bet $947 (which was a small
>>>>>>>> amount, indicating that it wasn't sure).
>>>>>>>>
>>>>>>>> But the category was "US Cities".  It certainly should
>>>>>>>> have checked.
>>>>>>>>
>>>>>>>> John
>>>>>>>>
>>>>>>>>
>>> _________________________________________________________________
>>> Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/
>>> Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/
>>> Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
>>> Shared Files: http://ontolog.cim3.net/file/
>>> Community Wiki: http://ontolog.cim3.net/wiki/
>>> To join: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J
>>> To Post: mailto:ontolog-forum@xxxxxxxxxxxxxxxx
>>>
>>>
>>>
>
> _________________________________________________________________
> Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/
> Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/
> Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
> Shared Files: http://ontolog.cim3.net/file/
> Community Wiki: http://ontolog.cim3.net/wiki/
> To join: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J
> To Post: mailto:ontolog-forum@xxxxxxxxxxxxxxxx
>
>
>    (04)


-- 
Mike Bennett
Director
Hypercube Ltd.
89 Worship Street
London EC2A 2BF
Tel: +44 (0) 20 7917 9522
Mob: +44 (0) 7721 420 730
www.hypercube.co.uk
Registered in England and Wales No. 2461068    (05)


_________________________________________________________________
Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/  
Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/  
Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
Shared Files: http://ontolog.cim3.net/file/
Community Wiki: http://ontolog.cim3.net/wiki/ 
To join: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J
To Post: mailto:ontolog-forum@xxxxxxxxxxxxxxxx    (06)

<Prev in Thread] Current Thread [Next in Thread>