As the discussion shifts to how are we going to establish the value
of "strong semantics" and other positions, we all have common ground. (01)
Mills
On Mar 12, 2006, at 12:18 AM, Cassidy, Patrick J. wrote: (02)
> My take on the "scientific" approach to ontology:
>
> One of the essential features of experimental science is the need to
> carefully control what is varied in an experiment, ideally varying
> only
> one factor at a time and assessing the result. In comparing different
> semantic systems, what is typically done is to vary thousands of
> factors (the ontology and/or program), even though **something**
> may be
> kept constant (e.g. a text to be interpreted). The result can tell us
> which aggregate system is quicker or more accurate, but the experiment
> itself cannot pinpoint **why** there is a difference. The
> experimenters are left to guess at high-level differences that may
> have
> caused a performance difference across systems. But it is not
> possible
> to rule out that a single axiom or method in one program may have made
> a major difference. You can't tell whether or which by that process.
>
> In ontology studies, the scientific method should also be applicable.
> There are at least two parts to an ontology-based reasoning system,
> the
> ontology and the reasoning methods. Unless each is tested separately,
> the results will be subject to varied interpretation.
>
> One can keep the ontology constant and test small changes in the
> reasoning technique to see how the performance or accuracy changes.
> More than one ontology can be tested against the same basic reasoning
> system, but the changes in each reasoning system would be the factors
> to be varied in the test, evaluated within each ontology environment,
> not across ontologies.
>
> One can keep the reasoning system constant and see how additions,
> deletions, or changes to an ontology affect the result - performance
> speed or accuracy. More than one ontology can be tested against the
> same reasoning system, but it is the changes in each ontology,
> considered separately, that would be the subject of such experiments.
>
> It's a lot of work but the results will be more objective and reliable
> than just comparing full systems. The latter is what is done in the
> marketplace (and, unfortunately, in a lot of funded "bake-off"
> contests, including Project Halo), but if we want a science we need a
> more analytical approach. Just setting up such a test might be the
> equivalent of a Master's thesis, but once set up, many informative
> tests may be performed.
>
> It might be useful to begin to explore such an approach by agreeing on
> some one or small set of ontologies that should be used to test
> reasoning engines. By "reasoning" I am thinking of something more
> complicated than just a DL reasoner or an FOL reasoner doing
> inferences
> over simple problems and being tested only for speed. Some heuristic
> methods may be tested this way.
>
> And conversely, we might start discussing the kinds of reasoning
> problems (or "applications") that can be used to test ontologies. A
> "test suite" that does not assume a particular ontological structure,
> but is designed to work within different ontological structures would
> be needed.
>
> I would hope that at least one such test suite would include the
> understanding of language. A start might be with a "controlled
> English" that has some ambiguity that needs resolution by semantics,
> and get more complicated until it reaches the language abilities of a
> six-year-old native speaker, at which time we can argue over who gets
> the Nobel prize. ;-)
>
> In the interim, however, as Sergei has suggested, such an interface
> could help enormously to make the ontologies easier to use, and get
> accurate and useful input from domain experts uninterested in the
> mechanics of ontology.
>
> Pat
>
> Patrick Cassidy
> MITRE Corporation
> 260 Industrial Way
> Eatontown, NJ 07724
> Mail Stop: MNJE
> Phone: 732-578-6340
> Cell: 908-565-4053
> Fax: 732-578-6012
> Email: pcassidy@xxxxxxxxx
>
>
> -----Original Message-----
> From: uos-convene-bounces@xxxxxxxxxxxxxxxx
> [mailto:uos-convene-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Mills Davis
> Sent: Saturday, March 11, 2006 10:58 PM
> To: Upper Ontology Summit convention
> Subject: Re: [uos-convene] The value of ULO
>
> Bill,
>
> Exactly. What if the ULO group can establish a test bed where each of
> these different formalisms can be represented, explored, compared,
> and evaluated in an executable environment (and in the context of
> specific problems to solve, or solutions to implement)? This would be
> valuable for everybody.
>
> Mills
>
> On Mar 11, 2006, at 5:58 PM, Bill Andersen wrote:
>
>> C'mon, Leo! You're spoiling my fun! And why do all of this work
>> if it isn't fun? :-D
>>
>> Actually, I am serious about that test I mentioned. If some entity
>> like DARPA or ARDA is looking for something worthwhile to do, I
>> would consider a test such as this worthy of funding.
>>
>> On Mar 11, 2006, at 17:55 , Obrst, Leo J. wrote:
>>
>>> Bill,
>>>
>>> We are ALL on your side, because we have all gone through this so
>>> many
>>> times. We all firmly believe in the value and ultimate necessity of
>>> Upper Ontologies -- because we have fought our way to this
> conclusion
>>> the hard way. We just need to impress this experience on others who
>>> have not yet undergone the baptism by fire. Teach them, not exorcise
>>> them.
>>>
>>> Leo
>>>
>>> ps. So does this mean that relational databases don't already
> provide
>>> everything that we need? ;)
>>> _____________________________________________
>>> Dr. Leo Obrst The MITRE Corporation, Information Semantics
>>> lobrst@xxxxxxxxx Center for Innovative Computing & Informatics
>>> Voice: 703-983-6770 7515 Colshire Drive, M/S H305
>>> Fax: 703-983-1379 McLean, VA 22102-7508, USA
>>>
>>>
>>> -----Original Message-----
>>> From: uos-convene-bounces@xxxxxxxxxxxxxxxx
>>> [mailto:uos-convene-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Bill
>>> Andersen
>>> Sent: Saturday, March 11, 2006 5:36 PM
>>> To: Upper Ontology Summit convention
>>> Subject: [uos-convene] The value of ULO
>>>
>>> Just a quick comment on Mills' comment:
>>>
>>> Before continuing, let me first say that I am happy that Mills is an
>>> advocate for ontologies in general. Ok... but...
>>>
>>>> Bottom line: I applaud and encourage the efforts of this UO
>>>> community. By coming together, I hope they deliver value. At the
>>>> same
>>>> time, I'm reserving judgement (or, remaining agnostic regarding the
>>>> value, pending evidence), and am harboring a supposition that in
> the
>>>> next year or so, technology may be emerging that will obviate UO
>>>> arguments by subsuming all of these disparate approaches and
>>>> subjecting them to tests of efficacy.
>>>
>>> I am having a very hard time understanding this comment. Short of
>>> HAL-9000, just what technology might that be that will subsume UO or
>>> "obviate UO arguments"?
>>>
>>> One of the things that has been systematically overlooked in these
>>> discussions is that almost all of the ULOs under discussion here are
>>> at least in part philosophically motivated. This includes at least
>>> SUMO, Cyc, DOLCE, and Matt West's 4D ontology and to a lesser extent
>>> PSL. They are so because the philosophers have been at this
> business
>>> for 2500 years, long before the advent of the W3C. The tools of
>>> mathematical logic and later computational logic were what were
>>> needed to make this 2500 years of work effective.
>>>
>>> As for efficacy, I have been arguing the need for this since my
> first
>>> posts to this mailing list. One comment that I have made more than
> a
>>> couple times in different guises is that the value of ULO lies NOT
>>> ONLY in semantic interoperability but has engineering advantages as
>>> well for building individual ontologies. Our experience at OW has
>>> been that we build better ontologies much faster (and thus at less
>>> cost) than those who take a tabula rasa approach, no matter what
>>> formalism they work in. That we have done this, you'll just have to
>>> take my word as the work was either classified or FOUO.
>>>
>>> The good news is you don't have to take my word for it. I suggest
> we
>>> set up an experiment by which teams of comparable expertise in their
>>> chosen formalisms, given the same domain description and a fixed
>>> amount of time, one using a ULO and one prohibited from doing so,
>>> build a domain ontology to meet the description. Then, given the
>>> same data sets, both are subject to a bank of blind competence
>>> questions based solely on the domain description by parties
>>> unfamiliar with the logical formalisms employed. We repeat the
>>> experiment several times on different domains. As an added bonus,
> we
>>> then take the resulting domain ontologies and, using them as
> starting
>>> points, ask each team (ULO, and non-ULO) to *extend* them to a
>>> different, but related domain. Then more competency questions.
>>>
>>> I know on which team I'll place my bet. Any takers?
>>>
>>> .bill
>>>
>>> Bill Andersen (andersen@xxxxxxxxxxxxxxxxx)
>>> Chief Scientist
>>> Ontology Works, Inc. (www.ontologyworks.com)
>>> 3600 O'Donnell Street, Suite 600
>>> Baltimore, MD 21224
>>> Office: 410-675-1201
>>> Cell: 443-858-6444
>>>
>>>
>>> _________________________________________________________________
>>> Message Archives: http://ontolog.cim3.net/forum/uos-convene/
>>> To Post: mailto:uos-convene@xxxxxxxxxxxxxxxx
>>> Community Portal: http://ontolog.cim3.net/
>>> Shared Files:
>>> http://ontolog.cim3.net/file/work/UpperOntologySummit/uos-convene/
>>> Community Wiki:
>>> http://ontolog.cim3.net/cgi-bin/wiki.pl?UpperOntologySummit
>>> _________________________________________________________________
>>> Message Archives: http://ontolog.cim3.net/forum/uos-convene/
>>> To Post: mailto:uos-convene@xxxxxxxxxxxxxxxx
>>> Community Portal: http://ontolog.cim3.net/
>>> Shared Files: http://ontolog.cim3.net/file/work/
>>> UpperOntologySummit/uos-convene/
>>> Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?
>>> UpperOntologySummit
>>>
>>
>> Bill Andersen (andersen@xxxxxxxxxxxxxxxxx)
>> Chief Scientist
>> Ontology Works, Inc. (www.ontologyworks.com)
>> 3600 O'Donnell Street, Suite 600
>> Baltimore, MD 21224
>> Office: 410-675-1201
>> Cell: 443-858-6444
>>
>>
>> _________________________________________________________________
>> Message Archives: http://ontolog.cim3.net/forum/uos-convene/
>> To Post: mailto:uos-convene@xxxxxxxxxxxxxxxx
>> Community Portal: http://ontolog.cim3.net/
>> Shared Files: http://ontolog.cim3.net/file/work/UpperOntologySummit/
>> uos-convene/
>> Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?
>> UpperOntologySummit
>>
>
> Mills Davis
> Managing Director
> Project10X
> 202-667-6400
> 202-255-6655 cel
> 202-667-6512 fax
> mdavis@xxxxxxxxxxxxxx
>
>
>
>
> _________________________________________________________________
> Message Archives: http://ontolog.cim3.net/forum/uos-convene/
> To Post: mailto:uos-convene@xxxxxxxxxxxxxxxx
> Community Portal: http://ontolog.cim3.net/
> Shared Files:
> http://ontolog.cim3.net/file/work/UpperOntologySummit/uos-convene/
> Community Wiki:
> http://ontolog.cim3.net/cgi-bin/wiki.pl?UpperOntologySummit
> _________________________________________________________________
> Message Archives: http://ontolog.cim3.net/forum/uos-convene/
> To Post: mailto:uos-convene@xxxxxxxxxxxxxxxx
> Community Portal: http://ontolog.cim3.net/
> Shared Files: http://ontolog.cim3.net/file/work/UpperOntologySummit/
> uos-convene/
> Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?
> UpperOntologySummit
> (03)
Mills Davis
Managing Director
Project10X
202-667-6400
202-255-6655 cel
202-667-6512 fax
mdavis@xxxxxxxxxxxxxx (04)
_________________________________________________________________
Message Archives: http://ontolog.cim3.net/forum/uos-convene/
To Post: mailto:uos-convene@xxxxxxxxxxxxxxxx
Community Portal: http://ontolog.cim3.net/
Shared Files: http://ontolog.cim3.net/file/work/UpperOntologySummit/uos-convene/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?UpperOntologySummit (05)
|