Re: [ontology-summit] Reasoners and the life cycle

[Michael F Uschold <uschold@xxxxxxxxx>]

There is a broader issue here that reasoners are only a part of. That is: "Where do you derive confidence from that your ontology is accurate"?

One time honored way is to apply systematic techniques to discover known bugs or likely errors.  A reasoner can help in different ways.  The examples below are in they context of OWL, but most generalize to other formalisms.

1. It can find logical inconsistencies 
2. It can find classes that cannot possibly have instances
3. It can make inferences that on manual inspection do not accord with the intended meaning.

But there are other very mundane ways to help root out errors or discrepancies from disired standard practice, e.g.:

1. printing out and scanning an alphabetized list of terms and finding misspellings like:  partOf and patrOf.
2. searching for properties that do not have inverses (if you like all your properties to have inverses)

There is a nice tool called Ontology Pitfall Scanner which allows you to upload an ontology and it will do a bunch of things like this.

Still other ways to derive confidence that the ontology is accurate is to have it checked by experts in the field.  

Two other important criteria that we use to evaluate our client's enterprise ontologies are: Completeness and Understandability

This discussed in my recent invited talk at EKAW in Galway in October.  It starts about 2/3 of the way through the 55 min talk.

You can see it from the following link, skip to the bottom of the page.

In the Trenches with Enterprise Ontology

 Tue, Dec 18, 2012 at 10:24 AM, Obrst, Leo J. <lobrst@xxxxxxxxx> wrote:

John, James, and all,

Of course you must evaluate what kind of reasoner you will use, and generally that includes knowledge compilation. But reasoning can indeed point out flaws or inadequacies in the ontology, a true part of ontology evaluation.

Another issue that must be raised is so-called "glass box vs. black box" evaluation, e.g., from [1]:

"One issue in evaluating ontologies is whether to perform glass box
(component-based) vs. black box (task-based) evaluation, the latter usually
applied to ontologies that are tightly integrated with an application
performing specific tasks [36]."

36. Hartmann, Jens; Peter Spyns; Alain Giboin; Diana Maynard; Roberta Cuel; Mari Carmen
Suárez-Figueroa. 2005. D1.2.3 Methods for ontology evaluation. EU-IST Network of
Excellence (NoE) IST-2004-507482 KWEB Deliverable D1.2.3 (WP 1.2).

Thanks,
Leo

[1] Obrst, Leo; Werner Ceusters; Inderjeet Mani; Steve Ray; Barry Smith. 2007 The Evaluation of Ontologies: Toward Improved Semantic Interoperability. Chapter in: Semantic Web: Revolutionizing Knowledge Discovery in the Life Sciences, Christopher J. O. Baker and Kei-Hoi Cheung, Eds., Springer, 2007.

>-----Original Message-----
>From: ontology-summit-bounces@xxxxxxxxxxxxxxxx [mailto:ontology-
>summit-bounces@xxxxxxxxxxxxxxxx] On Behalf Of John F Sowa
>Sent: Tuesday, December 18, 2012 1:12 PM
>To: ontology-summit@xxxxxxxxxxxxxxxx
>Subject: Re: [ontology-summit] Reasoners and the life cycle
>
>Leo and James,
>
>That is highly misleading as a requirement:
>
>Leo
>> Many ontology projects either have automated reasoning as a requirement
>> now or they will in the future, as applications evolve and new
>> applications arise to use/reuse the ontology.
>
>James D
>> If this read "Many projects which involve an ontology have/will have also
>> a requirement for a reasoner ..." I'd be happy. But ontologies and
>> reasoners are different things. It is plausible to include
>> "reasoner-suitability" as a criterion in evaluating ontologies, certainly.
>> ... But, since an ontology is not a reasoner, it's irrelevant for
>> the purpose of evaluating ontologies.
>
>Yes.  And I would emphasize just *one* of many criteria.
>
>Much more important is *compilation* from a very general, highly
>expressive ontology into formats that are appropriate for different
>purposes.  Among them are database constraints, which can be stated
>in full first-order logic and evaluated in polynomial time.
>
>Cyc uses compilation methods very heavily for translating a logic
>(CycL) that is even more expressive than FOL into formats that
>are appropriate for different kinds of reasoning engines that
>are tailored for different kinds of problems.
>
>Cyc has been in the ontology business for 28 years, and they have
>implemented the largest formal ontology on the planet.  It should
>be cited as an example of a *good* ontology: very general and stated
>in very expressive logic designed for ease of translation to formats
>for an open-ended variety of reasoners.
>
>That criterion does not imply that restricted logics, such as OWL DL,
>are necessarily bad.  But they should be put in the category of special-
>purpose ontologies designed for just one narrow kind of reasoner.
>
>For more detail, see the following article which was published in
>a journal for which Jim Hendler was the editor.  Jim said that he
>liked the article very much:
>
>    http://www.jfsowa.com/pubs/fflogic.pdf
>    Fads and fallacies about logic
>
>For another example, see the following note about a system that
>successfully used an *undecidable* logic in mission critical
>applications.  They did use OWL for the type hierarchy, but most
>of the ontology was in a more expressive logic.
>
>John
>
>-------- Original Message --------
>Subject: Mission Critical IT
>Date: Fri, 14 Dec 2012 10:47:52 -0500
>From: John F Sowa <sowa@xxxxxxxxxxx>
>To: '[ontolog-forum] ' <ontolog-forum@xxxxxxxxxxxxxxxx>
>CC: Michel Vanden Bossche
>
>Dear Michel,
>
>Thanks for the information about the SWESE system.
>
>I'm sending this note to Ontolog Forum because it has a larger audience
>than the original list, Ontology Summit.  I renamed the subject line
>to a pun on the name of your company and the elusive goal that the SW
>and AI in general have been groping toward -- with limited success.
>
>Following is the paper about SWESE:
>
>    http://www.missioncriticalit.com/pdfs/swese-2007.pdf
>    Ontology Driven Software Engineering for Real Life Applications
>
>I'd like to emphasize some of your points, but I recommend the entire
>note (copied below) and the above paper.
>
>MVB
> > The implementation is 100% Mercury (a concern for the customer), but
> > necessitates only 45,000 LOC of which 25,000 were automatically
>generated.
>
>A requirement for mainstream IT:  generate the AI stuff *automatically*
>from notations that the IT staff and the SMEs can read and write.
>
> > It took 5000 man.days to specify, design, implement, test and deploy.
> > A similar system, although much simpler (one product at a time, one
> > channel, etc.), built in Java EE, took 15,000 md.
>
>For the record, Mercury is a high-speed logic-programming language:
>
>    http://www.mercury.csse.unimelb.edu.au/information/features.html
>
>This increase in productivity for implementing the DSL (Domain Specific
>Language) is typical of Prolog and other logic programming languages.
>In fact, Alain Colmerauer, who designed the first Prolog system, adapted
>it from a parser he had originally written for machine translation.
>
> > The system was just starting in production (4000 users) when Winterthur
> > was acquired by Axa. Axa viewed our system as much too advanced,
>considering
> > that their IT staff was not competent enough to support it. AAMOF,
>Axa has
> > just cancelled a 3rd attempt to build a system like ours, using BaNCS,
> > a package from TCS (Tata Consultancy Services).  They pulled the plug
> > after spending 35 million USD.
>
>That is a sad but familiar story.  Our company, VivoMind Research, LLC,
>had similar experiences.  We implemented the system for oil and gas
>exploration described in the Future Directions paper (pp 15-17):
>
>    http://www.jfsowa.com/pubs/futures.pdf
>
>After we developed that prototype, the oil company spoke with a larger
>company that claimed they could do anything we could do.  They got a
>two-year contract with a lot more money than we did.  But they failed.
>
> > Mercury is really great but is sociologically impossible to sell
> >
> > So, we decided to developed a Java backend for Mercury. Today,
> > Mercury  compiled in Java is even faster than compiling to C.
>
>That is also a familiar story.  What you deliver to customers is Java.
>They have heard of Java.  Their programmers will never look at the code,
>but they'll feel comfortable that it's something they "know".
>
> > Rules (beyond OWL) are absolutely critical. We started with SWRL,
> > but  real life applications require at least aggregation and negation,
> > with a subtle balance between OWA and CWA. So, we worked on xSWRL
> > (extended SWRL, a very poor name as this is not SWRL anymore, but
> > it simplifies the marketing ;-)).
>
>That is the major strength of the SW:  it's a powerful hype machine.
>But what they hype is pitifully weak for serious applications.
>
> > It's a pity that W3C is not addressing the Business Rules issue
> > aggressively. RIF is like OSI: they couldn't decide between
> > Production Rules and Logic Rules (OSI couldn't decide between
> > connectionless and connectionfull).
>
>I spoke at a conference on Business Rules.  One of the sponsors
>was Experian -- a credit bureau that processes huge amounts of data
>about everybody in the world.  Their major language is Prolog.  They
>use it so heavily that they bought Prologia, the company founded by
>Alain Colmerauer.  Experian is truly mission critical, but they're
>so secretive that nobody knows how they specify their rules.
>
> > Same syndrome: W3C is not ready to bite the bullet, i.e. aggressively
> > support logic. They backtrack to Linked Data Object to erase the AI
> > from their story. And "Rules Interchange Format" is a terrible name.
>
>I don't blame Tim Berners-Lee.  But I do blame the academics who rammed
>decidability down the throats of people who had no clue about what it
>meant.  Look at Tim's original proposal from February 2000:
>
>    http://www.w3.org/2000/01/sw/DevelopmentProposal
>
>Tim discussed Prolog and many other AI systems. He emphasized diversity.
>And he proposed SWeLL (Semantic Web Logic Language) for communication
>among heterogeneous systems.  SWeLL was intended as a superset of
>propositional, first order, and higher order logic.  Unfortunately,
>the final report claimed that OWL was a renamed version of SWeLL.
>
> > Logic gives you the possibility to debug declaratively your model.
> > This is what we do with our ODASE workbench: you can validate the
> > model with the SME before writing the first LOC.
> >
> > And, when the system runs, you can trace and explain why the system
> > works as it is.
>
>Yes.  And to do that, you need a logic that can express anything
>that the system uses or does.  But no decidable logic can.
>
> > With our new reasoner, we can price a car contract with more than
> > 10 coverages in 83 ms, on a standard Xeon 3.4 GHz, on one core.
> > Batch is easily scaled-out on multiple cores.
> >
> > This system is mission-critical and we move to a phase of productization.
>
>Congratulations.  Note the time:  83 milliseconds.  That is the kind
>of timing that mission critical applications require.
>
>Decidability only guarantees *finite* times.  OWL guarantees finite
>time by limiting their models to trees.  But the size of a tree grows
>exponentially with the depth.  The algorithms can take longer than
>the age of the universe.  That's finite, but not mission critical.
>
> > fundamental objective: better software requires better science first,
> > sound engineering then, and real-life experimentation with real
>customers.
>
>Memo to W3C:  Don't standardize anything until *after* there is at least
>one real live customer that is happy with the results.
>
>John
>
>-------- Original Message --------
>Subject: Re: [ontology-summit] Potential Tracks for Ontology Summit 2013
>Date: Fri, 14 Dec 2012 12:12:29 +0100
>From: Michel Vanden Bossche
>
>Dear John,
>
>Thanks for your mail to Michael [Uschold] and thanks to Michael for
>copying me.
>
>First, I read many of your papers and I share your views. The last
>paper I read is "Future Directions for Semantic Systems".
>
> > I repeat the question:  What mission-critical applications has anybody
> > developed with that platform?  Are they actually being used on a daily
> > basis?  For any significant time (at least a year) after the software
> > was delivered?
>
>1. SWESE system
>
>The system described in our SWESE paper was really "mission-critical".
>That project started in July 2004 and was completed, successfully in
>October 2007. It is an eInsurance system with the following objectives:
>
>- customer centric
>- multi products
>- multi channels
>- fast turn around time of new products (2 weeks max)
>- fully web
>- < 3 sec response time for pricing (8 products, 5 variants each)
>- etc.
>
>It is ontology driven, using OWL (just released in February 2004). The
>ontology defines the P&C (Property and Casualty) insurance domain and
>insurance products as instances of product definitions in the
>ontology. The UI is Ajax (we used a kind of JSP for Mercury that we
>named MSP). The navigation is defined, in the ontology, using YAOWL, a
>variant of YAWL (van der Aalst), a CPN based workflow. RBAC (Role
>Based Access Control) defines Roles and Powers ontologically.
>
>Rating/pricing is executed by a DSL that was imposed by the customer
>(Winterthur) so that pricing in the mid-tier (our system) was
>identical with the mainframe pricing (zOS, COBOL, DB2). We couldn't
>price on the mainframe calling services because the mainframe died due
>to the many CICS calls related to multiple products and multiple
>variants. We reimplemented the DSL written in COBOL in Mercury (it
>took us 3 weeks) and the response time went from more than 1 minute to
>350 ms (on an HP Intel Xeon Box of that time).
>
>The implementation is 100% Mercury (a concern for the customer), but
>necessitates only 45,000 LOC of which 25,000 were automatically generated.
>
>It took 5000 man.days to specify, design, implement, test and deploy.
>A similar system, although much simpler (one product at a time, one
>channel, etc.), built in Java EE, took 15,000 md.
>
>The system was just starting in production (4000 users) when
>Winterthur was acquired by Axa. Axa viewed our system as much too
>advanced, considering that their IT staff was not competent enough to
>support it. AAMOF, Axa has just cancelled a 3rd attempt to build a
>system like ours, using BaNCS, a package from TCS (Tata Consultancy
>Services). They pulled the plug after spending 35 MUSD.
>
>2. Lessons learned
>
>a. Mercury is really great but is sociologically impossible to sell
>
>So, we decided to developed a Java backend for Mercury. Today, Mercury
>compiled in Java is even faster than compiling to C. This illustrates
>the fact that a great language with a great compiler which can
>leverage fundamental CS research (e.g. Abstract Data Interpretation) wins.
>
>Since then, we have also developed a C# backend, which works fine but
>is not yet as mature.
>
>We have also built, as an exercise, an Erlang backend. This one is
>only alpha.
>
>We think that we need three layers:
>
>- logic at the modeling level
>- logic programming
>      to consume the model
>      to mitigate with the accepted languages of the day
>- Java, C# today and something else in the future (unknown)
>
>The problem is to transcend time.
>
>b. Code generation
>
>We recognized the importance of combining code generation (Business
>API in Java or C# to access the ontology) and interpretation.
>
>c. Rules
>
>Rules (beyond OWL) are absolutely critical. We started with SWRL, but
>real life applications require at least aggregation and negation, with
>a subtle balance between OWA and CWA. So, we worked on xSWRL
>(extended
>SWRL, a very poor name as this is not SWRL anymore, but it simplifies
>the marketing ;-)).
>
>We observed that Pellet performance for SWRL was not adequate. Relying
>on a PR engine (JESS...) was not an option because we would loose
>declarativeness. So, we built our own reasoners, trading completeness
>for performance ; this is not a problem in real-life. As usual, sound
>engineering requires a good balance between science and pragmatism.
>
>We now exceed the performance requirements imposed on us by our last
>customer (see later).
>
>It's a pity that W3C is not addressing the Business Rules issue
>aggressively. RIF is like OSI: they couldn't decide between Production
>Rules and Logic Rules (OSI couldn't decide between connectionless and
>connectionfull).
>
>Same syndrom: W3C is not ready to bite the bullet, i.e. aggressively
>support logic. They backtrack to Linked Data Object to erase the AI
>from their story. And "Rules Interchange Format" is a terrible name.
>
>Silk is going in the right direction, but it's maybe too ambitious for
>the moment.
>
>d. Process
>
>Ontologies are "Sein" but real applications require "Sein und Zeit".
>Again, besides OWL-S (a very imperative approach), W3C is not
>addressing that issue.
>
>We moved from the imperative approach (CPN) to a declarative approach
>combining OWL, xSWRL and LTL. In practice, may processes issues are
>related to Data flow problems, much less than to Control flow problems.
>
>e. Model "debugging"
>
>Logic gives yoiu the possibility to debug declaratively your model.
>This is what we do with our ODASE workbench: you can validate the
>model with the SME before writing the first LOC.
>
>And, when the system runs, you can trace and explain why the system
>works as it is.
>
>f. IT ego
>
>There is a really big problem with IT people. Their expertise is like
>the one of the Byzantine priests. They know everything related to
>their practical domain of expertise, but if they don't know something
>(logic, OWL...) they'll fight like hell to avoid being not experts
>anymore. So, they are ultra conservatives under the costume of modernists.
>
>This is a difficult issue. We have the feeling that we can now deal
>with them without antagonizing them too much ;-)
>
>3. Last project
>
>We just completed the first phase of a project for AVIVA (another
>insurance company): externalizing the product definition and the
>pricing (product factory and rating engine).
>
>With our new reasoner, we can price a car contract with more than 10
>coverages in 83 ms, on a standard Xeon 3.4 GHz, on one core. Batch is
>easily scaled-out on multiple cores.
>
>This system is mission-critical and we move to a phase of productization.
>
>We are now investigating a parallel implementation of our reasoners. A
>language like Mercury makes that much less complex than Java or C#
>(maybe Java 8 will change that).
>
>4. Conclusion
>
>I started the "logic dream" in 1978 with Prolog. We moved to Mercury
>in 1995. We had our first commercial application written in Mercury
>deployed in 2000. It is not really ontology driven (although there is
>a DAML like model), there are no rules (à la SWRL), but there is a CPN
>at the heart. It still works after many changes, with a TCO that is 5
>times better than similar systems (Case Management). Our first ODIS
>was built experimentally in 2003 and our first commercial system
>delivered in 2006. And we are still at the beginning...
>
>The biggest issue is "Software Innovation": IT people are frighten.
>Sales and Marketing is a NP complete problem.
>
>So, it needs time, a lot of time and the capability of not deviating
>from the fundamental objective: better software requires better
>science first, sound engineering then, and real-life experimentation
>with real customers.
>
>Tough... and you don't have VC happy with a 30 year business plan.
>
>Hope it helps,
>
>Michel
>
>
>
>___________________________________________________________
>______
>Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
>Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-
>summit/
>Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
>Community Files: http://ontolog.cim3.net/file/work/OntologySummit2013/
>Community Wiki: http://ontolog.cim3.net/cgi-
>bin/wiki.pl?OntologySummit2013
>Community Portal: http://ontolog.cim3.net/wiki/

_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2013/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013
Community Portal: http://ontolog.cim3.net/wiki/

--

Michael Uschold
   Senior Ontology Consultant, Semantic Arts
   http://www.semanticarts.com
   LinkedIn: http://tr.im/limfu
   Skype, Twitter: UscholdM

_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/   
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/  
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2013/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013  
Community Portal: http://ontolog.cim3.net/wiki/     (01)