Re: [ontology-summit] FW: [ontology-summit-org] OntologySummit2013: Comm

[joseph simpson <jjs0sbw@xxxxxxxxx>]

This is an interesting organization of concepts.

The establishment of the two categories: Structural Intrinsic Evaluation and Domain Intrinsic Evaluation, suggests that there exists a set of structural components associated with an ontology that are domain independent.

The structural category lists mathematical and logical properties as the objects to be found in this category. 

Meta-properties such as transitivity, symmetry, reflexivity, and equivalence are singled out for special comment.

The "meta-properties" are mathematical and logical properties.

It may be a good idea to make a amplify the distinction between "meta-properties"  and "mathematical and logical properties."

Have fun,

Joe

On Sun, Mar 31, 2013 at 12:48 PM, Obrst, Leo J. <lobrst@xxxxxxxxx> wrote:

All, it was suggested that we submit our initial draft Communique statements (per Track) to the entire Ontology Summit 2013 list, for possible wider discussion.  As you may know, we Track Co-Champions are trying to synthesize our Track contributions this week in order to provide input to the Communique editors (Amanda Vizedom and Fabian Neuhaus), who are trying to consolidate a lot of material based on our virtual sessions, to complete the final Communique.

 

This is only a first cut of the Track A: Intrinsic Aspects of Ontology Evaluation, and we will be adding to it this week.

 

Thanks and Happy Easter!

 

Steve and Leo

 

From: ontology-summit-org-bounces@xxxxxxxxxxxxxxxx [mailto:ontology-summit-org-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Obrst, Leo J.
Sent: Friday, March 29, 2013 6:11 PM
To: Ontology Summit 2013 Organizing Committee
Subject: [ontology-summit-org] OntologySummit2013: Communique - Track A (draft) Contribution

 

Folks,

 

Steve and I had limited time to work on this this past week, so this is still just an initial draft. We promise to have more next week, especially after any subsequent bashing. ;)

 

We have a Word doc version of this, but are including it below as html:

 

Thanks,

Leo and Steve

 

=====

 

Ontology Summit 2013 Track A: Intrinsic Aspects of Ontology Evaluation

Steve Ray, Leo Obrst

 

Towards a Synthesis for the Communiqué

 

Comments for Introduction (Section B)

(2) What is the scope of this document?
This document has as scope the dimensions of ontology evaluation, methods, criteria, and the properties to measure to ensure better quality ontologies.

 

Intrinsic Aspects of Ontology Evaluation

Ontologies are built to solve problems, and ultimately an ontology’s worth can be measured by the effectiveness with which it helps in solving a particular problem. Nevertheless, as a designed artifact, there are a number of intrinsic characteristics that can be measured for any ontology that give an indication of how “well-designed” it is. Examples include the proper use of various relations found within an ontology, proper separation of concepts and facts (sometimes referred to as classes vs. instance distinctions), proper handling of data type declarations, embedding of semantics in naming (sometimes called “optimistic naming”), inconsistent range or domain constraints, better class/subclass determination, the use of principles of ontological analysis, and many more.

 

We focus in the communiqué on the evaluation of ontologies under the following intrinsic aspects:

-        Is the ontology free of obvious inconsistencies and errors in modeling?

-        Is the ontology structurally sound? How do we gauge that?

-        Is the ontology appropriately modular?

-        Is the ontology designed and implemented according to sound principles of logical, semantic, and ontological analysis?

-        Which intrinsic aspects of ontology evaluation are of greater value to downstream extrinsic ontology evaluation?

 

Section C (2) What are the desirable characteristics of ontologies? Do they depend on the intended use of the ontology? And how are they measured over the life cycle of the ontology?

 

1)     Partitioning the Ontology Evaluation Space:

a.      Intrinsic Evaluation Aspects: Intrinsic ontology evaluation, from our perspective, consists of two parts: Structural Intrinsic Evaluation and Domain Intrinsic Evaluation.

 

Structural Intrinsic Evaluation: Ontology evaluation that does not depend at all on knowledge of the domain being modeled, but does draw upon mathematical and logical properties such as graph-theoretic connectivity, logical consistency, model-theoretic interpretation issues, inter-modularity mappings and preservations, etc. Structural properties such as branching factor, density, counts of ontology constructs, averages, and the like are intrinsic. Some meta-properties such as transitivity, symmetry, reflexivity, and equivalence may also figure in intrinsic notions. 

 

In general, structural intrinsic criteria are focused only on domain-independent notions, mostly structural, and those based on the knowledge representation language.

 

Some examples of tools and methodologies that address intrinsic ontology evaluation:

                                     I.          Oops! Evaluation web site at http://oeg-lia3.dia.fi.upm.es/oops/index-content.jsp and described by MariaPovedaVillalon

                                   II.          OntoQA to develop metrics for any ontology based on structural properties and instance populations, described by SamirTartir

                                  III.          PatrickLambrix’s debugging of Isa-a taxonomic structures, especially with mappings between ontologies

 

Domain Intrinsic Evaluation: Evaluation where some understanding of the domain is needed in order to, for example, determine that a particular modeling construct is in alignment with the reality it is supposed to model. It may be that some meta-properties such as rigidity, identity, unity, etc., suggested by metaphysics, philosophical ontology, semantics, and philosophy of language are used to gauge the quality of the axioms of the ontology, including e.g., the subclass/isa taxonomic backbone of the ontology and other structural aspects of the ontology.  

 

Most of the aspects of this category focus on ontological content methods such as better ontological and semantic analysis, including meta-property analysis (such as provided by methodologies like OntoClean, etc.)

 

Domain knowledge and better ways to represent that knowledge do come into play here, though divorced as much as possible from application-specific domain requirements that come more explicitly from extrinsic evaluation issues. At the extrinsic edge of domain intrinsic evaluation, the context-independent measures from Structural Intrinsic evaluation begin to blend into the very context-dependent, application issues of Extrinsic evaluation.

 

Some examples of tools and methodologies that address domain intrinsic ontology evaluation:

                                     I.          OQuaRE framework described by AstridDuqueRamos

                                   II.          OntoClean (Guarino and Welty)

                                  III.          MariaCopeland: Ontology Evolution and Regression Testing

                                  IV.          MelissaHaendel: Ontology Utility from a biological viewpoint

                                   V.          EdBarkmeyer: Issues with mapping vocabularies (especially code-lists) to ontologies.

 

b.      Extrinsic Evaluation Aspects: Ontology evaluation where the structure and design of the ontology is opaque to the tester, and the evaluation is determined by the correctness of answers to various interrogations of the model. In general, application requirements and domain requirements that are specifically needed by particular applications are the focus of extrinsic evaluation.

 

2)  Evaluation Across the Ontology Lifecycle

 

Every criterion should be evaluated at each point in the ontology lifecycle, but with some criteria being more important (necessary/sufficient) at some points more than others. Therefore, a better ontology evaluation methodology might define necessary and sufficient criteria (and their measures) derived from both intrinsic and extrinsic aspects that apply to different points in the ontology lifecycle.

 

In addition, the determination of these necessary or sufficient criteria may be subject to constraints: for example, though initially an intrinsic criterion of logical consistency of the ontology may be imposed as a necessary property at the beginning of the first phase of ontology development, it might be relaxed subsequently when it is determined that a different semantics will apply in how the ontology is interpreted within a given application (e.g., if the application-specific reasoning will not observe the full FOL or description logic Open World Assumption, but instead interpret the ontology under a Closed World Assumption).

 

 

[ASIDE: also note that we could insert a lifecyle or suggested lifecycle graphic at this point.]

_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2013/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013
Community Portal: http://ontolog.cim3.net/wiki/

--
Joe Simpson

Sent From My DROID!!

_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/   
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/  
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2013/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013  
Community Portal: http://ontolog.cim3.net/wiki/     (01)