All, it was suggested that we submit our initial draft Communique statements (per Track) to the entire Ontology Summit 2013 list, for possible wider discussion. As you may know, we Track Co-Champions are trying
to synthesize our Track contributions this week in order to provide input to the Communique editors (Amanda Vizedom and Fabian Neuhaus), who are trying to consolidate a lot of material based on our virtual sessions, to complete the final Communique.
This is only a first cut of the Track A: Intrinsic Aspects of Ontology Evaluation, and we will be adding to it this week.
Thanks and Happy Easter!
Steve and Leo
Folks,
Steve and I had limited time to work on this this past week, so this is still just an initial draft. We promise to have more next week, especially after any subsequent bashing. ;)
We have a Word doc version of this, but are including it below as html:
Thanks,
Leo and Steve
=====
Ontology Summit 2013 Track A: Intrinsic Aspects of Ontology Evaluation
Steve Ray, Leo Obrst
Towards a Synthesis for the Communiqué
Comments for Introduction (Section B)
(2) What is the scope of this document?
This document has as scope the dimensions of ontology evaluation, methods, criteria, and the properties to measure to ensure better quality ontologies.
Intrinsic Aspects of Ontology Evaluation
Ontologies are built to solve problems, and ultimately an ontology’s worth can be measured by the effectiveness with which it helps in solving a particular problem. Nevertheless, as a designed artifact, there are a number of intrinsic characteristics
that can be measured for any ontology that give an indication of how “well-designed” it is. Examples include the proper use of various relations found within an ontology, proper separation of concepts and facts (sometimes referred to as classes vs. instance
distinctions), proper handling of data type declarations, embedding of semantics in naming (sometimes called “optimistic naming”), inconsistent range or domain constraints, better class/subclass determination, the use of principles of ontological analysis,
and many more.
We focus in the communiqué on the evaluation of ontologies under the following intrinsic aspects:
-
Is the ontology free of obvious inconsistencies and errors in modeling?
-
Is the ontology structurally sound? How do we gauge that?
-
Is the ontology appropriately modular?
-
Is the ontology designed and implemented according to sound principles of logical, semantic, and ontological analysis?
-
Which intrinsic aspects of ontology evaluation are of greater value to downstream extrinsic ontology evaluation?
Section C (2) What are the desirable characteristics of ontologies? Do they depend on the intended use of the ontology? And how are they measured over the life cycle of the ontology?
1)
Partitioning the Ontology Evaluation Space:
<image003.jpg>
a.
Intrinsic Evaluation Aspects: Intrinsic ontology evaluation, from our perspective, consists of two parts: Structural Intrinsic Evaluation and Domain Intrinsic Evaluation.
Structural Intrinsic Evaluation: Ontology evaluation that does not depend at all on knowledge of the domain being modeled, but does draw upon mathematical and logical properties such as graph-theoretic connectivity,
logical consistency, model-theoretic interpretation issues, inter-modularity mappings and preservations, etc. Structural properties such as branching factor, density, counts of ontology constructs, averages, and the like are intrinsic. Some meta-properties
such as transitivity, symmetry, reflexivity, and equivalence may also figure in intrinsic notions.
In general, structural intrinsic criteria are focused only on domain-independent notions, mostly structural, and those based on the knowledge representation language.
Some examples of tools and methodologies that address intrinsic ontology evaluation:
I. Oops! Evaluation web site at
http://oeg-lia3.dia.fi.upm.es/oops/index-content.jsp and described by MariaPovedaVillalon
II. OntoQA to develop metrics for any ontology based on structural properties and instance populations, described by SamirTartir
III. PatrickLambrix’s debugging of Isa-a taxonomic structures, especially with mappings between ontologies
Domain Intrinsic Evaluation: Evaluation where some understanding of the domain is needed in order to, for example, determine that a particular modeling construct is in alignment with the reality it is supposed
to model. It may be that some meta-properties such as rigidity, identity, unity, etc., suggested by metaphysics, philosophical ontology, semantics, and philosophy of language are used to gauge the quality of the axioms of the ontology, including e.g., the
subclass/isa taxonomic backbone of the ontology and other structural aspects of the ontology.
Most of the aspects of this category focus on ontological content methods such as better ontological and semantic analysis, including meta-property analysis (such as provided by methodologies like OntoClean, etc.)
Domain knowledge and better ways to represent that knowledge do come into play here, though divorced as much as possible from application-specific domain requirements that come more explicitly from extrinsic evaluation
issues. At the extrinsic edge of domain intrinsic evaluation, the context-independent measures from Structural Intrinsic evaluation begin to blend into the very context-dependent, application issues of Extrinsic evaluation.
Some examples of tools and methodologies that address domain intrinsic ontology evaluation:
I. OQuaRE framework described by AstridDuqueRamos
II. OntoClean (Guarino and Welty)
III. MariaCopeland: Ontology Evolution and Regression Testing
IV. MelissaHaendel: Ontology Utility from a biological viewpoint
V. EdBarkmeyer: Issues with mapping vocabularies (especially code-lists) to ontologies.
b.
Extrinsic Evaluation Aspects: Ontology evaluation where the structure and design of the ontology is opaque to the tester, and the evaluation is determined by the correctness of answers to various interrogations of the model. In
general, application requirements and domain requirements that are specifically needed by particular applications are the focus of extrinsic evaluation.
2) Evaluation Across the Ontology Lifecycle
Every criterion should be evaluated at each point in the ontology lifecycle, but with some criteria being more important (necessary/sufficient) at some points more than others. Therefore, a better ontology evaluation methodology might define
necessary and sufficient criteria (and their measures) derived from both intrinsic and extrinsic aspects that apply to different points in the ontology lifecycle.
In addition, the determination of these necessary or sufficient criteria may be subject to constraints: for example, though initially an intrinsic criterion of logical consistency of the ontology may be imposed as a necessary property at
the beginning of the first phase of ontology development, it might be relaxed subsequently when it is determined that a different semantics will apply in how the ontology is interpreted within a given application (e.g., if the application-specific reasoning
will not observe the full FOL or description logic Open World Assumption, but instead interpret the ontology under a Closed World Assumption).
[ASIDE: also note that we could insert a lifecyle or suggested lifecycle graphic at this point.]