[ontology-summit] Initial Track C input to Communique

["Matthew West" <matthew.west@xxxxxxxxxxxxxxxxxxxxxxxxx>]

Dear Colleagues,

Please find attached initial input from Track C for the Summit Communique. You are encouraged to review, add to and comment on this, especially if you were a contributor to one of our sessions.

Regards

Matthew West and Mike Bennett

Track C Co-Champions

A Executive Summary

B Introduction
This section should answer mainly these questions:
(1) Why is ontology evaluation important?

·       Establishing requirements (agreed between users and developers of an ontology) that an ontology needs to meet in order to meet the needs of its application means that those developing the ontology have a better chance of meeting those requirements (you can’t fail to meet unstated requirements).

·       Confirming that an ontology meets the requirements should be part of the acceptance of an ontology in a wider systems development context. There may be several stages of development and maintenance with different levels of requirements at different stages.

·       When looking to reuse rather than reinvent an ontology, an evaluation of the ontology in terms of what requirements it meets, will make it easier to identify an ontology that may be appropriately reused – in whole or in part – for some other purpose.

(2) What is the scope of this document?
We focus in the communique on the evaluation of ontologies under the following aspects
- Is the domain represented appropriately (given the requirements of the IT system)?
- Is the ontology human-intelligble?
- Is the ontology maintainable?
- Does the query/reasoning capability and performance meet the requirements of the IT system? 


C The State of the Art of Ontology Evaluation
This section should cover these topics:

(1) The terminological distinctions that we use in the rest of the text.  [all tracks]

(2) What are the desirable characteristics of ontologies and how are they measured? For each of the main kinds of ontology evaluation, it should highlight  desirable characteristics of ontologies (e.g., reusability) and measurable metrics (e.g., natural language definitions of classes and relations) linked to them. This communique should not strive for an exhaustive list, but should focus on the most important characteristics. [track A, track B]

Track C noted that for integrating ontologies, consistency was a critical property. Achieving consistency across large and potentially geographically and culturally diverse development and maintenance teams was a particular challenge in methodology development.

(3) What best practices should one adopt (across the whole ontology life cycle) to ensure that ontologies have the desired characteristics identified in C-2? Ideally this section should be organized by the characteristics mentioned in  C-2; at  a minimum there needs to be a clear correlation between the desirable characteristics and best practices. [track C]

The development process for an ontology needs to have a number of stages, just like the data model in a traditional information systems development process. Similarly requirements need to be identified in levels too, starting with the capabilities of the overall system that the ontology is a component of, to capabilities of the ontology itself in that setting, to high level requirements, like consistency, to detailed requirements, like conforming to naming standards. The ontology development needs to go through stages to match, equivlanet to conceptual, logical, and physical data model development in information systems. There are architectural decisions to be made in terms of the choices of ontological commitments the ontology needs to make and does make. There are choices of ontology language and implementation environment. There is little evidence of this in current practice, where ontology development seems to start with someone writing some OWL or CL.

(4) What tool-support is currently available to support the evaluation of the characteristics identified in C-2 and the best practices identified in C-3? Again, the point is not an exhaustive list of all available tools, but draw an explicit connection between the results of the other tracks and the findings of the tool track. [track D]

There is little or no integrated tool support for multilevel/multistage ontology development beyond some tools to directly support the development of ontologies at this physical level.

D Future Steps
What needs need to be addressed in order to improve the situation for ontology evaluation and, thus, indirectly, improve the quality of ontologies out there? This includes theoretical contributions (e.g., a better understanding of the characteristics or the development of better metrics) as well as lack of tool support. [all tracks]

·       A better understanding of the relationships between requirements at different levels and how low level requirements support higher level requirements.

·       Ontology development methodologies that align with and recognize similar stages to information systems development with distinct conceptual, logical, and physical stages, so that ontology development does not start at the physical level with the choice of an implementation language.

·       A clearer understanding of the architecture of ontology development and the different aspects of architecture that are relevant, from ontological commitments to language choices.

_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/   
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/  
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2013/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013  
Community Portal: http://ontolog.cim3.net/wiki/     (01)