Re: [ontology-summit] Ontology Summit 2013

[Amanda Vizedom <amanda.vizedom@xxxxxxxxx>]

+1

Well put, Matthew.

And is that focus that allows evaluation to be both a narrow enough focus and one that brings key other topics, but only insofar as they make a real difference to evaluation. 

The summit can start from a position that is neutral with respect to many subjects that are bottomless-cans-of-worms when approached without such a framework. We can work to support discussion that puts *requirements* at center. 

I'd like to add that since evaluation of quality, in this sense, encompasses whatever requirements have been identified, it can encompass both validation and verification, both global and local requirements. What matters is whether they have been identified, through a careful gathering and analysis, as requirements in a particular case. That's why "evaluation across the lifecycle" is a better defined topic than "evaluation" or "quality" without qualification. 

It will be important to pay attention to that requirements identification and analysis bit early and often, and to think in terms of a range of lifecyle exemplars.

Amanda

On Sat, Dec 8, 2012 at 4:29 AM, Matthew West <dr.matthew.west@xxxxxxxxx> wrote:

Dear John,

This is the right question.

> What is the purpose of an ontology?

MW: My answer to this is that an ontology (computer science artefact) is an
information system, it both is a repository of information and you can query
it to get the answers to questions.

MW: So now we can consider the purpose of information, which in business, is
to support decisions taking (information can also be entertainment or some
combination of the two). Better information leads to better decisions. Where
decisions appear in the processes of an enterprise.

MW: So now we have a basis for evaluating our ontologies. Do they provide
(some of) the information that meets the requirements to support a decision.
Of course there may be more than one decision that the ontology supports,
and this answers the question of scope. The (intended) scope of an ontology
is the set of decisions it is designed to support.

We can now evaluate an ontology in terms of the quality of the information
it provides. Here quality is not better or worse, but meeting agreed
requirements. An evaluation would be around the effectiveness and efficiency
with which these requirements are met.

Some properties that you might consider that determine effectiveness of
information include:
- Relevance - is the information relevant to the decision at hand
- Clarity - is the meaning of the information clear
- Accuracy - how close to the truth is the information?
- Completeness/timeliness - is all the information available when the
decision needs to be made?

Some properties that affect efficiency include:
- Cost - how much does the information cost to provide?
- consistency - is the same thing referred to in the same way? (otherwise
there are costs and timeliness issues of reconciliation)

> But focused on what?  Any kind of comparison, including evaluation,
> implies a criterion of some kind:  "X is better than Y according to
> some criterion Z."
>
> What criterion Z makes one ontology better than another?

MW: Therefore one ontology is better than another if it meets the
information requirements where the other does not. On the other hand, one
ontology is not better than another if it exceeds the information
requirements. Also if two ontologies meet the information requirements, then
one is better than another if it does so more efficiently, which usually
means at lower cost.

MW: One of the things that falls out from this is that an ontology is not of
poor quality if it fails to meet requirements that were not part of its
specification (you cannot fail to meet unstated requirements). This is often
one of the criticisms of ontologies (it's a bad ontology because it was
designed to do this, but it can't meet my requirement which is that). There
is no reason to suppose an ontology will meet wider requirements than it was
designed to meet, so if you intend it to meet broad requirements, that had
better be part of its design.

MW: So in essence it is all a matter of quality (fitness for purpose).

Regards

Matthew West
Information  Junction
Tel: +44 1489 880185
Mobile: +44 750 3385279
Skype: dr.matthew.west
matthew.west@xxxxxxxxxxxxxxxxxxxxxxxxx
http://www.informationjunction.co.uk/
http://www.matthew-west.org.uk/

This email originates from Information Junction Ltd. Registered in England
and Wales No. 6632177.
Registered office: 2 Brookside, Meadow Way, Letchworth Garden City,
Hertfordshire, SG6 3JE.

_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2013/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013
Community Portal: http://ontolog.cim3.net/wiki/

_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/   
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/  
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2013/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013  
Community Portal: http://ontolog.cim3.net/wiki/     (01)