Re: [ontology-summit] {quality-methodology} Ontology Summit Track A: Met

[Amanda Vizedom <amanda.vizedom@xxxxxxxxx>]

Megan, and All,

I took the liberty of editing the subject to note the connection to the Track C session Megan mentions.

There is an enormously important issue here, critical to the connection between evaluation and quality/suitability.

Megan's question is one way of getting at it. A few other ways the issue has appeared within the Ontology Summit context: 

- When is a metric or evaluation dimension relevant to ontology quality? For some metrics, the answer might be "always." For most metrics, the answer might be "when any or all of a set of requirements applies, derived from the intended application type and context."

- What kinds of evaluation have people used, and for what purposes? When did or didn't the evaluation outcomes correlate to successful use of ontologies? 

In my experience, we sometimes get data from particular projects, but not enough to begin to form a well-grounded picture of patterns of relevance. I believe that this is partly because a comparatively small portion of ontology-based projects currently devote substantial, explicit thought to evaluation or to requirements identification, and use case characterization. And there isn't enough communication between projects, or across the broader community, for good cross-pollination and comparison to occur. Without this, when people do evaluation, they tend to simply do whatever they know how to do and have the resources to do, rather than thinking through alternatives and what evaluation is really meaningful and relevant to their particular ontology evaluation (development, selection, etc.) problem.

For the quality cross-track of last year's summit, Mike Bennett, Simon Spero and I worked on a survey (on experiences with ontology quality assurance) that was aimed at just this knowledge gap. The complexity of the question, a late start, and other factors (including, in my case and Mike's, having little experience in the hard problems of survey design) challenged us enough that although we got a survey version out the door by the end of the summit, we did not collect enough data to be meaningfully analyzed.

There have been suggestions to revise/refactor the survey for this year's summit focus and try again. I can't devote enough time to do this well, especially while also serving as Communique co-editor and working to get and keep the group library up to date. However, I think such a survey (still) would be very interesting and useful, and take us a step toward addressing this knowledge gap. 

However, if there are others (Megan?) who would be interested and willing to pitch in reviving and revising this effort, last year's material is still around.  Anyone?

Best,

Amanda

On Tue, Feb 12, 2013 at 1:20 PM, Megan Katsumi <katsumi@xxxxxxxxxxxxxxx> wrote:

Hi All -

I'd like to pose a question regarding the (quality) evaluation metrics that we saw in session 3, and potentially for any related presentations that we may see in the next session for this track:

Have any associated methodologies or guidelines been developed to address how these metrics should be applied in practice?  Specifically, I am interested in what occurs after the metrics' values have been obtained.  How are we to interpret the results of these approaches in order to make decisions about the ontology that we are designing / have designed?  Is the ontology developer expected to attempt to optimize these metrics?  If so, how?  Are there any use cases that demonstrate the value of this sort of application of evaluation metrics?

Regards,
Megan Katsumi

_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2013/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013
Community Portal: http://ontolog.cim3.net/wiki/

_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/   
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/  
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2013/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013  
Community Portal: http://ontolog.cim3.net/wiki/     (01)