|
At Friday's Ontology Summit organizing
committee meeting, the conversation drifted into territories
that we felt were better explored with the whole of the Summit
community. Among other things, we started discussing the
content and organization of the communiqué. The current draft
of the communiqué is here.
To bring the rest of you into the discussion
(and as penance for being one of the straying), I accepted the
charge to extract appropriate highlights and exchanges from our
chat session and post them here. We hope they will generate
some discussion on this forum, as well as suggestions about the
final form and content of the communiqué. Some of the
conversation was not recorded in the chat, and in other places,
the chat wasn't in the order of the verbal exchanges, so this
ordering is my attempt to organize the threads for consistency.
I've used {braces} around editorial additions.
Terry Longstreth
Communiqué Discussion Threads
- Style and Structure:
- The proposed structure {being
discussed; the current version is at the URL in the
opening of this note } (Todd Schneider)
- Executive Summary
- Introduction
- Why is ontology
evaluation important?
- What is the scope of
this document?
- The State of the Art of
Ontology Evaluation
- What are the
desirable characteristics of ontologies and how are they
measured?
- What best practices
should one adopt to ensure that ontologies have the
desirable characteristics in (3.1)?
- What tool-support is
currently available to support the evaluation of the
characteristics and the best practices identified in
(3.1) and (3.2)?
- Suggest Scope (section
2) focus on uses of ontologies in system(s) perspectives
(Terry Longstreth)
- Tracks' framing works
against the type of example-based, cross-cutting discussion
described {by
Michael Gruninger, verbally} No such thing will be
within Track topics A, B, or D, certainly. (Amanda Vizedom)
- We should Illustrate
evaluation approaches with real ontologies (Michael
Gruninger)
- ... in order to elicit
such example applications, the champions would need to shift
emphasis explicitly in their communications with speakers
about what's most important (e.g., push the "intrinsic"
"extrinsic" issues back. (Amanda Vizedom)
- We are torn between a
"advance ontology research" agenda and a "advance ontology's
adoption into the main stream" agenda ... the recognition
(by Fabian and Amanda) that the Communique scope does not
have to totally match the OntologySummit2013 scope would
definitely help mitigate this dissonance - and I strongly
support that (Peter Yim)
- Some are approaching this
from the 'industrial use' perspective. Others are more
concerned with research. Could we separate the communique
into two sections based on usage? (Terry Longstreth)
- We always suggest that
the ontologist(s) provide natural language descriptions of
the primitives, and examples, and references/pointers to
related discussions (e.g., the different notions of event in
the literature; or other ontologies' use or closest
approximation). Given 2 ontologies, one may have that, one
may not, and it may actually be that the one that doesn't is
the better ontology. So perhaps necessary and sufficient
conditions are better? (Leo Obrst)
- I think that identifying
those, to the extent that we have consensus on them, is a
good idea. I also think it is important to make explicit
that these are not usually enough; for any given use, an
ontology should be evaluated according to these AND
additional characteristics derived from operational
requirements. (Amanda Vizedom)
- a generally sound process
can start from the operational needs, and demonstrate the
importance of the requirements they don't meet. (Amanda
Vizedom)
- Recommendations? How
about some general notions (of evaluation) that should be
applied at the different lifecycle phases? (Todd Schneider)
- Maybe focus on
(generally) necessary and sufficient evaluation criteria at
lifecycle points.(Leo Obrst)
- We need to differentiate
between the lifecycle of the ontology and role of (1 or
more) ontologies across systems and enterprise lifecycles.
(Terry Longstreth)
- Intrinsic/Extrinsic
Dichotomy:
- Track A found that
intrinsic and extrinsic are arbitrary endpoints perhaps, but
are not worth much for gauging criteria except at the
extreme endpoints, that most everything falls in-between,
and that really nearly every criterion applies at each point
in the lifecycle, but differently. (Leo Obrst)
- Disagree with the
implication that extrinsics have no standalone value.
Black Box testing may be a necessary precursor to problem
isolation and identification. Comparing the
performance/functional attributes of
one (ontology based) system to another (without ontology)
may provide insights into the strengths or weaknesses of
the ontology based solution. (Terry Longstreth)
- I think that an
engineering-sound requirements analysis of existing
semantic apps, including many we might intuitively say are
crap, will find that there are technical "intrinsic"
requirements entailed but not met
(Amanda Vizedom)
- The most important
extrinsic property is functional / requirements
satisfaction (Terry Longstreth)
- extrinsic
properties are dependent on intrinsic properties, and
perhaps distinct ones given distinct extrinsic criteria.
E.g., consistency may not be important, or adherence to
the general requirements of a knowledge representation
language if ultimately one will use different semantics,
say in FOL, logic programming, etc. So one may not
explicitly put in disjointness constraints, etc. (Leo
Obrst)
|
_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2013/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013
Community Portal: http://ontolog.cim3.net/wiki/ (01)
|