Dear All,
As you all know, discussions on this mailing list are supposed to
prepare the Ontology Summit at the end of April, which is dedicated to
the effort of building an Open Ontology Repository. Barry and I -- in
our role as 'champions' of the gatekeeping/quality discussion thread --
are supposed to give a presentation which summarizes the results of our
discussions. And with less than a month left until the Ontology
Summit we should really focus on producing tangible results. Otherwise
Barry and my presentation will have a title like: 'The Ontolog Community
agrees: Before we can build an OOR we need to answer "What is truth?"
and all related metaphysical questions.' :-) (01)
Let me try to summarize the results of the discussion so far. As before
I will distinguish between gatekeeping/minimal requirements on one hand
and stewardship/ quality control on the other hand. (Just to prevent a
misunderstanding that has occurred before: "Gatekeeping" in the context
of this discussion is about excluding stuff from the repository.) (02)
== Summary: Gatekeeping/ Minimal Requirements ==
We discussed a number of candidates for minimal requirements that any
ontology has to meet in order to be accepted in the repository. This is
the list of candidates of criteria that has been suggested: (03)
1. Openness (see below)
2. /The ontology is expressed in a formal language with a well-defined
syntax./
3. /The authors of the ontology provide the required metadata. /
4. /The ontology has a clearly specified and clearly delineated scope.
/5. /Successive versions of an ontology are clearly identified.
/6. /The ontology has passed certain dynamic tests. /
7. /The ontology has unique IDs for its terms. /
8. /The ontology is adequately labeled. /
9. /The ontology has a plurality of users.
/
Some additional comments can be found at
http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2008_QualityAndGatekeeping (04)
So far the most controversial suggested criterion has been the first,
"openess". To summarize a complicated debate: we need to distinguish
between different kinds of "openness", in particular between 'open'
development processes and 'open' software licenses. Different members of
the community have different preferences on which kinds of openness and
how much openness should be required. Some, e.g. Matthew West, would
like to cancel 'openness' as a gateway criterion and rather require the
developers of ontologies to provide metadata that allows potential users
to understand how 'open' (and in which senses of the word) an ontology is.
This issue is unresolved so far. (05)
Note that none of the proposed gatekeeping criteria puts restriction on
the content of the ontologies or is concerned with the quality -- maybe
except (6), but it is hard to tell. Barry could you spell out what
you meant by "certain dynamic tests"? (06)
== Summary: Stewardship/ Quality Control==
The community agrees that it is not sufficient for the OOR just to store
ontologies, but that it needs to provide the possibility to evaluate the
ontologies within it. There is no agreement on how to evaluate
ontologies; the main strategies suggested are: (i) A market driven
approach where ontologies are reviewed by users and ranked like items on
Amazon.com; and (ii) an editorial process where ontologies are reviewed
by experts in a similar way as papers which are submitted to scientific
journals. This conflict is not surprising, since the different members
of this community are using ontologies for different purposes and thus
have different perspectives on what ontologies are and how they are
supposed to be evaluated. However, there is agreement that the OOR
should accept ontologies regardless whether their developers see
ontologies as piece of software, as a representation of scientific
knowledge, or as a standardized vocabulary. Accordingly, the OOR needs
to enable the friendly co-existence of different styles of evaluation
and different standards for ontologies. Farrukh suggested a distributed
governance model where the OOR allows for subcommunities that provide
stewardship for their ontologies and are allowed to enforce membership
criteria that are much stricter than the OOR wide gateway criteria. For
example, there could be a SUMO subcommunity that only accepts ontologies
that use SUMO as upper ontology. Or there could be an OBO Foundry
subcommunity within the OOR that requires that all its ontologies don't
overlap, are interoperable with each other, and pass the review of
experts. Each subcommunity would be free to decide which evaluation
strategy is appropriate for their ontologies. (07)
Assuming that there is consensus to follow Farrukh's suggestion we will
have to do two things: flesh out the details of how the distributed
governance model of the OOR is supposed to work; and list the various
evaluation methods that people might want to apply to ontologies. The
result will inform the discussion on metadata and the repository
architecture. Ideally, by the end of April we have settled on a set of
requirement that the members of the OOR Initiative can use when they
build the repository. (08)
== Going forward ==
In order to keep discussion focused on topics that are relevant for the
preparation of the Ontology Summit I would suggest that we break up the
discussion in little threads devoted to a single topic instead of using
one big thread. Each of these little threads should be devoted to
exactly one open question, and the idea is to stay on topic until it is
resolved or it is decided that the question needs to be addressed during
the face to face meeting. Of course, I don't want to discourage the free
flow of ideas that is so characteristic for the ontolog community, but
in my opinion the ontolog forum is a more appropriate place for that. By
the way, when you start a new thread, please, don't forget to put
"[Quality]" in front of the subject, so that other people can easily
determine that the email is part of the quality/gatekeeping discussion. (09)
Best
Fabian (010)
_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2008/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2008
Community Portal: http://ontolog.cim3.net/ (011)
|