On Mar 20, 2008, at 8:56 PM, John F. Sowa wrote:
> There are two independent issues here: reviewing and publishing.
> Everybody would agree that reviewing is important, but ideally,
> the readers/users should have the option of making their own
> choices based on the reviews. When publication was expensive,
> the publishers became gatekeepers because it was economically
> impractical to publish everything. (01)
The analogy between peer review of journal articles and peer review of
ontologies has been applied too glibly, I believe. (02)
The best reviewers of a journal article are scientists who can
evaluate the methods described in the paper, judge whether the data
presented are plausibly consistent with the methods, and assess
whether the authors' interpretations of the data are reasonable. This
process is all done rather well by scientists who are experts in the
field and who can understand the work that is described in the paper.
Although the system does break down, sometimes in notorious ways, it
generally works rather well. (03)
Ontologies are not journal articles. Although there are many surface-
level distinctions that can be assessed purely by inspection (OBO-
Foundry criteria regarding representation language, namespaces,
textual definitions, and so on), the key question one wants answered
before using an ontology concerns whether the ontology makes the right
distinctions about the domain being modeled. This question cannot be
answered by inspection of the ontology; it can be answered only by
application of the ontology to some set of real-world problems and
discovering where things break down. The people best suited for
making the kinds of assessment that are needed are not necessarily the
best experts in the field, but the mid-level practitioners who
actually do the work. Any effective system of peer review has got to
capture the opinions of ontology users, and not just those of renowned
subject-matter experts or of curators. (04)
I think ontologies are much more like refrigerators than they are like
journal articles. I view ontologies as artifacts. Not surprisingly,
I am much more interested in the opinions of people who actually use
refrigerators than I am of experts in thermodynamics, product
manufacturing, or mechanical engineering. The latter are people who
can inspect a particular refrigerator very carefully for surface-level
flaws, but who may have no first-hand knowledge of what happens when
you actually plug it in. (05)
Mark (06)
_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2008/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2008
Community Portal: http://ontolog.cim3.net/ (07)
|