ontology-summit
[Top] [All Lists]

Re: [ontology-summit] [Quality] What means

To: Ontology Summit 2008 <ontology-summit@xxxxxxxxxxxxxxxx>
From: Mark Musen <musen@xxxxxxxxxxxx>
Date: Thu, 20 Mar 2008 22:08:28 -0700
Message-id: <CDD70C50-C4C6-4169-85C6-AA7053EFC4E1@xxxxxxxxxxxx>
On Mar 20, 2008, at 8:56 PM, John F. Sowa wrote:
> There are two independent issues here:  reviewing and publishing.
> Everybody would agree that reviewing is important, but ideally,
> the readers/users should have the option of making their own
> choices based on the reviews.  When publication was expensive,
> the publishers became gatekeepers because it was economically
> impractical to publish everything.    (01)

The analogy between peer review of journal articles and peer review of  
ontologies has been applied too glibly, I believe.    (02)

The best reviewers of a journal article are scientists who can  
evaluate the methods described in the paper, judge whether the data  
presented are plausibly consistent with the methods, and assess  
whether the authors' interpretations of the data are reasonable.  This  
process is all done rather well by scientists who are experts in the  
field and who can understand the work that is described in the paper.   
Although the system does break down, sometimes in notorious ways, it  
generally works rather well.    (03)

Ontologies are not journal articles.  Although there are many surface- 
level distinctions that can be assessed purely by inspection (OBO- 
Foundry criteria regarding representation language, namespaces,  
textual definitions, and so on), the key question one wants answered  
before using an ontology concerns whether the ontology makes the right  
distinctions about the domain being modeled.  This question cannot be  
answered by inspection of the ontology; it can be answered only by  
application of the ontology to some set of real-world problems and  
discovering where things break down.  The people best suited for  
making the kinds of assessment that are needed are not necessarily the  
best experts in the field, but the mid-level practitioners who  
actually do the work.  Any effective system of peer review has got to  
capture the opinions of ontology users, and not just those of renowned  
subject-matter experts or of curators.    (04)

I think ontologies are much more like refrigerators than they are like  
journal articles.  I view ontologies as artifacts.  Not surprisingly,  
I am much more interested in the opinions of people who actually use  
refrigerators than I am of experts in thermodynamics, product  
manufacturing, or mechanical engineering.  The latter are people who  
can inspect a particular refrigerator very carefully for surface-level  
flaws, but who may have no first-hand knowledge of what happens when  
you actually plug it in.    (05)

Mark    (06)

_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/ 
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/  
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2008/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2008 
Community Portal: http://ontolog.cim3.net/    (07)
<Prev in Thread] Current Thread [Next in Thread>