ontology-summit
[Top] [All Lists]

Re: [ontology-summit] [Quality] What means

To: Ontology Summit 2008 <ontology-summit@xxxxxxxxxxxxxxxx>
From: Bill Bug <wbug@xxxxxxxxxxxxxx>
Date: Fri, 21 Mar 2008 02:39:51 -0400
Message-id: <CF232AD8-6B46-4827-864C-115F2CA46525@xxxxxxxxxxxxxx>
On quick point I left out - I would not expect the "gatekeeper" to exclude any particular ontology from an "open" repository.  Rather - as others have said - the gatekeepers would help to establish and populate the required metadata needed to provide for the vetting capability.  In this way, the gatekeeper function we are discussing here is very much different from the process of peer review, where those manuscripts that don't pass muster never end up in the repository.

On Mar 21, 2008, at 2:35 AM, Bill Bug wrote:
Mark is making a very import point regarding the problematic nature of the analogy between journal peer reviewers and OOR gatekeepers.  To some extent this is an apples to oranges comparison, despite the apparent similarity in the gatekeeper function.

Having said that, I do believe there is another function journal reviewers provide which is very relevant to what I'd hope to see provided by an OOR gatekeeper - at least when it comes to ontologies used to help add reliable, highly-scalable automation to bioinformatic data processing.  To go back to the realm of advertising, this other gatekeeper role relates to the Google Scholar "slogan" - "Stand on the shoulders of giants".  The idea is the peer review process in the literature helps provide a certain level of distillation and reliable vetting to ensure every researcher need only read, absorb, and collate vetted manuscripts, as opposed to all the manuscripts submitted to the relevant journals.  As imperfect as this process, I still tend to transfer my confidence in the reviewers having competently vetted the articles to the articles themselves, so that I can accept - at a given point in time - the validity of the assertions in a given article and build on them by performing additional work on their "shoulders".

Perhaps - as others have suggested - it is too early in the evolution of ontology development practice for us to expect we can produce a vetting process capable of functioning in this way, but until we do, some of the expectations informaticists - and funding agencies - have for the use of ontologies are probably not achievable. Until there is a reliable vetting procedure, we cannot expect to re-use and extend existing ontologies effectively or with confidence for the purpose of bringing like data together in novel ways from across the biomedical data diaspora.  Without vetting, we cannot expect to provide other developers with clear advise on what are the reliable ontological shoulders to build on.  If the OOR has 3 ontologies covering a domain of interest at roughly the same scope and level of granularity such as the physiology of mammalian electrolyte balance or the assembly of peptides into functional multimeric macromolecular receptor complexes, how would a bioinformatics application developer determine which one to use? Even more importantly, if users pick at random from amongst the 2 or more ontologies covering the same domain, who will maintain the maps and software required to make deductions or inferences across the annotated data repositories which use these different ontologies to cover the same domain?

Another one of my expectations for using ontologies in biomedical informatics is as the data representation gets richer and more expressive, the nature of the software each application developer needs create can focus more on application-specific issues.  Community tools capable of parsing the expressivity (reasoners) can provide more of the "smarts", so that the custom tools don't need to hard code it - and can exclusively focus on the application specific algorithms.

This partly goes back to a point I made earlier in this thread.  Using ontologies to power broadly scoped, federated inferencing brings with it a distinct set of requirements that differ from those of applications focused on providing decision support built on a more narrowly focussed data warehouse.  In that case, it can be perfectly acceptable for developers to pick the ontology they like the best from the several covering the domain(s) they require, so long as there is no expectation the resulting knowledge repository will be easily shared with other informatics projects. 

These may all be expectations so narrowly linked to biomedical informatics they do not hold sway in a more generic OOR.  That's one of the issues I'm hoping to better understand by participating in this discussion and the upcoming summit.

Thanks again to all for the stimulating and detailed dissection of this important topic.

Cheers,
Bill


On Mar 21, 2008, at 1:08 AM, Mark Musen wrote:
On Mar 20, 2008, at 8:56 PM, John F. Sowa wrote:
There are two independent issues here:  reviewing and publishing.
Everybody would agree that reviewing is important, but ideally,
the readers/users should have the option of making their own
choices based on the reviews.  When publication was expensive,
the publishers became gatekeepers because it was economically
impractical to publish everything.

The analogy between peer review of journal articles and peer review of  
ontologies has been applied too glibly, I believe.

The best reviewers of a journal article are scientists who can  
evaluate the methods described in the paper, judge whether the data  
presented are plausibly consistent with the methods, and assess  
whether the authors' interpretations of the data are reasonable.  This  
process is all done rather well by scientists who are experts in the  
field and who can understand the work that is described in the paper.   
Although the system does break down, sometimes in notorious ways, it  
generally works rather well.

Ontologies are not journal articles.  Although there are many surface- 
level distinctions that can be assessed purely by inspection (OBO- 
Foundry criteria regarding representation language, namespaces,  
textual definitions, and so on), the key question one wants answered  
before using an ontology concerns whether the ontology makes the right  
distinctions about the domain being modeled.  This question cannot be  
answered by inspection of the ontology; it can be answered only by  
application of the ontology to some set of real-world problems and  
discovering where things break down.  The people best suited for  
making the kinds of assessment that are needed are not necessarily the  
best experts in the field, but the mid-level practitioners who  
actually do the work.  Any effective system of peer review has got to  
capture the opinions of ontology users, and not just those of renowned  
subject-matter experts or of curators.

I think ontologies are much more like refrigerators than they are like  
journal articles.  I view ontologies as artifacts.  Not surprisingly,  
I am much more interested in the opinions of people who actually use  
refrigerators than I am of experts in thermodynamics, product  
manufacturing, or mechanical engineering.  The latter are people who  
can inspect a particular refrigerator very carefully for surface-level  
flaws, but who may have no first-hand knowledge of what happens when  
you actually plug it in.

Mark

_________________________________________________________________
Community Portal: http://ontolog.cim3.net/


William Bug, M.S., M.Phil.                                          email: wbug@xxxxxxxxxxxxxx
Ontological Engineer work: (610) 457-0443
Biomedical Informatics Research Network (BIRN)
and
National Center for Microscopy & 
Imaging Research (NCMIR)
Dept. of Neuroscience, School of Medicine
University of California, San Diego
9500 Gilman Drive
La Jolla, CA 92093

Please note my email has recently changed




William Bug, M.S., M.Phil.                                          email: wbug@xxxxxxxxxxxxxx
Ontological Engineer work: (610) 457-0443
Biomedical Informatics Research Network (BIRN)
and
National Center for Microscopy & 
Imaging Research (NCMIR)
Dept. of Neuroscience, School of Medicine
University of California, San Diego
9500 Gilman Drive
La Jolla, CA 92093

Please note my email has recently changed




_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/ 
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/  
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2008/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2008 
Community Portal: http://ontolog.cim3.net/    (01)
<Prev in Thread] Current Thread [Next in Thread>