ontology-summit
[Top] [All Lists]

Re: [ontology-summit] [Quality] Gatekeeping: Democratic Ranking vs. Peer

To: Ontology Summit 2008 <ontology-summit@xxxxxxxxxxxxxxxx>, Ontology Summit 2008 <ontology-summit@xxxxxxxxxxxxxxxx>
From: Barry Smith <phismith@xxxxxxxxxxx>
Date: Fri, 21 Mar 2008 11:55:48 -0400
Message-id: <20080321155843.INRB22170.mta13.adelphia.net@xxxxxxxxxxxxxxxxxxxxxxx>
At 11:28 AM 3/21/2008, Holger Lewen wrote:
>HL
> >> Why do people invest time here to discuss this matter? Because some
> >> people dedicate their time to deal with such problems. Also, if the
> >> impact becomes high enough, you can give jobs to your best reviewers,
> >> thereby making it worth their while.
> >    (01)


BS
> > Try taking this to a university Dean:
> >
> > Give this man a job; he posted 4000 comments to
> > the NeON ontology ranking system just last week
> > alone, and his reviews came top in 79% of cases!
> >
> > Try convincing a peer review commitee of a
> > funding agency to accept this as part of the
> > argument why a project should be funded.    (02)

HL
>Since you suggest doing something similar for your closed reviewing
>system (having people that allocate some part of their funded time to
>assess the quality of ontologies for your registry), I do not see why
>this should not work.    (03)

BS
We both face a chicken and egg problem. We will see empirically which 
approach survives. Perhaps both.    (04)

HL
>Your reviewers would also have to justify
>spending resources on reviewing the ontologies, and thus it will cost
>some funding agencies money. If the open review system gives birth to
>a brilliant reviewer, who emerges from the crowd and was not part of
>your team before, I do not see why funding a reviewer selected by you
>should be rather funded than a reviewer that 79% of the users of the
>system think of as a good reviewer.    (05)

BS
You do not see the thorny nature of the chicken and egg problem you 
are facing, I'm afraid.    (06)

HL
>  I also think that no branding is
>needed, so I would rather see an open system where everyone could
>collaborate than a closed and branded NeOn system. If we start having
>competing systems, it will be harder for either system to gather the
>necessary reviews.
>    (07)

BS
Science thrives on competition.
...    (08)

> >    (09)


BS
> >>> Crudely put: experts are motivated to review
> >>> ontologies in their relevant domains of expertise
> >>> because they get career-related credit for serving as reviewers.    (010)


HL
> >> As they could in an open review system, given it were as accepted as
> >> the closed system. Being the top reviewer in such an open system
> >> might
> >> even prove to provide more credibility, since not only your peers but
> >> also a large community of users agree with your assessment.
> >    (011)

BS
> > People like this, who have some knowledge of
> > biology, would indeed almost certainly be invited
> > to join the OBO Foundry review process.    (012)


HL
>I guess finding them will be more difficult in your scenario,    (013)

BS
We are not facing problems in this regard. But in any case, we have 
no objections to drawing on the fruits of your scenario.\    (014)

BS
>The open
> > process favored by you and Mark would, I believe,
> > face more formidable difficulties in managing two
> > sets of moving targets (ontologies, ranking clouds).    (015)


HL
>If there is a solution that works for your scenario, I do not see why
>it should not be adaptable to our scenario. Since your reviewing
>process also consists of multiple reviews per ontology, it seems that
>the solution could be applied rather similarly.    (016)

BS
We will see, empirically.    (017)

> >    (018)

BS
> >
> >>> The publishing
> >>> process we have in mind will have the incidental
> >>> advantage that it will allow the multiple
> >>> developers typically involved in serious ontology
> >>> endeavors to get appropriate credit, which they
> >>> can use to justify spending the time and effort involved.
> >>>    (019)

HL
> >> Again, I do not see why this should not be possible in an open
> >> reviewing system. If your ontology gets good acceptance there, it
> >> should count at least as much.
> >    (020)


BS
> > See above, re: getting a job. Do we wish ontology
> > to become one day a professional endeavor,
> > involving such things as levels of expertise? Or
> > do we wish ontology to remain a cottage industry?
> >    (021)

HL
>It seems the recent development in the IT world go towards utilizing
>the wisdom of the masses. I think the problem you mention about
>perception of the work by funding agencies is just a frame of mind. If
>the open review system becomes accepted, work done there should count
>no less than work done for a closed reviewing system.    (022)

BS
Touching optimism, again.
'Should' does not imply 'will'.    (023)


BS
> >>> Both reviewers and developers will be further
> >>> motivated to participate in this process because
> >>> they can thereby directly influence the standard
> >>> set of ontology resources which will be available
> >>> to them, thereby also motivating the creation of:
> >>> related ontology-based software, useful bodies of
> >>> data annotated in terms of these resources, etc.
> >>>    (024)

HL
> >> Again, in my opinion also possible in an open reviewing system.    (025)


You can't have it both ways -- on the one hand you claim multiple 
benefits from having lots of reviewers, lots of perspectives, people 
can find precisely the ontology which will suit their purposes from 
the wisdom of crowds, etc.
Now, you say that lone reviewers lost in these crowds will be able 
'to directly influence the standard set of ontology resources which 
will be available to them'.    (026)


BS
> > Good journals allow anyone to submit.
> > Also the system of journal publishing which we
> > take as our starting-point provides a rather
> > elegant way to divide up the potentially
> > unlimited domain that is available for ontology
> > development into limited disciplines and subdisciplines.
> >    (027)

HL
>Yes, they do, but this is because it still results in limited
>submissions. If everybody would start submitting to journals, it would
>only take short time until there would be some kind of restriction
>(e.g. only people working in science may submit).
>I agree that your approach is valuable if taken one discipline at a
>time with dedicated reviewers for that discipline. This is again a
>question of scale. How do you determine with which subcategory to
>start. If I have the best possible ontology for a subdomain you are
>not considering yet, what do I do? It might still be extremely
>valuable for the community to see my ontology straight away.
>
BS
Register it immediately with the OBO Foundry!
This will help people to find it.
You will benefit from expert review.
If there is already an OBO Foundry ontology available for your topic, 
you can start working on a merger, to reap the benefits listed by 
Bill (post copied below).
What, after all, can you lose?    (028)

BS    (029)

Delivered-To: phismith@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Received: (qmail 15153 invoked from network); 21 Mar 2008 06:36:02 -0000
Received: from unknown (HELO mailscan4.acsu.buffalo.edu) (128.205.6.136)
   by mail3 with SMTP; 21 Mar 2008 06:36:02 -0000
Received: (qmail 21960 invoked by uid 26149); 21 Mar 2008 06:36:01 -0000
Delivered-To: phismith@xxxxxxxxxxx
Received: (qmail 21829 invoked from network); 21 Mar 2008 06:36:00 -0000
Received: from ns3.cim3.com (HELO mccarthy.cim3.com) (64.62.192.10)
    by front2.acsu.buffalo.edu with SMTP; 21 Mar 2008 06:36:00 -0000
Received: from ns3.cim3.com (localhost.localdomain [127.0.0.1])
         by mccarthy.cim3.com (Postfix) with ESMTP id DB556108B62;
         Thu, 20 Mar 2008 23:35:59 -0700 (PDT)
X-Original-To: ontology-summit@xxxxxxxxxxxxxxxx
Delivered-To: ontology-summit@xxxxxxxxxxxxxxxxx
Received: from mail3.sea5.speakeasy.net (mail3.sea5.speakeasy.net
         [69.17.117.5])
         by mccarthy.cim3.com (Postfix) with ESMTP id 0C12E108B62
         for <ontology-summit@xxxxxxxxxxxxxxxx>;
         Thu, 20 Mar 2008 23:35:57 -0700 (PDT)
Received: (qmail 24930 invoked from network); 21 Mar 2008 06:35:57 -0000
Received: from vpn137-110-192-21.ucsd.edu (HELO [192.168.1.101])
         (bbug@[137.110.192.21]) (envelope-sender <wbug@xxxxxxxxxxxxxx>)
         by mail3.sea5.speakeasy.net (qmail-ldap-1.03) with SMTP
         for <ontology-summit@xxxxxxxxxxxxxxxx>; 21 Mar 2008 06:35:53 -0000
Mime-Version: 1.0 (Apple Message framework v753)
In-Reply-To: <CDD70C50-C4C6-4169-85C6-AA7053EFC4E1@xxxxxxxxxxxx>
References: <52043.1206018194@xxxxxxxxxxx>
         <p06230900c40844a2ffe4@[192.168.1.2]>
         <47E331DC.1060709@xxxxxxxxxxx>
         <CDD70C50-C4C6-4169-85C6-AA7053EFC4E1@xxxxxxxxxxxx>
Message-Id: <5E8366C6-E410-47F5-9EA6-4BB7715067F3@xxxxxxxxxxxxxx>
From: Bill Bug <wbug@xxxxxxxxxxxxxx>
Date: Fri, 21 Mar 2008 02:35:51 -0400
To: Ontology Summit 2008 <ontology-summit@xxxxxxxxxxxxxxxx>    (030)

Mark is making a very import point regarding the problematic nature 
of the analogy between journal peer reviewers and OOR 
gatekeepers.  To some extent this is an apples to oranges comparison, 
despite the apparent similarity in the gatekeeper function.    (031)

Having said that, I do believe there is another function journal 
reviewers provide which is very relevant to what I'd hope to see 
provided by an OOR gatekeeper - at least when it comes to ontologies 
used to help add reliable, highly-scalable automation to 
bioinformatic data processing.  To go back to the realm of 
advertising, this other gatekeeper role relates to the Google Scholar 
"slogan" - "Stand on the shoulders of giants".  The idea is the peer 
review process in the literature helps provide a certain level of 
distillation and reliable vetting to ensure every researcher need 
only read, absorb, and collate vetted manuscripts, as opposed to all 
the manuscripts submitted to the relevant journals.  As imperfect as 
this process, I still tend to transfer my confidence in the reviewers 
having competently vetted the articles to the articles themselves, so 
that I can accept - at a given point in time - the validity of the 
assertions in a given article and build on them by performing 
additional work on their "shoulders".    (032)

Perhaps - as others have suggested - it is too early in the evolution 
of ontology development practice for us to expect we can produce a 
vetting process capable of functioning in this way, but until we do, 
some of the expectations informaticists - and funding agencies - have 
for the use of ontologies are probably not achievable. Until there is 
a reliable vetting procedure, we cannot expect to re-use and extend 
existing ontologies effectively or with confidence for the purpose of 
bringing like data together in novel ways from across the biomedical 
data diaspora.  Without vetting, we cannot expect to provide other 
developers with clear advise on what are the reliable ontological 
shoulders to build on.  If the OOR has 3 ontologies covering a domain 
of interest at roughly the same scope and level of granularity such 
as the physiology of mammalian electrolyte balance or the assembly of 
peptides into functional multimeric macromolecular receptor 
complexes, how would a bioinformatics application developer determine 
which one to use? Even more importantly, if users pick at random from 
amongst the 2 or more ontologies covering the same domain, who will 
maintain the maps and software required to make deductions or 
inferences across the annotated data repositories which use these 
different ontologies to cover the same domain?    (033)

Another one of my expectations for using ontologies in biomedical 
informatics is as the data representation gets richer and more 
expressive, the nature of the software each application developer 
needs create can focus more on application-specific 
issues.  Community tools capable of parsing the expressivity 
(reasoners) can provide more of the "smarts", so that the custom 
tools don't need to hard code it - and can exclusively focus on the 
application specific algorithms.    (034)

This partly goes back to a point I made earlier in this 
thread.  Using ontologies to power broadly scoped, federated 
inferencing brings with it a distinct set of requirements that differ 
from those of applications focused on providing decision support 
built on a more narrowly focussed data warehouse.  In that case, it 
can be perfectly acceptable for developers to pick the ontology they 
like the best from the several covering the domain(s) they require, 
so long as there is no expectation the resulting knowledge repository 
will be easily shared with other informatics projects.    (035)

These may all be expectations so narrowly linked to biomedical 
informatics they do not hold sway in a more generic OOR.  That's one 
of the issues I'm hoping to better understand by participating in 
this discussion and the upcoming summit.    (036)

Thanks again to all for the stimulating and detailed dissection of 
this important topic.    (037)

Cheers,
Bill    (038)




_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/ 
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/  
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2008/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2008 
Community Portal: http://ontolog.cim3.net/    (039)
<Prev in Thread] Current Thread [Next in Thread>