At 11:28 AM 3/21/2008, Holger Lewen wrote:
>HL
> >> Why do people invest time here to discuss this matter? Because some
> >> people dedicate their time to deal with such problems. Also, if the
> >> impact becomes high enough, you can give jobs to your best reviewers,
> >> thereby making it worth their while.
> > (01)
BS
> > Try taking this to a university Dean:
> >
> > Give this man a job; he posted 4000 comments to
> > the NeON ontology ranking system just last week
> > alone, and his reviews came top in 79% of cases!
> >
> > Try convincing a peer review commitee of a
> > funding agency to accept this as part of the
> > argument why a project should be funded. (02)
HL
>Since you suggest doing something similar for your closed reviewing
>system (having people that allocate some part of their funded time to
>assess the quality of ontologies for your registry), I do not see why
>this should not work. (03)
BS
We both face a chicken and egg problem. We will see empirically which
approach survives. Perhaps both. (04)
HL
>Your reviewers would also have to justify
>spending resources on reviewing the ontologies, and thus it will cost
>some funding agencies money. If the open review system gives birth to
>a brilliant reviewer, who emerges from the crowd and was not part of
>your team before, I do not see why funding a reviewer selected by you
>should be rather funded than a reviewer that 79% of the users of the
>system think of as a good reviewer. (05)
BS
You do not see the thorny nature of the chicken and egg problem you
are facing, I'm afraid. (06)
HL
> I also think that no branding is
>needed, so I would rather see an open system where everyone could
>collaborate than a closed and branded NeOn system. If we start having
>competing systems, it will be harder for either system to gather the
>necessary reviews.
> (07)
BS
Science thrives on competition.
... (08)
> > (09)
BS
> >>> Crudely put: experts are motivated to review
> >>> ontologies in their relevant domains of expertise
> >>> because they get career-related credit for serving as reviewers. (010)
HL
> >> As they could in an open review system, given it were as accepted as
> >> the closed system. Being the top reviewer in such an open system
> >> might
> >> even prove to provide more credibility, since not only your peers but
> >> also a large community of users agree with your assessment.
> > (011)
BS
> > People like this, who have some knowledge of
> > biology, would indeed almost certainly be invited
> > to join the OBO Foundry review process. (012)
HL
>I guess finding them will be more difficult in your scenario, (013)
BS
We are not facing problems in this regard. But in any case, we have
no objections to drawing on the fruits of your scenario.\ (014)
BS
>The open
> > process favored by you and Mark would, I believe,
> > face more formidable difficulties in managing two
> > sets of moving targets (ontologies, ranking clouds). (015)
HL
>If there is a solution that works for your scenario, I do not see why
>it should not be adaptable to our scenario. Since your reviewing
>process also consists of multiple reviews per ontology, it seems that
>the solution could be applied rather similarly. (016)
BS
We will see, empirically. (017)
> > (018)
BS
> >
> >>> The publishing
> >>> process we have in mind will have the incidental
> >>> advantage that it will allow the multiple
> >>> developers typically involved in serious ontology
> >>> endeavors to get appropriate credit, which they
> >>> can use to justify spending the time and effort involved.
> >>> (019)
HL
> >> Again, I do not see why this should not be possible in an open
> >> reviewing system. If your ontology gets good acceptance there, it
> >> should count at least as much.
> > (020)
BS
> > See above, re: getting a job. Do we wish ontology
> > to become one day a professional endeavor,
> > involving such things as levels of expertise? Or
> > do we wish ontology to remain a cottage industry?
> > (021)
HL
>It seems the recent development in the IT world go towards utilizing
>the wisdom of the masses. I think the problem you mention about
>perception of the work by funding agencies is just a frame of mind. If
>the open review system becomes accepted, work done there should count
>no less than work done for a closed reviewing system. (022)
BS
Touching optimism, again.
'Should' does not imply 'will'. (023)
BS
> >>> Both reviewers and developers will be further
> >>> motivated to participate in this process because
> >>> they can thereby directly influence the standard
> >>> set of ontology resources which will be available
> >>> to them, thereby also motivating the creation of:
> >>> related ontology-based software, useful bodies of
> >>> data annotated in terms of these resources, etc.
> >>> (024)
HL
> >> Again, in my opinion also possible in an open reviewing system. (025)
You can't have it both ways -- on the one hand you claim multiple
benefits from having lots of reviewers, lots of perspectives, people
can find precisely the ontology which will suit their purposes from
the wisdom of crowds, etc.
Now, you say that lone reviewers lost in these crowds will be able
'to directly influence the standard set of ontology resources which
will be available to them'. (026)
BS
> > Good journals allow anyone to submit.
> > Also the system of journal publishing which we
> > take as our starting-point provides a rather
> > elegant way to divide up the potentially
> > unlimited domain that is available for ontology
> > development into limited disciplines and subdisciplines.
> > (027)
HL
>Yes, they do, but this is because it still results in limited
>submissions. If everybody would start submitting to journals, it would
>only take short time until there would be some kind of restriction
>(e.g. only people working in science may submit).
>I agree that your approach is valuable if taken one discipline at a
>time with dedicated reviewers for that discipline. This is again a
>question of scale. How do you determine with which subcategory to
>start. If I have the best possible ontology for a subdomain you are
>not considering yet, what do I do? It might still be extremely
>valuable for the community to see my ontology straight away.
>
BS
Register it immediately with the OBO Foundry!
This will help people to find it.
You will benefit from expert review.
If there is already an OBO Foundry ontology available for your topic,
you can start working on a merger, to reap the benefits listed by
Bill (post copied below).
What, after all, can you lose? (028)
BS (029)
Delivered-To: phismith@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Received: (qmail 15153 invoked from network); 21 Mar 2008 06:36:02 -0000
Received: from unknown (HELO mailscan4.acsu.buffalo.edu) (128.205.6.136)
by mail3 with SMTP; 21 Mar 2008 06:36:02 -0000
Received: (qmail 21960 invoked by uid 26149); 21 Mar 2008 06:36:01 -0000
Delivered-To: phismith@xxxxxxxxxxx
Received: (qmail 21829 invoked from network); 21 Mar 2008 06:36:00 -0000
Received: from ns3.cim3.com (HELO mccarthy.cim3.com) (64.62.192.10)
by front2.acsu.buffalo.edu with SMTP; 21 Mar 2008 06:36:00 -0000
Received: from ns3.cim3.com (localhost.localdomain [127.0.0.1])
by mccarthy.cim3.com (Postfix) with ESMTP id DB556108B62;
Thu, 20 Mar 2008 23:35:59 -0700 (PDT)
X-Original-To: ontology-summit@xxxxxxxxxxxxxxxx
Delivered-To: ontology-summit@xxxxxxxxxxxxxxxxx
Received: from mail3.sea5.speakeasy.net (mail3.sea5.speakeasy.net
[69.17.117.5])
by mccarthy.cim3.com (Postfix) with ESMTP id 0C12E108B62
for <ontology-summit@xxxxxxxxxxxxxxxx>;
Thu, 20 Mar 2008 23:35:57 -0700 (PDT)
Received: (qmail 24930 invoked from network); 21 Mar 2008 06:35:57 -0000
Received: from vpn137-110-192-21.ucsd.edu (HELO [192.168.1.101])
(bbug@[137.110.192.21]) (envelope-sender <wbug@xxxxxxxxxxxxxx>)
by mail3.sea5.speakeasy.net (qmail-ldap-1.03) with SMTP
for <ontology-summit@xxxxxxxxxxxxxxxx>; 21 Mar 2008 06:35:53 -0000
Mime-Version: 1.0 (Apple Message framework v753)
In-Reply-To: <CDD70C50-C4C6-4169-85C6-AA7053EFC4E1@xxxxxxxxxxxx>
References: <52043.1206018194@xxxxxxxxxxx>
<p06230900c40844a2ffe4@[192.168.1.2]>
<47E331DC.1060709@xxxxxxxxxxx>
<CDD70C50-C4C6-4169-85C6-AA7053EFC4E1@xxxxxxxxxxxx>
Message-Id: <5E8366C6-E410-47F5-9EA6-4BB7715067F3@xxxxxxxxxxxxxx>
From: Bill Bug <wbug@xxxxxxxxxxxxxx>
Date: Fri, 21 Mar 2008 02:35:51 -0400
To: Ontology Summit 2008 <ontology-summit@xxxxxxxxxxxxxxxx> (030)
Mark is making a very import point regarding the problematic nature
of the analogy between journal peer reviewers and OOR
gatekeepers. To some extent this is an apples to oranges comparison,
despite the apparent similarity in the gatekeeper function. (031)
Having said that, I do believe there is another function journal
reviewers provide which is very relevant to what I'd hope to see
provided by an OOR gatekeeper - at least when it comes to ontologies
used to help add reliable, highly-scalable automation to
bioinformatic data processing. To go back to the realm of
advertising, this other gatekeeper role relates to the Google Scholar
"slogan" - "Stand on the shoulders of giants". The idea is the peer
review process in the literature helps provide a certain level of
distillation and reliable vetting to ensure every researcher need
only read, absorb, and collate vetted manuscripts, as opposed to all
the manuscripts submitted to the relevant journals. As imperfect as
this process, I still tend to transfer my confidence in the reviewers
having competently vetted the articles to the articles themselves, so
that I can accept - at a given point in time - the validity of the
assertions in a given article and build on them by performing
additional work on their "shoulders". (032)
Perhaps - as others have suggested - it is too early in the evolution
of ontology development practice for us to expect we can produce a
vetting process capable of functioning in this way, but until we do,
some of the expectations informaticists - and funding agencies - have
for the use of ontologies are probably not achievable. Until there is
a reliable vetting procedure, we cannot expect to re-use and extend
existing ontologies effectively or with confidence for the purpose of
bringing like data together in novel ways from across the biomedical
data diaspora. Without vetting, we cannot expect to provide other
developers with clear advise on what are the reliable ontological
shoulders to build on. If the OOR has 3 ontologies covering a domain
of interest at roughly the same scope and level of granularity such
as the physiology of mammalian electrolyte balance or the assembly of
peptides into functional multimeric macromolecular receptor
complexes, how would a bioinformatics application developer determine
which one to use? Even more importantly, if users pick at random from
amongst the 2 or more ontologies covering the same domain, who will
maintain the maps and software required to make deductions or
inferences across the annotated data repositories which use these
different ontologies to cover the same domain? (033)
Another one of my expectations for using ontologies in biomedical
informatics is as the data representation gets richer and more
expressive, the nature of the software each application developer
needs create can focus more on application-specific
issues. Community tools capable of parsing the expressivity
(reasoners) can provide more of the "smarts", so that the custom
tools don't need to hard code it - and can exclusively focus on the
application specific algorithms. (034)
This partly goes back to a point I made earlier in this
thread. Using ontologies to power broadly scoped, federated
inferencing brings with it a distinct set of requirements that differ
from those of applications focused on providing decision support
built on a more narrowly focussed data warehouse. In that case, it
can be perfectly acceptable for developers to pick the ontology they
like the best from the several covering the domain(s) they require,
so long as there is no expectation the resulting knowledge repository
will be easily shared with other informatics projects. (035)
These may all be expectations so narrowly linked to biomedical
informatics they do not hold sway in a more generic OOR. That's one
of the issues I'm hoping to better understand by participating in
this discussion and the upcoming summit. (036)
Thanks again to all for the stimulating and detailed dissection of
this important topic. (037)
Cheers,
Bill (038)
_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2008/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2008
Community Portal: http://ontolog.cim3.net/ (039)
|