Dear Barry, (01)
comments inline: (02)
Am 21.03.2008 um 15:50 schrieb Barry Smith: (03)
> At 10:10 AM 3/21/2008, Holger Lewen wrote:
>> Dear Barry, comments inline
>>
>> Am 21.03.2008 um 14:25 schrieb Barry Smith:
>>
>>> Mark Musen, below, makes a number of valuable
>>> points, which are made all the more interesting
>>> in virtue of the fact that the NCBO's BioPortal,
>>> an ontology repository for which Mark is
>>> responsible,
>>> http://www.bioontology.org/bioportal.html, is
>>> carrying out an experimental test of the benefits
>>> of democratic ranking-based approach to ontology assessment.
>>>
>> I agree with you that up to now no experimental study has proven that
>> user-based evaluation does indeed work for ontologies. We will also
>> carry out experiments in the context of the NeOn project, which
>> together with the NCBO's experiments should provide some insights.
>>
>>
>>> Specifically, the BioPortal will test a thesis to
>>> the effect that democratic ranking based on user
>>> comments can 1. provide a valuable service which
>>> will scale as the population of ontologies grows
>>> and 2. allow true openness (no gatekeeping at
>>> all) of a repository (thus perhaps even allowing
>>> admission to the BioPortal of
>>> http://www.schemaweb.info/schema/SchemaDetails.aspx?id=163,
>>> which is, as I understand it, a bio-ontology-like
>>> artifact pertaining to organisms with more than two legs).
>>>
>>> However, his main argument against the
>>> alternative (expert peer review-based) approach
>>> currently being tested by the OBO Foundry, has
>>> been addressed already in earlier postings to
>>> this list: the committee of peer reviewers used
>>> by the Foundry will in every case involve expert
>>> users from the specific user communities.
>>>
>> As was argued before, ontologies can be used in different contexts
>> and
>> for different applications / use cases. What might be perfectly
>> valuable in one scenario might not work in the other. Therefore
>> having
>> a team testing an ontology for one specific setting or according to a
>> specific set of rules/guidelines might not be enough. Unless you can
>> get the team to tackle all the scenarios and usecases an open
>> platform
>> could address by application specialists commenting on usefulness in
>> practice, I think expert-based reviews can only be ONE of many
>> aspects
>> to consider when reusing an ontology.
>
> But a valuable one, foresooth!
Which is why it would be included in an open review system. (04)
>
>
>
>>> Mark thinks that ontologies are much more like
>>> refrigerators than they are like journal
>>> articles. I think that most of them are in fact
>>> still much more like collections of refrigerator
>>> magnets. The question is: how can we motivate
>>> ontology creators (and potential ontology evaluators) to do a better
>>> job?
>>>
>> By having many users provide feedback and insides, not only some
>> experts. I could imagine that in practice you will be rather simply
>> rejecting an ontology not up to expected standard than providing
>> detailed feedback as to why it was rejected and what has to be
>> improved. As you said, reviewers are busy people. So different
>> reviews
>> focussing on different errors or aspects might provide valuable
>> feedback, similar to the whole process of open source software
>> development and bug-reporting. Of course you could also implement
>> other incentive mechanism, be it fame, money or whatever ontology
>> developers might desire.
>>
>>> This question is also not addressed by Holger
>>> Lewen, in another interesting and useful post
>>> that is also appended below. Indeed, Holger
>>> expresses a touching optimism to the effect that
>>> large bodies of intelligent user comments will
>>> form around the ontologies submitted to a
>>> potential OOR; that software will allow potential
>>> new users to navigate through these comments to
>>> help them find the answers to just the questions
>>> they need; and that intelligent evaluators will
>>> keep on submitting new comments as ever new
>>> collections of refrigerator magnet products
>>> (sorry: ontologies) come onto the market.
>>>
>> I guess you need optimism to try writing your PhD about that topic. I
>> am glad I could touch you with it! As mentioned above, of course it
>> has to be shown to work in that area, but there have been promising
>> examples of people collaborating without monetary incentives. I would
>> claim that large ontology development projects are not entirely
>> different from large open source software development problems. Also
>> there people collaborate out of free will, and bad contributions are
>> quickly removed by other users. I think one point to note is that it
>> is not necessary for user comments to come from intelligent users. By
>> means of the underlying Web of Trust, bad reviews will be filtered
>> according to personal preferences. As long as we do not expect the
>> majority of collaborators to purposely game the system, it should be
>> stable enough to handle content of varying quality. So even if your
>> gatekeeping team would be the only trustworthy body in the system, it
>> would still work at least as good as your gatekeeping approach alone.
>> But potentially it can work better by addressing more scenarios.
>
> Ah, such touching optimism!
>
>>> Why should they invest this time and effort?
>>> Skilled users of ontologies are, I can imagine,
>>> busy people. They also form a rather small
>>> community, with limited resources to spend e.g.
>>> on training teams of document inspectors as proposed (also below) by
>>> Matthew.
>>>
>> Why do people invest time here to discuss this matter? Because some
>> people dedicate their time to deal with such problems. Also, if the
>> impact becomes high enough, you can give jobs to your best reviewers,
>> thereby making it worth their while.
>
> Try taking this to a university Dean:
>
> Give this man a job; he posted 4000 comments to
> the NeON ontology ranking system just last week
> alone, and his reviews came top in 79% of cases!
>
> Try convincing a peer review commitee of a
> funding agency to accept this as part of the
> argument why a project should be funded.
Since you suggest doing something similar for your closed reviewing
system (having people that allocate some part of their funded time to
assess the quality of ontologies for your registry), I do not see why
this should not work. Your reviewers would also have to justify
spending resources on reviewing the ontologies, and thus it will cost
some funding agencies money. If the open review system gives birth to
a brilliant reviewer, who emerges from the crowd and was not part of
your team before, I do not see why funding a reviewer selected by you
should be rather funded than a reviewer that 79% of the users of the
system think of as a good reviewer. I also think that no branding is
needed, so I would rather see an open system where everyone could
collaborate than a closed and branded NeOn system. If we start having
competing systems, it will be harder for either system to gather the
necessary reviews. (05)
>
>
>> And the incentives in such a
>> system should not differ too much from the incentive to participate
>> as
>> a reviewer in the closed gatekeeping approach. Of course, there will
>> be more competition from outside reviewers and one could see which
>> reviews the masses prefer and if the "gatekeepers'" reviews indeed
>> rank top.
>>
>>
>>> The OBO aims to test one potential answer to this
>>> motivation question, at least for ontologies
>>> developed to aid science. This answer has the
>>> advantage of resting on a methodology -- the
>>> methodology of peer review -- that has enjoyed
>>> some three hundred years of success that is
>>> roughly co-terminous with the advance of modern science. .
>>>
>> Well, I cannot follow how peer review as a methodology can lead to
>> superior reviews. In the end it is not the process of reviewing that
>> makes this work (more or less), but the choice of the reviewers.
>> There
>> is no reason why having a similar methodology for open review (as
>> also
>> proposed for the scientific domain by some) should not work, given at
>> least the same reviewers. Also we have no comparison as to how
>> science
>> might have involved given a more open review process for scientific
>> publications. So I do not know whether we can claim the success of
>> scientific advance to be due to the methodology of peer review or the
>> idea of reviewing in some way.
>>
>
> By the same token we could prove that the process
> of using standard arithmetic does not contribute
> to making physics (engineering, etc.) work,
> because there might have been another,
> non-standard arithmetic, perhaps developed on the
> basis of democratic vote on how addition works,
> which would have done much better.
>
I do not see how that comparison is related to my claim. People on
this mailing lists have already stated that there might not be a
single way to review ontologies, because the assessment will vary
based on the intended use-case. As for arithmetic and physics, there
seem to be some general and universal truths, that could be researched
and discovered. If you will discover the single right way and the
single truth to assessing the quality of an ontology, I would be more
than glad to integrate it as the only reviewing measure into the
review system. (06)
>
>>> Crudely put: experts are motivated to review
>>> ontologies in their relevant domains of expertise
>>> because they get career-related credit for serving as reviewers.
>> As they could in an open review system, given it were as accepted as
>> the closed system. Being the top reviewer in such an open system
>> might
>> even prove to provide more credibility, since not only your peers but
>> also a large community of users agree with your assessment.
>
> People like this, who have some knowledge of
> biology, would indeed almost certainly be invited
> to join the OBO Foundry review process.
I guess finding them will be more difficult in your scenario, (07)
>
>>>
>>>
>>> Ontology developers are motived to create better
>>> ontologies because they get career-related credit
>>> for having their ontologies included (published)
>>> in what, if the peer-review process is successful
>>> will count as analogous to a peer-reviewed
>>> scientific journal. (We are working on the many
>>> tough problems which must be tackled to make this
>>> possible -- including all the problems mentioned
>>> by Mark and Holger, below.)
>> I would like to see that come true. Especially since you have so many
>> changes in the science ontologies, especially the big ones, that I
>> would not even know which version to "publish" in the repository.
>> If I
>> chose to publish every new delta, will you restart the peer review
>> process again? Or will you just give a carte blanche? Any change
>> could
>> potentially make an ontology inconsistent and unusable. So not
>> reviewing again would not be the solution. I know that the same
>> problem arises with the open review system as well, and I am
>> currently
>> thinking about different scenarios to tackle it. But because of your
>> limited resources (reviewers), it might slow down your publishing
>> process considerably.
>>
>>
>
> This is one of the problems where we are, I
> believe, close to a good understanding of the
> issues involved. The Gene Ontology has been
> practicing good versioning policies for several
> years and it is updated on a nightly basis. We
> have no intention to review it every morning. Our
> process will, I believe, provide a way to tackle
> this problem that is still manageable. The open
> process favored by you and Mark would, I believe,
> face more formidable difficulties in managing two
> sets of moving targets (ontologies, ranking clouds).
If there is a solution that works for your scenario, I do not see why
it should not be adaptable to our scenario. Since your reviewing
process also consists of multiple reviews per ontology, it seems that
the solution could be applied rather similarly. (08)
>
>
>>> The publishing
>>> process we have in mind will have the incidental
>>> advantage that it will allow the multiple
>>> developers typically involved in serious ontology
>>> endeavors to get appropriate credit, which they
>>> can use to justify spending the time and effort involved.
>>>
>> Again, I do not see why this should not be possible in an open
>> reviewing system. If your ontology gets good acceptance there, it
>> should count at least as much.
>
> See above, re: getting a job. Do we wish ontology
> to become one day a professional endeavor,
> involving such things as levels of expertise? Or
> do we wish ontology to remain a cottage industry?
>
It seems the recent development in the IT world go towards utilizing
the wisdom of the masses. I think the problem you mention about
perception of the work by funding agencies is just a frame of mind. If
the open review system becomes accepted, work done there should count
no less than work done for a closed reviewing system. (09)
>
>>> Both reviewers and developers will be further
>>> motivated to participate in this process because
>>> they can thereby directly influence the standard
>>> set of ontology resources which will be available
>>> to them, thereby also motivating the creation of:
>>> related ontology-based software, useful bodies of
>>> data annotated in terms of these resources, etc.
>>>
>> Again, in my opinion also possible in an open reviewing system.
>>
>>
>>> Note that I am not recommending this as an
>>> approach to be adopted by the OOR. It rests on
>>> too many features peculiar to the domain of
>>> science. However, if Patrick Hayes is right -
>>> that people like him can just as well publish
>>> their ontologies on the web - then this suggests
>>> the need for a real raison d'être for the OOR,
>>> and I am suggesting non-trivial and evolving
>>> gatekeeping constraints in the cause of
>>> incrementally raising the quality of ontologies as one such raison
>>> d'être.
>>>
>>
>> I agree that you approach will work fine in a very limited domain
>> with
>> a limited number of submissions and a good choice of reviewers.
>> However, as you mentioned, this would be more similar to a "journal"
>> of ontologies rather than an open collection.
>
> Good journals allow anyone to submit.
> Also the system of journal publishing which we
> take as our starting-point provides a rather
> elegant way to divide up the potentially
> unlimited domain that is available for ontology
> development into limited disciplines and subdisciplines.
>
Yes, they do, but this is because it still results in limited
submissions. If everybody would start submitting to journals, it would
only take short time until there would be some kind of restriction
(e.g. only people working in science may submit).
I agree that your approach is valuable if taken one discipline at a
time with dedicated reviewers for that discipline. This is again a
question of scale. How do you determine with which subcategory to
start. If I have the best possible ontology for a subdomain you are
not considering yet, what do I do? It might still be extremely
valuable for the community to see my ontology straight away. (010)
>
>> I totally agree that
>> there can be value in having this journal like collection, especially
>> with people knowing its reviewing process and constraints. Still, it
>> might just as well be implemented in an open fashion which
>> additionally allows other users to provide feedback.
>
> We provide for constant feedback of all types --
> ontologies work well, in scientific research at
> least, to the degree that they have large numbers
> of users who trust them -- hence consensus is
> important. We receive huge amounts of feedback of
> all types. We could, if we wish, use this
> feedback to rank the ontologies (4,000 emails
> alone last week on the
> <http://www.schemaweb.info/schema/SchemaDetails.aspx?id=292>Bio-Zen
> Ontology, so it must be good).
> (011)
Which is great news. This means that the people writing those 4000
emails spend enough time caring about the quality of the ontology. I
would assume they would not mind changing the way they provide their
feedback, for example writing reviews instead of writing emails. At
least now I am convinced that we could fill an open review system with
feedback from the community. (012)
>> You could easily
>> implement your journal like approach in an open system. Just name 5
>> reviewers of your choice and designate a special role to them, e.g.
>> expert in that domain. Given these reviewers had some sort of
>> consensus on the quality of the ontology, you mark it as approved by
>> XYZ and give it the status of a high profile submission. And you
>> could
>> even see if end users agree with the assessment of your experts. You
>> could think of this as a way to evaluate the validity of expert-based
>> review.
>
> We have been thinking carefully about such
> approaches for some time, and what we end up with
> will certainly involve aspects of what you
> suggest. Some (not me) might worry, however, that
> what you suggest still bears certain traces of elitism.
>
Which would be fine if it is an optional way and not the only to look
at the ontologies. You could easily imaging features like: Show only
reviews of certified OBO reviewers or similar. Also, if the users
voted some reviewers to be the best, this is a democratic approach,
and as such should not be criticized for elitism. (013)
>
>> And so far, I am not aware of any study proving that the peer-
>> reviewed
>> ontology repository provides benefits for the end user actually
>> employing the ontologies. Are you planning such a study on end user
>> satisfaction? I think it would be a really valuable experiment to let
>> users compare the proposed open review solution including the
>> gatekeepers' reviews with the closed solution.
>
> The GO was built using a subset of the principles
> now formalized by the OBO Foundry (of which the
> GO forms the central part). The GO ranks 5th on
> google; the 2nd most highly ranked ontology (GeoNames), is ranked
> 16th.
> We have documented some of the qualitative
> benefits in http://www.nature.com/nbt/journal/v25/n11/pdf/nbt1346.pdf
> Some of us are engaging in research on
> quantitative metrics to measure these benefits
> e.g. at http://www.org.buffalo.edu/RTU/papers/CeustersFois2006.pdf
>
>
I would argue that the ontologies you mention have certainly
benefitted from your involvement of aligning them with a top-level
ontology and providing critique. Providing guidelines and
methodologies for building proper ontologies is certainly important
work and I am glad that your dissemination and training efforts have
shown effects. The question would still be whether the developers
benefitted from the feedback given by the reviewers and the training
efforts or the fact that it was a closed peer based scenario. I would
rather see the positive effects of reviews on the quality of
ontologies as an indication that reviewing is needed and the feedback
is taken into consideration by the developers. Now the point I do not
see as clear yet is why this should not work in a scenario where the
developers would not only get feedback from the gatekeepers but also
from actual users, each with potentially different usage scenarios. (014)
Best,
Holger (015)
> BS (016)
_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2008/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2008
Community Portal: http://ontolog.cim3.net/ (017)
|