On Tue, 15 Apr 2008, John F. Sowa wrote: (01)
> Denise and Ken,
>
> DB> How can the evaluation and registry parameters be more
> > closely aligned?
>
> The only things that can be evaluated on a linear scale are ones
> that are already expressed in numbers. But even those can cause
> serious misunderstandings -- for example, comparing miles per
> gallon of a sports car, a lawn mower, and a snow plow.
>
> I realize that managers constantly ask for some kind of magic
> number, but giving them numbers is often irresponsible. The
> greatest danger is that they might actually make a decision
> based on those numbers. (02)
I agree. The analysis is useful to give a global picture in general
terms, but it would be a bad idea to make any decisions based on the
assessments. (03)
> KB> With one of my students, I have evaluated 40 large ontologies
> > using the OntologySummit2007 Framework Dimensions. I have
> > written up some conclusions about the results and could present
> > them at some point.
> >
> > I have posted it to
> >
> > http://www.ccs.neu.edu/home/kenb/OntologySummit2007Assessment
>
> The "radar diagrams" demonstrate the problems with that kind of
> factor analysis. For example, the diagram for WordNet and the
> diagram for Human Disease - OBO are almost identical. Yet those
> two systems are totally different in purpose, representation, and
> methodology. But the web page makes the following claim:
> (04)
The assessments were based on published descriptions of the ontologies.
As these are almost entirely informal, the assessments suffer from the
same lack of formality. If the OOR is successful, it should encourage
developers to supply a formal description. (05)
>
> > The factor analysis is statistically significant with p-value 0.00137.
>
> That statement makes the approach sound extremely significant, but
> I would consider any method that would produce almost identical
> numbers for WordNet and the Human Disease OBO as hopelessly misleading. (06)
The significance is only about the analysis not about the data. (07)
No doubt many of the assessments are inaccurate. We considered asking the
developers of the ontologies to help us with the assessments, but it isn't
clear that it would have improved the accuracy since every individual
would have interpreted the framework dimensions somewhat differently. (08)
There are many other issues that one can raise about the statistical
analysis. For example, it was not a random sample; we identified one
kind of selection bias. So the statistical analysis is only about the
sample itself, not about some hypothetical population of ontologies
(whatever that would be). (09)
> The list of 40 different ontologies/terminologies/resources (see below)
> is useful. But instead of those radar diagrams (or the numbers from
> which they were derived), the following info would be far more valuable:
>
> 1. A URL that points to the web site for that resource.
>
> 2. A one-paragraph description derived from the documentation on
> that web site.
>
> 3. One sentence that describes the subject matter of the resource.
>
> 4. A statement of the notation used for the resource (i.e., whether
> it uses a natural language, a formalized language, or some
> combination of one or more languages and/or diagrams).
>
> 5. Some measure of size in terms of the number of items defined
> (i.e., categories, concepts, words, terms, predicates, relations,
> types, classes, or whatevers).
>
> If I were looking for an ontology, I would find the above information
> useful as a *starting point*. Those radar diagrams are misguided
> because they give the impression that there is some kind useful
> evaluation embodied in them. (010)
I agree that this information is essential when one is looking for an
ontology. It would be even more valuable if the information were
specified using a formal annotation. IMHO the framework dimensions serve
a different purpose and are not appropriate for search and decision making
tasks. (011)
-- Ken (012)
>
> John Sowa
> _____________________________________________________________________
>
> 1. MeSH
> 2. Gene Ontology
> 3. Basic Formal Ontology
> 4. SUMO
> 5. Engineering Math
> 6. COSMO
> 7. PSL
> 8. Environmental Data Coding Specification
> 9. Geospatial ML
> 10. International Classification of Diseases
> 11. Unified Medical Language System Metathesaurus
> 12. World Bank MetaThesaurus
> 13. Spase Ontology
> 14. FOAF
> 15. AKTiveSAOntology
> 16. CMMI Smallscale Ontology
> 17. Beer Ontology
> 18. BioMoby Services
> 19. Bio-Zen
> 20. UNSPCS
> 21. IMDB Mapping Movie Ontology
> 22. Federal Enterprise Architecture Reference Model Ontology (FEA-RMO)
> 23. Cyc
> 24. Encyclopedia of Life
> 25. Ontology for Biomedical Investigation
> 26. Functional Genomics Investigation Ontology
> 27. ProMed Mail
> 28. Agricola
> 29. GenBank
> 30. Medline
> 31. GBIF
> 32. WHO health reports
> 33. Human Disease - OBO
> 34. Core Software Ontology
> 35. Cell System Ontology
> 36. CIA Factbook
> 37. Evidence Code
> 38. WordNet
> 39. Ontology for Geographic Information ISO 19115:2003
> 40. General Ontology for Linguistic Description
>
>
> _________________________________________________________________
> Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
> Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
> Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
> Community Files: http://ontolog.cim3.net/file/work/OntologySummit2008/
> Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2008
> Community Portal: http://ontolog.cim3.net/
> (013)
_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2008/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2008
Community Portal: http://ontolog.cim3.net/ (014)
|