OntologySummit2013: (Track-B) "Extrinsic Aspects of Ontology Evaluation" Community Input (3KAA)
Track Co-champions: TerryLongstreth & ToddSchneider (3KAK)
Mission Statement: (3KAB)
The intent is to explore, clarify, and identify gaps, practical and theoretical, in the of evaluation of ontology from a systems perspective using the paradigm of blackbox evaluation. T (3KAC)
Extrinsic aspects of ontology evaluation includes subjective factors, measures or metrics, and the range of values of quantifiable attributes. In a systems context evaluations are derived from examination of inputs or stimuli (to the blackbox) and the outputs or externally measurable attributes or behaviors, where those behaviors are controlled or influenced by an ontology. T (3KCN)
The ontology in question may be fully embedded/encapsulated within an entity or system, or may be externally accessible (and potentially shared) among multiple entities or systems. The separation of system or entity behaviors which are not governed by an ontology must be accounted for in any ontology evaluation process. T (3KCO)
Extrinsic aspects to be considered include, T (3KCM)
- interoperability among ontologies (3KCB)
- requirements and their verification (3KCC)
- how metrics can be derived from requirements (3KCD)
- how 'good' requirements relevant to ontology can be crafted (3KCE)
- fitness for purpose (3KCF)
- query performance (3KCG)
- relevant relational database evaluation methods, metrics and techniques (3KCH)
- differences in evaluation among an ontology and instance data (3KCI)
- how evaluation metrics can be derived from examination of test inputs or stimuli (3KCJ)
- how evaluations can be used to revise requirements (3KCK)
- how evaluations can be used to correct an ontology (3KCL)
see also: OntologySummit2013_Extrinsic_Aspects_of_Ontology_Evaluation_Synthesis (3KAD)
Enter your input below ... (please identify yourself and date your entry) (3KAE)
- (Longstreth- 21Jan2013) For a Case Study of industrial use of ontology see http://www.youtube.com/watch?v=lbR8UzlF9B8 , which is a recent talk by Michael Uschold . At about minute 44, be discusses evaluation. (3KAF)
Evaluation Dimensions (3L55)
Please place your suggestions for evaluation dimensions here. (3L56)
From Todd - 26Jan2013 (3LEW)
Performance (3LCM)
- Inference (3LEH)
User (3LDY)
- Correctness (3LD9)
- Input Mechanism (3LDA)
- Understandability (3LDB)
- Terminology (3LDC)
- Domain Completeness (3LDD)
Adequacy (3LDZ)
Security (3LE0)
Cost (3LE1)
- Development (3LDL)
- Integration (3LDM)
- Testing (3LDN)
- Training (3LDO)
- Operations (3LDP)
- Maintenance (3LDQ)
Legal (3LE2)
End Todd (3LEX)
Subject: Chat Gleanings from 24 Jan (3LF1)
As Todd mentioned in the chat for the 24January2013 Track B session, "...We will need definitions, context, and possibly intent. But first I'd like to conduct a simple gathering exercise." (3LF2)
I've gone through the chat and gathered the comments that I believe represent possible properties or characteristics for evaluating an ontology extrinsically. In the email sent to the Ontolog-Summit list, I've placed these in a formatted table which shows provenance by associating the items with quotations from the chat. Beyond that I've made no serious attempt to organize (Ontologize?) these items. (3LF3)
- Terry Longstreth (3LF4)
- Property / Characteristic (3LF5)
- Reusefulness, (3LF6)
- Quality, (3LF7)
- Parsimony, (3LF8)
- Beauty (3LF9)
- License (3LFA)
- Relevance, (3LFB)
- Clarity, (3LFC)
- Consistency, (3LFD)
- Accessibility, (3LFE)
- timeliness, (3LFF)
- completeness, (3LFG)
- accuracy, (3LFH)
- costs (development, maintenance), (3LFI)
- Benefits, (3LFJ)
- Provenance, (3LFK)
- Modularity (3LFL)
- complexness (3LFM)
- Reliability, (3LFN)
- Availability, (3LFO)
- Maintainability, (3LFP)
- Performance, (3LFQ)
- Scalability, (3LFR)
- Security. (3LFS)
- domain integrity, (3LFT)
- referential integrity, (3LFU)
- semantic integrity, (3LFV)
- Precision With_Respect_To(domain D, requirement R) (3LFW)
- Effectiveness, (3LFX)
- Beauty (3LFY)
- Requirements Satisfaction (3LFZ)
- consistency; (3LG0)
- correctness, (3LG1)
- completeness (3LG2)
- Goodness (3LG3)
- Elegance (3LG4)
- Simplicity (3LG5)
- Minimality (3LG6)
- Normalized (3LG7)
- Granularity (3LG8)
- Update Impedance/ complexity/ cost (3LG9)
- Degree of stability (3LGA)
- Error Discovery Profile (3LGB)