OntologySummit2013: Panel Session-14 - Thu 2013-04-18    (3OR8)

Summit Theme: "Ontology Evaluation Across the Ontology Lifecycle"    (3OR9)

Session Topic: OntologySummit2013 Communique: Final Draft Review    (3ORA)

Communique Co-Lead Editors & Session Co-chairs: ... intro slides (from the first review session)    (3QKP)

Reference:    (3QKR)

Archives:    (3QKY)

Abstract:    (3QMR)

OntologySummit2013 Session-14: OntologySummit2013 Communique: Final Draft Review - intro slides    (3QMS)

This is our 8th Ontology Summit, a joint initiative by NIST, Ontolog, NCOR, NCBO, IAOA & NCO_NITRD with the support of our co-sponsors. The theme adopted for this Ontology Summit is: "Ontology Evaluation Across the Ontology Lifecycle."    (3QMT)

Currently, there is no agreed methodology for development of ontologies, and there are no universally agreed metrics for ontology evaluation. At the same time, everybody agrees that there are a lot of badly engineered ontologies out there, thus people use -- at least implicitly -- some criteria for the evaluation of ontologies.    (3QMU)

During this OntologySummit, we seek to identify best practices for ontology development and evaluation. We will consider the entire lifecycle of an ontology -- from requirements gathering and analysis, through to design and implementation. In this endeavor, the Summit will seek collaboration with the software engineering and knowledge acquisition communities. Research in these fields has led to several mature models for the software lifecycle and the design of knowledge-based systems, and we expect that fruitful interaction among all participants will lead to a consensus for a methodology within ontological engineering. Following earlier Ontology Summit practice, the synthesized results of this season's discourse will be published as a Communique.    (3QMV)

As a continuation from last week's session, in this 14th virtual session of the Summit, the co-lead editors will present the final draft of the summit communique. It should be noted that, while suggestions on smaller, more bounded items will still be accepted by the Communique co-lead editors until April 25, this (April 18th) is the last day for additional comments/suggestions of substance. During the session, the session chairs will moderate an open discussion among contributing editors and the community, in an effort to get this collaboratively authored work completed before the OntologySummit2013_Symposium, where this Communique will be presented and adopted.    (3QMW)

More details about this OntologySummit is available at: OntologySummit2013 (homepage for this summit)    (3QMX)

Agenda:    (3QMY)

OntologySummit2013 - Virtual Session-14    (3QMZ)

Proceedings:    (3QN5)

Please refer to the above    (3QN6)

IM Chat Transcript captured during the session:    (3QN7)

 see raw transcript here.    (3QN8)
 (for better clarity, the version below is a re-organized and lightly edited chat-transcript.)
 Participants are welcome to make light edits to their own contributions as they see fit.    (3QN9)
 -- begin in-session chat-transcript --    (3QNA)
	[9:09] PeterYim: Welcome to the    (3RBY)
	 = OntologySummit2013: Virtual Session-14 - Thu 2013-04-18 =    (3RBZ)
	Summit Theme: Ontology Evaluation Across the Ontology Lifecycle    (3RC0)
	Session Topic: OntologySummit2013 Communique: Final Draft Review    (3RC1)
	* Session Co-chairs 
	 - Dr. FabianNeuhaus (NIST) and Dr. AmandaVizedom (Ind. Consultant)    (3RC2)
	Agenda:    (3RC3)
	1. Opening (co-chairs)    (3RC4)
	2. Presentation of plans and second draft of the communique (co-lead editors)    (3RC5)
	3. Review by section and open discussion [Co-lead editors, co-editors & All]    (3RC6)
	4. Wrap-up / Announcements (co-chairs)    (3RC7)
	Logistics:    (3RC8)
	* Refer to details on session page at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_04_18    (3RC9)
	* (if you haven't already done so) please click on "settings" (top center) and morph from "anonymous" to your RealName (in WikiWord format)    (3RCA)
	* Mute control: *7 to un-mute ... *6 to mute    (3RCB)
	* Can't find Skype Dial pad?
	** for Windows Skype users: it may be under the "Call" dropdown menu as "Show Dial Pad"
	** for Linux Skype users: please note that the dial-pad is only available on v4.1 (or later or the earlier Skype versions 2.x,)
	   if the dialpad button is not shown in the call window you need to press the "d" hotkey to enable it.    (3RCC)
	* Note: ... it has come to our attention that our conference bridge provider is running into some problems with the "joinconference" skype connections. 
	  In case anyone gets in trouble, please try to call the phone numbers instead (e.g. from your phone, skype-out, google-voice, etc.)    (3RCD)
	Attendees: FabianNeuhaus (co-chair), AmandaVizedom (co-chair), AstridDuqueRamos, BobSmith, 
	BobbinTeegarden, DougFoxvog, JackRing, JoelBender, JohnMcClure, JulienCorman, KenBaclawski, 
	LamarHenderson, MatthewWest, MichaelGruninger, MikeDean, PeterYim, SteveRay, TerryLongstreth    (3RCE)
	 == Proceedings: ==    (3RCF)
	[9:22] JoelBender: howdy!    (3RCG)
	[9:32] FabianNeuhaus: working draft is at: 
	https://docs.google.com/document/d/1jeU5DL9JuNiyxv7BDOMr0VgFIWmSKVHXcnLlcHF-Zg0/edit#    (3RCH)
	[9:35] anonymous morphed into DougFoxvog    (3RCI)
	[9:43] JohnMcClure: hi DougFoxvog!    (3RCJ)
	[9:48] TerryLongstreth: [in agreement with SteveRay's suggestion that we should include some 
	pointers to tools on the communique] A URL pointing to a Tool Summary page should also be in the 
	(TBS) Executive summary    (3RCK)
	[9:50] TerryLongstreth: Amanda / Fabian - Is the Purpose supposed to be part of the Exec summary, or 
	a separate section?    (3RCL)
	[9:57] TerryLongstreth: There are often two reasons for an executive summary: 1) to allow someone to 
	quickly glean the gist of the summarized document 2) to entice them to read the rest. It's not easy 
	to strike a balance between the two    (3RCM)
	[10:00] AmandaVizedom: Terry, "purpose" is not meant to be a separate section. It is now discussed 
	in the Introduction, but we had comments last week asking for it to be addressed in the Exec 
	Summary.    (3RCN)
	[10:27] TerryLongstreth: @Amanda - re: Purpose - I suggest it include quoted or paraphrastic text 
	referring to the original intent of this summit ("The goal for Ontology Summit 2013 is to identify 
	best practices for ontology development and evaluation." That phrasing suggests absolutes ("best" 
	practices) when we really ended up with a survey of current practices, the associated tools, and 
	possible avenues for future research on practices and tools.    (3RCO)
	[9:51] MichaelGruninger: [in response to Peter's remark that there are only a few responses to the 
	software survey so far] I think that even five tools is good enough for a start. However, we should 
	try to engage more tool developers and users over time so that this page remains alive rather than 
	collect cyberdust when the Summit is over.    (3RCP)
	[10:15] anonymous morphed into LamarHenderson    (3RCQ)
	[10:33] JohnMcClure: 1) waterfall v agile: "systemic" requirements lead, by themselves, to 
	significant ontology commitments that may be implemented apart from a specific domain 2) reuse v 
	reusuability: the paper seems to focus on evaluating reusability, little on reuse of published 
	ontologies to satisfy the needs of a domain and the impact of choosing one path over another (this 
	is the "not invented here' syndrome) 3) properties v classes v individuals - paper does not discuss 
	evaluating properties of an ontology v its classes - the differences in evaluation criteria that are 
	to be applied. 4) metrics - more discussion about the state of ontology metrics would be nice to see    (3RCR)
	[10:21] AstridDuqueRamos: [in response to JohnMcClure's remark that "metrics" is not covered] An 
	approach to metrics realated to OQuaRE is here 
	http://miuras.inf.um.es/evaluation/oquare/Contenido.html    (3RCS)
	[10:29] PeterYim: ref. "metrics" the HC-03 FIBO Clinic group did some really good work, which I 
	think we might make reference to, they (FICO, OOPS!, OQuaRE & OntoQA folks) are collaboratively 
	working on a document - see: 
	http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013_Hackathon_Clinics_FIBO_OOPS_OQuaRE#nid3QX 
	F    (3RCT)
	[10:28] AstridDuqueRamos: 
	https://docs.google.com/document/d/1ErbZV0IFj890lHFcnygsw6n93dxub1AamOu9oBnHdOo/edit#heading=h.kqnhd 
	tvmz5vq    (3RCU)
	[10:26] SteveRay: @John: No problem. I believe that this communique should really be viewed as a 
	portal to the large body of material present on the Ontolog wiki pages and the associated 
	presentations, discussions, and library, all archived and freely available.    (3RCV)
	[10:23] JohnMcClure: thanks steve ray    (3RCW)
	[10:28] SteveRay: @Terry: To avoid the impression this is a "how to" guide for ontology development, 
	maybe we should emphasize the "evaluation" focus more, and less on the "development" aspect.    (3RCX)
	[10:30] TerryLongstreth: @Steve - I'm just quoting from the summit home page. I'd certainly agree 
	that improving/studying operations are the real goal of evaluation.    (3RCY)
	[10:30] TerryLongstreth: c/real/one real/    (3RCZ)
	[10:33] TerryLongstreth: Requirements aren't necessarily an early point in the lifecycle, if it's 
	truly a cycle.    (3RD0)
	[10:35] AmandaVizedom: We need to say something about the *general* relationship between 
	requirements and evaluation criteria, and through there to metrics, etc.. From this we can link to 
	the metrics document cited above. Where should this go?    (3RD1)
	[10:35] AmandaVizedom: Note in diagram and text: everything feeds back into requirements!    (3RD2)
	[10:28] MichaelGruninger: @JohnMcClure: Can you give an example that illustrates your comment about 
	classes/properties?    (3RD3)
	[10:37] JohnMcClure: @MichaelGruninger: OWL attributes for properties are vastly different from 
	those for classes of course - specification of these seem a natural criteria. Which are the ones 
	most critical, which are important, which are nice-to-have? I'd hope this paper would explicitly 
	explore the *data model for ontologies*. I also believe you should eat your own dog food - using the 
	guidleines from the paper, EVALUATE RDF    (3RD4)
	[10:45] MichaelGruninger: @JohnMcClure: You explicitly refer to OWL and RDF, which is why Amanda 
	said that this issue refers to the Formal Modeling phase.    (3RD5)
	[10:50] JohnMcClure: MichaelGruninger, the section on Formal Modeling does not discuss reusing other 
	ontologies that I can see... it seems that the document presumes "not invented here' is at play in 
	designs. I disagree!    (3RD6)
	[10:51] AmandaVizedom: @Michael, exactly. Also, it's important to have a broader perspective on the 
	vast range of ontologies that are developed, used, and reused, and the range of formal languages and 
	applications in play. I said that discussion of properties and classes in OWL is "in the weeds" 
	precisely because it is a fine-grained area of consideration within a particular implementation 
	language, and even then the requirements for such an OWL ontology will be partially dependent on the 
	intended usage, including operational environment. That's more detailed than we can get - doing this 
	for each aspect of each implementation language in common use would require a book-length document. 
	What we can do is refer to collected resources that help with such details.    (3RD7)
	[10:38] JohnMcClure: Does this document point to an evaluation that's been done in the manner such 
	as it recommends? No? Why not?    (3RD8)
	[10:38] JohnMcClure: otherise the paper sounds a bit academic    (3RD9)
	[10:40] PeterYim: @JohnMcClure - the reference to the HC-03 FIBO Clinic will address this issue you 
	are bringing up    (3RDA)
	[10:40] JohnMcClure: looking for it, cannot find such reference    (3RDB)
	[10:41] PeterYim: @JohnMcClure - ref. above [10:28] & [10:29]    (3RDC)
	[10:43] JohnMcClure: thanks peter yim, (doc still trying to load....!)    (3RDD)
	[10:40] AmandaVizedom: I think that our mental models are closer to agile or continuous than to 
	waterfall or other (slower, less efficient, more unidirectional) life cycles. We want to support a 
	wide variety of entry points and paths, noting where there are dependencies and interactions between 
	phases, and emphasizing the role of evaluation everywhere.    (3RDE)
	[10:42] MatthewWest: Apologies for being so late.    (3RDF)
	[10:42] JohnMcClure: no ontologies are being reused? what about all the W3 ontologies????    (3RDG)
	[10:44] JohnMcClure: there is no glossary in the doc    (3RDH)
	[10:44] JoelBender: (as I'm in the thick of the weeds of developing my first ontology, I can't +1 
	what Amanda just said enough!)    (3RDI)
	[10:46] PeterYim: @JoelBender - can you recap that ("what Amanda just said") to provide context for 
	your comment ... (for archival purposes)    (3RDJ)
	[10:49] JoelBender: @Peter - I will review the audio recording and extract the statement for you to 
	splice into the chat log    (3RDK)
	[10:50] PeterYim: Thanks, Joel [you might want to start around time-point 1:07:40]    (3RDL)
	[10:49] DougFoxvog: [refering to the poll on choice of terminology we were about to take] Please put 
	the text options in the chat    (3RDM)
	[10:51] MichaelGruninger initiated a vote - please click the Vote button to cast your ballot: 
	Selection of Terminology (1) Quality (2) Fidelity (3) Validity This is a single choice vote.    (3RDN)
	[10:51] FabianNeuhaus voted for: Quality    (3RDO)
	[10:51] MichaelGruninger voted for: Fidelity    (3RDP)
	[10:51] PeterYim voted for: Fidelity    (3RDQ)
	[10:51] JoelBender voted for: Fidelity    (3RDR)
	[10:51] SteveRay voted for: Validity    (3RDS)
	[10:51] Bob Smith voted for: Validity    (3RDT)
	[10:52] AmandaVizedom voted for: Fidelity    (3RDU)
	[10:52] BobbinTeegarden voted for: Quality    (3RDV)
	[10:52] JohnMcClure: 2 & 3 are subclasses of Quality    (3RDW)
	[10:52] JohnMcClure voted for: Quality - base class of other two    (3RDX)
	[10:53] AstridDuqueRamos: @agree with John    (3RDY)
	[10:52] DougFoxvog voted for: Fidelity - Quality seems too broad, & validity seems too narrow    (3RDZ)
	[10:52] TerryLongstreth voted for: Validity - determine that an ontology is an accurate 
	representation of the target domain.    (3RE0)
	[10:53] SteveRay: Fidelity, to me, focuses on alignment with the domain, but doesn't bring to mind 
	the model consistency/quality. Hence I voted Validity.    (3RE1)
	[10:58] LamarHenderson abstains    (3RE2)
	[10:54] KenBaclawski voted for: Validity - Meanings of fidelity in general use: Faithfulness to a 
	person, cause, or belief, demonstrated by continuing loyalty and support. Sexual faithfulness to a 
	spouse or partner. So validity would be better.    (3RE3)
	[10:55] PeterYim: @Ken - indeed, faithfulness, as in Hi-Fi    (3RE4)
	[10:55] DougFoxvog: @Ken, i take "fidelity" to mean "faithfulness to a standard"    (3RE5)
	[10:55] MichaelGruninger: @Steve: I thought that we were voting on the term for aspect (2), which 
	explicitly mentions the domain    (3RE6)
	[10:56] SteveRay: @Michael: My problem is that (2) is the only one where Fabian/Amanda have included 
	model consistency, so I'm trying to bring that to the surface.    (3RE7)
	[10:56] JohnMcClure: i agree with the speaker    (3RE8)
	[10:57] PeterYim: @John - that's MatthewWest    (3RE9)
	[10:57] JohnMcClure: oh!    (3REA)
	[10:57] AmandaVizedom: I think Matthews point is a good one, that it isn't obvious why the things 
	gathered under (2) go together    (3REB)
	[10:57] DougFoxvog: Aren't we discussing a term for aspect 2 -- " Does the ontology represent the 
	domain appropriately"?    (3REC)
	[11:00] JohnMcClure: DougFoxvog, this use of the term "appropriately" bothers me.... I think 
	minimally is better    (3RED)
	[10:58] AmandaVizedom: @doug, yes, but Fabian's definition goes beyond that...    (3REE)
	[10:58] AmandaVizedom: see p. 8    (3REF)
	[10:58] SteveRay: @Doug: Yes, but also that's where model consistency is put.    (3REG)
	[10:58] SteveRay: I agree with Amanda that we should separate the domain stuff and the internal 
	consistency stuff.    (3REH)
	[10:58] DougFoxvog: If so, "fidelity" seems right. Validity and quality bring up other issues.    (3REI)
	[10:59] SteveRay: I think we should add a 5th high-level characteristic to capture internal quality.    (3REJ)
	[10:59] DougFoxvog: @Steve: model consistency seems to refer to an internal property -- not on 
	whether it is consistent with the domain.    (3REK)
	[11:00] SteveRay: @Doug: I completely agree, hence my desire for the 5th focus.    (3REL)
	[11:01] MichaelGruninger: Shall i close the ballot?    (3REM)
	[11:01] MichaelGruninger ended the vote - results: Selection of Terminology    (3REN)
	Tally Choice 3 Quality 5 Fidelity 4 Validity 1 Abstains    (3REO)
	[11:01] KenBaclawski: @Peter - Indeed, there are many meanings for fidelity and validity in 
	different domains. I was just showing the meanings that are commonly used. Since both fidelity and 
	validity could be used in this context, it is difficult to come up with a criterion to distinguish 
	them. Perhaps the scientific meaning is the best way to distinguish them. "In science and 
	statistics, validity is the extent to which a concept, conclusion or measurement is well-founded and 
	corresponds accurately to the real world."    (3REP)
	[11:02] DougFoxvog: Various aspects have to do with quality. I wouldn't restrict that word to a 
	single aspect.    (3REQ)
	[11:02] TerryLongstreth: @Ken - +1    (3RER)
	[11:03] JohnMcClure: KenBaclawski, that's great - let's have a glossary in the document?    (3RES)
	[11:04] KenBaclawski: In the fields of scientific modelling and simulation, fidelity refers to the 
	degree to which a model or simulation reproduces the state and behaviour of a real world object, 
	feature or condition. Fidelity is therefore a measure of the realism of a model or simulation.    (3RET)
	[11:04] PeterYim: @Ken - nice ... as Fabian said, we'll pick the term, and define it clearly    (3REU)
	[11:04] AmandaVizedom: Yes, @Ken, that is the notion of fidelity that I have in mind.    (3REV)
	[11:04] FabianNeuhaus: Domain fidelity.    (3REW)
	[11:04] DougFoxvog: I like "domain fidelity".    (3REX)
	[11:04] AmandaVizedom: Separate into two characteristics? One that has been suggested is "domain 
	fidelity" -- what about the other?    (3REY)
	[11:05] JohnMcClure: "domain coverage" works better for me, relates to overlapping sets    (3REZ)
	[11:05] SteveRay: The other one could be: Model quality, or Model well-builtedness (!)    (3RF0)
	[11:06] KenBaclawski: Perhaps "model fidelity"?    (3RF1)
	[11:06] AmandaVizedom: There is also a compatible notion of the fidelity of an information exchange 
	or translation: the degree to which the end version contains all and only the information in the 
	beginning version. applies well to modeling & representation.    (3RF2)
	[11:06] SteveRay: @Michael, what do you call what your student checks for?    (3RF3)
	[11:09] MichaelGruninger: @Steve: we refer to "ontology verification", since we are determining 
	whether or not the ontology satisfies the requirements that are formalized as intended models    (3RF4)
	[11:10] SteveRay: @Michael: So a noun capturing the property measured during "ontology verification" 
	would be a candidate for the 5th criterion.    (3RF5)
	[11:12] MichaelGruninger: @Steve: although now that you mention it, we often refer to the question 
	of whether or not we have the right requirements (intended models) for the domain as "ontology 
	validation", analogous to software validation. Perhaps I should change my vote to "validity" from 
	fidelity :-)    (3RF6)
	[11:11] JohnMcClure: ok, here is one: few "best practices" have been defined by the ontology 
	community, which can be leveraged here. There are some however, eg the W3 has identified some - 
	where do these fit in with this communique?    (3RF7)
	[11:13] JohnMcClure: If this were a peer-reviewed paper, I think it would be thrown back at ya 
	because there is no glossary, there are no references, it reads like a discussion    (3RF8)
	[11:13] JohnMcClure: this is a flawed paper at this point, sorry    (3RF9)
	[11:15] MichaelGruninger: @JohnMcClure: this isn't intended to be an academic paper, but rather the 
	synthesis of discussions that have taken place during the Summit.    (3RFA)
	[11:17] JohnMcClure: an "academic paper" === SCIENCE.... so there's little science here, sorry    (3RFB)
	[11:20] MichaelGruninger: @John: I think the better analogy here is that the Communique is a "white 
	paper" rather than a scientific paper, particularly since the intended audience is not academia    (3RFC)
	[11:23] JohnMcClure: suggest you look at w3 recs as a model    (3RFD)
	[11:14] AmandaVizedom: Deadline for substantial comments and suggestions to be considered: Tuesday, 
	23rd April 2013.    (3RFE)
	[11:14] AmandaVizedom: Prose clean-up, typos, and such, can go on a bit longer.    (3RFF)
	[11:24] PeterYim: Join us again, same time next week, for OntologySummit2013 session-15: 
	Pre-symposium preparation and review - Co-chairs: MikeDean & RamSriram - developing session 
	details at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_04_25    (3RFG)
	[11:30] PeterYim: ... For those who will be presenting at the symposium, please note that slides are 
	due on Monday Apr-29 (not Tuesday, as we would normally have it!) ... please come to the session 
	next Thursday too, so we can tie down final details!    (3RFH)
	[11:30] PeterYim: great session!    (3RFI)
	[11:31] PeterYim: -- session ended 11:29am PDT --    (3RFJ)
 -- end of in-session chat-transcript --    (3QNB)

Additional Resources:    (3QNI)

For the record ...    (3QNR)

How To Join (while the session is in progress)    (3QNS)

Conference Call Details    (3QL7)

Attendees    (3QM4)