ppy/OntologySummit2013_s14_chat-transcript_unedited_20130418a.txt ------ Chat transcript from room: summit_20130418 2013-04-18 GMT-08:00 [PDT] ------ [9:09] PeterYim: Welcome to the = OntologySummit2013: Virtual Panel Session-14 - Thu 2013-04-18 = Summit Theme: Ontology Evaluation Across the Ontology Lifecycle Session Topic: OntologySummit2013 Communique: Final Draft Review * Session Co-chairs - Dr. FabianNeuhaus (NIST) and Dr. AmandaVizedom (Ind. Consultant) Agenda: 1. Opening (co-chairs) 2. Presentation of plans and second draft of the communique (co-lead editors) 3. Review by section and open discussion [Co-lead editors, co-editors & All] 4. Wrap-up / Announcements (co-chairs) Logistics: * Refer to details on session page at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_04_18 * (if you haven't already done so) please click on "settings" (top center) and morph from "anonymous" to your RealName (in WikiWord format) * Mute control: *7 to un-mute ... *6 to mute * Can't find Skype Dial pad? ** for Windows Skype users: it may be under the "Call" dropdown menu as "Show Dial Pad" ** for Linux Skype users: please note that the dial-pad is only available on v4.1 (or later or the earlier Skype versions 2.x,) if the dialpad button is not shown in the call window you need to press the "d" hotkey to enable it. * Note: ... it has come to our attention that our conference bridge provider is running into some problems with the "joinconference" skype connections. In case anyone gets in trouble, please try to call the phone numbers instead (e.g. from your phone, skype-out, google-voice, etc.) Attendees: AmandaVizedom, AstridDuqueRamos, BobSmith, BobbinTeegarden, DougFoxvog, FabianNeuhaus, JackRing, JoelBender, JohnMcClure, JulienCorman, KenBaclawski, LamarHenderson, MatthewWest, MichaelGruninger, MikeDean, PeterYim, SteveRay, TerryLongstreth, anonymous, . == Proceedings: == . [9:22] JoelBender: howdy! [9:32] FabianNeuhaus: https://docs.google.com/document/d/1jeU5DL9JuNiyxv7BDOMr0VgFIWmSKVHXcnLlcHF-Zg0/edit# [9:35] anonymous morphed into dougFoxvog [9:43] John McClure: hi dougfoxvog! [9:48] TerryLongstreth: A URL pointing to a Tool Summary page should also be in the (TBS) Executive summary [9:50] TerryLongstreth: Amanda/Fabian - Is the Purpose supposed to be part of the Exec summary, or a separate section? [9:51] MichaelGruninger: I think that even five tools is good enough for a start. However, we should try to engage more tool developers and users over time so that this page remains alive rather than collect cyberdust when the Summit is over. [9:57] TerryLongstreth: There are often two reasons for an executive summary: 1) to allow someone to quickly glean the gist of the summarized document 2) to entice them to read the rest. It's not easy to strike a balance between the two [10:00] AmandaVizedom: Terry, purpose is not meant to be a separate section. It is now discussed in the Introduction, but we had comments last week asking for it to be addressed in the Exec Summary. [10:15] anonymous morphed into LamarHenderson [10:21] AstridDuqueRamos: An approach to metrics realated to OQuaRE [10:21] AstridDuqueRamos: is here http://miuras.inf.um.es/evaluation/oquare/Contenido.html [10:23] John McClure: thanks steve ray [10:26] SteveRay: @John: No problem. I believe that this communique should really be viewed as a portal to the large body of material present on the Ontolog wiki pages and the associated presentations, discussions, and library, all archived and freely available. [10:27] TerryLongstreth: @Amanda - re: Purpose - I suggest it include quoted or paraphrastic text referring to the original intent of this summit ("The goal for Ontology Summit 2013 is to identify best practices for ontology development and evaluation." That phrasing suggests absolutes ("best" practices) when we really ended up with a survey of current practices, the associated tools, and possible avenues for future research on practices and tools. [10:28] MichaelGruninger: @John McClure: Can you give an example that illustrates your comment about classes/properties? [10:28] SteveRay: @Terry: To avoid the impression this is a "how to" guide for ontology development, maybe we should emphasize the "evaluation" focus more, and less on the "development" aspect. [10:28] AstridDuqueRamos: https://docs.google.com/document/d/1ErbZV0IFj890lHFcnygsw6n93dxub1AamOu9oBnHdOo/edit#heading=h.kqnhdtvmz5vq [10:29] PeterYim: ref. "metrics" the HC-03 FIBO Clinic group did some really good work, which I think we might make reference to, they (FICO, OOPS!, OQuaRE & OntoQA folks) are collaboratively working on a document - see: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013_Hackathon_Clinics_FIBO_OOPS_OQuaRE#nid3QXF [10:30] TerryLongstreth: @Steve - I'm just quoting from the summit home page. I'd certainly agree that improving/studying operations are the real goal of evaluation. [10:30] TerryLongstreth: c/real/one real/ [10:33] John McClure: 1) waterfall v agile: "systemic" requirements lead, by themselves, to significant ontology committments that may be implemented apart from a specific domain 2) reuse v reusuability: the paper seems to focus on evaluating reusability, little on reuse of published ontologies to satisfy the needs of a domain and the impact of choosing one path over another (this is the "not invented here' syndrome) 3) properties v classes v individs - paper does not discuss evaluating properties of an onotlogy v its classes - the differences in evaluation critera that are to be applied. 4) metrics - more discussion about the state of ontology metrics would be nice to see [10:33] TerryLongstreth: Requirements aren't necessarily an early point in the lifecycle, if it's truly a cycle. [10:35] AmandaVizedom: We need to say something about the *general* relationship between requirements and evaluation criteria, and through there to metrics, etc.. From this we can link to the metrics document cited above. Where should this go? [10:35] AmandaVizedom: Note in diagram and text: everything feeds back into requirements! [10:37] John McClure: Michael Gruninger: OWL attributes for properties are vastly different from those for classes of course - specification of these seem a natural criteria. Which are the ones most critical, which are important, which are nice-to-have? I'd hope this paper would explicitly explore the *data model for ontologies*. I also believe you should eat your own dog food - using the guidleines from the paper, EVALUATE RDF [10:38] John McClure: Does this document point to an evaluation that's been done in the manner such as it recommends? No? Why not? [10:38] John McClure: otherise the paper sounds a bit academic [10:40] PeterYim: @JohnMcClure - the reference to the HC-03 FIBO Clinic will address this issue you are bringing up [10:40] John McClure: looking for it, cannot find such reference [10:40] AmandaVizedom: I think that our mental models are closer to agile or continuous than to waterfall or other (slower, less efficient, more unidirectional) life cycles. We want to support a wide variety of entry points and paths, noting where there are dependencies and interactions between phases, and emphasizing the role of evaluation everywhere. [10:41] PeterYim: ref. above [+1:28] & [+1:29] [10:41] PeterYim: ^(above is for JohnMcClure) [10:42] MatthewWest: Apologies for being so late. [10:42] John McClure: no ontologies are being reused? what about all the W3 ontologues???? [10:43] John McClure: thanks peter yim, (doc still trying to load....!) [10:44] JoelBender: (as I'm in the thick of the weeds of developing my first ontology, I can't +1 what Amanda just said enough!) [10:44] John McClure: there is no glossary in the doc [10:45] MichaelGruninger: @John McClure: You explicitly refer to OWL and RDF, which is why Amanda said that this issue refers to the Formal Modeling phase. [10:46] PeterYim: @JoelBender - can you recap that ("what Amanda just said") to provide context for your comment ... (for archival purpposes) [10:49] dougFoxvog: Please put the text options in the chat [10:49] MichaelGruninger initiated a vote - please click the Vote button to cast your ballot: Selection of Terminology (1) Quality (2) Fidelity (3) Validity This is a single choice vote. [10:49] JoelBender: @Peter - I will review the audio recording and extract the statement for you to splice into the chat log [10:50] PeterYim: Thanks, Joel [10:50] MichaelGruninger voted for: Fidelity [10:50] LamarHenderson ended the vote - results: Selection of Terminology Tally Choice 0 Quality 1 Fidelity 0 Validity 0 Abstains [10:50] John McClure: MichaelGruninger, the section on Formal Modeling does not discuss reusing other ontologies that I can see... it seems that the document presumes "not invented here' is at play in designs. I disagree! [10:51] AmandaVizedom: @Michael, exactly. Also, it's important to have a broader perspective on the vast range of ontologies that are developed, used, and reused, and the range of formal languages and applications in play. I said that discussion of propoerties and classes in OWL is "in the weeds" precisely because it is a fine-grained area of consideration within a particular implementation language, and even then the requirements for such an OWL ontology will be partially dependent on the intended usage, including operational environment. That's more detailed than we can get - doing this for each aspect of each implementation language in common use would require a book-length document. What we can do is refer to collected resources that help with such details. [10:51] MichaelGruninger initiated a vote - please click the Vote button to cast your ballot: Selection of Terminology (1) Quality (2) Fidelity (3) Validity This is a single choice vote. [10:51] FabianNeuhaus voted for: Quality [10:51] MichaelGruninger voted for: Fidelity [10:51] PeterYim voted for: Fidelity [10:51] JoelBender voted for: Fidelity [10:51] SteveRay voted for: Validity [10:51] Bob Smith voted for: Validity [10:52] AmandaVizedom voted for: Fidelity [10:52] BobbinTeegarden voted for: Quality [10:52] John McClure: 2 & 3 are subclasses of Quality [10:52] dougFoxvog voted for: Fidelity - Quality seems too broad, & validity seems too narrow [10:52] John McClure voted for: Quality - base class of other two [10:52] TerryLongstreth voted for: Validity - determine that an ontology is an accurate representation of the target domain. [10:53] AstridDuqueRamos: @agree with John [10:53] SteveRay: Fidelity, to me, focuses on alignment with the domain, but doesn't bring to mind the model consistency/quality. Hence I voted Validity. [10:54] KenBaclawski voted for: Validity - Meanings of fidelity in general use: Faithfulness to a person, cause, or belief, demonstrated by continuing loyalty and support. Sexual faithfulness to a spouse or partner. So validity would be better. [10:55] PeterYim: @Ken - indeed, faithfulness, as in Hi-Fi [10:55] dougFoxvog: @Ken, i take "fidelity" to mean "faithfulness to a standard" [10:55] MichaelGruninger: @Steve: I thought that we were voting on the term for aspect (2), which explicitly mentions the domain [10:56] PeterYim: @LamarHenderson - Michael re-started the poll, so please vote again [10:56] John McClure: i agree with the speaker [10:56] SteveRay: @Michael: My problem is that (2) is the only one where Fabian/Amanda have included model consistency, so I'm trying to bring that to the surface. [10:57] PeterYim: @John - that's MatthewWest [10:57] John McClure: oh! [10:57] AmandaVizedom: I think Matthews point is a good one, that it isn't obvious why the things gathered under (2) go together [10:57] dougFoxvog: Aren't we discussing a term for aspect 2 -- " Does the ontology represent the domain appropriately"? [10:58] AmandaVizedom: @doug, yes, but Fabian's definition goes beyond that... [10:58] SteveRay: @Doug: Yes, but also that's where model consistency is put. [10:58] AmandaVizedom: see p. 8 [10:58] SteveRay: I agree with Amanda that we should separate the domain stuff and the internal consistency stuff. [10:58] dougFoxvog: If so, "fidelity" seems right. Validity and quality bring up other issues. [10:58] LamarHenderson abstains [10:59] SteveRay: I think we should add a 5th high-level characteristic to capture internal quality. [10:59] dougFoxvog: @Steve: model consistency seems to refer to an internal property -- not on whether it is consistent with the domain. [11:00] SteveRay: @Doug: I completely agree, hence my desire for the 5th focus. [11:00] John McClure: dougfoxvog, this use of the term "appropriately" bothers me.... I think "minimally" is better [11:01] MichaelGruninger: Shall i close the ballot? [11:01] MichaelGruninger ended the vote - results: Selection of Terminology Tally Choice 3 Quality 5 Fidelity 4 Validity 1 Abstains [11:01] KenBaclawski: @Peter - Indeed, there are many meanings for fidelity and validity in different domains. I was just showing the meanings that are commonly used. Since both fidelity and validity could be used in this context, it is difficult to come up with a criterion to distinguish them. Perhaps the scientific meaning is the best way to distinguish them. "In science and statistics, validity is the extent to which a concept, conclusion or measurement is well-founded and corresponds accurately to the real world." [11:02] dougFoxvog: Various aspects have to do with quality. I wouldn't restrict that word to a single aspect. [11:02] TerryLongstreth: @Ken - +1 [11:03] John McClure: KenB, that's great - let's have a glossary in the document? [11:04] AmandaVizedom: Separate into two characteristics? One that has been suggested is "domain fidelity" -- what about the other? [11:04] FabianNeuhaus: Domain fidelity. [11:04] KenBaclawski: In the fields of scientific modelling and simulation, fidelity refers to the degree to which a model or simulation reproduces the state and behaviour of a real world object, feature or condition. Fidelity is therefore a measure of the realism of a model or simulation. [11:04] dougFoxvog: I like "domain fidelity". [11:04] PeterYim: @Ken - nice ... as Fabian said, we'll pick the term, and define it clearly [11:04] AmandaVizedom: Yes, @Ken, that is the notion of fidelity that I have in mind. [11:05] SteveRay: The other one could be: Model quality, or Model well-builtedness (!) [11:05] John McClure: domain coverage works better for me, relates to overlapping sets [11:05] John McClure: "domain coverage" i mean [11:06] KenBaclawski: Perhaps "model fidelity"? [11:06] SteveRay: @Michael, what do you call what your student checks for? [11:06] AmandaVizedom: There is also a compatible notion of the fidelity of an information exchange or translation: the degree to which the end version contains all and only the information in the beginning version. applies well to modeling & representation. [11:09] MichaelGruninger: @Steve: we refer to "ontology verification", since we are determining whether or not the ontology satisfies the requirements that are formalized as intended models [11:10] SteveRay: @Michael: So a noun capturing the property measured during "ontology verification" would be a candidate for the 5th criterion. [11:11] John McClure: ok, here is one: few "best practices" have been defined by the ontology community, which can be leveraged here. There are some however, eg the W3 has identified some - whete do these fit in with this communique? [11:12] MichaelGruninger: @Steve: although now that you mention it, we often refer to the question of whether or not we have the right requirements (intended models) for the domain as "ontology validation", analogous to software validation. Perhaps I should change my vote to "validity" from "fidelity" :-) [11:13] John McClure: If this were a peer-reviewed paper, I think it would be thrown back at ya because there is no glossary, there are no references, it reads like a discussion [11:13] John McClure: this is a flawed paper at this point, sorry [11:14] AmandaVizedom: Deadline for substantial comments and suggestions to be considered: Tuesday, 23rd April 2013. [11:14] AmandaVizedom: Prose clean-up, typos, and such, can go on a bit longer. [11:15] MichaelGruninger: @John McClure: this isn't intended to be an academic paper, but rather the synthesis of discussions that have taken place during the Summit. [11:17] John McClure: an "academic paper" === SCIENCE.... so there's little sciece here, sorry [11:20] MichaelGruninger: @John: I think the better analogy here is that the Communique is a "white paper" rather than a scientific paper, particularly since the intended audience is not academia [11:23] John McClure: suggest you look at w3 recs as a model [11:24] PeterYim: Join us again, same time next week, for OntologySummit2013 session-15: "Pre-symposium preparation and review" - Co-chairs: MikeDean & RamSriram - developing session details at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_04_25 [11:30] PeterYim: ... For those who will be presenting at the symposium, please note that slide are due on Monday Apr-29 (not Tuesday, as we would normally have it!) ... please come to the session next Thursday too, so we can tie down final details! [11:30] PeterYim: great session! [11:31] PeterYim: -- session ended 11:29am PDT -- [11:31] List of attendees: AmandaVizedom, AstridDuqueRamos, Bob Smith, BobbinTeegarden, FabianNeuhaus, JackRing, JackRing1, JoelBender, John McClure, JulienCorman, KenBaclawski, LamarHenderson, MatthewWest, MichaelGruninger, MikeDean, PeterYim, SteveRay, TerryLongstreth, anonymous, dougFoxvog, vnc2 ------