ppy/OntologySummit2013_s14_chat-transcript_unedited_20130418a.txt ------ Chat transcript from room: summit_20130418 2013-04-18 GMT-08:00 [PDT] ------ [9:09] PeterYim: Welcome to the = OntologySummit2013: Virtual Session-14 - Thu 2013-04-18 = Summit Theme: Ontology Evaluation Across the Ontology Lifecycle Session Topic: OntologySummit2013 Communique: Final Draft Review * Session Co-chairs - Dr. FabianNeuhaus (NIST) and Dr. AmandaVizedom (Ind. Consultant) Agenda: 1. Opening (co-chairs) 2. Presentation of plans and second draft of the communique (co-lead editors) 3. Review by section and open discussion [Co-lead editors, co-editors & All] 4. Wrap-up / Announcements (co-chairs) Logistics: * Refer to details on session page at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_04_18 * (if you haven't already done so) please click on "settings" (top center) and morph from "anonymous" to your RealName (in WikiWord format) * Mute control: *7 to un-mute ... *6 to mute * Can't find Skype Dial pad? ** for Windows Skype users: it may be under the "Call" dropdown menu as "Show Dial Pad" ** for Linux Skype users: please note that the dial-pad is only available on v4.1 (or later or the earlier Skype versions 2.x,) if the dialpad button is not shown in the call window you need to press the "d" hotkey to enable it. * Note: ... it has come to our attention that our conference bridge provider is running into some problems with the "joinconference" skype connections. In case anyone gets in trouble, please try to call the phone numbers instead (e.g. from your phone, skype-out, google-voice, etc.) Attendees: FabianNeuhaus (co-chair), AmandaVizedom (co-chair), AstridDuqueRamos, BobSmith, BobbinTeegarden, DougFoxvog, JackRing, JoelBender, JohnMcClure, JulienCorman, KenBaclawski, LamarHenderson, MatthewWest, MichaelGruninger, MikeDean, PeterYim, SteveRay, TerryLongstreth . == Proceedings: == . [9:22] JoelBender: howdy! [9:32] FabianNeuhaus: working draft is at: https://docs.google.com/document/d/1jeU5DL9JuNiyxv7BDOMr0VgFIWmSKVHXcnLlcHF-Zg0/edit# [9:35] anonymous morphed into DougFoxvog [9:43] JohnMcClure: hi DougFoxvog! [9:48] TerryLongstreth: [in agreement with SteveRay's suggestion that we should include some pointers to tools on the communique] A URL pointing to a Tool Summary page should also be in the (TBS) Executive summary [9:50] TerryLongstreth: Amanda / Fabian - Is the Purpose supposed to be part of the Exec summary, or a separate section? [9:57] TerryLongstreth: There are often two reasons for an executive summary: 1) to allow someone to quickly glean the gist of the summarized document 2) to entice them to read the rest. It's not easy to strike a balance between the two [10:00] AmandaVizedom: Terry, "purpose" is not meant to be a separate section. It is now discussed in the Introduction, but we had comments last week asking for it to be addressed in the Exec Summary. [10:27] TerryLongstreth: @Amanda - re: Purpose - I suggest it include quoted or paraphrastic text referring to the original intent of this summit ("The goal for Ontology Summit 2013 is to identify best practices for ontology development and evaluation." That phrasing suggests absolutes ("best" practices) when we really ended up with a survey of current practices, the associated tools, and possible avenues for future research on practices and tools. [9:51] MichaelGruninger: [in response to Peter's remark that there are only a few responses to the software survey so far] I think that even five tools is good enough for a start. However, we should try to engage more tool developers and users over time so that this page remains alive rather than collect cyberdust when the Summit is over. [10:15] anonymous morphed into LamarHenderson [10:33] JohnMcClure: 1) waterfall v agile: "systemic" requirements lead, by themselves, to significant ontology commitments that may be implemented apart from a specific domain 2) reuse v reusuability: the paper seems to focus on evaluating reusability, little on reuse of published ontologies to satisfy the needs of a domain and the impact of choosing one path over another (this is the "not invented here' syndrome) 3) properties v classes v individuals - paper does not discuss evaluating properties of an ontology v its classes - the differences in evaluation criteria that are to be applied. 4) metrics - more discussion about the state of ontology metrics would be nice to see [10:21] AstridDuqueRamos: [in response to JohnMcClure's remark that "metrics" is not covered] An approach to metrics realated to OQuaRE is here http://miuras.inf.um.es/evaluation/oquare/Contenido.html [10:29] PeterYim: ref. "metrics" the HC-03 FIBO Clinic group did some really good work, which I think we might make reference to, they (FICO, OOPS!, OQuaRE & OntoQA folks) are collaboratively working on a document - see: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013_Hackathon_Clinics_FIBO_OOPS_OQuaRE#nid3QXF [10:28] AstridDuqueRamos: https://docs.google.com/document/d/1ErbZV0IFj890lHFcnygsw6n93dxub1AamOu9oBnHdOo/edit#heading=h.kqnhdtvmz5vq [10:26] SteveRay: @John: No problem. I believe that this communique should really be viewed as a portal to the large body of material present on the Ontolog wiki pages and the associated presentations, discussions, and library, all archived and freely available. [10:23] JohnMcClure: thanks steve ray [10:28] SteveRay: @Terry: To avoid the impression this is a "how to" guide for ontology development, maybe we should emphasize the "evaluation" focus more, and less on the "development" aspect. [10:30] TerryLongstreth: @Steve - I'm just quoting from the summit home page. I'd certainly agree that improving/studying operations are the real goal of evaluation. [10:30] TerryLongstreth: c/real/one real/ [10:33] TerryLongstreth: Requirements aren't necessarily an early point in the lifecycle, if it's truly a cycle. [10:35] AmandaVizedom: We need to say something about the *general* relationship between requirements and evaluation criteria, and through there to metrics, etc.. From this we can link to the metrics document cited above. Where should this go? [10:35] AmandaVizedom: Note in diagram and text: everything feeds back into requirements! [10:28] MichaelGruninger: @JohnMcClure: Can you give an example that illustrates your comment about classes/properties? [10:37] JohnMcClure: @MichaelGruninger: OWL attributes for properties are vastly different from those for classes of course - specification of these seem a natural criteria. Which are the ones most critical, which are important, which are nice-to-have? I'd hope this paper would explicitly explore the *data model for ontologies*. I also believe you should eat your own dog food - using the guidleines from the paper, EVALUATE RDF [10:45] MichaelGruninger: @JohnMcClure: You explicitly refer to OWL and RDF, which is why Amanda said that this issue refers to the Formal Modeling phase. [10:50] JohnMcClure: MichaelGruninger, the section on Formal Modeling does not discuss reusing other ontologies that I can see... it seems that the document presumes "not invented here' is at play in designs. I disagree! [10:51] AmandaVizedom: @Michael, exactly. Also, it's important to have a broader perspective on the vast range of ontologies that are developed, used, and reused, and the range of formal languages and applications in play. I said that discussion of properties and classes in OWL is "in the weeds" precisely because it is a fine-grained area of consideration within a particular implementation language, and even then the requirements for such an OWL ontology will be partially dependent on the intended usage, including operational environment. That's more detailed than we can get - doing this for each aspect of each implementation language in common use would require a book-length document. What we can do is refer to collected resources that help with such details. [10:38] JohnMcClure: Does this document point to an evaluation that's been done in the manner such as it recommends? No? Why not? [10:38] JohnMcClure: otherise the paper sounds a bit academic [10:40] PeterYim: @JohnMcClure - the reference to the HC-03 FIBO Clinic will address this issue you are bringing up [10:40] JohnMcClure: looking for it, cannot find such reference [10:41] PeterYim: @JohnMcClure - ref. above [10:28] & [10:29] [10:43] JohnMcClure: thanks peter yim, (doc still trying to load....!) [10:40] AmandaVizedom: I think that our mental models are closer to agile or continuous than to waterfall or other (slower, less efficient, more unidirectional) life cycles. We want to support a wide variety of entry points and paths, noting where there are dependencies and interactions between phases, and emphasizing the role of evaluation everywhere. [10:42] MatthewWest: Apologies for being so late. [10:42] JohnMcClure: no ontologies are being reused? what about all the W3 ontologies???? [10:44] JohnMcClure: there is no glossary in the doc [10:44] JoelBender: (as I'm in the thick of the weeds of developing my first ontology, I can't +1 what Amanda just said enough!) [10:46] PeterYim: @JoelBender - can you recap that ("what Amanda just said") to provide context for your comment ... (for archival purposes) [10:49] JoelBender: @Peter - I will review the audio recording and extract the statement for you to splice into the chat log [10:50] PeterYim: Thanks, Joel [you might want to start around time-point 1:07:40] [10:49] DougFoxvog: [refering to the poll on choice of terminology we were about to take] Please put the text options in the chat [10:51] MichaelGruninger initiated a vote - please click the Vote button to cast your ballot: Selection of Terminology (1) Quality (2) Fidelity (3) Validity This is a single choice vote. [10:51] FabianNeuhaus voted for: Quality [10:51] MichaelGruninger voted for: Fidelity [10:51] PeterYim voted for: Fidelity [10:51] JoelBender voted for: Fidelity [10:51] SteveRay voted for: Validity [10:51] Bob Smith voted for: Validity [10:52] AmandaVizedom voted for: Fidelity [10:52] BobbinTeegarden voted for: Quality [10:52] JohnMcClure: 2 & 3 are subclasses of Quality [10:52] JohnMcClure voted for: Quality - base class of other two [10:53] AstridDuqueRamos: @agree with John [10:52] DougFoxvog voted for: Fidelity - Quality seems too broad, & validity seems too narrow [10:52] TerryLongstreth voted for: Validity - determine that an ontology is an accurate representation of the target domain. [10:53] SteveRay: Fidelity, to me, focuses on alignment with the domain, but doesn't bring to mind the model consistency/quality. Hence I voted Validity. [10:58] LamarHenderson abstains [10:54] KenBaclawski voted for: Validity - Meanings of fidelity in general use: Faithfulness to a person, cause, or belief, demonstrated by continuing loyalty and support. Sexual faithfulness to a spouse or partner. So validity would be better. [10:55] PeterYim: @Ken - indeed, faithfulness, as in Hi-Fi [10:55] DougFoxvog: @Ken, i take "fidelity" to mean "faithfulness to a standard" [10:55] MichaelGruninger: @Steve: I thought that we were voting on the term for aspect (2), which explicitly mentions the domain [10:56] SteveRay: @Michael: My problem is that (2) is the only one where Fabian/Amanda have included model consistency, so I'm trying to bring that to the surface. [10:56] JohnMcClure: i agree with the speaker [10:57] PeterYim: @John - that's MatthewWest [10:57] JohnMcClure: oh! [10:57] AmandaVizedom: I think Matthews point is a good one, that it isn't obvious why the things gathered under (2) go together [10:57] DougFoxvog: Aren't we discussing a term for aspect 2 -- " Does the ontology represent the domain appropriately"? [11:00] JohnMcClure: DougFoxvog, this use of the term "appropriately" bothers me.... I think "minimally" is better [10:58] AmandaVizedom: @doug, yes, but Fabian's definition goes beyond that... [10:58] AmandaVizedom: see p. 8 [10:58] SteveRay: @Doug: Yes, but also that's where model consistency is put. [10:58] SteveRay: I agree with Amanda that we should separate the domain stuff and the internal consistency stuff. [10:58] DougFoxvog: If so, "fidelity" seems right. Validity and quality bring up other issues. [10:59] SteveRay: I think we should add a 5th high-level characteristic to capture internal quality. [10:59] DougFoxvog: @Steve: model consistency seems to refer to an internal property -- not on whether it is consistent with the domain. [11:00] SteveRay: @Doug: I completely agree, hence my desire for the 5th focus. [11:01] MichaelGruninger: Shall i close the ballot? [11:01] MichaelGruninger ended the vote - results: Selection of Terminology Tally Choice 3 Quality 5 Fidelity 4 Validity 1 Abstains [11:01] KenBaclawski: @Peter - Indeed, there are many meanings for fidelity and validity in different domains. I was just showing the meanings that are commonly used. Since both fidelity and validity could be used in this context, it is difficult to come up with a criterion to distinguish them. Perhaps the scientific meaning is the best way to distinguish them. "In science and statistics, validity is the extent to which a concept, conclusion or measurement is well-founded and corresponds accurately to the real world." [11:02] DougFoxvog: Various aspects have to do with quality. I wouldn't restrict that word to a single aspect. [11:02] TerryLongstreth: @Ken - +1 [11:03] JohnMcClure: KenB, that's great - let's have a glossary in the document? [11:04] KenBaclawski: In the fields of scientific modelling and simulation, fidelity refers to the degree to which a model or simulation reproduces the state and behaviour of a real world object, feature or condition. Fidelity is therefore a measure of the realism of a model or simulation. [11:04] PeterYim: @Ken - nice ... as Fabian said, we'll pick the term, and define it clearly [11:04] AmandaVizedom: Yes, @Ken, that is the notion of fidelity that I have in mind. [11:04] FabianNeuhaus: Domain fidelity. [11:04] DougFoxvog: I like "domain fidelity". [11:04] AmandaVizedom: Separate into two characteristics? One that has been suggested is "domain fidelity" -- what about the other? [11:05] JohnMcClure: "domain coverage" works better for me, relates to overlapping sets [11:05] SteveRay: The other one could be: Model quality, or Model well-builtedness (!) [11:06] KenBaclawski: Perhaps "model fidelity"? [11:06] AmandaVizedom: There is also a compatible notion of the fidelity of an information exchange or translation: the degree to which the end version contains all and only the information in the beginning version. applies well to modeling & representation. [11:06] SteveRay: @Michael, what do you call what your student checks for? [11:09] MichaelGruninger: @Steve: we refer to "ontology verification", since we are determining whether or not the ontology satisfies the requirements that are formalized as intended models [11:10] SteveRay: @Michael: So a noun capturing the property measured during "ontology verification" would be a candidate for the 5th criterion. [11:12] MichaelGruninger: @Steve: although now that you mention it, we often refer to the question of whether or not we have the right requirements (intended models) for the domain as "ontology validation", analogous to software validation. Perhaps I should change my vote to "validity" from "fidelity" :-) [11:11] JohnMcClure: ok, here is one: few "best practices" have been defined by the ontology community, which can be leveraged here. There are some however, eg the W3 has identified some - where do these fit in with this communique? [11:13] JohnMcClure: If this were a peer-reviewed paper, I think it would be thrown back at ya because there is no glossary, there are no references, it reads like a discussion [11:13] JohnMcClure: this is a flawed paper at this point, sorry [11:15] MichaelGruninger: @JohnMcClure: this isn't intended to be an academic paper, but rather the synthesis of discussions that have taken place during the Summit. [11:17] JohnMcClure: an "academic paper" === SCIENCE.... so there's little science here, sorry [11:20] MichaelGruninger: @John: I think the better analogy here is that the Communique is a "white paper" rather than a scientific paper, particularly since the intended audience is not academia [11:23] JohnMcClure: suggest you look at w3 recs as a model [11:14] AmandaVizedom: Deadline for substantial comments and suggestions to be considered: Tuesday, 23rd April 2013. [11:14] AmandaVizedom: Prose clean-up, typos, and such, can go on a bit longer. [11:24] PeterYim: Join us again, same time next week, for OntologySummit2013 session-15: "Pre-symposium preparation and review" - Co-chairs: MikeDean & RamSriram - developing session details at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_04_25 [11:30] PeterYim: ... For those who will be presenting at the symposium, please note that slides are due on Monday Apr-29 (not Tuesday, as we would normally have it!) ... please come to the session next Thursday too, so we can tie down final details! [11:30] PeterYim: great session! [11:31] PeterYim: -- session ended 11:29am PDT -- ------