ppy/OntologySummit2013_s13_chat-transcript_edited_20130411b.txt ------ Chat transcript from room: summit_20130411 2013-04-11 GMT-08:00 [PDT] ------ [8:55] PeterYim: Welcome to the = OntologySummit2013: Virtual Panel Session-13 - Thu 2013-04-11 = Summit Theme: Ontology Evaluation Across the Ontology Lifecycle Session Topic: OntologySummit2013 Communique: First Draft Review * Session Co-chairs - Dr. AmandaVizedom (Ind. Consultant) and Dr. FabianNeuhaus (NIST) Agenda: 1. Opening (co-chairs) ... ref.: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_04_11#nid3QHP 2. Presentation of plans and first draft of the communique (co-lead editors) 3. Review by section and open discussion [Co-lead editors, co-editors & All] 4. Wrap-up / Announcements (co-chairs) Logistics: * Refer to details on session page at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_04_11 * (if you haven't already done so) please click on "settings" (top center) and morph from "anonymous" to your RealName (in WikiWord format) * Mute control: *7 to un-mute ... *6 to mute * Can't find Skype Dial pad? ** for Windows Skype users: it may be under the "Call" dropdown menu as "Show Dial pad" ** for Linux Skype users: please note that the dial-pad is only available on v4.1 (or later or the earlier Skype versions 2.x,) if the dialpad button is not shown in the call window you need to press the "d" hotkey to enable it. * Note: ... it has come to our attention that our conference bridge provider is running into some problems with the "joinconference" skype connections. In case anyone gets in trouble, please try to call the phone numbers instead (e.g. from your phone, skype-out, google-voice, etc.) Attendees: AmandaVizedom (co-chair), FabianNeuhaus (co-chair), AliHashemi, AstridDuqueRamos, BobSmith, BobbinTeegarden, BruceBray, CarmenChui, ClarePaul, DougFoxvog, FrancescaQuattri, JoanneLuciano, JoelBender, JulienCorman, KenBaclawski, LamarHenderson, MatthewWest, MeganKatsumi, MichaelGruninger, MikeBennett, MikeDean, NancyWiegand, PavithraKenjige, PeterYim, RamSriram, ShariThurow, SimonSpero, SteveRay, TerryLongstreth, ToddSchneider, TorstenHahmann, == Proceedings: == [9:23] anonymous morphed into CarmenChui [9:29] anonymous morphed into ClarePaul [9:31] PeterYim: == FabianNeuhaus starts the session on behalf of the session co-chairs ... see: [ 0-Chair ] slides [9:34] anonymous morphed into FrancescaQuattri [9:34] List of members: AliHashemi, AmandaVizedom, AstridDuqueRamos, BobSmith, BruceBray, CarmenChui, ClarePaul, DougFoxvog, FabianNeuhaus, FrancescaQuattri, JoelBender, JulienCorman, KenBaclawski, MatthewWest, MeganKatsumi, MichaelGruninger, MikeDean, PeterYim, RamSriram, SteveRay, TerryLongstreth, vnc2 [9:36] SteveRay: My suggestion for a 5th high-level question for the bottom of page 1 on the communique: We should include something that covers intrinsic evaluation, such as: -Is the ontology free of obvious inconsistencies and errors in modeling? [9:39] AmandaVizedom: Slide 6 - schedule -- is off. Next week, April 18, is the 2nd draft review. Final draft will be complete by May 1. [9:42] ToddSchneider: Change 'IT' to 'information'. [9:43] PeterYim: +1 re ToddSchneider suggestion above [9:44] anonymous morphed into ShariThurow [9:45] AmandaVizedom: Fabian notes: We are going to have some terminology clashes. We will need to discuss what are the best terms to use for some items; we can then do a global search & replace. [9:46] ToddSchneider: Amanda, who is the intended audience? That should drive terminology issues. [9:46] MatthewWest: On Quality. Some of what we are doing here is rediscovering things that have been well established over some 50 years in other areas. It is useful for us to use other language when we are discovering that we are doing the same thing as others have done before, but once we have understood it is the same thing, we should adopt the language that is established and try to learn from their experience rather than re-invent it. I would hope we had reached that point during this summit. [9:52] SimonSpero: @MatthewWest: Are any of the definition sections of ISO 9K/9K1 available in the public domain? [10:05] SimonSpero: ASQ glossary: http://asq.org/glossary/ [10:05] MatthewWest: @Simon how about this for quality related definitions: http://www.praxiom.com/iso-definition.htm these look pretty good to me. You will noticed that we have independently identified the need for many of these things. [9:51] AmandaVizedom: @Matthew, I agree. We discussed this, but I failed to come up with a satisfactory alternative. So, we decided instead to put this question, of what terminology to use where we have "model quality". Suggestions? [9:55] MatthewWest: @Amanda I'm quite happy to review the document and make specific suggestions so that the text is consistent with standard quality management terminology if that would be helpful. However, there is a wider point. I take the adoption of the language we use as an indicator of the communities maturity in this area. So if the community is not yet comfortable with quality terminology, then it is better to note this as an issue, than adopt the language when it does not reflect where the community is. [9:56] ToddSchneider: The term 'quality' should be banished from the communique (sorry Matthew, but use of this term does [not] help). [10:09] MatthewWest: @Todd Interestingly when you talk about quality management, you tend not to use the term quality very much, beyond the name of the process, mostly you talk about meeting requirements. [9:48] ShariThurow: If this helps, my Information Architecture Institute colleague came up with an RDF vocab for content inventories: http://privatealpha.com/ontology/content-inventory/1# [9:57] ToddSchneider: Amanda, if ontology development is so similar to software, why not state it. [10:04] AmandaVizedom: Todd, we do. [9:49] ToddSchneider: The diagram in the Lifecycle Overview section should be better aligned to more common systems engineering 'stages' (e.g., the SE V). [9:51] SteveRay: @Todd: I think I agree with you. Or at least we should make reference to the classic SE phases. [10:00] ToddSchneider: Have to go. I'll be sending 'suggestions'. [9:58] JoanneLuciano: jumping in with a comment (i have not been following - so it this is off or redundant just ignore): in the interim (at least) in the Exec summary -- What is the purpose of this document would be useful (esp since it is for a wider audience) [10:00] JoanneLuciano: RE: life cycle....figure -- the figure doesn't depict a "cycle" shouldn't it? [10:02] JoanneLuciano: If not, then maybe developmental stage or some other analogy would be better [10:02] FabianNeuhaus: @Joanne: true. I always thought that lifecycle is a misnomer [10:06] TerryLongstreth: @Joanne - +1.. The cycle should show feedback from operational experience to requirements changes and resulting ontology changes [10:09] PeterYim: the ontolology lifecycle diagram seems a bit incomplete, I suggest looking at the 7 ontology lifecycle phases from MikeDenny as used in the OntologySummit2013 Survey of software capabilities: Exploration Phase Management Phase Design Phase Build Phase General Development Concept Development Relationship Development Validation Phase Integration & Use Phase Maintenance Phase [10:26] PeterYim: @Amanda & Fabian - my issue is with the diagram only ... I can see that you guys have covered that well in the "Model Build Phase" writeup [10:10] ShariThurow: Where is user/usability testing in this list? [10:20] PeterYim: Further to the above [10:09] MikeDenny's seven ontology lifecycle phases, even that set is missing the phases where one implements - formalizes and reduces the design into a representation in an ontology language. I think that should be added into the diagram two ... and presumably, there are very specific ontology evaluation needs at that phase of the lifecycle [10:17] SimonSpero: @PeterYim: This presupposes the older forms of software engineering. In more recent approaches, testing and integration occurs at all times, even during requirements gathering, (which is iterative) [10:26] SimonSpero: http://commons.wikimedia.org/wiki/File:Agile_Software_Development_methodology.svg [10:22] PeterYim: @SimonSpero - totally agree ... of course MikeDenny's focus was on software tools/environments (and so he was light on the non-tools related work ... and probably just group them under the "design phase") [10:25] FabianNeuhaus: @Peter: is there a document where the phases listed above are explained? [10:30] PeterYim: @FabianNeuhaus [10:25] - see what got captured at the "Ontology of Ontology Evaluation" Hackathon HC-05 - (p.11~15) of http://ontolog.cim3.net/file/work/OntologySummit2013/Hackathon-Clinics/HC-05_Ontology-of-OntologyEvaluation/wip/HC-05_doc-snapshot_at-end-day-20130331/2013-03-30RequirementsGatheringWorkingFile_20130331_at-end-day2.rtf [10:28] TerryLongstreth: on Denny's lifecycle - I recommend Integration and Use be two phases (Delivery/Deployment and Operational Use) From the latter we should show (repair and new requirements requests) going into Maintenance, and a from there (change actions) splitting into separate lines to management, design, and build. [10:31] SimonSpero: @TerryLongsteth: Is Integration necessarily a separate phase or is that just a feature of some lifecycle [10:39] TerryLongstreth: @Simon [10:31] - That's why I re-labeled it Delivery/deployment. In general, Final integration is essentially packaging up the product for deployment. It's assumed to include a final factory integration test. [10:39] AmandaVizedom: NB: these steps need not be separate in time or cleanly separated. In fact, they will often overlap and blur together. Ontological Analysis and Modelling (informal or formal) may support each other and happen together. But they are conceptually separate and have different requirements. [10:11] AmandaVizedom: We are open to suggestions for specific areas in which existing software evaluation methods handle some aspect of the ontology evaluation without need for ontology-specific modification. If they can so be applied as-is, we should say that. If they can be so applied in principal, but the tools do not exist to do this for ontologies, that is an important point to make in our future/recommendations section. [10:12] JoanneLuciano: @Peter - I was just about to paste in the same thing! would someone point me to where the **purpose** of the communicate is stated -- i.e. beyond a summary of the presentations -- [10:12] JoanneLuciano: Can we put "who the audience is" in the exec summary? [10:13] PeterYim: @Joanne - check out Fabian & Amanda's opening slides for today - http://ontolog.cim3.net/file/work/OntologySummit2013/OntologySummit2013_Communique/2013-04-11_18_draft-review/OntologySummit2013_Communique_first-draft-review--FabianNeuhaus-AmandaVizedom_20130411.pdf ... that's what they are intending to do (this year) [10:13] JoanneLuciano: Thanks (i had to join late) [10:14] JoanneLuciano: + agree with amanda - eval isnt being used when it could be. [10:15] SteveRay: Good responses, Amanda & Fabian. Let's state clearly up front who we are talking to, such as Amanda's observation of "people who are building ontologies but are not evaluating them". [10:13] MatthewWest: It's not just about building a good one, but building one reliably, and not be surprised by the results. [10:15] FabianNeuhaus: @ Matthew: yes, I agree. I meant "good" in the sense of high quality, where quality encompasses all of the characteristics (intelligibility, well-designed model, fitness to operational requirements) [10:34] AliHashemi: @Fabian, @Steve - doesn't it depend on how you've implemented your ontology development processes? [10:32] AstridDuqueRamos1 morphed into AstridDuqueRamos [10:15] TerryLongstreth: + with amanda and Joanne, but much of today's discussion is from the perspective of an ontologist evaluating their own work. Can we throw a bone to those who want to evaluate an ontology as a 'found' object? [10:17] ShariThurow: I agree with Terry. [10:32] AmandaVizedom: Terry, we agree that the reuser is important and needs to be discussed and supported. We indicate in some places that this is meant to be included, but I think that you are pointing out that we need to make this clearer. The idea is that the exploration-ontological analysis - design phases need to be carried out to *some* degree in order for the ontology consumer to be able to tell, even roughly, whether an existing model might suit there needs. [10:34] anonymous morphed into LamarHenderson [10:35] SimonSpero: SteveRay describes a modeling mistake of something being made A subclass that is really a property; can you clarify this? [10:36] SimonSpero: ok.... that's pretty wrong... [10:37] SimonSpero: BooksByShippingMethod [10:38] SimonSpero: BooksThatMustBeShippedByElephant ; BooksThatCanBeShippedByBicycle ; BooksThatWillFitInASedan [10:37] JoanneLuciano: Great question by Steve Ray -- when does this get caught (very common) - "Shipping method" is_a "Book" [10:38] JoanneLuciano: The Manchester paper on Common errors and pitfalls is well-worth a read (before implementation) -- but good point [10:40] SteveRay: @Joanne: +1. Perhaps our communique could at least provide some pointers to resources that are a good start, such as the paper you mention. [10:37] MikeBennett: Isn't there a related issue here, that there is ontology with a small o - capturing the domain knowledge formally, in which some fundamental errors need to be caught as per Steve's example, and then there's Ontology with a bigO in OWL or CLIF as a technical artifact. These may happen at the same time, but there may also be development lifecycles in which the ontology is the formal capture of domain knowledge for some implementation technology other than reasoning applications. [10:38] MikeBennett: So the kinds of error that relate to meaning itself would be captured at potentially a different stage to the trapping of errors in DL-safe reasoning requirements or other ontology application requirements. Semantics v syntax essentially. [10:37] anonymous1 morphed into PavithraKenjige [10:37] anonymous morphed into NancyWiegand [10:39] anonymous morphed into TorstenHahmann [10:39] PavithraKenjige: Superclass vs Subclass .. for something to be subclass of a superclass, it has to meet certain criteria. [10:39] PavithraKenjige: Set theory is used to verify subclass and superclass verification [10:40] PavithraKenjige: Dr. JohnSowa would use one of kind of logic to verify such properties .. ( first or second order logic??) [10:42] SteveRay: In addition, some pointers to some of the tools we have experienced during the Summit, or at least to some of the wiki pages where a reader can go, would be useful in a concrete way. I just worry that we end with a nice academic paper but that doesn't provide some good tips and links. [10:43] SimonSpero: TerryLongstreth: Deploy cycles are getting faster in some models (this becomes possible when testing is taking place all the time). [10:43] SimonSpero: http://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912/ref=sr_1_1?s=books&ie=UTF8&qid=1365702215&sr=1-1&keywords=continuous+delivery [10:48] TerryLongstreth: @Simon - I agree, but however it's done, you still need to package and deliver something [10:52] SimonSpero: @TerryLongstreth: delivery in many cases is QA pushing the button to allow applications to move from QA to production. [10:52] SimonSpero: @Terry: That can be weekly or less depending on the enterprise [10:53] TerryLongstreth: If QA's doing its job, it has validated the package as having conformed to all forms and procedures, including appropriate tests of the to-be-delivered package [10:55] SimonSpero: @Terry: right - else back it goes. The important thing is that testing is automated throughout the cycle, so that most old-fashioned QA work is done; the same QA resources are doing the stuff that is human only [10:56] TerryLongstreth: @Simon - looks like we're in violent agreement. [10:56] TerryLongstreth: I have to leave now. [10:44] AliHashemi: I'm imagining a scenario, where the ontology has already largely been developed, so it is "further along" the phase, but I want to implement an auditing mechanism to ensure that that evaluation is complete [10:45] AliHashemi: the current language is geared / tailored to those who are just embarking on an ontology development process, as opposed to those who may have already developed to some degree. [10:45] SimonSpero: @AliHashemi: define "evaluation is complete"? [10:46] SimonSpero: WRT? [10:46] AliHashemi: @Simon - so that the evaluation procedure that is utilized covers all the points that are elaborated in the lifecycle phases [10:47] AliHashemi: complete in evaluating all the various ways that we can evaluate ontologies [10:47] MatthewWest: The evaluation is complete when conformance (or not) to all stated requirements has been established. [10:48] AmandaVizedom: Look at first full paragraph on page 3 for discussion of blending cycles, variation in sequence, iterations, etc. [10:49] AliHashemi: @Amanda, that comes much closer to addressing my question. Apologies if it was pointed out earlier. Will add a comment to the document if appropriate. [10:53] AmandaVizedom: @Ali, no problem, and thanks for feedback. This session format is difficult because we are only skimming through the document and briefly mentioning things we say, so the detailed way that we may have addressed points is not conveyed. [10:52] MichaelGruninger: Since notions of ontological errors and correctness are not addressed in the "Ontological Analysis" section, we may want to include a forward pointer to say that such notions will be addressed later in the "Formal Modeling" section. [10:56] JoanneLuciano: I have to leave now too. [10:56] AmandaVizedom: Perhaps we need a extension of the diagram that shows entry by different kinds of potential evaluators: (1) folks at the beginning of an ontology lifecycle, (2) folks with an ontology existing, (3) folks looking to reuse an existing ontology. [10:59] SteveRay: Again, in the System Design Phase, I would suggest some references to standard tomes on system design. [11:02] AmandaVizedom: @Steve, agreed. We have a note on our off-line working document that we need such references. We're open to suggestions. [11:04] SteveRay: Uh oh. The Model Build Phase / Informal Modelling / Evaluation section makes a backward reference to "Ontological Analysis", where I thought that you had just agreed with Michael's suggestion that the ontological analysis section makes reference to the Model Build Phase. Circularity? [10:59] PavithraKenjige: For that we have to map the concepts to Ontology usage. If it is software, during and after deployment .. it is testing phases and maintenance phace. [10:59] JoanneLuciano: to all... I'm wondering if the semantic methodology and diagram we use at TWC RPI (Fox and McGuinness) (the diagram is in the presentation on GOEF) that might be a better fit for the communique (if we keep a diagram). sorry i have to go now - [11:07] PeterYim: @Joanne [10:59] - are you referring to the slide that is labeled: "Semantic Web Development Methodology"? (slide#2 on the slide deck - http://ontolog.cim3.net/file/work/OntologySummit2013/2013-03-28_OntologySummit2013_Hackathon-Clinics_Launch/OntologySummit2013_HC-02_GOEF-iChoose_HackathonClinic--JoanneLuciano-et-al_20130328.pdf [11:01] MatthewWest: Apologies. I need to leave now. [11:02] PeterYim: @Fabian - why are we calling it "Model Build" Phase and not "Ontology Build" [11:02] PeterYim: I think the latter is more appropriate [11:12] AmandaVizedom: re: "Ontology" vs "Model" We don't object to this replacement. Amanda favors it. [11:14] PeterYim: +1 "Ontology Build" please [11:16] FabianNeuhaus: concerning "model", just as historical explanation: in some previous version I tried to make the distinction between the track A view and track B view by talking about ontology-as-model and ontology-as-system, during the editing process this was shortened [11:08] SteveRay: Minor point under Formal Modeling: Suggest not using the term "OWL Full" which is no longer used with OWL 2.0. [11:09] SimonSpero: SteveRay: ""OWL 2 Full" is used informally to refer to RDF graphs considered as OWL 2 ontologies and interpreted using the RDF-Based Semantics." - http://www.w3.org/TR/owl2-overview/ [11:10] anonymous morphed into PavithraKenjige [11:11] FabianNeuhaus: Michael: add concrete example for ontology adaption [11:14] SteveRay: Page 7: Suggest editing "Structural properties include branching factor, density, counts of ontology constructs, averages, and the like are intrinsic." to "Structural metrics include branching factor, density, counts of ontology constructs, averages, and the like." [11:15] JulienCorman: Just a reaction to the definition on page 7 : The domain is represented accurately if all axioms within the ontology are true. In order to accurately represent the domain, one may also require this axiomatization to be strong enough, i.e. the set of interpretations satisfying this ontology to be restricted enough, so that the ontology actually rules out unintended usage, or misunderstanding of its concepts. Page 4, it is suggested that the documentation should be unambiguous enough, but this could be a requirement for the ontology as well. [11:27] AmandaVizedom: @Julien, yes, thank you for making that point. It is one we noted as needing further discussion but not get to. One of our top-level questions is about intelligibility. This must include whether the interpretations supported by the ontology are those intended! [11:15] PavithraKenjige: Are you all establishing that Ontology is separate from software all the time or in some cases? [11:21] PeterYim: @PavithraKenjige - that would depend on what you mean by "separate" ... if that means "not equivalent to," then, I'd say the answer is: "yes, all the time" [11:21] FabianNeuhaus: @pavithra: we try to accommodate the fact that there is a view of ontologies/knowledge base as data bases. This is the view from track B. In that sense ontologies are software. Then there is a view of ontology as a model of reality. In that sense it is not software, but just a set of declarative statements written in some formal language. [11:31] DougFoxvog: @Fabian, @Pavithra: It is useful to define different kinds of ontology: basic definitional ontologies, theory ontologies, and knowledge bases. [11:24] PavithraKenjige: Ontology is conceptual .. but most of the time, it is used to build or by systems.. [11:26] PavithraKenjige: One outcome of Ontology can be logical and physical design and development stages and deployment & usage are referred to as databases .. [11:35] PavithraKenjige: @DougFoxvog, you have a good point. We have discussed domain and usage.. . [11:27] ToddSchneider: Fabian, Track B presented the notion that there is a large similarity between ontology and relational databases, hence many of the evaluation techniques from that field maybe used, with modification, for ontology evaluation. [11:29] PeterYim: (ref. verbal suggestion) - Recommendations should be in a separate section ... maybe putting them with (as proposed in the outline) "Future Steps What issues need to be addressed to improve ontology evaluation and its adoption by ontology developers?" would be fine (... but not with support or tools) [11:24] SteveRay: I think we could help a lot by providing links to the tools we have uncovered during this Summit. [11:31] FabianNeuhaus: link to summary pages wiki [11:31] FabianNeuhaus: link to wiki page tools [11:29] SteveRay: Logistics question: Can we somehow allow comments by all us reviewers on the Google Doc? [11:32] AliHashemi: @Amanda, i thought you can highlight text within the document to add a comment to a specific word/sentence/paragraph [11:32] AliHashemi: You should be able to have the Share setting be "add comment" with link [11:32] AmandaVizedom: Comments on the Google doc are now enabled. [11:37] AmandaVizedom: To comment, select the text you want to comment on, then right click and select "comment". Enter your comment an save. [11:35] TorstenHahmann: @Peter: (responding to Peter's call to tools developers to respond to the Survey) I did update some of the information on the survey, please check that. [11:35] PeterYim: @Torsten - this is great! I didn't check lately! [11:37] DougFoxvog: Goodbye all! [11:37] PeterYim: great session! [11:37] PavithraKenjige: thank you [11:37] SimonSpero: ;;;; good work [11:37] JoelBender: Thank you! [11:37] PeterYim: Great job, Fabian and Amanda! [11:39] PeterYim: -- session ended: 11:37am PDT -- ------