ppy/chat-transcript_edited_201201213b.txt --------- Chat transcript from room: ontolog_20121212 2012-12-12 GMT-08:00 [PST] --------- [09:00] PeterYim: Welcome to the = OntologySummit2013 (Pre-launch) Community Input and Planning Session - Thu 2012-12-13 = * Summit Theme: Ontology Evaluation Across the Ontology Lifecycle * Co-chair: Professor MichaelGruninger & Dr. MatthewWest * Agenda: This is a (pre-launch) communitywide brainstorming and planning session for OntologySummit2013 Logistics: * Refer to details on session page at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2012_12_13 * (if you haven't already done so) please click on "settings" (top center) and morph from "anonymous" to your RealName (in WikiWord format) * Mute control: *7 to un-mute ... *6 to mute * Can't find Skype Dial pad? ** for Windows Skype users: it's under the "Call" dropdown menu as "Show Dial pad" ** for Linux Skype users: please note that the dial-pad is only available on v4.1 (or later or the earlier Skype versions 2.x,) if the dialpad button is not shown in the call window you need to press the "d" hotkey to enable it. == Proceedings: == Attendees: AidaGandara, AlanRector, AlexShkotin, AmandaVizedom, AnatolyLevenchuk, AsmaMiniaoui, DavidMendes, DeanaPennington, EdLowry, FabianNeuhaus, FrankOlken, GaryBergCross, JackRing, Javier G, JoanneLuciano, JohnBateman, KenBaclawski, LeoObrst, MarcelaVegetti, MatthewWest, MichaelGruninger, MikeBennett, MikeDean, OnnoPaap, PavithraKenjige, PeterYim, RamSriram, Richard Martin, RobertRovetto, RosarioUcedaSosa, SimonSpero, SteveRay, TerryLongstreth, TillMossakowski, ToddSchneider, TomTinsley, TrishWhetzel, VictorAgroskin, ZubeidaCasmodDawood ---------- [09:20] MatthewWest: @Peter: Michael will present, I will do process. [09:23] anonymous morphed into RobertRovetto [09:25] anonymous morphed into Javier G [09:29] anonymous1 morphed into AmandaVizedom [09:29] anonymous morphed into SimonSpero [09:33] PeterYim: == MatthewWest starts the session ... [09:35] anonymous morphed into OnnoPaap [09:37] PeterYim: == self-introductions by everyone in the session ... [09:39] anonymous morphed into ZubeidaCasmodDawood [09:38] GaryBergCross: David M Let's talk something about the NLP help you might provide to extract concepts from text as a way of starting ontology work. [09:40] DavidMendes: Please start to contact me via skype: (diverzulu [at] gmail.com) or mail: dmendes [at] uevora.pt . Whenever suits you best. Thank you very much in advance :) [09:42] DavidMendes: Please provide me with your contacts Gary ! [09:43] AnatolyLevenchuk: We have experiments with NLP for ISO 15926 -- http://www.slideshare.net/vvagr/ontology-modelling-of-an-engineering-document-perspectives-of-linguistics-analysis (but now we have more). [09:43] anonymous2 morphed into AlanRector [09:43] anonymous morphed into JackRing [09:43] anonymous1 morphed into DeanaPennington [09:44] AlanRector: Something bizarre has happened to the wiki. It seems to be serving things so thatthey only show up in raw HTML, -= three browsers on two machines tried. [09:45] RamSriram: @Alan: Same thing happened to me on the Mac. Try clicking on the link in the jumbled version and you will be taken to the right page. [09:46] PeterYim: @AlanRector - thank you for notifying us of the issue ... fixed now! (a bug in the wiki - happens when there is a write conflict.) [09:44] DavidMendes: Seems to be working fine here, Alan ! [09:44] anonymous morphed into JoanneLuciano [09:48] anonymous morphed into TomTinsley [09:49] anonymous morphed into PavithraKenjige [09:54] anonymous morphed into MarcelaVegetti [09:55] PeterYim: on the call (but not in the chat-room yet): EdLowry, ... [09:56] JoanneLuciano: @LeoObrst - Leo, I've just gotten back from travel. I got your email and would like to follow up. [09:58] PeterYim: RamSriram ask what day of week is preferred for the symposium (in April) [09:45] RamSriram: I will need to leave at 1pm for another meeting. Will need know the potential dates in April for the face-face meeting at NIST. [09:42] FabianNeuhaus: Ram: you meant April (not January, right?) [09:58] PeterYim: general preference seems to be Thursday & Friday ... but then locals in the Washington DC area prefers that we don't pick Fridays [09:58] DavidMendes: Any date is fine for me ! [10:00] AmandaVizedom: Suggest leaving a day after Symposium for associated / follow-up workshops, vocamps, etc. [10:01] anonymous morphed into TrishWhetzel [10:02] MichaelGruninger: Possible Symposium Dates: a) 16,17 b) 17,18 c)11,12 d) 18,19 [10:03] MatthewWest: (after much deliberation) Proposed dates are for the Symposium, in order of preference, are: April 16-17 (Tue-Wed), 17-18 (Wed-Thu), 11-12 (Thu-Fri). [10:10] List of members in the chat now: AidaGandara, AlanRector, AlexShkotin, AmandaVizedom, AnatolyLevenchuk, DavidMendes, EdLowry, FabianNeuhaus, FrankOlken, GaryBergCross, JackRing, Javier G, JoanneLuciano, JohnBateman, KenBaclawski, LeoObrst, MarcelaVegetti, MatthewWest, MichaelGruninger, MikeBennett, MikeDean, OnnoPaap, PavithraKenjige, PeterYim, Richard Martin, RobertRovetto, RosarioUcedaSosa, SimonSpero, SteveRay, TerryLongstreth, TillMossakowski, ToddSchneider, TomTinsley, TrishWhetzel, ZubeidaCasmodDawood, vnc2, [10:09] PeterYim: == MichaelGruninger walking us through the slides ... Theme: "Ontology Evaluation Across the Ontology Lifecycle" ( ref. http://ontolog.cim3.net/forum/ontology-summit/2012-12/msg00000.html#nid03 ) Proposed Tracks: 1. Dimensions of Ontology Evaluation - addresses notions of verification, validation, quality, ranking, ... (02) 2. Evaluation and the Ontology Application Framework - looks at the problem of ontology evaluation from the perspective of the applications that use the ontologies. This Framework was one of the outcomes of Ontology Summit 2011 (http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2011_ApplicationFramework_Synthesis) (03) 3. Best Practices in Ontological Analysis - focuses on the ontology evaluation based on the ontology itself, such as logical criteria (consistency, completeness, modularity) and ontological analysis techniques (e.g. OntoClean). (04) 4. Requirements for Ontologies - how do we specify the requirements against which we evaluate ontologies? (05) 5. Environments for Developing and Evaluating Ontologies - what are best practices for evaluation that we can adapt from software engineering, particularly with distributed open-source software development? (06) ( ref. http://ontolog.cim3.net/forum/ontology-summit/2012-12/msg00042.html#nid02 ) [10:18] PeterYim: == starting Discussion segment: Review of current proposals for topics and tracks for this year's theme [10:12] GaryBergCross: Michael why not call it "Ontology Development and Evaluation across...."? That is add Development explicitly. [10:17] TerryLongstreth: @Gary - Track 4 talks to evolvability, implying that development may not have a fixed completion point, in contrast to systems development [10:13] JackRing: The characterization of life cycle can be improved. [10:16] RosarioUcedaSosa: Seems like tracks 2 and 4 are closely related. [10:16] MikeBennett: Ontology Application Framework is important as it challenges the assumptions some practitioners may have to the effect that what they assume an ontology is "for" is the only thing they are for. Sometimes the development and QA questions are framed with one specific "for" in mind. [10:18] AmandaVizedom: +1 MikeBennett [10:17] RosarioUcedaSosa: From what I hear Michael propose, we're dealing with Ontology specification and goals, Ontology Quality and Ontology Usability (how it's easier to use) [10:17] AlexShkotin: For evaluation we need metrics. [10:17] TerryLongstreth: @Alex +1 [10:17] AnatolyLevenchuk: Track 4: Better "requirements engineering" because "requirements management" is only about configuration management and distribution of already created requirements. [10:18] JackRing: there may be more to learn from system engineering than from software engineering. [10:19] AnatolyLevenchuk: @JackRing: we should take from both Software and Systems Engineering. [10:18] GaryBergCross: Comment on track 5. I think the issue of including ontology development as part of app/system development is important. [10:19] AmandaVizedom: Re: requirements -- not only specification but *identification* [10:20] AmandaVizedom: that is, the process of looking at a use case and figuring out what about it has implications for ontology requirements... [10:21] AmandaVizedom: ... and then, figuring out what those implications *are*... [10:22] AmandaVizedom: Requirements ID is necessary background for evaluation. It is often skipped, precisely because people too often make the assumption Mike Bennett mentions above. [10:21] AlexShkotin: Other dimension - what kind of methodology has been used to create ontology? If any;-) [10:21] JoanneLuciano: How many of the candidates do we select, or is it just one? [10:23] JoanneLuciano: HOW MANY TRACKS do we select? [10:25] LeoObrst: @Joanne: We can decide that all 5 tracks will be selected, and probably max of 5 tracks for logistical purposes. We are also considering whether these tracks are the ones we agree to. [10:24] FabianNeuhaus: I have a clarification question concerning track 1 to Michael. The list verification, validation ..., metrics seems to include very different notions. What do you mean by "dimension"? [10:30] MichaelGruninger: @Fabian: re: dimension I was trying to convey the idea that there are different sets of criteria that are independent of each other. The notion of metrics was added because it was appearing in many postings, but the other notions (such as verification and validation) were more qualitative. [10:30] SteveRay: @Mike: +1 [10:21] MichaelGruninger: What makes an ontology usable by domain experts? (Rosario) [10:24] RosarioUcedaSosa: Three dimensions associated with the ontology lifecycle: Ontology specification (roughly track 1) Ontology building (roughly track 3) and then ontology querying and navigation (usability) by domain users (not sure what this track would be). Mostly a re-factoring to follow the ontology lifecycle theme. [10:27] RobertRovetto: If I may suggest an answer to the question 'What makes an ontology usable by domain experts?'... Perhaps this is an obvious or simple answer but an ontology is usable by a domain expert if (U1) the ontology accurately reflects the universe of discourse (the domain subject matter) that is the expertise of the domain expert, (U2) the ontology supports the expert in their research/work,...(Un) [10:25] JackRing: The 'ontology life cycle' is naive. There will be numerous ontologies in any system of non-trivial size and each will likely evolve greatly so the notion of begin-middle-end is misleading. [10:26] AmandaVizedom: @Jack: I don't think "lifecycle" assumes begin-middle-end... there are still lifecycles in continuous processes. [10:26] SimonSpero: @AmandaVizedom it's why it's called a cycle [10:27] TerryLongstreth: @Todd, Jack, Amanda: Ontology lifecycle probably more like the data lifecycle, which may extend into eternity (i.e. well passed the cycle of any one system) [10:28] AmandaVizedom: @Terry, I'd suggest that it's often a hybrid, and also varies with application [10:29] PavithraKenjige: Life cycle refers to stages of development, maintenance, usage, disposal ... and Reuse too, which is part Usage.. [10:33] AnatolyLevenchuk: @PavithraKenjige: before development there are stage of conception (before requirements engineering and architectural design of development substage). Decision to invest in development. Loooong stage usually :-) [10:29] RosarioUcedaSosa: @Jack We're trying to categorize the validation criteria for an ontology at different stages, not necessarily thinking that these stages are linear or iterative. [10:27] GaryBergCross: As information products ontologies have some of the usual phases from Req and anlysis to design to building, evaluation etc. [10:28] MatthewWest: I'd like to see how Information quality Management can help us frame and justify how we evaluate ontologies since ontologies ultimately provide information to support decisions. [10:28] ToddSchneider: First, need to include operations or operational use. [10:29] PeterYim: +1 on Todd and others, on the consideration for the "full" lifecycle, and the need to ground our discourse to domain applications, too ... Within the application track, I suggest sub-tracks (to the extent that collaborating communities and champions are available) covering specific domains, against whose requirements and use case scenarios, we will craft the evaluation criteria - candidate domains may include (a) use of ontology in standards, (b) use of ontology in harnessing data (bigdata? ... if that's too "big" try "research data), (c) ontology in earth science, (d) ontology in bio of medical informatics, (e) ...etc. [10:30] GaryBergCross: Ontology content will have requirements, but also the ontology product may have architectural and interface requirements to work within a system. [10:30] SteveRay: The larger issue is one of scope for the summit. We need to decide if we are going to address "evaluation", or a broader "ontology lifecycle" that could include development, requirements gathering, etc. etc. One risk is that many people argue that xyz is important, so we should include it, at the expense of focus for the summit. [10:33] MichaelGruninger: @SteveRay: The focus is still on Ontology Evaluation. The tracks are intended to explore this in more detail by identifying the different ways in which people evaluate ontologies, particularly with respect to applications and the original intention of the ontology [10:30] ToddSchneider: What are the differences among 'ontology lifecycle' and engineering lifecycle? Should they be considered equivalent and hence adopted as an organizing paradigm for this summit? [10:30] TomTinsley: Lifecycle in software and hardware means cradle to grave. Since ontology is capturing knowledge, is there a grave? [11:01] AmandaVizedom: @TomTinsley, it could be used that way, but it could also be used to measure fitness of ontologies to uses, and ontologists' ability to develop usable ontologies for particular kinds of uses. [10:30] AmandaVizedom: I have worked on cases where ontology is system component, developed and test incrementally everyday along with other system components, with periodic freezes with more formal testing that then, when sufficiently QAed, become the live system. The next live system is never not under development and testing and the most recent one to be frozen & QAed is never not live. [10:30] TerryLongstreth: @Amanda: of course, but I'm pushing back against the engineering lifecycle, which ends in disposal [10:32] AmandaVizedom: @Terry, Ok, get it. I'd say that applies to some and not others. If dev continues, with versioning, then any individual version does have that kind of cycle. But not all. [10:32] MikeBennett: @Terry Amanda Surely any lifecycle leads to some sort of success, whether that is then built on for some subsequent success or not? [10:32] JackRing: @All, the sooner you shed lifecycle and focus on usage scenarios the quicker you will become relevant to and appreciated by practitioners. [10:32] RosarioUcedaSosa: It may be worth describing the different ways to specify an ontology (whether it's existing or not) for a particular goal. Also, there needs to be a disciplined way to 'build' or change the ontology. The considerations here may be different, as we're trying to build (or modify, or integrate) a high quality ontology. The third discussion is around how easy it is to use. This may require usability studies, or comparisons with existing vocabulary. [10:32] LeoObrst: Lifecycle has to include how the ontology interacts with other components of the architecture and the data/software deployments, e.g., mapping vocabularies to the ontologies, linking data sources to the ontologies, application interface and reasoning services, etc. An excellent ontology does not mean that a given ontology application effort will succeed; so we need to evaluate the ontology within its emerging application environment(s), maintenance, and reuse, redeployment, etc. [10:33] SimonSpero: @Rosario: Of possible relevance is some of the work that Hollie White did for her dissertation, where she compared the metadata & term assignment behavior of domain scientists with that of information professionals: http://sites.duke.edu/holliewhite/dissertation-research/ [10:33] AlanRector: It's critical to evaluate the ontology against its purposes and its claims and what would count as evidence. We have lots of claims in the biomedical community for criteria for quality, frequently without explicit purpose or agreement of what counts as evidence [10:41] JoanneLuciano: @ AlanRector's "It's critical to evaluate the ontology against its purposes and its claims and what would count as evidence" ... The little research I did, when I took the framework approach - that included extrinsic and intrinsic quality metrics was that these can be articulated in use case formalization. I would like to see use case as part of the evaluation criteria. So, agree and +1 with Amanda in the @Gary post [10:33] RosarioUcedaSosa: thanks, Simon. I'll check it out. [10:31] AlexShkotin: Ontology classification theme should be very important. As we should have many different kinds. And evaluate differently:-) [10:33] AmandaVizedom: @Alex, IME, there is to much variety and uniqueness to make classification of ontologies as whole very useful. I think it more useful to talk in terms of *features* (some of which may come in degrees) which ontologies may or may not have... [10:36] AlexShkotin: @Amanda, consider different kind of databases - operational, analytical... We should have even more, I think. [10:33] JackRing: Seems like we need an ontology for "ontology" [10:34] GaryBergCross: Some people may evaluate an ontology based on how easy it might be for their group to maintain. So modularity might come into play, but also whether the formalism are understood and well supported etc. [10:34] TerryLongstreth: I'm probably going against the flow here, but I believe that a successful ontology is an organic object, that grows and evolves in step with the human (or cultural, or world, or sentient) knowledge that informs it. [10:34] MikeBennett: Sorry I have to go! [10:34] RosarioUcedaSosa: @Gary, that's part of the usability criteria. [10:35] AmandaVizedom: ... and how ontology requirements point to the ontology features that matter... and then, hugely important, how (and to what extent) those features can be evaluated. [10:35] AlanRector: I would like to second this. What counts as evidence? [10:35] GaryBergCross: @Rosario yes, usability criteria sounds like a good dimension. [10:37] AmandaVizedom: @Gary, I would relate that to the extent to which a use case requires expert verification and/or human end-user support. Either of these adds understandability, by some group of humans, to the requirements. [10:41] JoanneLuciano: +1 with Amanda in the @Gary post [10:35] MichaelGruninger: Track 3 will be "Best Practices and Techniques in Ontology Evaluation" (Fabian) [10:37] JackRing: I suggest that Track 3 include instances of Worst Practices as well. Mistakes are the prime root of learning. [10:38] SimonSpero: @JackRing: These are usually referred to as AntiPatterns [10:38] AlexShkotin: @Jack Yes! [10:40] JohnBateman: @JackRing and some other comments: Note also results of the OntologySummit2007 on the classification of distinct types (dimensions) of ontologies: seems that this might also come in as a way of characterising what *can* be done with an ontology and so whether it might be expected to meet some requirements - http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2007_Communique [10:40] AmandaVizedom: @Jack - this is something I wanted to evoke in the survey last year. I know that there are many, many examples out there of people doing evaluation that is not predictive of whether the ontology works for intended use. I hoped to get folks to share them, along with more successful evaluations, so that we could learn. Unfortunately, building survey ended up harder/longer than worked with summit timeline. But I still thing that the very large pile of community wide experiences, and/or lessons learned, is a gold mine for this. [10:41] TerryLongstreth: Pavithra has a standard model of Ontology development that parallels system development; is it possible that ontologies might be products (at least partially) of system execution? [10:38] GaryBergCross: @Rosario as with any Info product its documentation across the cycle is important. [10:41] RosarioUcedaSosa: @Gary Yes, but we have to be careful. Requiring a full specification as in a software lifecycle may be too complex for most practitioners (and has had limited success in the wider software engineering community) There may be alternate -easier- requirements in the linked data community... I don't claim to have the answers, but would advocate keeping the validation and formal aspects as simple as possible. [10:43] MichaelGruninger: I didn't mean for the Ontology Lifecycle to be the focus; rather, it was meant to draw attention to a broader approach to evaluation that includes requirements and applications. If the phrase "Ontology Lifecycle" is getting in the way, we can drop it from the title. The important thing is the set of topics covered in the tracks. [10:50] LeoObrst: @Michael: I think it's important to keep emphasizing the Ontology "Lifecycle" in the Summit, since the ontology may have to morph or separate into components, get refined, etc., based on evolution of the application(s). [10:41] SimonSpero: @GaryBergCross: Documentation-and-software-engineering "Ron Jeffries: "I was taught by Kent Beck that need for a comment is the code's way of asking to be more clear. I strive to make the code more clear and to see if the need for the comment goes away. Usually it does." [10:41] JackRing: @Simon, "usually" doesn't cover most of the 1 million people engaged in system engineering, the 6 million software engineering or the few who are concerned with semantic accuracy. Where I come from Anti- means Not or As Contrasted To whereas worst practices are patterns, too. See John Gall, Systemantics. [10:46] SimonSpero: @jackRing: Koenig, A. (1995). "Patterns and Antipatterns". Journal of Object-Oriented Programming 8 (1): 4648. See also http://c2.com/cgi/wiki?AntiPattern [10:44] GaryBergCross: @Simon I'm not sure that I agree with this idea of clean code speaking as clearly as documentation. In ontology development one often needs to explain scoping and commitment choices and what went into that. [10:43] TillMossakowski: Considering candidate Track 5: the open ontology repository (OOR) community has developed an architecture for ontology development in distributed, federated, heterogeneous repositories, generalising what is there in Bioportal. This includes ontology development, logical reasoning, ontology review and workflows. Heterogeneous means that not only OWL, but also RDF, CommonLogic, UML,... are possible languages. Ontohub is a current open-source effort that tries to implement this architecture in a style inspired by github. Would there be interest in discussing the design principles of ontohub and also in doing some coding? [10:46] PeterYim: +1 to TillMossakowski's comment ... using OOR as a platform/environment to instantiate ontology evaluation tools [10:46] PeterYim: also suggesting, we form implementation teams for, for example, (i) extreme programming team(s) to do things on, say, on some ontology evaluation platform, (ii) a semantic wiki team (we may start using the OntologPSMW (purple semantic media wiki) this year, as a beta for this summit, (iii) integration and demo team (in preparation for something we might possibly do, as a segment of the symposium, (iv) a Public Relations / Marketing team that gets the word out, so this summit can make an impact ... some of the above tasks may be merged and streamlined (so it may only be a couple of teams we would need to create) [11:02] PeterYim: For the PSMW (purple semantic mediawiki) effort, we are planning on making a debut of the OntologPSMW next week - see: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2012_12_19 ... please mark your calendars [10:50] SimonSpero: @PeterYim: XP/Agile bib entries from last year: https://www.zotero.org/groups/ontologysummit2012/items/collectionKey/VFKU526K [10:56] PeterYim: @SimonSpero - thank you for the references to Agile Methodologies and Extreme Programming [10:58] AmandaVizedom: RE: sharing reference resources: Last year we built the zotero library @Simon just pointed to, as an auxiliary, collaborative library/bibliography. We started it mid-summit, though, and wished we'd started from the beginning. I can start one for this year this moment, if folks would find it useful... we can cross list items from last year over to it without duplicating resources. [10:44] PeterYim: for "best practice" or "environments" - who (person, community, ...) in Software Engineering, or Systems Engineering, can we engage and collaborate with, to help us advance the cause of this summit? ... need specific names of champions/volunteers who can push this forward [10:44] JoanneLuciano: Don't know if this is appropriate here, but I'm also looking for anyone who want to collaborate on developing the framework further. I have an implementation started but need other ontogeeks to play with to help it develop [10:45] AlexShkotin: Ontology itself is a piece of knowledge. And should be some methods of knowledge evaluation. Other point - the place, role of ontology in information system - like database... [10:49] RobertRovetto: @Alex Depending on what you mean by 'a piece of knowledge'... Ontology, itself (and historically), is roughly the study of existence/being, of what exists. It is not a piece of knowledge. :) [10:50] AmandaVizedom: @Robert that is true of *philosophical ontology*. Applied, formal ontology, as practice or artifact, is a different beast. [10:51] AlexShkotin: @Amanda, another way to classify ontology - domain of knowledge is very important. [10:51] MatthewWest lowered your hand [10:53] RobertRovetto: @Amanda Thanks for reiterating the distinction and possibly clarifying the intended meaning of the statement. I know the distinction, but the phrasing and my reading of the statement suggested something more general. :) No worries. [10:53] AmandaVizedom: @Alex: true, but in this case I was speaking not of the domain of knowledge *covered by an ontology* but as "ontology" being a term for two very different, though historically related, fields of study and practice. [10:47] JackRing: A few years ago a friend spent five years in Cyprus evolving a way for the Greeks and Turks to pursue mutual ends. This largely evolved an ontology of their mental and emotional models. Are we thinking of this scope or only regarding IT? [10:49] JoanneLuciano: Does it make sense to evaluate an ontology outside of a (any) use case? Is that different from asking "is x big enough?" [big enough for what? does the shoe fit? (which foot?)] these questions require asking and answering these questions for each use case. And if there isn't a use case in mind (at the least), then why is the ontology being built? [10:54] JackRing: In one use case we are concerned with automatically composing an ontology with respect to an intended usage by surveying ontologies and selecting modules that can be harmonized. The method of composing is now known. Now we need to state the rules for qualifying any candidate module. [10:54] LeoObrst: One issue, e.g., is if you expose the ontology too early to domain experts or users who do not have sufficient understanding of what ontology is, they will confuse the ontology with the emerging application that uses it, dig in their heels, and potentially cause the effort to fail. Representation is not presentation, and you can have many application "presentations" of the same underlying ontology representation. [10:53] GaryBergCross: @Anatoly I take this meaning of standard to mean something like "ontology X is the standard for Hydrology objects." [10:55] TerryLongstreth: @Anatoly - I agree. I'd suggest you're describing an Ontology as a standard model of something else [10:55] SimonSpero: The standard for standards: "Rough Consensus and Running Code". Consensus is measured by humming. [10:58] LeoObrst: Thanks, all: must leave for another meeting. [10:58] PeterYim: thanks, Leo, bye! [10:58] MichaelGruninger: Ontology evaluation is closely tied to ontology testing -- ideally, ontology evaluation criteria should be testable in some way [10:59] TerryLongstreth: @Todd - Agile still has a pretty well-defined end point: what about an Ontology that models an evolving natural process? [10:59] GaryBergCross: Thanks, all: I also must leave for another meeting. [11:00] JoanneLuciano: @MichaelGruninger -- it's tied to requirements, they define the test [11:00] JoanneLuciano: @Tom +1 governance [11:01] AlanRector: But who is the "user". In many of our scenarios the primary users are either configuration engineers or software engineers trying to develop software that uses an ontology. Whether they can understand and use it - e.g. whether it is easy to relate to UML or their OO designs may be critical for them but of no interest to the end users of the application. In fact if the ontology is really successful the end users may never know it's there. There was a large semantic web technologies including ontologies implementation behind the BBC Olympics web site, but if you were browsing the system, all you knew was that it worked (almost always). [11:01] JoanneLuciano: Propose SADI services as a registry ( SADI - Semantic Automated Discovery and Integration - http://sadiframework.org ) [11:01] SimonSpero: ICANN came after the Internet had grown; it took over the role of IANA, which was basically Jon and Joyce. [11:01] MichaelGruninger: Question 1: Are there any topics missing in the current tracks? [11:02] MichaelGruninger: Question 2: Are there any missing tracks? [11:03] MichaelGruninger: Question 3: Should any of the Tracks be split/merged? [11:03] MichaelGruninger: If additional topics can fit into an existing Track, then we can let participants of that Track start their own discussions [11:04] TomTinsley: Establishing quality can lead to certification. This could lead to establishing an organization to internationally govern ontology development in a similar way ICANN manages the internet. [11:03] DavidMendes: @Michael: I would suggest mentioning specifically "metrics for evaluation" in some tracking ... [11:04] MichaelGruninger: @DavidMendes: "Metrics" is included in Track 1 (slide 7) [11:04] DavidMendes: metrics in ontology, or anywhere else, have to be discussed for something to be comparable that is fundamental for evaluating. [11:06] DavidMendes: Yes, I missed that. Thank you ! [11:04] AsmaMiniaoui: @Michael :I suggest Evaluation Context , Quality is not property of something but a judgment, so must be relative to some purpose according to a given context [11:07] JoanneLuciano: Yes, capturing context. Use case provides context. Some metrics are context free and some context sensitive. What are the outcomes -- think these should be articulated at the outset. [11:12] AsmaMiniaoui: yes, in fact , for each context we have different evaluators,different evaluation metrics, different goals..etc [11:13] AmandaVizedom: +1 @AsmaMiniaoui [11:07] ToddSchneider: Where are evaluation techniques represented? [11:07] JoanneLuciano: I hear - max of 5 tracks [11:07] JoanneLuciano: thank you [11:08] SteveRay: Suggest we consider merging tracks 2 and 4 [11:09] Richard Martin: It seems that Track 5 could be subsumed by the other tracks. [11:09] JoanneLuciano: suggestion -- Track 4 could be folded into 1 [11:09] ToddSchneider: Suggest dropping track 4. [11:09] AmandaVizedom: @Steve is that essentially treating application/use case features as input to requirements identification? [11:09] DavidMendes: Is there anywhere specific directions about Multilingual or Internationalization problems or issues in evaluation ? [11:10] AmandaVizedom: @David, I would treat that a specific case of a cluster use case features that have particular implications for ontology requirements. [11:11] DavidMendes: TY @Amanda [11:10] JackRing: Apparently my Skype has a BS filter. ;-) I am concerned about the use of the singular throughout this discussion. There are 7 billion ontologies today and some of these people create a local, mutual ontology which doesn't interoperate with other local ones. Is this community presuming to pursue one standard ontology or a standard for producing multiple ontologies or what? [11:12] AmandaVizedom: @Jack: I certainly don't think most of us are. The very notion of understanding the variety of use cases, the variety of ontology requirements that can emerge from such use cases, and need to evaluate ontologies along the dimensions that matter for the use case -- all of that speaks against the kind of presumption you mention. [11:14] AlexShkotin: @JackRing, do you have a reference to "7 billion ontologies"? [11:15] MichaelGruninger: @JackRing: We are talking about evaluating many different ontologies. Or do you mean that there should be more emphasis on evaluation of a set of ontologies all at once (e.g. how well do ontologies play with each other)? If so, I agree, and this can be addressed somewhere. [11:13] FabianNeuhaus: (Summary of what I said earlier) Different ontology evaluation methodologies are appropriate at different times during the ontology life-cycle. During the development of an ontology the developers have the need to measure progress and whether the ontology meets the requirements. After deployment one can evaluate the use of ontology; in particular for ontologies that are used for annotating texts this essential. A different (potential) phase in an ontology life-cycle is reuse: people evaluating existing ontologies with the goal to decide which one they should use (if any). [11:05] AmandaVizedom: +1 @Fabian [11:14] AlanRector: I'd argue that in many cases, we find that, in collaborative development as in large medical ontologies/terminologies, different people have different requirements and making those differences explicit is one key to resolving the resulting disagreements. [11:14] JoanneLuciano: One question I'd like to see addressed (in some track) is how does one determine the expressivity and language needed in the application that one is presuming one needs an ontology developed for? and how can one evaluate whether is is correct before the investment is made? [11:15] TerryLongstreth: In any living ontology based upon requirements, there will be a continuing requirements evolution driving ontology evolution [11:15] PavithraKenjige: I think we should evaluate each phase of life cycle and agree upon them. Requirements are one phase of the life cycle [11:15] JoanneLuciano: And how does one make public the requirements that the ontology was designed to meet? [11:15] AmandaVizedom: IMHO, what we need is (a) better understanding of the relationship between use case/context features and ontology requirements, and (b) better understanding of how to evaluate ontologies along particular dimensions that correspond to those requirements. [11:16] JoanneLuciano: (agree with Amanda's HO) [11:17] ToddSchneider: Amanda, definitely! [11:15] JoanneLuciano: The sound went dead... [11:15] JohnBateman: I think we lost the call? [11:16] ToddSchneider: Peter we've lost audio. [11:16] AmandaVizedom: skype says "joinconference is offline" [11:16] AlexShkotin: me too. [11:16] MatthewWest: Sorry, my turn to go offline [11:16] JoanneLuciano: "The moderator has left the conference." [11:16] SteveRay: Ah, I thought it was my end... [11:16] JoanneLuciano: Nope. [11:16] TerryLongstreth: We're waiting for the moderator [11:17] AmandaVizedom: Bring Your Own Hold Music... [11:17] JoanneLuciano: Hmm, now there's an idea that has market potential! [11:17] JoanneLuciano: select 1 for classical, 2 for folk, 3 for jazz, 4 for wrap, 5 to upload your own, 6 for talk radio.... [11:17] MichaelGruninger: I'm cutoff, and I am unable to reconnect [11:17] ToddSchneider: Nice filler music. [11:17] SteveRay: Is anybody not using Skype? Do they have audio? [11:17] PeterYim: looks like there is some problem with the phone bridge network, and a whole bunch of people have been dropped (including myself) ... please hang on [11:18] ToddSchneider: Not using Skype; and lost audio. [11:18] AnatolyLevenchuk: skype connect disappear for me [11:18] PavithraKenjige: We got music .. indicating we are on hold [11:18] ToddSchneider: So, have we defined the tracks and their subjects? [11:19] MichaelGruninger: @Todd: There seems to be a rough consensus on the Tracks, although their contents will still need to be developed by the Champions [11:20] SteveRay: I suggest we move forward with these tracks, and if merging or adding is needed, we can do that (like last year). [11:18] MichaelGruninger: Ok, let's continue in the chat. [11:18] MichaelGruninger: Are there any volunteers to be Track Champions? [11:18] PeterYim: == who is coming to the organizing committee call tomorrow? ... please indicate (as an rsvp) below - please make sure you go through details at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2012_12_13#nid3J88 ... beware that being part of the organizing committee is a fairly serious commitment [11:19] ToddSchneider: I'll be there. [11:19] PavithraKenjige: I would like to attend [11:20] JoanneLuciano: i am on [11:20] TomTinsley: I am on. [11:21] MatthewWest: OK I'm being told that joinconference is Offline on Skype. [11:21] PeterYim: I am back ... calling into the phone number still works ... Phone (US): +1 (206) 402-0100 ... PIN 141184# [11:23] DavidMendes: I can't call the States from here ! I will wait for joinconference to get back online, it was working very well for me. [11:24] AlexShkotin: Sorry, it's too late in Moscow:-) Bye. [11:21] PeterYim: Please try to call in or stay on the chat [11:21] JoanneLuciano: @StevenRay No, I'm dialed in from my office landline [11:23] MichaelGruninger: If anyone knows other people who would make good Track Champions, please make your suggestions [11:20] SteveRay: @Joanne: On Skype? [11:24] JoanneLuciano: I will consider being on the organizing committee - I think the track champion role may be too time consuming for me (it was last time, but will discuss and consider various options) - given I'm doing active research in a generalized framework of methods and metrics in ontology evaluation, for use and reuse [11:25] MatthewWest: We generally had co-champions last year, and I know I found that very helpful. [11:24] PeterYim: == if you can planning to join the organizing committee, please post your name below now [11:24] SteveRay: I plan to participate on the organizing committee call tomorrow. [11:24] TerryLongstreth: I'm happy to participate in the organizing committee [11:25] AmandaVizedom: I will be in the call tomorrow. [11:25] PavithraKenjige: I will participate tomorrow [11:25] JoanneLuciano: what time is it tomorrow? [11:25] FabianNeuhaus: I plan to participate tomorrow. [11:25] ToddSchneider: I'll be there. [11:25] JoanneLuciano: I plan to participate if I don't have a pre-existing conflict (i.e. one I can't move or skip) [11:25] MichaelGruninger: Given the technical difficulties, we will need to finish today's meeting. If anyone wants to participate in the Organizing Committee, please join tommorrow's meeting. [11:26] TerryLongstreth: I suggest you put the call for participation on the mailing list. [11:26] JoanneLuciano: Thanks everyone! [11:26] MarcelaVegetti: I will be in the call tomorrow. But I consider that the champion role is too big for me [11:26] MikeBennett: I have other meetings at that time. [11:27] PeterYim: sorry about the technical difficulty ... after collecting names of people who have signed up for the organizing committee, we will adjourn this meeting. Michale & Matthew will synthesize the input collected today, and we can tie down the tracks, and who will be co-championing them at the meeting tomorrow - http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013/GettingOrganized#nid3J33 [11:27] MatthewWest: Time to call time I think. Thank you everyone for your participation today. I hope to "see" many of you tomorrow. [11:27] DavidMendes: IMHO, I don't think you need me in the Organizing Committee. [11:28] MatthewWest: @David: I'm sure you would be valuable, the question is whether you can make the commitment. [11:29] DavidMendes: I am here to help should any need arises [11:28] MarcelaVegetti: Sorry I have to go. Thanks to all [11:29] PeterYim: Meeting adjourned [11:29] PeterYim: -- session ended 11:29am PST -- [11:29] MichaelGruninger: OK, everyone -- we will officially conclude today's meeting. Talk to people at tomorrow's Organizing Committee meeting [11:29] DavidMendes: Thank You All [11:31] List of attendees: AidaGandara, AlanRector, AlexShkotin, AmandaVizedom, AnatolyLevenchuk, AsmaMiniaoui, DavidMendes, DeanaPennington, EdLowry, FabianNeuhaus, FrankOlken, GaryBergCross, JackRing, Javier G, JoanneLuciano, JohnBateman, KenBaclawski, LeoObrst, MarcelaVegetti, MatthewWest, MichaelGruninger, MikeBennett, MikeDean, OnnoPaap, PavithraKenjige, PeterYim, RamSriram, RichardMartin, RobertRovetto, RosarioUcedaSosa, SimonSpero, SteveRay, TerryLongstreth, TillMossakowski, ToddSchneider, TomTinsley, TrishWhetzel, VictorAgroskin, ZubeidaCasmodDawood, vnc2 -----------------