OntologySummit2013: Panel Session-09 - Thu 2013-03-14    (3LHC)

Summit Theme: "Ontology Evaluation Across the Ontology Lifecycle"    (3LHD)

Summit Track Title: Track-C: Building Ontologies to Meet Evaluation Criteria    (3LHE)

Session Topic: Ontology Development Methodologies for Reasoning Ontologies    (3NP4)

Session Co-chairs: Mr. MikeBennett (EDM Council; Hypercube, UK) and Dr. MatthewWest (Information Junction, UK) - intro slides    (3NP5)

Panelists / Briefings:    (3NP6)

Archives:    (3NPC)

Abstract:    (3NR8)

OntologySummit2013 Session-09: "Ontology Development Methodologies for Reasoning Ontologies" - intro slides    (3NR9)

This is our 9th Ontology Summit, a joint initiative by NIST, Ontolog, NCOR, NCBO, IAOA & NCO_NITRD with the support of our co-sponsors. The theme adopted for this Ontology Summit is: "Ontology Evaluation Across the Ontology Lifecycle."    (3NRA)

Currently, there is no agreed methodology for development of ontologies, and there are no universally agreed metrics for ontology evaluation. At the same time, everybody agrees that there are a lot of badly engineered ontologies out there, thus people use -- at least implicitly -- some criteria for the evaluation of ontologies.    (3NRB)

During this OntologySummit, we seek to identify best practices for ontology development and evaluation. We will consider the entire lifecycle of an ontology -- from requirements gathering and analysis, through to design and implementation. In this endeavor, the Summit will seek collaboration with the software engineering and knowledge acquisition communities. Research in these fields has led to several mature models for the software lifecycle and the design of knowledge-based systems, and we expect that fruitful interaction among all participants will lead to a consensus for a methodology within ontological engineering. Following earlier Ontology Summit practice, the synthesized results of this season's discourse will be published as a Communique.    (3NRC)

At the Launch Event on 17 Jan 2013, the organizing team provided an overview of the program, and how we will be framing the discourse around the theme of of this OntologySummit. Today's session is one of the events planned.    (3NRD)

In this 9th virtual panel session of the Summit, we will look at methodologies for developing ontologies used for reasoning applications.    (3NRE)

More details about this OntologySummit is available at: OntologySummit2013 (homepage for this summit)    (3NRF)

Briefings:    (3NRG)

Agenda:    (3NRR)

OntologySummit2013 - Panel Session-09    (3NRS)

Proceedings:    (3NRY)

Please refer to the above    (3NRZ)

IM Chat Transcript captured during the session:    (3NS0)

 see raw transcript here.    (3NS1)
 (for better clarity, the version below is a re-organized and lightly edited chat-transcript.)
 Participants are welcome to make light edits to their own contributions as they see fit.    (3NS2)
 -- begin in-session chat-transcript --    (3NS3)
	------
	Chat transcript from room: summit_20130314
	2013-03-14 GMT-08:00 [PDT]
	------    (3OK4)
	[10:10] PeterYim: Welcome to the    (3OK5)
	 = OntologySummit2013: Virtual Panel Session-09 - Thu 2013-03-14 =    (3OK6)
	Summit Theme: Ontology Evaluation Across the Ontology Lifecycle    (3OK7)
	* Summit Track Title: Track-C: Building Ontologies to Meet Evaluation Criteria    (3OK8)
	Session Topic: Ontology Development Methodologies for Reasoning Ontologies    (3OK9)
	* Session Co-chairs: 
	  Mr. MikeBennett (EDM Council; Hypercube, UK) and Dr. MatthewWest (Information Junction, UK)    (3OKA)
	Panelists / Briefings:    (3OKB)
	* Dr. JoanneLuciano (RPI-TWC, US) - "A Generalized Framework for Ontology Evaluation (GOEF)"    (3OKC)
	* Dr. LeoObrst (MITRE, US) - "Developing Quality Ontologies Used for Reasoning"    (3OKD)
	Logistics:    (3OKE)
	* Refer to details on session page at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_03_14    (3OKF)
	* (if you haven't already done so) please click on "settings" (top center) and morph from "anonymous" to your RealName (in WikiWord format)    (3OKG)
	* Mute control: *7 to un-mute ... *6 to mute    (3OKH)
	* Can't find Skype Dial pad?
	** for Windows Skype users: it's under the "Call" dropdown menu as "Show Dial pad"
	** for Linux Skype users: please note that the dial-pad is only available on v4.1 (or later or the earlier Skype versions 2.x,) if the dialpad button is not shown in the call window you need to press the "d" hotkey to enable it.    (3OKI)
	Attendees: AliHashemi, AmandaVizedom, AstridDuqueRamos, BobSmith, BobSchloss, BobbinTeegarden, 
	DaliaVaranka, DennisWisnosky, DougFoxvog, FabianNeuhaus, FredHosea, GaryBergCross, HansPolzer, 
	JamesMichaelis, JoanneLuciano, JoelBender, JohnBilmanis, KenBaclawski, LamarHenderson, LeoObrst, 
	LudgerJansen, MatthewWest, MeganKatsumi, MichaelDenny, MichaelGruninger, MichelDumontier, 
	MikeBennett, MikeDean, PavithraKenjige, PeterYim, RichardMartin, RosarioUcedaSosa, SamirTartir, 
	SteveRay, TerryLongstreth, ToddSchneider, TorstenHahmann, YuriyMilov, vnc2    (3OKJ)
	 == Proceedings: ==    (3OKK)
	[10:19] anonymous morphed into AstridDuqueRamos    (3OKL)
	[10:23] anonymous morphed into BobSchloss    (3OKM)
	[10:24] Astrid: Hello Peter    (3OKN)
	[10:24] Astrid: I am not able to speak    (3OKO)
	[10:24] Astrid: I have problems with my microphone    (3OKP)
	[10:24] Astrid: But I will listen.    (3OKQ)
	[10:25] Lamar Henderson morphed into LamarHenderson    (3OKR)
	[10:25] anonymous morphed into MichelDumontier    (3OKS)
	[10:26] Astrid morphed into Astrid Duque    (3OKT)
	[10:27] PeterYim: No problem, Astrid ... I was suggesting that you (and everyone else who hasn't 
	already) to morph your name into the nominal WikiWord name (as per your identity on our wiki) - like 
	AstridDuqueRamos, or JoanneLuciano, BobSmith, etc. ... thanks (that helps the automatic link 
	generation on the wiki)    (3OKU)
	[10:29] Astrid Duque morphed into AstridDuqueRamos    (3OKV)
	[10:29] MichelDumontier: hi!    (3OKW)
	[10:30] PeterYim: Hi Michel, welcome!    (3OKX)
	[10:30] SteveRay: Lots of hiss on the line.    (3OKY)
	[10:30] BobSchloss: On the telecon audio bridge, there is a heavy buzzing which is making it hard 
	for me to hear you, Peter, as well as the other person who is speaking.    (3OKZ)
	[10:30] BobSchloss: If I am the only person who hears the hissing, I can call in again.    (3OL0)
	[10:30] MikeBennett: We all hear it. Also someone talking.    (3OL1)
	[10:30] MikeBennett: This will clear when Peter mutes everyone    (3OL2)
	[10:31] BobSchloss: It sounds like 2 telecon lines are crossed.....    (3OL3)
	[10:31] anonymous morphed into AliHashemi    (3OL4)
	[10:31] AmandaVizedom: Yes, in the meantime, perhaps those who aren't speaking can self-mute (*6)    (3OL5)
	[10:31] anonymous1 morphed into TorstenHahmann    (3OL6)
	[10:32] anonymous morphed into DougFoxvog    (3OL7)
	[10:32] anonymous1 morphed into MichaelDenny    (3OL8)
	[10:34] anonymous morphed into LudgerJansen    (3OL9)
	[10:34] PeterYim: == MikeBennett opening the session on behalf of the co-chairs ... ... see: the 
	[0-Chair] slides    (3OLA)
	[10:35] anonymous morphed into RosarioUcedaSosa    (3OLB)
	[10:35] List of members: AliHashemi, AmandaVizedom, AstridDuqueRamos, BobSmith, BobSchloss, 
	DaliaVaranka, DougFoxvog, FabianNeuhaus, HansPolzer, JamesMichaelis, JoanneLuciano, JohnBilmanis, 
	LamarHenderson, LeoObrst, LudgerJansen, MatthewWest, MeganKatsumi, MichaelDenny, MichaelGruninger, 
	MichelDumontier, MikeBennett, MikeDean, PeterYim, RichardMartin, RosarioUcedaSosa, SamirTartir, 
	SteveRay, TerryLongstreth, TorstenHahmann, vnc2    (3OLC)
	[10:37] Samir Tartir morphed into SamirTartir    (3OLD)
	[10:38] anonymous: where can one download the slides    (3OLE)
	[10:39] PeterYim: links to slides can be found under: 
	http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_03_14#nid3NPF    (3OLF)
	[10:41] PeterYim: == JoanneLuciano presenting ... see: the [ 1-Luciano ] slides    (3OLG)
	[10:48] GaryBergCross: Is the link for SADI service (sadi.org) on the page the correct???    (3OLH)
	[11:00] MikeDean: @Gary SADI service is at http://sadiframework.org    (3OLI)
	[11:03] GaryBergCross: Mike, yes that is what I thought. The session page has it wrong.    (3OLJ)
	[11:03] PeterYim: @GaryBergCross, @MikeDean - thank you, the link has been updated on the session 
	page now    (3OLK)
	[10:51] TerryLongstreth: Joanne - can you give an open reference /url for the Cockburn Method?    (3OLL)
	[11:04] JoanneLuciano: http://alistair.cockburn.us/Basic+use+case+template the use case template 
	appears to be broken there --we have a local copy (and I'll send the to Alistair to let him know. I 
	think I forgot to do this when I was in contact with him a while ago: 
	http://tw.rpi.edu/media/latest/UseCase-Template_SeS (someone please try this RPI link and verify it 
	works). thanks    (3OLM)
	[11:05] MikeBennett: That link works.    (3OLN)
	[11:06] MikeDean: @Joanne http://tw.rpi.edu/media/latest/UseCase-Template_SeS works, downloading a 
	Microsoft Word template    (3OLO)
	[11:07] MeganKatsumi: Thanks!    (3OLP)
	[11:07] JoanneLuciano: @MikeBennett Thanks for checking.    (3OLQ)
	[11:23] JoanneLuciano: For those downloading the use case template -- recall that is the "starting 
	point" -- that is what needs to be formalized so that an ontology (or part of one) can be evaluated 
	with respect to it.    (3OLR)
	[10:55] anonymous morphed into BobbinTeegarden    (3OLS)
	[10:48] HansPolzer: Regarding Joanne's presentation, I would suggest looking at a range of contexts 
	for ontology evaluation, not just a use case. Another way to look at this is as a set of use cases 
	that span the range of ontology application contexts    (3OLT)
	[10:51] MikeBennett: @Hans I think that defines a difference between an ontology which is to be 
	developed for a specific application, and an ontology that is to be developed as a standard, which 
	would necessarily be use case agnostic. Though of course, if ontologies developed for applications 
	are designed to be broader than just one use case, then they can be reused by others. I think this 
	opens up an interesting line of discussion we can cover in the discussion.    (3OLU)
	[10:53] MatthewWest: I know people who argue that Use Cases are inadequate as a statement of 
	requirements, because they are point requirements, whereas what you really want is to develop a 
	required capability. Use cases can be very helpful in informing such a required capability, and 
	later illustrating it, but are not a substitute.    (3OLV)
	[10:54] AmandaVizedom: @Hans - alternative but compatible view (I think): identify elements or 
	characteristics of use cases that matter (or are hypothesized to matter) for ontology suitability, 
	so that these elements can be reused to analyze and describe other use cases. Some of these will 
	clearly be contextual, but the path to reuseful reasoning is better defined. As I understand it 
	(Joanne can confirm/disconfirm) this is part of the first-stage support that is envisioned for GOEF.    (3OLW)
	[10:56] ToddSchneider: Use cases can be used to discover requirements (i.e., an analysis tool).    (3OLX)
	[10:57] HansPolzer: Use cases also have lots of implicit scope, which can lead to overlooked 
	requirement assumptions. On the other hand, a set of use cases can help understanding of a range of 
	application contexts, since "range" is often difficult to conceptualize while use cases are 
	typically presented as quasi-concrete examples.    (3OLY)
	[10:59] AmandaVizedom: @Matthew: I would agree with that; there is an analysis stage between use 
	case description and technical requirements specification that utilizes knowledge of both the 
	operational background/need and the technical specifics.    (3OLZ)
	[10:59] TerryLongstreth: Joanne question - Does your methodology necessarily reveal the existence of 
	an Ontology?    (3OM0)
	[11:12] JoanneLuciano: @TerryLongstreth, I don't understand your question. Let's discuss during the 
	discussion.    (3OM1)
	[11:00] MeganKatsumi: @Amanda: Can you clarify what you mean by elements or characteristics of use 
	cases?    (3OM2)
	[11:06] AmandaVizedom: @Megan - I should probably let Joanne talk about what she means first. ;-) At 
	least, if I'm correct in saying that this is a reasonable way to describe what she is talking about 
	in the description of the 1st stage of the GOEF framework, in which the use case is analyzed and 
	specified. I think, from Joanne's description (and previous discussions), that this specification is 
	not open-ended but focuses on capturing particular aspects of the use case that are known or 
	believed to be needed as input in order to evaluate how well various candidate ontologies might fit 
	that use case.    (3OM3)
	[11:06] JoanneLuciano: my original presentation from 2008 is on slideshare (you'll notice I used 
	Leo's template for that (I was at MITRE).    (3OM4)
	[11:50] JoanneLuciano: should it come up. here's the link to the slideshare presentation: 
	http://www.slideshare.net/joanneluciano/luciano-pr-08849ontologyevaluationmethodsmetrics 
	(from 2008!) it was a proposal for internal research at MITRE.    (3OM5)
	[11:07] JoanneLuciano: I am listening to Leo and I still have a bad cold, so I'm happy to discuss 
	after during discussion.    (3OM6)
	[11:00] PeterYim: == LeoObrst presenting ... see: the [ 2-Obrst ] slides    (3OM7)
	[11:03] FabianNeuhaus: Leo: could you tell us when you switch slides?    (3OM8)
	[11:09] DougFoxvog: There can be problems with a separation into T-Box & A-Box. In nature guides, 
	for example, lots of statements are made of properties of classes (e.g., types of living thing). 
	These classes, themselves are instances of higher-order classes (Species, Genus, ... Phylum, etc.). 
	These types are both instances & classes. The same thing happens when talking about part types. The 
	classic Wine Ontology had huge problems in expressing properties of types of wine, ending up 
	defining the narrowest classes as instances of the next broader class, because of restrictions of 
	the A/T box restrictions it was operating under.    (3OM9)
	[11:10] AmandaVizedom: @Megan, @Joanne, I'd expect, for example, that the kinds of things Leo is 
	discussing now (slides 2-3: how complex is the reasoning needed for the use case, if any?) would be 
	a subgroup of the use case characteristics that can be described and matter for ontology 
	suitability.    (3OMA)
	[11:11] DougFoxvog: Leo discussing DL not being able to represent many rule types. I suggest that 
	this is a reason to avoid using DL languages.    (3OMB)
	[11:12] MikeBennett: @Doug avoid in general, or avoid for specific requirements / usages of the 
	ontology?    (3OMC)
	[11:14] AmandaVizedom: @doug +1 regarding the Wine Ontology example. It is a good example of the 
	approach suggested, and choices forced, by a particular kind of DL-based language. It is also a good 
	example of why for most reasoning, or cross-specialty integration, applications, that approach is 
	crippling.    (3OMD)
	[11:14] DougFoxvog: @Mike: I suggest avoiding unless the only use of the "ontology" is to be used as 
	a taxonomy. If you use rules or want to integrate, a more powerful language is called for, imho.    (3OME)
	[11:15] MikeBennett: @Doug - thanks. I guess that comes under the heading of what Leo calls "not an 
	ontology" :)    (3OMF)
	[11:15] DougFoxvog: @Mike: +1    (3OMG)
	[11:17] anonymous morphed into PavithraKenjige    (3OMH)
	[11:19] DougFoxvog: Re Slide 7, "tractable reasoning": even a higher order language can exhibit 
	tractable reasoning, when the rules reasoned over permit it. Note that normal computer languages are 
	intractable, but people still program using them. A programmer would not want to program in a 
	guaranteed tractable language.    (3OMI)
	[11:18] PavithraKenjige: Requirements are written in statements can be shown as use cases and 
	scenarios to capture how user use a system to meet those requirements    (3OMJ)
	[11:20] PavithraKenjige: So in my opinion, Use Cases and scenarios and test cases that are based on 
	Use Cases & scenarios are verification or evaluation of meeting the requirements    (3OMK)
	[11:21] JoanneLuciano: I think SADI services can help - I'd like to incorporate GOEF development 
	into https://github.com/timrdf/DataFAQs/wiki I'd love to have some "clinics" to discuss and develop 
	the implementation. I have a paper in the works that will use the iChoose example throughout. The 
	current draft version doesn't have a single example (because these ideas have only been able to be 
	developed in fits and starts). It includes some mock up screen shots to help get ideas across.    (3OML)
	RE: not an ontology -- the 'spectrum' seems to help cover bases and avoid "semantic" arguments 
	(which is a challenge in a "semantic" community)    (3OMM)
	[11:22] DougFoxvog: Re slide 8. 3D/4D reasoning can be left out of most ontologies. If a given 
	application requires committing to one or the other, it can also inherit a small ontology that 
	specifies the 3D+1 or 4D theory.    (3OMN)
	[11:25] DougFoxvog: re slide 9: There is a huge ontology of "part-of". Transitivity often won't work 
	if one switches between different types of "part-of".    (3OMO)
	[11:26] MikeBennett: @Doug presumably part terms like "nearside front wheel" versus part terms like 
	Wheel have different requirements in this regard.    (3OMP)
	[11:29] DougFoxvog: @Mike: Yes.    (3OMQ)
	[11:29] GaryBergCross: @MikeBennett wheels may be "components" rather than a simple part.    (3OMR)
	[11:31] MikeBennett: @Doug thought so. I think distinguishing between those usages (e.g. what is a 
	Component) per Gary, depends on using a suitable upper ontology that distinguishing 
	something-in-a-role from something-in-itself. So there is a dependency between these two 
	considerations I think.    (3OMS)
	[11:31] DougFoxvog: @Gary: "componentOf" would be a specialization of "physicalSubpartOf".    (3OMT)
	[11:33] GaryBergCross: I sometimes think of these bridging, integrative ontologies as simpler 
	bridging schemas.    (3OMU)
	[11:33] DougFoxvog: @Mike: Yes. the truth of (componentOf Car2087532408 Wheel234752347) is time 
	dependent.    (3OMV)
	[11:34] GaryBergCross: @Doug Yes that is a good specialization.    (3OMW)
	[11:29] DougFoxvog: re slide 10: Do you need human capability to formulate queries using the 
	computer's query language? A user interface should obviate this. One does not require a human 
	capability to formulate SQL queries for someone to ask for information from a database.    (3OMX)
	[11:33] anonymous morphed into FredHosea    (3OMY)
	[11:35] DougFoxvog: Re slide 12: "Anyone we know" might be Insectivore.    (3OMZ)
	[11:36] DougFoxvog: The Venn diagram does not intersect Reptile and Mammal, so the "Anyone we know" 
	circle includes some mammals, some reptiles, and some other living things.    (3ON0)
	[11:35] AmandaVizedom: Relating Leo's talk to my own (two weeks ago,) 
	about getting from business requirements to ontology evaluation): 
        most of the questions Leo is talking about are the sorts of things that should be asked 
	at the stage when "business requirements" are being transformed into technical requirements 
	(for the ontology and/or system). The answers should: figure into ontology/system design; 
	be used to development constraints, goals, steps; be used to identify what should be evaluated 
	when to make sure requirements are being met and maintained. I would say that this is the 
	knowing-what-to-develop/knowing-what-to-check-for aspect of the Big Issues in Ontology 
	Evaluation; the how-to-check-for-X/evaluation methods aspect is a parallel Big Issue.    (3ON1)
	[11:38] MikeBennett: @Amanda +1 - these kinds of technical considerations are a parallel to what in 
	conventional technology development would be non functional requirements - but they clearly deliver 
	benefits to the business integrity of a model of the business domain. That opens up a lot of 
	interesting questions.    (3ON2)
	[11:39] MatthewWest: @LeoObrst - please go ahead and cover you backup slides as well.    (3ON3)
	[11:40] AmandaVizedom: re: collaborative development: big tool need in the OWL world: tunable 
	automatic checking on addition (commit-time; rejection if inconsistent). More comprehensive 
	automatic bookkeeping on assertions and inferences so that problems can be debugged, fixed and 
	modules/packages re-submitted.    (3ON4)
	[11:40] JoanneLuciano: I'm wondering how many times Leo has been around the circles and items on 
	slides#18-21. I don't think I want to count. BTW, Happy Pi Day (3.14 in the way the US does month 
	day).    (3ON5)
	[11:46] AmandaVizedom: @Leo - slide 20 - yes indeed, often looped around, etc., *and* often there 
	are interdependencies with other components of a larger system. At various points, there is or 
	should be cross-checking to see whether things are (still) in sync, whether they are on-track to 
	work together as desired.    (3ON6)
	[11:47] MikeBennett: @Amanda +1 - not only for collaborative development but I think we need a tool 
	that does for ontologies what UML does for software designs, including what you have there, and also 
	visualizations both to business domain and to implementers.    (3ON7)
	[11:48] AmandaVizedom: @MikeBennett: Agreed.    (3ON8)
	[11:49] AmandaVizedom: @MikeBennett: I should note that the capabilities I mentioned already do 
	exist in some development environments, especially Cyc and to a degree in some in-house, specialized 
	systems. But not, to my knowledge, in OWL tools generally.    (3ON9)
	[11:40] PavithraKenjige: @Amanda, Use Cases follow the requirement or use cases & scenarios 
	represent detail requirements.. which is followed by design phase. In UML, Object diagrams are done 
	as design stage..    (3ONA)
	[11:41] PavithraKenjige: Ontology development is a phase that can be mapped to design stage ..    (3ONB)
	[11:50] PeterYim: == Q & A and Open Discussion, MatthewWest moderating ...    (3ONC)
	[11:50] MatthewWest: In the previous talk in Track C we were looking at methodologies for 
	integrating ontologies. The first thing that struck me was that easily the biggest priority was 
	achieving consistency in the ontology, particularly in the face of large ontologies with many 
	contributors who are geographically dispersed. The other thing that came across was the need for an 
	architectural approach, providing structure to taking the right decisions in the right order.    (3OND)
	[11:51] MatthewWest: Today's session on development methods for reasoning ontologies also impressed 
	on me the need for an architectural approach, although the emphasis was slightly different here. The 
	other thing that struck me is that ontology development is often done directly at the implementation 
	level. You start by developing your ontology in CL or OWL, already in its implementation 
	environment. This is how it was once in database development, but that was a long time ago. It would 
	now be considered normal to develop data models at multiple levels, a conceptual level which just 
	gives the outline, the logical level that is complete, including all the rules, but still 
	independent of any implementation environment. Finally there is a data model that takes account of 
	the implementation environment, and the processing needs of the application. This separation would 
	indicate a level of maturity in understanding the ontology development process, but will need tool 
	support that does not exist yet as far as I am aware.    (3ONE)
	[11:52] DougFoxvog: Ref the capabilities Amanda referred to. The OBO ontologies have obviously not 
	had such tools, since, e.g., disjoint classes had had common subclasses in several posted 
	ontologies.    (3ONF)
	[11:52] MikeBennett: Oops! (pun intended :) )    (3ONG)
	[11:56] AmandaVizedom: It is good to see more tools for evaluating ontologies, including OWL if for 
	no other reason than that so many people are developing ontologies in it and are in severe need of 
	evaluation help. :-) It's important to note, though, that the infrastructure for easy/automated 
	testing of collaborative additions to a large ontology, or addition of new ontologies to a 
	repository of ontologies that are meant to be compatible, is also very much needed.    (3ONH)
	[11:56] MikeBennett: I should add that we have the same problems in the FIBO development - we have 
	defined upper ontology partitions to distinguish e.g. independent v relative things (like the parts 
	examples) but don't know when these have been misapplied until we can run external checks on it. 
	Something like what UML tools do, where illegal model efforts are flagged up as non compliant, would 
	be useful in a tool.    (3ONI)
	[11:59] AmandaVizedom: And, along the lines of Joanne's GOEF ideas and Leo's exposition of important 
	requirements questions and design decisions, tools are also very much needed to support this kind of 
	analysis, and recording of the results, so that people can figure out what they need and what to 
	test for in the first place.    (3ONJ)
	[12:00] JoanneLuciano: I agree w Leo - need many kinds of testing (at many levels)    (3ONK)
	[12:01] JoanneLuciano: Agree w/ Amanda's comments    (3ONL)
	[12:02] JoanneLuciano: The reason I like the "component" aspect is it facilitates testing (and Leo 
	just mentioned unit testing). an important "component"    (3ONM)
	[12:01] BobbinTeegarden: @Mike is it possible that when one starts with tree structures 
	(categorization) when modeling something essentially graph shaped, that we end up in unintended 
	stovepipes?    (3ONN)
	[12:02] DougFoxvog: @Bobbin: yes. Modeling a graph structure as a tree will almost always lead to 
	trouble.    (3ONO)
	[12:02] MikeBennett: @Bobbin I think that is a real danger - unless some serious imagination is used 
	in defining abstractions "What kind of thing is this" asked iteratively until you get to a very 
	atomic meaningful concept. I don't know how you would validate / verify for that.    (3ONP)
	[12:04] MikeBennett: @Bobbin plus you need to apply faceted classification, which would require some 
	additional notation. There is no reason for any given class to only have one parent (except, of 
	course, when you are designing for an application and need to think about the reasoning overhead of 
	multiple inheritance).    (3ONQ)
	[12:08] DougFoxvog: re Mike's reference to faceted classification: specifying the facets is 
	difficult, and ensuring those facets that should be coverings or partitions are such can be hard to 
	model in a simple ontology language. Higher-level languages such as Cyc, enable this by reifying the 
	facets as meta-classes.    (3ONR)
	[12:10] MikeBennett: @Doug thanks - I've been trying to figure out if there's a way of defining some 
	OWL annotation properties to signify facets - the obvious basis for a classification facet seems to 
	be the OWL Union Class - but need to decorate that somehow to identify what property of the parent 
	class has different values in the child classes in that facet.    (3ONS)
	[12:13] DougFoxvog: @Mike: Using OWL-Full, one can create Facets & define classes as instances of 
	the facets. If several types of facets are defined.    (3ONT)
	[12:14] MikeBennett: @Doug thanks    (3ONU)
	[12:16] DougFoxvog: ... If several types of facets are defined: Partition, Covering, ..., with rules 
	attached, it could force the appropriate disjointnesses and would be good for annotation, even if 
	later use of the ontology would drop such facets as mere documentation in order to allow for a 
	simpler (e.g., DL) reasoning scheme.    (3ONV)
	[12:21] MikeBennett: @Doug I'll look into that - this is going to be very useful in classifying 
	financial instruments for example. Different facets are suited to different use cases, so it would 
	make sense to extract a single-inheritance taxonomy for a given use case - but different ones for 
	different use cases (e.g. investment management versus risk and compliance).    (3ONW)
	[12:02] AmandaVizedom: Re: Todd's question: debugging when things don't go right - This is one 
	reason why ontology development / management systems should have much, much more automated 
	bookkeeping and inference traceability than many do. Again, Cyc & some other systems have this, but 
	it is lacking in many commonly-used environments, with real consequences.    (3ONX)
	[12:05] MichaelGruninger: Competency questions are great for evaluating whether or not there are 
	enough axioms in our ontology, but there are still two outstanding issues. 1. Are these the right 
	competency questions? 2. The competency questions themselves introduce their own ontological bias    (3ONY)
	[12:05] JoanneLuciano: Agree with what Michael is saying - the competency questions aren't enough, 
	and yes the 2nd level addresses the external requirements (Compliance standards, for example) which 
	again is a different level than the OWL / intrinsic level.    (3ONZ)
	[12:06] AmandaVizedom: @Michael +1 Questions embed some ontological commitments. That can be OK *if 
	it does done intentionally*, i.e., if you design your questions to test for compatibility with those 
	commitments. But it's a real problem with many published test approaches - they embed *assumed* 
	commitments that may not be shared in real cases in which people attempt to apply those questions.    (3OO0)
	[12:06] LeoObrst: @Matthew's comments: re: ontology development still seems to be research. Yes, 
	that's why ontology training (a previous Ontology Summit) is very important.    (3OO1)
	[12:07] ToddSchneider: To what extent are the discussions about ontology evaluations assuming a 
	non-dynamic environment (of ontology changes)?    (3OO2)
	[12:07] JoanneLuciano: not assumed in the GOEF approach    (3OO3)
	[12:08] JoanneLuciano: interesting though about autonomous systems    (3OO4)
	[12:12] AmandaVizedom: @Todd: I assume that dynamic is in fact more typical, perhaps biased by 
	environments I have worked in. I think that much research and tool development assumes a more static 
	model, though, and/or that individual ontologies reach a "done" stage, after which they are rarely 
	changed and new work is on other ontologies.    (3OO5)
	[12:09] AmandaVizedom: @Matthew, @Leo -- true, but consider also the role of capturing our (ontology 
	community) dispersed knowledge and lessons learned. Some things are better understood than others, 
	but the understanding is unevenly distributed and redundant research continues. Or, at least, mature 
	hypotheses can be formed, rather than the sort proto-wheel ones that still get run up the flagpole.    (3OO6)
	[12:12] LeoObrst: @Fabian: I agree. Please place that in the chat, so we don't lose the comment, 
	i.e., the 3-n things you need for evaluating ontologies.    (3OO7)
	[12:18] FabianNeuhaus: My two questions are: (1) Assuming you need to make a recommendation to 
	somebody who develops an ontology. What are the three most important aspects that you think that the 
	person should evaluate and how? (2) Could you identify the kind of tools that would make it easier 
	for ontology developers to perform the recommended ontology evaluation?    (3OO8)
	[12:11] DougFoxvog: Does anyone want to create an ontology of the ontology evaluation & development 
	concepts and issues that we have been discussing? With such a tool, individual ontologies could have 
	their properties specified using this ontology.    (3OO9)
	[12:14] AmandaVizedom: @doug - that is one of the hackathon & clinic proposals, though I don't know 
	that there will be enough interest or participation for it to pass the selection gate.    (3OOA)
	[12:15] AmandaVizedom: Ontology of Ontology Evaluation proposal is at: 
	http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013_Hackathon_Clinics#nid3O90    (3OOB)
	[12:17] LeoObrst: Folks, I must go to another meeting. Thank you for your comments and 
	participation!    (3OOC)
	[12:17] MikeBennett: Thanks @Leo    (3OOD)
	[12:18] JoanneLuciano: @LeoObrst THANK YOU -- always!    (3OOE)
	[12:19] TerryLongstreth: USE cases vs. capabilities wrt evaluation - If we assume that Use cases are 
	elaborated into capabilities, then evaluation based on use cases would be elaborated into more 
	detailed evaluations, and so on until we arrive at evaluation    (3OOF)
	[12:19] TerryLongstreth: based upon implementation    (3OOG)
	[12:18] JoanneLuciano: When I say "Function" I include in that "Capability"    (3OOH)
	[12:19] JoanneLuciano: The feedback is useful - I'll make it more explicit next time    (3OOI)
	[12:20] JoanneLuciano: words alone will often get us into trouble. -- I don't like that I use 
	standard for the "second" level, for example    (3OOJ)
	[12:23] JoanneLuciano: We used to use functional specifications, they worked    (3OOK)
	[12:22] FabianNeuhaus: @Pavithra, Matthew: maybe you could take this discussion offline?    (3OOL)
	[12:23] PeterYim: join us again, same time next week (Thu 2013.03.21), for OntologySummit2013 
	session-10: "Software Environments for Evaluating Ontologies - II" - Co-chairs: MikeDenny & PeterYim 
	- see developing session page at http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_03_21 
	... please pay special attention to the start-time (for the folks in the US and Canada, unlike this 
	particular session, we will back to our normal start-time next week)!    (3OOM)
	[12:23] PeterYim: great talks, thank you Joanne & Leo    (3OON)
	[12:23] AmandaVizedom: Thanks all!    (3OOO)
	[12:23] DougFoxvog: Thanks, all    (3OOP)
	[12:23] SamirTartir: Thanks all    (3OOQ)
	[12:24] JoanneLuciano: I want to acknowledge JamesMichaelis and Nicolau Depaula from CTG SUNY Albany 
	for their contributions to my presentation    (3OOR)
	[12:24] PeterYim: -- session ended: 12:23pm PDT --    (3OOS)
 -- end of in-session chat-transcript --    (3NS4)

Additional Resources:    (3NSB)

For the record ...    (3NSI)

How To Join (while the session is in progress)    (3NSJ)

Conference Call Details    (3NPL)

Attention: Please take special note on the start time of the event, as the US will be on daylight saving (summer) time on this day, while the EU is still on standard (winter) time! ... For participants from the US and Canada, for example, this session will start an hour later than usual, whereas for those from the EU, the start-time will be the same as last week's.    (3O89)

Attendees    (3NQI)