OntologySummit2013: Panel Session-06 - Thu 2013-02-21    (3LH2)

Summit Theme: "Ontology Evaluation Across the Ontology Lifecycle"    (3LH3)

Session Topic: Ontology Summit 2013: Synthesis-I    (3MT3)

Briefings:    (3MT5)

Archives:    (3MTD)

Abstract:    (3MV6)

OntologySummit2013 Session-06: "Synthesis-I" - intro slides    (3MV7)

This is our 8th Ontology Summit, a joint initiative by NIST, Ontolog, NCOR, NCBO, IAOA & NCO_NITRD with the support of our co-sponsors. The theme adopted for this Ontology Summit is: "Ontology Evaluation Across the Ontology Lifecycle."    (3MV8)

This is our 6th event, and we are a third of the way through the virtual sessions for the Summit. Each of the four tracks have hosted very exciting presentations that address the key Summit themes -- Intrinsic Aspects of Ontology Evaluation, Extrinsic Aspects of Ontology Evaluation, Building Ontologies to Meet Evaluation Criteria, and Software Environments for Evaluating Ontologies.    (3MV9)

Currently, there is no agreed methodology for development of ontologies, and there are no universally agreed metrics for ontology evaluation. At the same time, everybody agrees that there are a lot of badly engineered ontologies out there, thus people use -- at least implicitly -- some criteria for the evaluation of ontologies.    (3MVA)

During this Ontology Summit, we seek to identify best practices for ontology development and evaluation. We will consider the entire lifecycle of an ontology -- from requirements gathering and analysis, through to design and implementation. In this endeavor, the Summit will seek collaboration with the software engineering and knowledge acquisition communities. Research in these fields has led to several mature models for the software lifecycle and the design of knowledge-based systems, and we expect that fruitful interaction among all participants will lead to a consensus for a methodology within ontological engineering. Following earlier Ontology Summit practice, the synthesized results of this season's discourse will be published as a Communiqué.    (3MVB)

In today's session, we will focus on the synthesis of all of these ideas as input into an initial draft of the Summit Communiqué. We will also have a discussion on some new features of this year's Symposium -- a hackathon for ontology evaluation software support and the idea of ontology evaluation clinics for ontologies that people want evaluated.    (3MVC)

More details about this OntologySummit is available at: OntologySummit2013 (homepage for this summit)    (3MVD)

Agenda:    (3MVE)

OntologySummit2013 - Panel Session-06    (3MVF)

Proceedings:    (3MVS)

Please refer to the above    (3MVT)

IM Chat Transcript captured during the session:    (3MVU)

 see raw transcript here.    (3MVV)
 (for better clarity, the version below is a re-organized and lightly edited chat-transcript.)
 Participants are welcome to make light edits to their own contributions as they see fit.    (3MVW)
 -- begin in-session chat-transcript --    (3MVX)
	------	
	Chat transcript from room: summit_20130221
	2013-02-21 GMT-08:00 [PST]	
	------    (3NBI)
	[8:36] PeterYim: Welcome to the    (3NBJ)
	 = OntologySummit2013: Panel Session-06 - Thu 2013-02-21 =    (3NBK)
	Summit Theme: Ontology Evaluation Across the Ontology Lifecycle    (3NBL)
	* Summit General Co-chairs: Professor MichaelGruninger (U of Toronto) and Dr. MatthewWest (Information Junction)    (3NBM)
	Session Topic: Ontology Summit 2013: Synthesis-I    (3NBN)
	* Session Chair: Dr. MatthewWest (Information Junction, UK)    (3NBO)
	Panelists / Briefings:    (3NBP)
	* Dr. MatthewWest (Information Junction, UK) & Professor MichaelGruninger (U of Toronto, Canada)
	  - "General Assessment & Fine-tuning of OntologySummit2013 Direction & Approach"    (3NBQ)
	* Dr. LeoObrst (MITRE) & Dr. SteveRay (CMU)
	  - "Track-A: Intrinsic Aspects of Ontology Evaluation - Synthesis-1"    (3NBR)
	* Mr. TerryLongstreth (Ind. Consultant) & Dr. ToddSchneider (Raytheon)
	  - "Track-B: Extrinsic Aspects of Ontology Evaluation - Synthesis-1"    (3NBS)
	* Dr. MatthewWest (Information Junction) & Mr. MikeBennett (EDM Council; Hypercube)
	  - "Track-C: Building Ontologies to Meet Evaluation Criteria - Synthesis-1"    (3NBT)
	* Dr. MichaelDenny (MITRE) & Mr. PeterYim (Ontolog; CIM3)
	  - "Track-D: Software Environments for Evaluating Ontologies - Synthesis-1"    (3NBU)
	* Dr. AmandaVizedom (Ind. Consultant) & Dr. FabianNeuhaus (NIST)
	  - "Approach to the OntologySummit2013 Communique"    (3NBV)
	* Mr. MikeDean (Raytheon BBN) & Mr. PeterYim (Ontolog; CIM3)
	  - "Approach to the "Hackathon" & "Clinics" Activities"    (3NBW)
	Logistics:    (3NBX)
	* Refer to details on session page at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_02_21    (3NBY)
	* (if you haven't already done so) please click on "settings" (top center) and morph from "anonymous" to your RealName (in WikiWord format)    (3NBZ)
	* Mute control: *7 to un-mute ... *6 to mute    (3NC0)
	* Can't find Skype Dial pad?
	** for Windows Skype users: it's under the "Call" dropdown menu as "Show Dial pad"
	** for Linux Skype users: please note that the dial-pad is only available on v4.1 (or later or the earlier Skype versions 2.x,)
	   if the dialpad button is not shown in the call window you need to press the "d" hotkey to enable it.    (3NC1)
	Attendees: AliHashemi, AmandaVizedom, AnatolyLevenchuk, AstridDuqueRamos, BruceBray, ClarePaul, 
	DavidLeal, DougFoxvog, DuaneNickull, FabianNeuhaus, FrankLoebe, GaryBergCross, JeanneHolm, JieZheng, 
	JoanneLuciano, JoelBender, KenBaclawski, LamarHenderson, LeoObrst, MarcelaVegetti, MariaPoveda, 
	MatthewWest, MichaelDenny, MikeBennett, MikeDean, MikeRiben, PeterYim, RichardMartin, SimonSpero, 
	SteveRay, TerryLongstreth, TillMossakowski, ToddSchneider, TomTinsley, YuriyMilov    (3NC2)
	 == Proceedings: ==    (3NC3)
	[9:25] anonymous morphed into MariaPoveda    (3NC4)
	[9:27] SteveRay: Hi Maria, glad you could join us today.    (3NC5)
	[9:28] anonymous morphed into MichaelDenny    (3NC6)
	[9:29] MariaPoveda: Hi all    (3NC7)
	[9:30] anonymous morphed into LamarHenderson    (3NC8)
	[9:30] SteveRay: Skype is acting up again. Drops me after about 2 seconds...    (3NC9)
	[9:31] SteveRay: Google Voice works...    (3NCA)
	[9:42] JoanneLuciano: can't get in on skype :-(    (3NCB)
	[9:45] PeterYim: @Joanne - please try restarting skype (or restarting your machine) or call one of 
	the phone numbers ... skype should be working as quite a few (over 15) are connected via skype as we speak    (3NCC)
	[9:32] DuaneNickull: Good Morning all!    (3NCD)
	[9:33] AnatolyLevenchuk: To Duane: we have 21:13 here in Moscow. Good night! :-)    (3NCE)
	[9:35] PeterYim: == MatthewWest opens the session on behalf of the General Co-chairs 
	... see: the [0-Chair] slides    (3NCF)
	[9:35] List of members: AliHashemi, AmandaVizedom, AnatolyLevenchuk, Astrid, ClarePaul, DavidLeal, 
	DougFoxvog, DuaneNickull, FabianNeuhaus, FrankLoebe, JoelBender, LeoObrst, MariaPoveda, MatthewWest, 
	MichaelDenny, MikeBennett, MikeDean, PeterYim, RichardMartin, SimonSpero, SteveRay, TerryLongstreth, 
	ToddSchneider, vnc2    (3NCG)
	[9:39] AmandaVizedom: re: Matthew's slide 4: ...or, it may be conscious knowledge on the part of an 
	individual reviewer, but it isn't shared knowledge. Thus, issues of consistency and guidance for the 
	field, etc.    (3NCH)
	[9:42] DougFoxvog: @Amanda: the ref. to "Matthew's slide 4" should be to Steve/Leo's slide 4.    (3NCI)
	[9:48] AmandaVizedom: @Doug: No, in that comment I was responding to Matthew's comment about how we 
	*do* evaluation (of ontology papers).    (3NCJ)
	[9:35] SimonSpero morphed into SimonSpero    (3NCK)
	[9:44] anonymous morphed into TomTinsley    (3NCL)
	[9:39] PeterYim: == LeoObrst / SteveRay presenting ... see: the [1-Track-A] slides    (3NCM)
	[9:41] AmandaVizedom: re: Leo's slide 2: In-Between > Both    (3NCN)
	[9:44] FabianNeuhaus: @Leo: slide 2: I think there is an important difference between relationship 
	between ontology and world (e.g, whether the ontology represents reality accurately) and whether the 
	ontology meets black box requirements of an IT system. The first can be evaluated independently of 
	requirements, the second is always relative to the requirements from an IT system. In the first case 
	the ontology is not opaque to the tester, in the second it isn't [correction: "it might be" (ref. 
	below)]. Thus, these should not be lumped together as "extrinsic"    (3NCO)
	[9:58] SteveRay: @Fabian, your first comment at 9:44 you had two negatives. Could you restate? (The 
	sentence beginning "In the first case...")    (3NCP)
	[9:59] FabianNeuhaus: Sorry. I meant to write: "In the first case the ontology is not opaque to the 
	tester, in the second it might be (as blackbox testing). Thus, these should not be lumped together 
	as "extrinsic"    (3NCQ)
	[10:00] LeoObrst: @Fabian: (slide 2): Yes, indeed. I don't think we lump these together as 
	extrinsic, if you look at the other slides. The first focuses on mostly intrinsic-->land of 
	in-between. The second focuses on the extrinsic regions.    (3NCR)
	[10:01] JoanneLuciano: @LeoObrst --> I would say land of in-betweenS (plural)    (3NCS)
	[10:02] SteveRay: OK. As Leo said, he and I are using "Intrinsic" and "Extrinsic" as useful extreme 
	concepts, and few if any evaluations will be at either extreme.    (3NCT)
	[9:51] AmandaVizedom: @Fabian: I think that there are aspects of the first that can be evaluated 
	independently, but not nearly enough to select/eval ontologies for most uses -- they don't just need 
	to represent the world, but the parts and aspects of the world with which the domain / users 
	interact.    (3NCU)
	[9:52] FabianNeuhaus: @Amanda. I agree. However, ontology evaluation is not only done for the 
	purpose of choice. It is, for example, done during the development process.    (3NCV)
	[9:53] MichaelDenny: @FabianNeuhaus +1 Fitness for an application versus fitness as conformance to 
	world reality.    (3NCW)
	[9:54] FabianNeuhaus: @MichaelDenny: Exactly!    (3NCX)
	[9:49] MikeBennett: I think Fabian's point has interesting implications for the creation of formal 
	methodologies for ontology development and evaluation - in particular the ontology-world 
	relationship should be fundamental to what process paths to follow in such a methodology.    (3NCY)
	[9:57] AmandaVizedom: @Fabian, Michael: I agree that such elements of "conformance to world reality" 
	can be independently assessed. I argue, however, that in order to evaluate "conformance to world 
	reality" usefully, you need to know what portion of world is supposed to be modeled. As with 
	scientific theories, ontologies cover not only what we recognize at large scale as domains, but 
	particular kinds of relationships and interactions, and the characteristics of things that partake 
	of those.    (3NCZ)
	[9:58] MikeBennett: @Amanda +1 - there's the basic model theoretic relation of whether it's really 
	an ontology of the world or some application, and for the former, there's the scope and the 
	ontological commitments that would be appropriate to that scope.    (3ND0)
	[9:59] JoanneLuciano: +100 (have to know purpose before can evaluate)    (3ND1)
	[9:45] AmandaVizedom: @Leo: would you put some aspects of reasoning support in Region 1 (Intrinsic)? 
	I think I would, even though performing the reasoning requires more than the ontology. Probably 
	somewhat about the language and somewhat about what content is actually represented. I'm thinking 
	about questions like: is there support for representation of (& reasoning about) uncertainty? Is 
	there support for provenance information capture, such that it, too can be reasoned about?    (3ND2)
	[10:02] LeoObrst: @Amanda: yes, real reasoning figures across the 3 regions.    (3ND3)
	[9:47] AmandaVizedom: @Leo: I would also see adequacy of coverage as Region 2, insofar as you can't 
	tell what aspects of the world are wanted without knowing about the domain / usage.    (3ND4)
	[10:04] LeoObrst: @Amanda: (re. your second point) Yes, adequacy of coverage primarily falls under 
	Region 2, where domain knowledge and ontology-world correspondence is very important.    (3ND5)
	[9:46] ToddSchneider: Leo, Steve, Why are 'Transitivity, symmetry, reflexivity, equivalence' listed 
	as meta-properties [to an ontology]?    (3ND6)
	[9:48] DougFoxvog: @Todd: "transitivity, symmetry, & reflexivity" are properties of properties; thus 
	meta-properties. Equivalence can be a property of properties as well as one of types and individuals.    (3ND7)
	[9:48] ToddSchneider: Doug, okay.    (3ND8)
	[9:48] anonymous morphed into GaryBergCross    (3ND9)
	[9:51] DougFoxvog: Region 3 (purely extrinsic) would disallow a query as to whether two classes are 
	disjoint. However, since it allows queries, couldn't it ask if a hypothetical thing (perhaps by 
	reifying it) could be an instance of those two classes -- in order to determine disjointness?    (3NDA)
	[10:05] FabianNeuhaus: @Leo: I just don't think that this is a helpful way to slice up the cake. The 
	way you describe it there is a sliding scale between two extremes with considering only internal 
	properties on one side and considering behavior on the other side. Evaluating the ontology on 
	whether the ontology describes reality properly is not "in the middle between the extremes" it is a 
	different thing entirely.    (3NDB)
	[10:08] SteveRay: @Fabian: I don't agree - this is one of the performance characteristics a user 
	would want to assure is met.    (3NDC)
	[10:07] ToddSchneider: All, from an IT perspective use of the term 'reality' to describe intended 
	interpretations or uses (of the IT system) is misleading.    (3NDD)
	[10:11] LeoObrst: @Fabian: well, we discussed this and felt that by providing pole perspectives, 
	that this would help. One would probably say that conformance of the ontology to reality is truly 
	something that spans all 3 regions, since by definition that is what an ontology as an engineering 
	construct is all about. One might gauge that in different ways. For example, if one has 2 predicates 
	or 2 classes and 1 property for an ontology that is supposed to represent a complex domain, one 
	might gauge it from a narrow intrinsic perspective as being insufficient. That of course is a simple 
	case one hardly ever finds.    (3NDE)
	[10:14] AmandaVizedom: @Fabian, @Leo: I would agree that it doesn't feel like a scale. The second 
	kind of evaluation isn't really "between" the other two. Rather, it's a different kind of activity 
	that incorporates some of the kinds of knowledge that go into the first and third (i.e., ontology 
	theory and understanding of the intended use, respectively), plus some other kinds. To some extent, 
	you could view this in a Venn diagram manner, where each area represent the kinds of knowledge and 
	tools you need in order to do one of these three types of evaluation. In that case, 1 and 3 would 
	intersect, and 2 would include, but not be limited to, that intersection. It might also include 
	knowledge of human factors research, experimental techniques, and various ways of achieving SME 
	validation, etc.    (3NDF)
	[10:16] FabianNeuhaus: @Amanda: I agree. You can easily build an ontology that scores well on 
	intrinsic metrics and shows the intended behavior in the sense of answering queries fine, but 
	contains factual errors.    (3NDG)
	[10:11] FrankLoebe: @Fabian: Are you aware of any methodology / approach for evaluating an ontology 
	regarding its performance of describing reality? Or was anything discussed in the previous summit 
	sessions (not all of which I could follow / catch up with)?    (3NDH)
	[10:13] FabianNeuhaus: @Frank: WernerCeusters addresses that to a degree in his methodology.    (3NDI)
	[10:08] FabianNeuhaus: @Todd: that's not what I mean by reality.    (3NDJ)
	[10:09] ToddSchneider: Fabian, so much for interpretation clarity:) But I was also referring to 
	Leo's & Steve's slides and in general.    (3NDK)
	[10:28] TillMossakowski: a question about the track A talk (slide 7): is it possible to download 
	OntoQA somewhere?    (3NDL)
	[10:33] LeoObrst: @Till: I'm not sure. We'll query SamirTartir and the other OntoQA folks.    (3NDM)
	[9:52] anonymous morphed into KenBaclawski    (3NDN)
	[9:54] PeterYim: == TerryLongstreth / ToddSchneider presenting ... see: the [2-Track-B] slides    (3NDO)
	[10:04] PeterYim: == MikeBennett / MatthewWest presenting ... see: the [3-Track-C] slides    (3NDP)
	[10:08] GaryBergCross: I'm surprised that people haven't mention the difference between evaluating 
	light weight ontologies vs. 'heavy" ones with lots of axioms.    (3NDQ)
	[10:09] MariaPoveda: @Gary could it be the first one a subset of the second option?    (3NDR)
	[10:09] MariaPoveda: I mean in a heavyweight ontology you can make the mistake you do in a taxonomy 
	for example, and many other    (3NDS)
	[10:13] SimonSpero: @Maria: @What is the mistake of a taxonomy    (3NDT)
	[10:14] SimonSpero: @Maria: Apart from assuming that a taxonomy is necessarily an ontology    (3NDU)
	[10:14] MariaPoveda: @simon for example to set a subclassOf axiom wrong or to include a class in two 
	levels of the hierarchy    (3NDV)
	[10:14] MariaPoveda: @Simon, some authors also consider cycles as errors, they are not always, but 
	in some cases they are    (3NDW)
	[10:16] SimonSpero: Maria: Cycles are errors in controlled vocabularies other than synonym rings    (3NDX)
	[10:12] SimonSpero: @Gary How do you measure the weight of an ontology    (3NDY)
	[10:12] GaryBergCross: @maria It might be more useful to speculate that light weight ones are 
	transparent to the end user who have an understanding of those requirements rather than formal 
	requirements that a Knowledge Engineer understands as part of development.    (3NDZ)
	[10:14] GaryBergCross: @Simon see http://en.wikipedia.org/wiki/Lightweight_ontologies for a 
	discussion....    (3NE0)
	[10:15] SimonSpero: @Gary: That's what I thought, but I've heard the, er, term misused    (3NE1)
	[10:17] GaryBergCross: @Simon Yes, I am more comfortable with the idea of a light ontology than a 
	heavy one, which is why I use quotes.    (3NE2)
	[10:27] LeoObrst: @Gary et al: granularity is a factor distinguishing lightweight from heavyweight 
	ontologies, and is dependent on the intended application. If one does not need to distinguish 
	between 2 subclasses, e.g., one only needs to represent their parent class. This might be good for a 
	simple search and indexing application.    (3NE3)
	[10:33] GaryBergCross: @Leo Granularity might not be the right concept, but I think that I know what 
	you mean. The reason I don't think this as granular is that one might have 20 sub-types or parts or 
	scale levels in a light model but only get to 3-4 in a more formal one that is better modeled. 
	Concepts get detailed in more formal ontologies.    (3NE4)
	[10:39] LeoObrst: @Gary: yes, you always have that distinction: a very large ontology could be very 
	simple, but cover a lot of simple classes, whereas a very precise ontology may just cover 3-4 of 
	those classes, so a kind of zooming in. It really is granularity and precision. Think of a map and 
	the actual region it maps.    (3NE5)
	[10:36] MariaPoveda: @Leo @Gary, could it be "expressivity"? I'n not sure about the term either...    (3NE6)
	[10:39] MariaPoveda: something like this 
	http://www-ksl.stanford.edu/people/dlm/papers/ontologies-come-of-age-mit-press-(with-citation)_files/image005.gif ?    (3NE7)
	[10:41] LeoObrst: @Maria: Or this: ;) 
	http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2006_01_12#nidIT2. See slide 9, the Ontology Spectrum.    (3NE8)
	[10:46] MariaPoveda: @Leo thanks    (3NE9)
	[10:16] ToddSchneider: Leo, Steve. Terry, Has anybody discussed the notions of 'intrinsic' and 
	extrinsic' criteria w.r.t. particular lifecycle phases?    (3NEA)
	[10:18] ToddSchneider: It would seem that intrinsic criteria/evaluations are more relevant during 
	early lifecycle phases, while extrinsic are more relevant to later lifecycle phases.    (3NEB)
	[10:20] GaryBergCross: @Todd I can see it working the other way. One starts by having a lighter wt 
	ontology that used terms from the domain and is very extrinsic and only later is formalized. You 
	wind up with something in between.    (3NEC)
	[10:20] MichaelDenny: @ToddSchneider Perhaps just the opposite. One may begin with concerns about 
	the domain semantics (extrinsic) and then worry about how best to implement those in a formal 
	ontology (intrinsic).    (3NED)
	[10:21] GaryBergCross: @Michael +1 we agree.    (3NEE)
	[10:19] SteveRay: @Todd: Agreed    (3NEF)
	[10:19] TerryLongstreth: @Todd Not necessarily more relevant, but certainly more available    (3NEG)
	[10:19] FabianNeuhaus: @Todd. I think inconsistency might be relevant at any stage :-)    (3NEH)
	[10:20] SteveRay: @Fabian: But you would have eliminated the inconsistency early in the lifecycle...    (3NEI)
	[10:21] TerryLongstreth: @Todd (rephrase) Intrinsics (of which I would include syntax checking) are 
	probably tested from the outset of a project.    (3NEJ)
	[10:21] ToddSchneider: Fabian, Yes, but the manifestation of inconsistency will be different, hence 
	the associated evaluation criteria should be expected to be different.    (3NEK)
	[10:20] FabianNeuhaus: @Steve: any change to the ontology might potentially lead to inconsistency. 
	That can happen at any stage of the life cycle.    (3NEL)
	[10:21] SteveRay: @Fabian: Of course. I was half serious.    (3NEM)
	[10:22] FabianNeuhaus: @Steve. Sorry. the humor got lost in the medium :-)    (3NEN)
	[10:23] SteveRay: @Fabian: I'll try to use :) more often.    (3NEO)
	[10:21] AmandaVizedom: @Todd: I don't think so; I've seen too many projects go off on the (wildly) 
	wrong track because they haven't defined their requirements (or shared them across sub-teams) up 
	front. I have been a pained neighbor-observer to one that when through three different start-end 
	year-long contracts like this, during each round of which the new contractor went off building 
	something with no potential to meet the need even if perfectly executed.    (3NEP)
	[10:22] LeoObrst: @Fabian, Amanda: Yes, one might have a perfectly consistent ontology with good 
	structural metrics that is just nonsense, because it doesn't conform to the world. When we made this 
	a scale, we really weren't necessarily thinking of a 2D scale. It really was more of a rhetorical 
	device to talk about the space.    (3NEQ)
	[10:26] ToddSchneider: Amanda, MichaelDenny, There is the design phase, prior to any development, 
	where the issues you each raised would be addressed.    (3NER)
	[10:28] AmandaVizedom: @Todd: Yes, and it's important to make explicit that a good methodology 
	includes such a phase (design and/or requirements identification). Too many people never even think 
	of doing it for ontologies.    (3NES)
	[10:23] DougFoxvog: @Simon @Maria: cycles in a controlled vocabulary comprise what WordNet calls a 
	synset. In ontological terms, they are equivalence sets. One can have narrow contexts in which the 
	only instances of the more general concept are also instances of the narrower one. This can be 
	expressed in a domain ontology by creating a subclass cycle -- which defines all classes in the 
	cycle to be equivalent.    (3NET)
	[10:24] MariaPoveda: http://oa.upm.es/6456/1/Evaluation_of_Taxonomic_K.pdf here there are some 
	example of what have been identify as errors in taxonomies    (3NEU)
	[10:24] GaryBergCross: Q. Do these evaluation approaches and concepts apply equally to Ontology 
	Design Patterns or are there additional factors to consider for ODPs?    (3NEV)
	[10:25] AmandaVizedom: @Gary: Yes. ;-)    (3NEW)
	[10:25] MariaPoveda: yes, as in load of situations, if the goal of the developer is to do that it is 
	correct, the problem is when people (most of the time not ontologists) end up with that models by mistake    (3NEX)
	[10:26] MariaPoveda: so IMHO I would not look for things that are always and error because it is 
	going to be almost impossible because some one might want to do that    (3NEY)
	[10:26] MariaPoveda: but find situation that might be an error, identify them and decide whether 
	they should be corrected or not    (3NEZ)
	[10:27] PeterYim: == MichaelDenny / PeterYim presenting ... see: the [4-Track-D] slides    (3NF0)
	[10:36] AmandaVizedom: @MichaelDenny: I agree and think your point is important; it seems as though 
	many (most?) evaluation factors will be relevant at many (most?) points in the lifecycle, but 
	perhaps call for different treatment at different points.    (3NF1)
	[10:40] AmandaVizedom: @MichaelDenny: (ref. MichaelDenny's remarks during the presentation, that he 
	may not consider "visualization" a factor in the software that addresses ontology quality) Visual 
	building, or visualization, might be important for developer understanding OR for SME validation. 
	The survey could ask about this kind of capability at different levels of specificity, though (i.e., 
	render ontology in format intelligible to non-ontologists and facilitate input or review from same 
	vs. "ontology visualization" and/or "visual ontology construction."    (3NF2)
	[10:40] TerryLongstreth: @Amanda +1    (3NF3)
	[10:42] MikeBennett: @Amanda +2. Presentation to SMEs may take one of several forms, since non 
	technical people, being people, have one of several modalities they may be comfortable with: visual 
	Boxes and lines per Vision/whiteboard; tables or spreadsheets of terms and definitions; and 
	controlled natural language.    (3NF4)
	[10:43] TerryLongstreth: about extrinsic vs intrinsic (or the hybrid) is inspecting and judging a 
	visual representation of the CLIF based ontology an extrinsic or intrinsic test?    (3NF5)
	[10:56] AmandaVizedom: @Terry: it depends what you are inspecting and judging it for.    (3NF6)
	[10:43] PeterYim: == Open Discussion about the tracks' syntheses ...    (3NF7)
	[10:52] SteveRay: @Fabian: It's hard to imagine good performance for an ontology-based system if the 
	ontology doesn't align with reality.    (3NF8)
	[10:52] DougFoxvog: @Leo: the need is not always conformance to the "real world", but conformance to 
	the world of the domain. One could create an ontology for a role-playing game that uses ontology for 
	real-world physics, but adds fictional biological creatures, and fantastical powers.    (3NF9)
	[10:53] DougFoxvog: @Steve: see my above statement.    (3NFA)
	[10:53] SteveRay: @Doug: Yes, let me amend my use of the word "reality" to mean "the domain being modeled"    (3NFB)
	[10:56] LeoObrst: @Doug: yes, this gets into the "reality" of the domain, and so might be better 
	considered as verisimilitude to the domain, if the domain is e.g., fictional.    (3NFC)
	[10:55] AmandaVizedom: @Steve: Some "ontologies" do fairly well in their initial the context for 
	which they are originally created, because of implicit assumption shared in that context, but are 
	impossible to reuse in part because of basic problems, including conformance to reality *when just 
	the explicitly captured ontology is considered*.    (3NFD)
	[10:57] SteveRay: @Amanda: What you describe sounds to me like the ontology is simply incomplete, if 
	there are assumptions that are not reflected in the model. That, to me, is different from not being 
	aligned with the domain being modeled.    (3NFE)
	[10:57] MikeBennett: All these considerations about the reality of the domain, have two possible 
	implications: quality criteria for the thing you are developing for; and considerations when others 
	are developing something and want to consider whether or not to reuse that ontology.    (3NFF)
	[10:58] AliHashemi: @Steve - this comes back to the point of granularity and precision, right?    (3NFG)
	[10:58] AmandaVizedom: @Terry: *assuming*, that is, that the visualization technique preserves the 
	structure and content of the ontology.    (3NFH)
	[10:58] AliHashemi: a lightweight ontology will almost certainly leave many things left unsaid    (3NFI)
	[10:59] SteveRay: @Ali: Yes, I agree. All of this falls within the "correctness" of an ontology, 
	IMHO.    (3NFJ)
	[10:59] AliHashemi: From the verbal discussion that ensued, it's unclear to me how the "conformance 
	to the reality of the target domain" is reflected in our tracks.    (3NFK)
	[10:59] DougFoxvog: @Ali: *Every* ontology will certainly leave many things unsaid.    (3NFL)
	[10:59] AliHashemi: @Doug, agreed. So I'm not sure it's useful to simply state that an ontology is 
	complete if they didn't "completely" model their domain    (3NFM)
	[10:59] SteveRay: @Ali: See my comment at [10:52]    (3NFN)
	[11:00] AliHashemi: incomplete*    (3NFO)
	[11:00] MikeBennett: Apologies, I have to drop off now.    (3NFP)
	[11:01] TerryLongstreth: An important reason to continue to talk about implicit vs. explicit is to 
	minimize the opportunity for misinterpreting results. If the ontology picture (from my [10:14]) is 
	judged by the SME to be logically correct and aesthetically pleasing, it still needs to be validated 
	against the CLIF representation for logical equivalence, and the CLIF version has to be 
	intrinsically evaluated, to put bounds on the range of valid inferences that can be drawn from it.    (3NFQ)
	[11:02] AmandaVizedom: @Steve: original use might not have included machine reasoning; artifact may 
	therefor have inconsistencies in the explicit content that aren't caught (the original users may 
	interpret not according to the formal semantics, but rather according to some conventional treatment 
	local to them. They often don't realize this and put their artifact out there as a reusable ontology. 
	As soon as the formal semantics are applied, e.g., by machine validation, inconsistencies are apparent.    (3NFR)
	[11:04] SteveRay: @Amanda: In that case, the incompleteness is in the evaluation.    (3NFS)
	[11:05] AmandaVizedom: @Steve, in the initial evaluation, yes. When a new party considers this 
	artifact for reuse, *their* evaluation should catch it.    (3NFT)
	[11:02] DougFoxvog: Note that in a *knowledge base*, that uses an ontology, data need not be limited 
	to identification and properties of individuals. Additional properties of classes and rules for the 
	context can also be defined.    (3NFU)
	[11:04] TerryLongstreth: @SteveRay : you might want to say correctly constructed, or perhaps well-formed.    (3NFV)
	[11:05] AliHashemi: @Terry, I like the notion of a well-formed ontology    (3NFW)
	[11:11] TerryLongstreth: Why concern for well-formedness? It's necessary to insure that the semantic 
	properties/extrinsics are correctly represented    (3NFX)
	[11:06] DougFoxvog: @Amanda: It's true that inconsistencies can be made apparent when formal 
	semantics are applied. But these can often be "automatically" determined when missing statements 
	(e.g., disjointness) are specified. The Cell Line Ontology until Dec. 2011 had a number of common 
	subclasses of PlantCell and AnimalCell (and likewise of other disjoint class pairs). Upon specifying 
	disjointness in a reasoning system, these all popped out.    (3NFY)
	[11:08] DougFoxvog: A system can count the use of disjointness and similar assertions as indicators 
	and suggest that low use indicates incompleteness of an ontology.    (3NFZ)
	[10:55] PeterYim: == FabianNeuhaus / AmandaVizedom presenting ... see: the [5-Communique] slides    (3NG0)
	[11:03] SteveRay: @Fabian: (Regarding the communique). On slide 6, you don't seem to include "Is the 
	ontology well constructed?" which is what all the intrinsic evaluation is about.    (3NG1)
	[11:04] AliHashemi: @Fabian, will there also be a finer-grained discussion of how the various 
	metrics and evaluation approaches map to the various types of ontologies?    (3NG2)
	[11:07] PeterYim: == Open Discussion about the Communique Approach ...    (3NG3)
	[11:08] MatthewWest: One of the things that I think that would be very useful in structuring the 
	Communique is the use of a net of problems/net of solutions approach. It can help to gather together 
	and sift through the mass of detail we have.    (3NG4)
	[11:09] AmandaVizedom: @Matthew: thanks for that suggestion; we will look into that.    (3NG5)
	[11:15] MatthewWest: You can find a very simple net of problems here on Page 12 
	http://www.matthew-west.org.uk/documents/princ03.pdf A simple net of solutions is on P40.    (3NG6)
	[11:09] ToddSchneider: Amanda, Fabian, Were does the notion of lifecycle come in?    (3NG7)
	[11:19] ToddSchneider: To provide value this summit should provide some guidance as to when a 
	particular evaluation/validation, or collection there of, should be applied w.r.t. the system 
	lifecycle.    (3NG8)
	[11:10] AliHashemi: @All - will we be referring back to the Ontology Usage that was developed in the 
	previous Summits? This would help select subsets of the very broad ranges of tools, approaches and 
	metrics developed in this summit.    (3NG9)
	[11:11] AliHashemi: cf - 
	http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2011_ApplicationFramework_Synthesis    (3NGA)
	[11:12] FabianNeuhaus: It seems the call was dropped    (3NGB)
	[11:13] DougFoxvog: The conference hung up in the middle of Amanda's talking    (3NGC)
	[11:13] SteveRay: Did everyone just lose a connection? I suppose so.    (3NGD)
	[11:13] SimonSpero: Please hold. operators are standing by.    (3NGE)
	[11:13] LeoObrst: Yikes! Peter must be dropped.    (3NGF)
	[11:13] AmandaVizedom: Hopefully Peter will return soon...    (3NGG)
	[11:13] SteveRay: Calming music.    (3NGH)
	[11:13] DougFoxvog: We're back    (3NGI)
	[11:13] SimonSpero: Everyone stop talking about Peter    (3NGJ)
	[11:14] anonymous morphed into LamarHenderson    (3NGK)
	[11:14] AmandaVizedom: My point is: I think the lifecycle will appear most in the context of best practices.    (3NGL)
	[11:16] DougFoxvog: Should lifecycle be orthogonal to ontology validation/verification feature? If 
	not, could we specify which components are needed at which points in the lifecycle?    (3NGM)
	[11:20] AmandaVizedom: @DougFoxvog, we might, but we would not want to try to enumerate this in the 
	Communique. This is similar to the question of what evaluation factors are relevant to what sorts of uses.    (3NGN)
	[11:20] MichaelDenny: @DougFoxvog This is what the survey attempts to do by sorting some 100 
	software capabilities adressing ontology quality/fitness into seven phases of the lifecycle.    (3NGO)
	[11:22] AmandaVizedom: Both are areas of high value. I hope we make progress on them during the 
	Summit. I don't think we can or should try to enumerate all specifics of such factors in the 
	Communique. The findings and conclusions, however, should be reflected in the best practices and 
	future content.    (3NGP)
	[11:22] AliHashemi: @Amanda, might they be enumerated on the synthesis pages?    (3NGQ)
	[11:23] AmandaVizedom: @Ali, yes, I hope so. They might also turn into additional summit outputs, in 
	one form or another!    (3NGR)
	[11:14] PeterYim: == PeterYim / MikeDean presenting ... see: the [6-Hackathon-Clinics ] slides    (3NGS)
	[11:19] AnatolyLevenchuk: We issued today version 1.2 of .15926 Editor (this is like Protege for ISO 
	15926 ontology, while RDF/OWL is also supported): http://techinvestlab.ru/dot15926Editor -- it will 
	be used during hackathon/clinics for multiple ontology evaluation activities within our 
	Russian-speaking community (http://dot15926.livejournal.com/40280.html -- hackathon/clinics 
	announced in Russian).    (3NGT)
	[11:21] GaryBergCross: Evaluating an ontology developed by building out from an ODP might be an 
	interesting exercise.    (3NGU)
	[11:23] DougFoxvog: I note that sending "gold standard" ontologies through evaluation tools may very 
	well determine problems with the ontologies labeled "gold standard". Such use will not only be 
	useful for evaluating the tools, but also the "gold standards".    (3NGV)
	[11:25] TerryLongstreth: @Peter: I'd be careful with the term SME. It conjures images of greybeards, 
	but in an IT environment, the people most interested in the correctness of function are usually 
	called operators or users.    (3NGW)
	[11:25] DavidLeal: The CEN SERES workshop will produce an ontology for materials data before May. We 
	have participants also involved with the US Material Genome Initiative, so this ontology may have a 
	broad role. The ontology may be presented as an extension of ISO 15926. It would be very good if the 
	ontology validation tools could be used on this ontology.    (3NGX)
	[11:28] DavidLeal: Materials ontologies are a bit of a challenge for industry understandability, 
	because the word "material" is used to mean both batch of stuff and a type of stuff - sometimes in 
	the same sentence.    (3NGY)
	[11:29] GaryBergCross: EarthCube is interested in various kinds of "material entities."    (3NGZ)
	[11:31] DougFoxvog: @DavidLeal: Materials ontologies need to realize that some properties of 
	materials are properties of the "stuff" independent of the state of affairs (temperature, pressure, 
	...), others are "intrinsic" as they do not depend upon the amount of material, but may depend on 
	temp/pressure/other environment, while others are extrinsic, depending on the physical object    (3NH0)
	[11:31] DavidLeal: and the depend upon the history of the environment - materials have memory!    (3NH1)
	[11:32] DougFoxvog: @David: the material memory is in its microscopic structure. Since that is hard 
	to specify, specifying the history of events helps.    (3NH2)
	[11:26] GaryBergCross: It might be interesting to take a fairly loose, prototype or light ontology 
	and see if the evaluation gives us a way of understanding how to forge it into something good. Sort 
	of agile engineering approach.    (3NH3)
	[11:27] GaryBergCross: @ Amanda..perhaps we can work up the Hydro model to a form that would be 
	submitted for a test.    (3NH4)
	[11:29] AmandaVizedom: @Gary: Seems like a good candidate.    (3NH5)
	[11:30] AmandaVizedom: @Gary ... we'd need to get unstuck on the use case specification.    (3NH6)
	[11:30] GaryBergCross: @Amanda Let us see if we can get USGS interested in moving forward from their data.    (3NH7)
	[11:28] LeoObrst: Wow, we certainly have a lot here!    (3NH8)
	[11:30] PeterYim: == Q&A and Open Discussion ...    (3NH9)
	[11:31] MariaPoveda: bye :-)    (3NHA)
	[11:33] PeterYim: join us again, same time next week, for OntologySummit2013 session-07: "Extrinsic 
	Aspects of Ontology Evaluation - II" - Co-chairs: ToddSchneider & TerryLongstreth - 
	http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_02_28    (3NHB)
	[11:33] JoanneLuciano: thanks!    (3NHC)
	[11:33] PeterYim: -- session ended: 11:31 am PST --    (3NHD)
 -- end of in-session chat-transcript --    (3MVY)

Additional Resources:    (3MW5)

For the record ...    (3MWC)

How To Join (while the session is in progress)    (3MWD)

Conference Call Details    (3MTM)

Attendees    (3MUJ)