OntologySummit2013: Panel Session-08 - Thu 2013-03-07    (3LH9)

Summit Theme: "Ontology Evaluation Across the Ontology Lifecycle"    (3LHA)

Summit Track Title: Track-A: Intrinsic Aspects of Ontology Evaluation    (3LHB)

Session Topic: Intrinsic Aspects of Ontology Evaluation - II    (3NLP)

Session Co-chairs: Dr. LeoObrst (Ontolog; MITRE) and Dr. StevenRay (CMU) - intro slides    (3NLQ)

Panelists / Briefings:    (3NLR)

Archives:    (3NLW)

Abstract:    (3NNP)

OntologySummit2013 Session-08: "Intrinsic Aspects of Ontology Evaluation-II" - intro slides    (3NNQ)

This is our 8th Ontology Summit, a joint initiative by NIST, Ontolog, NCOR, NCBO, IAOA & NCO_NITRD with the support of our co-sponsors. The theme adopted for this Ontology Summit is: "Ontology Evaluation Across the Ontology Lifecycle."    (3NNR)

Currently, there is no agreed methodology for development of ontologies, and there are no universally agreed metrics for ontology evaluation. At the same time, everybody agrees that there are a lot of badly engineered ontologies out there, thus people use -- at least implicitly -- some criteria for the evaluation of ontologies.    (3NNS)

During this OntologySummit, we seek to identify best practices for ontology development and evaluation. We will consider the entire lifecycle of an ontology -- from requirements gathering and analysis, through to design and implementation. In this endeavor, the Summit will seek collaboration with the software engineering and knowledge acquisition communities. Research in these fields has led to several mature models for the software lifecycle and the design of knowledge-based systems, and we expect that fruitful interaction among all participants will lead to a consensus for a methodology within ontological engineering. Following earlier Ontology Summit practice, the synthesized results of this season's discourse will be published as a Communique.    (3NNT)

At the Launch Event on 17 Jan 2013, the organizing team provided an overview of the program, and how we will be framing the discourse around the theme of of this OntologySummit. Today's session is one of the events planned.    (3NNU)

In this 8th virtual panel session of the Summit, we continue to explore intrinsic ontology evaluation, from basic structural evaluation through exploration of the "is-a" relation with PatrickLambrix, software engineering concepts of regression testing with MariaCopeland, a biologist's perspective with MelissaHaendel, and finishing with some suggestions of the use of common elements with EdBarkmeyer. We hope that all of the participants in the open discussion and chat will join us in helping to flesh out the ingredients and methods for intrinsic evaluation.    (3NXP)

More details about this OntologySummit is available at: OntologySummit2013 (homepage for this summit)    (3NNW)

Briefings:    (3NNX)

Agenda:    (3NO6)

OntologySummit2013 - Panel Session-08    (3NO7)

Proceedings:    (3NOD)

Please refer to the above    (3NOE)

IM Chat Transcript captured during the session:    (3NOF)

 see raw transcript here.    (3NOG)
 (for better clarity, the version below is a re-organized and lightly edited chat-transcript.)
 Participants are welcome to make light edits to their own contributions as they see fit.    (3NOH)
 -- begin in-session chat-transcript --    (3NOI)
	------
	Chat transcript from room: summit_20130307
	2013-03-07 GMT-08:00 [PST]
	------    (3OCR)
	[09:26] PeterYim: Welcome to the    (3OCS)
	 = OntologySummit2013: Virtual Panel Session-08 - Thu 2013-03-07 =    (3OCT)
	Summit Theme: Ontology Evaluation Across the Ontology Lifecycle    (3OCU)
	* Summit Track Title: Track-A: Intrinsic Aspects of Ontology Evaluation    (3OCV)
	Session Topic: Intrinsic Aspects of Ontology Evaluation - II    (3OCW)
	* Session Co-chairs: Dr. LeoObrst & Dr. StevenRay    (3OCX)
	Panelists / Briefings:    (3OCY)
	* Professor PatrickLambrix (Linkping University, Sweden) - "Debugging is-a structure in ontologies"    (3OCZ)
	* Ms. MariaCopeland (University of Manchester) - "Ontology Evolution and Regression Testing"    (3OD0)
	* Dr. MelissaHaendel (Oregon Health & Science University) - "A biologists' perspective on ontology utility"    (3OD1)
	* Mr. EdBarkmeyer (NIST) - "Core components for an ontology: Modeling Codes and Code Lists"    (3OD2)
	Logistics:    (3OD3)
	* Refer to details on session page at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_03_07    (3OD4)
	* (if you haven't already done so) please click on "settings" (top center) and morph from "anonymous" to your RealName (in WikiWord format)    (3OD5)
	* Mute control: *7 to un-mute ... *6 to mute    (3OD6)
	* Can't find Skype Dial pad?
	** for Windows Skype users: it's under the "Call" dropdown menu as "Show Dial pad"
	** for Linux Skype users: please note that the dial-pad is only available on v4.1 (or later or the 
	earlier Skype versions 2.x,) if the dialpad button is not shown in the call window you need to press 
	the "d" hotkey to enable it.    (3OD7)
	Attendees: AliHashemi, AmandaVizedom, AnatolyLevenchuk, AstridDuqueRamos, BijanParsia, BobSmith, 
	BobbinTeegarden, DavidWhitten, DennisWisnosky, DmitryBorisoglebsky, DougFoxvog, EdBarkmeyer, 
	FabianNeuhaus, FranLightsom, HansPolzer, JoanneLuciano, JoelBender, JohnBilmanis, KenBaclawski, 
	KevinSimkins, LeoObrst, LudgerJansen, MariaCopeland, MeganKatsumi, MelissaHaendel, MichaelGruninger, 
	MikeDean, PatrickLambrix, PavithraKenjige, PeterYim, RamSriram, SteveRay, TerryLongstreth, 
	ToddSchneider, TorstenHahmann, ValentinaIvanova, VictorAgroskin, vnc2    (3OD8)
	 == proceedings: ==    (3OD9)
	[9:09] anonymous morphed into PatrickLambrix    (3ODA)
	[9:23] anonymous morphed into KevinSimkins    (3ODB)
	[9:29] PeterYim: Attn ALL: ... it has come to our attention that our conference bridge provider is 
	running into some problems with the "joinconference" skype connections. In case anyone gets in 
	trouble, please try to call the phone numbers instead (e.g. from your phone, skype-out, 
	google-voice, etc.)    (3ODC)
	[9:30] anonymous morphed into MelissaHaendel    (3ODD)
	[9:30] KevinSimkins: The IEEE Virtual World Standard Working Group (P1828) is focused on the 
	development of common standards for virtual environments. From assets to protocols and general 
	working models in order to facilitate the common interoperability scenario for all virtual 
	environments in the future. Our working group members are dedicated to multiple phases within this 
	process in order to accumulate further details and application methodologies over time. ... see: 
	http://www.metaversestandards.org/index.php?title=Main_Page    (3ODE)
	[9:32] JoelBender: @Peter - attempting to enter the conference ID is failing, the bridge is missing 
	digits - I'm calling from a land line    (3ODF)
	[9:38] AnatolyLevenchuk: wrt skype: always mute microphone when you enter ID to skype (mic take 
	sound from your tone-digits and then can be "strange" processes of echo and digits suppressing or 
	doubling of click and "listened" digits).    (3ODG)
	[9:34] anonymous morphed into BobSmith    (3ODH)
	[9:35] MelissaHaendel: yes, screen share doesn't work for me    (3ODI)
	[9:36] FabianNeuhaus: Melissa, the screen share might not work if you are behind a firewall    (3ODJ)
	[9:37] FabianNeuhaus: just download the slides from the session page 
	http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_03_07    (3ODK)
	[9:39] MelissaHaendel: thanks Fabian, I have the slides    (3ODL)
	[9:35] anonymous morphed into TorstenHahmann    (3ODM)
	[9:38] PeterYim: == SteveRay opening the session on behalf of the co-chairs ... ... see: the 
	[0-Chair] slides    (3ODN)
	[9:41] List of members: AmandaVizedom, AnatolyLevenchuk, AstridDuqueRamos, DennisWisnosky, 
	EdBarkmeyer, FabianNeuhaus, FranLightsom, HansPolzer, JoanneLuciano, JoelBender, JohnBilmanis, 
	KevinSimkins, LeoObrst, MariaCopeland, MelissaHaendel, MichaelGruninger, MikeDean, PatrickLambrix, 
	PeterYim, RamSriram, SteveRay, TerryLongstreth, ToddSchneider, TorstenHahmann    (3ODO)
	[9:41] PeterYim: == PatrickLambrix presenting ... see: the [ 1-Lambrix ] slides    (3ODP)
	[9:44] anonymous morphed into DougFoxvog    (3ODQ)
	[9:51] MelissaHaendel: Question, so NCI is largely a human representation whilst MA is mouse. How do 
	you know there are not species-specific is_a absences or presence? Examples are correct, though :-).    (3ODR)
	[11:24] PatrickLambrix: @Melissa [9:51]: For our experiments we had the help of a domain expert. 
	(This doesn't mean all the validations are correct. We have also noticed that a domain expert may 
	change his/her mind about the correctness of an is-a relation during the debugging session when more 
	is-a relations were debugged.)    (3ODS)
	[11:28] MelissaHaendel: @patrick. I understand biologists changing their mind! Also depends on the 
	way in which they are viewing the ontology and the debugging changes. Would be interested to view 
	this sometime, will look at your papers.    (3ODT)
	[9:54] SteveRay: @Melissa: My understanding from his talk is that is why the repairs must be 
	validated by a domain expert.    (3ODU)
	[11:24] PatrickLambrix: @Steve [9:54]: Yes, repairs need to be validated by a domain expert. For 
	missing is-a relations a system can compute repairs that guarantee that the missing is-a relations 
	will be logically derivable form the repaired ontology, but this does not necessarily mean that the 
	repair is correct according to the domain.    (3ODV)
	[9:54] AmandaVizedom: @PatrickLambrix - I like your focus on a high-level division into syntactic, 
	semantic, defects; your calling out that detection and debugging of the modeling defects requires 
	domain knowledge, and your subsequent emphasis on very practical ways to detect and debug this third 
	category. In my experience, the syntactic and semantic evaluation and correction, while necessary, 
	are rarely sufficient. There seems, however, to be a widespread impression, or fear, that practical, 
	reliable evaluation of model accuracy is not possible. Thanks for showing this not to be true.    (3ODW)
	[11:24] PatrickLambrix: @Amanda [9:54]: If I remember well, the different kinds of defects were 
	defined in Aditya Kalyanpur's PhD thesis. However, there is not so much work on systems for the 
	modeling defects yet.    (3ODX)
	[9:58] DougFoxvog: For the example, concluding "limb_joint" is-a "joint" because all of its 
	subclasses are-a joint although it is not in any of the ontologies, the conclusion must be verified 
	by a human. It is quite possible that that common is-a is too general a term -- which may be the 
	reason the is-a was not asserted.    (3ODY)
	[9:58] PeterYim: @PatrickLambrix - are there tools already implemented for the process outlined in 
	slide#26 ... if so, are they openly accessible (what is url?)    (3ODZ)
	[9:58] MelissaHaendel: It would be great to have tools that present these things easily to the 
	domain expert. Actually, I would not classify a hip joint as a limb joint (sorry, I work on this 
	stuff ;-)).    (3OE0)
	[11:24] PatrickLambrix: @Peter,@Melissa: [9:58]: We have systems and plan to make the first system 
	available soon. There are screenshots in the papers in the reference list of the talk.    (3OE1)
	[10:05] DougFoxvog: @Melissa: That depends upon the ontology's definition of "limb joint". The term 
	could mean a joint within a limb or a joint which is to at least one bone in a limb. Bone joints vs. 
	body region joints are also a contrast; these two types are disjoint.    (3OE2)
	[10:01] AmandaVizedom: @PatrickLambrix - I would note that "is-a" in your examples appears to be a 
	subClass or kind-of relationship (or, if also used for instances, it may be an under-specified 
	narrower concept used for both instancehood and subclass relationships, as found in some taxonomies, 
	less commonly ontologies, and other semantic models). I call attention to this simply because it can 
	be confusing to modelers who follow most contemporary ontology languages, in which "is-a" is used 
	only for the instancehood relationship, and a different relationship (subClassOf, kind-of, #$genls) 
	is used for the sub-class relationships in your examples.    (3OE3)
	[10:06] MelissaHaendel: well, many anatomists define limb as the free limb, as opposed to the limb 
	plus girdle. but that is perhaps neither here nor there.    (3OE4)
	[10:06] MelissaHaendel: @Patrick - I would love to have you experiment with our work on Uberon - see 
	http://uberon.org.    (3OE5)
	[11:25] PatrickLambrix: @Melissa [10:06]: I would be interested at looking at Uberon.    (3OE6)
	[10:09] DougFoxvog: @Amanda: many people use "is-a" to mean subclass of. Cyc uses #$isa to mean 
	instance of. I (growing up with Cyc) also find "is-a" used for subclass-of as grating. 
	Grammatically, it should be "a-is-a".    (3OE7)
	[11:24] PatrickLambrix: @Amanda,@DougFoxvog [10:01]: Yes, is-a isSubClass or kind-of. Our system 
	works currently only on the concept level. For is-a we assume relexivity and transitivity.    (3OE8)
	[10:10] AmandaVizedom: @doug, far fewer than used to. The two relationships are teased out, for 
	example, even in OWL.    (3OE9)
	[10:05] JoanneLuciano: is is-a a kind-of-kind-of is-a relationship? or is it a-kind-of-like 
	relation?    (3OEA)
	[11:24] PatrickLambrix: @Joanne [10:05]: exactly :-)    (3OEB)
	[10:05] JoanneLuciano: :-) nice presentation!    (3OEC)
	[10:10] LeoObrst: @Patrick: do repair actions ever cause additional problems for other parts of the 
	ontologies not yet analyzed?    (3OED)
	[11:25] PatrickLambrix: @Leo [10:10]: Repairing actions could cause additional problems. Therefore, 
	the system allows a user to switch between the different phases of detecting, validating and 
	repairing, as well as taxonomies and iteratively debug the whole network. The system also checks 
	whether a domain expert would want to remove a previously validated to be correct is-a relation. In 
	this case the user needs to retract the previous choice or the repair is not allowed. Similar when 
	the domain expert wants to add an is-a relation, that was previously validated as wrong.    (3OEE)
	[10:06] PeterYim: == MariaCopeland presenting ... see: the [ 2-Copeland ] slides    (3OEF)
	[10:14] EdBarkmeyer: @maria: Only some tested elements have meaning to the user. Many test elements 
	are intended to validate that the inferences work as expected for known cases. In many cases, the 
	user cannot recognize an erroneous inference, as Patrick mentioned.    (3OEG)
	[10:21] EdBarkmeyer: @maria: Is it clear that these changes are "truth" and "bugs", or just 
	differences in educated opinion prevailing at different times in the maintenance cycle?    (3OEH)
	[10:21] AliHashemi: @Maria - are the graphs on slides 30 and 34(b) correct? It seems to me it should 
	be Effectually added (solid line) Ineffectually Removed (dotted line) Effectually Removed (no line) 
	Effectually Added ... otherwise, what do the lines represent?    (3OEI)
	[10:23] DougFoxvog: @Maria: could these changes be a result of different users having different 
	ideas of what the terms (should) mean?    (3OEJ)
	[10:23] MelissaHaendel: We (try to) keep track of such edits in the ontology metadata, it would be 
	fantastic to have a human readable version provided to the ontology editor/domain editor whilst they 
	are editing.    (3OEK)
	[10:23] anonymous morphed into DavidWhitten    (3OEL)
	[10:26] AmandaVizedom: @MariaCopeland - could you explain your effectually / ineffectually 
	distinction a little more? I think I understood you to say that "effectual" presence means presence 
	as a directly asserted axiom, while "ineffectual" presence means absence from the directly asserted 
	axioms, but continued entailment by the ontology. Is that correct? If so, is this, as the label 
	ineffectual suggests, considered a fault? I ask because I imagine that presence as entailment only 
	would be desirable in some applications and undesirable in others. Specifically, if little to know 
	reasoning is done in an application and only directly asserted applications are usable, then 
	entailments are indeed "ineffectual". In other cases, where some degree of reasoning is used and is 
	efficient, assertions that are redundant with entailments may be removed for efficiency, if there is 
	not additional, provenance-related reason to make explicit assertions that rely on different 
	sources.    (3OEM)
	[11:24] MariaCopeland: @EdBarkmeyer, @AliHashemi, @DougFoxvog, @Melissa: Thanks for your questions 
	and comments. I will follow up offline.    (3OEN)
	[10:23] PeterYim: == MelissaHaendel presenting ... see: the [ 3-Haendel ] slides    (3OEO)
	[10:30] HansPolzer: (ref. slide#4) The example of fruit fly limbs vice human limbs underscores the 
	need for more explicit representation of context as it relates to ontologies    (3OEP)
	[10:32] DougFoxvog: @Hans: the fruit fly vs. human Tibia presents an example of using NL words as 
	terms for concepts in an ontology. There should be mappings from the NL terms to (multiple) 
	concepts. But annotation of the concept should make its intended meaning crystal clear.    (3OEQ)
	[10:34] AmandaVizedom: @Hans - I don't disagree, exactly, but I'd add that it is also a good 
	illustration of why it is important not to confuse expressions and concepts. Arguably, the fruit fly 
	tibia is a subclass of some class of animal body parts that are not skelatal; the human tibia is a 
	subclass of some class of animal body parts that are skelatal. "limb" may be an expression used for 
	both of those classes, but the classes are different.    (3OER)
	[10:35] DougFoxvog: @Melissa : Having a key separation in an ontology whether something is 
	detectable seems to be setting oneself up to have the feature expire when new technology allows 
	something not previously detectable to become detectable.    (3OES)
	[10:36] DougFoxvog: I had trouble hearing for a while. But the sound has now come back    (3OET)
	[10:36] HansPolzer: But in some contexts the term "limb" represents the identical concept, for 
	example, when considering locomotion rather than anatomy (and not focusing on how exactly the 
	movement of the limb is activated/energized.)    (3OEU)
	[11:02] MelissaHaendel: @hans. we have disjoint axioms to prevent fruitfly tibia being equated with 
	human tibia. I wonder how much such disjoint axioms are leveraged in the ontology alignment 
	strategies? or if we can be providing them better for such purposes.    (3OEV)
	[11:04] DougFoxvog: +1 Melissa. Disjointness axioms are crucial. Especially if they are enforced on 
	modification of an ontology to prevent the creation of classes that are disjoint with themselves.    (3OEW)
	[10:37] AmandaVizedom: @Melissa -- Don't worry, it's not just biologists! I've lost track of the 
	number of domains I've worked in or with, and I don't think I've yet met a community of practice 
	that doesn't have these features (moving conceptual targets, reuse of labels, and expert 
	disagreements on definitions). I suspect there are fundamentals of human cognition, language, and 
	community knowledge development at work . :-)    (3OEX)
	[10:38] DougFoxvog: One need to define multiple limbs: tree limb, animal limb, subclasses of animal 
	limb by exo/endo-skeleton, etc.    (3OEY)
	[10:38] AmandaVizedom: @Hans - agreed. And that is, arguably a different concept, for which the same 
	expression is used.    (3OEZ)
	[10:39] DougFoxvog: @Amanda: Of course. Replace disagreement on definitions by modeling all 
	definitions with different (maybe overlapping) concepts.    (3OF0)
	[10:39] SteveRay: This raises the point that different communities still want to use their own 
	terminologies, and they should, but we can separate those terminologies (expressions or labels) from 
	the concepts.    (3OF1)
	[10:41] EdBarkmeyer: @melissa: As you say, people think ontology terms are words, and they think 
	definitions are circular when those words are reused, but the fact is that the definitions are in 
	natural language, the ontology symbols are "words" in a formal language. The issue is whether the 
	definitions themselves are circular.    (3OF2)
	[10:42] DougFoxvog: OBO still has problems with different people having different ideas of what a 
	term is, even when the name is numeric. The NL description is not specific enough, so for example, 
	the plant people assume it means the plant definition of the term and the animal people assume it is 
	an animal definition. As a result, in the Cell (Line) Ontology, multiple cell types were subclasses 
	of both animal cell and plant cell.    (3OF3)
	[10:41] TerryLongstreth: @Melissa - Last bullet on slide 11 implies that speculative biology is not 
	in scope.    (3OF4)
	[11:04] MelissaHaendel: @terry I think what I was saying is that actually speculative biology is in 
	scope, but we just need to understand how to apply the ontology consistently when we are potentially 
	being speculative. (or maybe I misunderstand)    (3OF5)
	[10:41] HansPolzer: We have an aversion to specifying context because it seems unnecessary in most 
	situations we find ourselves in - yet the internet and networked world exposes us to otherwise alien 
	contexts that are left implicit, thus leading to the arguments about definitions    (3OF6)
	[10:42] HansPolzer: We want an absolute frame of reference, but usually assume our own frame of 
	reference is that absolute frame of reference.    (3OF7)
	[10:44] HansPolzer: We need to learn to tolerate that there are multiple frames of reference and 
	multiple perspectives on those frames of reference and associated scope.    (3OF8)
	[10:44] DougFoxvog: @Hans: You say "we" have an aversion to specifying context. *I* don't. 8)# I 
	find context crucial.    (3OF9)
	[10:45] HansPolzer: It was the editorial "we" :-)    (3OFA)
	[10:45] HansPolzer: Humans are very good at detecting context, except when it fails us - and thus 
	results in great literature    (3OFB)
	[10:46] HansPolzer: and puns    (3OFC)
	[10:43] anonymous morphed into ValentinaIvanova    (3OFD)
	[10:44] SteveRay: Gee, I'm thinking about a transiently transfected DNA expression construct and I'm 
	coming up blank :)    (3OFE)
	[10:45] AmandaVizedom: @Melissa - slide 16: very nice illustration of refinement to include more 
	specific classes, corresponding with important functional distinctions.    (3OFF)
	[10:46] EdBarkmeyer: @melissa: this is a great contribution! thanks.    (3OFG)
	[10:50] PeterYim: @MelissaHaendel: do you have tools to augment the process(es) that you are 
	describing    (3OFH)
	[11:07] MelissaHaendel: @Peter - we have some scripts that help. We have been using Jenkins to help 
	too.    (3OFI)
	[11:12] PeterYim: @MelissaHaendel - thank you, reason I am asking is because we are running 
	Ontology Clinic activities (as part of this Summit) by putting together ontology developers and 
	people with ontology evaluation methodologies and tool in the same (virtual) room to get some 
	interesting outcome, and we'll love to have you (and the other panelists today) to join us - see: 
	http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013_Hackathon_Clinics    (3OFJ)
	[11:17] MelissaHaendel: @peter, it would be great to see if you could get Chris Mungall to 
	participate regarding his development of Jenkins. see http://wiki.geneontology.org/index.php/Jenkins 
	. Most of our work on the metadata is in the form of scripts and not integrated into tooling (though 
	we have some protege plugins).    (3OFK)
	[11:19] PeterYim: @Melissa - thank you for the pointer ... this is helpful as we are doing a survey 
	on software and tools, as part of the Summit activities, too    (3OFL)
	[10:47] DougFoxvog: Re slide 15. An ontology should not have synonyms. It should have mappings from 
	NL terms to the ontology terms. One NL term can map to multiple ontology terms and vice versa.    (3OFM)
	[10:47] AmandaVizedom: @Melissa: slide 15: Do you mean that experts cannot articulate the 
	distinctions? Or that the modeling language is not expressive enough to capture it clearly?    (3OFN)
	[10:52] SteveRay: @Amanda: How would you handle the example of the Senator who said "I can't define 
	it but I'll know it when I see it". ...in the context of porn. Quite seriously, it seems that 
	perhaps we sometimes operate with definitions only by example and are unable to define concepts.    (3OFO)
	[10:58] AmandaVizedom: @Steve: I did mean to ask for clarification, not to imply that there are no 
	undefinable concepts. In fact, it is very common for us to use concepts operationally and regularly 
	without there being any set of necessary and sufficient conditions to accompany them. The less 
	technical the concept, I'd argue, the more this is likely to be true. IME, textual definitions that 
	mention and guide against potentially confused concepts can be very useful, though inherently 
	incomplete.    (3OFP)
	[11:00] SteveRay: @Amanda: Agreed.    (3OFQ)
	[10:54] AmandaVizedom: @Melissa - slide 18, "true path" violations -- common problem indeed; human 
	brains are not good at keeping that kind of entailment trace. ;-) One technique that can help catch 
	these is the creation of tests that instantiate classes and draw the transitive inferences, 
	presenting the results to the experts, who are more likely to catch it at *that* point if they've 
	created either inconsistencies or undesired implications.    (3OFR)
	[11:10] MelissaHaendel: @amanda: I would love your idea "One technique that can help catch these is 
	the creation of tests that instantiate classes and draw the transitive inferences, presenting the 
	results to the experts, who are more likely to catch it at *that* point if they've created either 
	inconsistencies or undesired implications." This would help biologists enormously.    (3OFS)
	[10:49] TerryLongstreth: @Melissa Slide 17 - Curation status include provenance trace?    (3OFT)
	[10:47] PeterYim: Ed & All: please provide some context wrt to your comment ... otherwise the 
	remarks won't mean much in the proceedings    (3OFU)
	[10:53] HansPolzer: Peter, Melissa: my comments were aimed at the issue of ambiguity and controversy 
	in concept definitions. From a pragmatic/realism approach it may be worthwhile to make more of the 
	context assumptions specific to a given definition, allowing more flexibility in applying a specific 
	ontology or term in an ontology in a given situation/perspective/context.    (3OFV)
	[10:55] PeterYim: @Hans - my comment is strictly aimed at messages people are typing into this 
	chat-room (like, "this is great!" would mean much less than "ref. your slide#12, this is great 
	insight!") since we captured the chat-transcript as part of the proceedings of the session.    (3OFW)
	[10:57] MelissaHaendel: (just lost of her voice connection as she got to the last slide) I'm all 
	done :-) ... Thank you.    (3OFX)
	[10:58] FabianNeuhaus: @ Melisssa, the standards for documentation: is this an OBO Foundry effort?    (3OFY)
	[11:06] MelissaHaendel: @fabian this is a Melissa's team effort ;-). I would like it to be an OBO 
	effort, and I think we are getting traction.    (3OFZ)
	[10:58] PeterYim: == EdBarkmeyer presenting ... see: the [ 4-Barkmeyer ] slides    (3OG0)
	[11:00] anonymous morphed into LudgerJansen    (3OG1)
	[11:02] JoelBender: Units! °F!    (3OG2)
	[11:04] LeoObrst: @Ed: would you say that code-lists are controlled vocabularies, and that these map 
	to concepts/classes in a given ontology? Another example are digraphs and trigraphs for 
	countries/states.    (3OG3)
	[11:04] SteveRay: @Ed: I'm interpreting your Code List to be what many call an Enumeration. Yes?    (3OG4)
	[11:05] PeterYim: @EdBarkmeyer and All - some members of the Ontolog community had actually worked 
	on a project "CCT-Representation" back in 2004/2005 to map the ebXML Core Component Types ("CCT") to 
	a First Order Logic ontology (SUMO, MILO, QoS) - see: An Ontological Basis for ebXML Core Component 
	Types - http://ontolog.cim3.net/cgi-bin/wiki.pl?CctRepresentation    (3OG5)
	[11:06] DougFoxvog: Code lists are controlled vocabularies. However, as EdBarkmeyer is saying, the 
	meanings of the codes changes. So, dated uses of the codes is important.    (3OG6)
	[11:11] LeoObrst: @Doug: yes, we see this all the time. There are extreme versioning issues involved 
	in maintaining the mappings from these controlled vocabularies to the ontology/ies.    (3OG7)
	[11:06] anonymous morphed into PavithraKenjige    (3OG8)
	[11:07] DougFoxvog: I note that ISO country codes occasionally change.    (3OG9)
	[11:08] HansPolzer: and countries change and come into being and go away    (3OGA)
	[11:11] DougFoxvog: @Hans: I remember once finding an ISO code for an expired country being re-used 
	over a decade later for a new country. I forget what the example was.    (3OGB)
	[11:14] AmandaVizedom: @EdBarkmeyer - Thanks for this presentation. I've seen this topic persist as 
	a point of conflict on ontology projects -- whether to go the string route or the 
	expression-for-thing route. Least-immediate-effort often pulls people toward the string route, but 
	if there is integration to be done, this is usually a mistake (or at best, pushes the effort down 
	the road, when it will have to be redone).    (3OGC)
	[11:14] HansPolzer: Ref slide 6 in Ed's talk, it would seem to have some temporal context 
	specification - I suppose that's in the administrative records somewhere    (3OGD)
	[11:18] HansPolzer: Looking up the code list is becoming more practical as things become more 
	connected over the internet. But there are also practical constraints on doing these lookups in real 
	time while processing transactions    (3OGE)
	[11:18] DougFoxvog: @EdBarkmeyer: re slides 7 & 8. Having a term mapped to a code & code list is 
	intrinsically a ternary concept. An object that is an instance in a code list can have two mappings 
	from item, one to the code & the other to the code list. However, this only would work if the item 
	maps only to a single code & single code list.    (3OGF)
	[11:30] VictorAgroskin: To do such code models consistently is much easier if your ontology language 
	contains concept of class_of_class. Coding models of this type are very common for systems 
	implemented on ISO 15926. And ISO 15926-6 is partially a mapping of ISO 11179-3.    (3OGG)
	[11:18] == Q & A and Open Discussion ...    (3OGH)
	[11:18] AmandaVizedom: I need to drop off -- thanks to all presenters! Good stuff.    (3OGI)
	[11:19] DougFoxvog: Ciao, Amanda!    (3OGJ)
	[11:24] LeoObrst: Gramm Richardson et al gave a talk at STIDS 2012 on an 11179 registry addressing 
	these issues: 
	http://stids.c4i.gmu.edu/papers/STIDSPapers/STIDS2012_T04_RichardsonSchwarz_Constellation.pdf. STIDS 
	2012: http://stids.c4i.gmu.edu/agenda2012.php.    (3OGK)
	[11:25] PeterYim: I am soliciting help from everyone here: -- for software environment stewards and 
	tool developers, please make sure you participate in the upcoming survey ---and help us get these 
	colleagues of yours to respond to the survey too (they'll be on a wiki, so everyone will know who 
	has or hasn't responded) ... or provide us with pointers so we can reach out to them -- we need 
	ontology evaluation experts and tool developers to participate in the "hackathon" and "clinics" 
	activities    (3OGL)
	[11:26] PeterYim: also ... Join us at the "hackathon" and "clinics" activities - 
	http://ontolog.cim3.net/forum/ontology-summit/2013-02/msg00056.html & 
	http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013_Hackathon_Clinics    (3OGM)
	[11:35] VictorAgroskin: @Peter, @Patrick, @Valentina - we'll be really happy to see Patrick's team 
	to collaborate in our Clinics activity 
	http://ontolog.cim3.net/forum/ontology-summit/2013-03/msg00011.html ISO 15926 Reference Data Library 
	is a nice combination of taxonomies, quite suitable for exploration by methods described by Patrick.    (3OGN)
	[11:28] PavithraKenjige: About intrinsic and extrinsic evaluation .. those parts that are 
	manufactured by others .. is that considered extrinsic evaluation ??    (3OGO)
	[11:35] LeoObrst: Thanks, all! Good session.    (3OGP)
	[11:35] JoanneLuciano: Would you repeat Peter's last point?    (3OGQ)
	[11:35] PeterYim: join us again, same time next week (Thu 2013.03.14), for OntologySummit2013 
	session-09: "Building Ontologies to Meet Evaluation Criteria - I" - Co-chairs: MikeBennett & 
	MatthewWest - see developing session page at 
	http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_03_14    (3OGR)
	[11:35] JoelBender: Thank you!    (3OGS)
	[11:35] MelissaHaendel: Thanks everyone.    (3OGT)
	[11:36] KevinSimkins: Thanks to all speakers and host.    (3OGU)
	[11:36] PeterYim: great session ... thanks everyone!    (3OGV)
	[11:36] JoanneLuciano: Very interesting, thanks!    (3OGW)
	[11:36] anonymous morphed into BijanParsia    (3OGX)
	[11:36] AliHashemi: thank you all.    (3OGY)
	[11:36] DavidWhitten: Fascinating insights. Well Done all of you.    (3OGZ)
	[11:36] PeterYim: ... session ended ==    (3OH0)
 -- end of in-session chat-transcript --    (3NOJ)

Additional Resources:    (3NOQ)

For the record ...    (3NOX)

How To Join (while the session is in progress)    (3NOY)

Conference Call Details    (3NM5)

Attendees    (3NN2)