OntologySummit2013: Panel Session-10 - Thu 2013-03-21    (3NSO)

Summit Theme: "Ontology Evaluation Across the Ontology Lifecycle"    (3NSP)

Summit Track Title: Track-D: Software Environments for Evaluating Ontologies    (3NSQ)

Session Topic: Software Environments for Evaluating Ontologies - II    (3NSR)

Session Co-chairs: Dr. MichaelDenny (MITRE) and Mr. PeterYim (Ontolog; CIM3) - intro slides    (3NSS)

Panelists / Briefings:    (3NST)

Archives:    (3NSZ)

Abstract:    (3NUV)

OntologySummit2013 Session-10: "Software Environments for Evaluating Ontologies - II" - intro slides    (3NUW)

This is our 8th Ontology Summit, a joint initiative by NIST, Ontolog, NCOR, NCBO, IAOA & NCO_NITRD with the support of our co-sponsors. The theme adopted for this Ontology Summit is: "Ontology Evaluation Across the Ontology Lifecycle."    (3NUX)

Currently, there is no agreed methodology for development of ontologies, and there are no universally agreed metrics for ontology evaluation. At the same time, everybody agrees that there are a lot of badly engineered ontologies out there, thus people use -- at least implicitly -- some criteria for the evaluation of ontologies.    (3NUY)

During this OntologySummit, we seek to identify best practices for ontology development and evaluation. We will consider the entire lifecycle of an ontology -- from requirements gathering and analysis, through to design and implementation. In this endeavor, the Summit will seek collaboration with the software engineering and knowledge acquisition communities. Research in these fields has led to several mature models for the software lifecycle and the design of knowledge-based systems, and we expect that fruitful interaction among all participants will lead to a consensus for a methodology within ontological engineering. Following earlier Ontology Summit practice, the synthesized results of this season's discourse will be published as a Communique.    (3NUZ)

At the Launch Event on 17 Jan 2013, the organizing team provided an overview of the program, and how we will be framing the discourse around the theme of of this OntologySummit. Today's session is one of the events planned.    (3NV0)

In this 10th virtual panel session of the Summit, we have invited the stewards of some exemplary ontology software tools and environments out there to join us on the panel. They will share with us their work, experience and insights - particularly on how the software environments/systems/tools they represent, support evaluating intrinsic aspects and extrinsic aspects of ontologies, and how they help assure quality across the lifecycle of the development (of the ontology or the ontology-driven application). We will also hear about what they think are the major obstacles now, toward quality excellence, relevant work they are embarking on, and wish lists of what have yet to be done in term of improving ontology quality.    (3NV1)

We will also expecting to be "officially" launching the OntologySummit2013 Survey of Software Support for Ontology Quality and Fitness, which we alluded to earlier, at this session.    (3ORN)

More details about this OntologySummit is available at: OntologySummit2013 (homepage for this summit)    (3NV2)

Briefings:    (3NV3)

Agenda:    (3NVE)

OntologySummit2013 - Panel Session-10    (3NVF)

Proceedings:    (3NVL)

Please refer to the above    (3NVM)

IM Chat Transcript captured during the session:    (3NVN)

 see raw transcript here.    (3NVO)
 (for better clarity, the version below is a re-organized and lightly edited chat-transcript.)
 Participants are welcome to make light edits to their own contributions as they see fit.    (3NVP)
 -- begin in-session chat-transcript --    (3NVQ)
	------
	Chat transcript from room: summit_20130321
	2013-03-21 GMT-08:00 [PDT]
	------    (3OYS)
	[9:16] PeterYim: Welcome to the    (3OYT)
	 = OntologySummit2013: Virtual Panel Session-10 - Thu 2013-03-21 =    (3OYU)
	Summit Theme: Ontology Evaluation Across the Ontology Lifecycle    (3OYV)
	* Summit Track Title: Track-D: Software Environments for Evaluating Ontologies    (3OYW)
	Session Topic: Software Environments for Evaluating Ontologies - II    (3OYX)
	* Session Co-chairs: Dr. MichaelDenny (MITRE) & Mr. PeterYim (Ontolog; CIM3)    (3OYY)
	Panelists / Briefings:    (3OYZ)
	* Mr. AdamPease (Articulate Software)
	  - "The Sigma Knowledge Engineering Environment"    (3OZ0)
	* Prof. Dr. TillMossakowski (University of Bremen)
	  - "Ontohub.org - a web platform for distributed and heterogeneous ontologies"    (3OZ1)
	* Dr. TaniaTudorache (Stanford-BMIR)
	  - "WebProtege and BioPortal as Infrastructures for Ontology Evaluation"    (3OZ2)
	* Professor MichelDumontier (Carleton University)
	  - "Evaluation of ontology-powered scientific research as a means to assess and improve ontology quality"    (3OZ3)
	* Mr. KingsleyIdehen (OpenLink Software)
	  - "Ontology life cycles and Linked Open Data (LOD)"    (3OZ4)
	Logistics:    (3OZ5)
	* Refer to details on session page at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_03_21    (3OZ6)
	* (if you haven't already done so) please click on "settings" (top center) and morph from "anonymous" to your RealName (in WikiWord format)    (3OZ7)
	* Mute control: *7 to un-mute ... *6 to mute    (3OZ8)
	* Can't find Skype Dial pad?
	** for Windows Skype users: it's under the "Call" dropdown menu as "Show Dial pad"
	** for Linux Skype users: please note that the dial-pad is only available on v4.1 (or later or the earlier Skype versions 2.x,)
	   if the dialpad button is not shown in the call window you need to press the "d" hotkey to enable it.    (3OZ9)
	Attendees: AdamPease, AmandaVizedom, AnatolyLevenchuk, AstridDuqueRamos, BobSmith, BrianHaugh, 
	BriceSommacal, BruceBray, DanielCoutoVale, DougFoxvog, FabianNeuhaus, FranLightsom, JorgeCruanes, 
	JoelBender, JohnBilmanis, K. Z. Watkins, KenBaclawski, KingsleyIdehen, LamarHenderson, MarcelloBax, 
	MariaPovedaVillalon, MatthewWest, MeganKatsumi, MichaelDenny, MichaelGruninger, MichelDumontier, 
	MikeDean, MikeRiben, OliverKutz, PavithraKenjige, PeterYim, RamSriram, RichMarkeloff, RobertYao, 
	SamirTartir, SimonSpero, SteveRay, TaniaTudorache, TerryLongstreth, TillMossakowski, ToddSchneider, 
	TomTinsley, TorstenHahmann, anonymous, ... vnc2    (3OZA)
	 == Proceedings: ==    (3OZB)
	[9:17] PeterYim: Attn ALL: ... it has come to our attention that our conference bridge provider is 
	running into some problems with the "joinconference" skype connections. In case anyone gets in 
	trouble, please try to call the phone numbers instead (e.g. from your phone, skype-out, 
	google-voice, etc.)    (3OZC)
	[7:42] anonymous morphed into K. Z. Watkins    (3OZD)
	[9:22] anonymous morphed into MichaelDenny    (3OZE)
	[9:26] anonymous morphed into BrianHaugh    (3OZF)
	[9:27] anonymous morphed into anonymous2    (3OZG)
	[9:28] MichelDumontier: howdy    (3OZH)
	[9:28] anonymous1 morphed into TorstenHahmann    (3OZI)
	[9:29] anonymous morphed into AdamPease    (3OZJ)
	[9:29] MichaelDenny: The survey of software capabilities for ontology quality & fitness across the 
	ontology life cycle has launched at http://ontolog-02.cim3.net/wiki/OntologySummit2013_Survey. 
	Spread the word!    (3OZK)
	[9:29] MariaPovedaVillalon morphed into MariaPoveda    (3OZL)
	[9:30] anonymous morphed into FranLightsom    (3OZM)
	[9:30] MariaPoveda morphed into MariaPovedaVillalon    (3OZN)
	[9:30] anonymous morphed into RichMarkeloff    (3OZO)
	[9:30] anonymous1 morphed into TaniaTudorache    (3OZP)
	[9:31] anonymous3 morphed into DougFoxvog    (3OZQ)
	[9:31] anonymous morphed into BriceSommacal    (3OZR)
	[9:32] anonymous morphed into LamarHenderson    (3OZS)
	[9:33] K. Z. Watkins: are slides only on skype?    (3OZT)
	[9:37] MeganKatsumi: @K.Z. You can download the slides here: 
	http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_03_21#nid3NT2    (3OZU)
	[9:38] anonymous morphed into TomTinsley    (3OZV)
	[9:39] PeterYim: == MikeDenny opens the session on behalf of the co-chairs ... see: the [0-Chair] slides    (3OZW)
	[9:42] PeterYim: === slide#3 - MichaelDenny announcing the OntologySummit2013 Survey on software 
	capabilities for quality and fitness of ontologies ...    (3OZX)
	[9:44] PeterYim: see also: http://ontolog.cim3.net/forum/ontology-summit/2013-03/msg00197.html    (3OZY)
	[9:45] PeterYim: the survey is at: http://ontolog-02.cim3.net/wiki/OntologySummit2013_Survey    (3OZZ)
	[9:47] BrianHaugh: Clap, Clap, Clap, Clap, ...    (3P00)
	[10:23] TorstenHahmann: @MichaelDenny: For the survey, I can't seem to login with my ontolog 
	credentials - is there any extra signup required?    (3P01)
	[10:24] MichaelDenny: @Torsten If you are having problems, please use guest and ontolog-guest.    (3P02)
	[10:24] PeterYim: @TorstenHahmann & ALL: anyone having issues with the login (for the survey) can 
	login with: username=guest ; password=ontolog-guest    (3P03)
	[10:26] TorstenHahmann: @PeterYim & @MichaelDenny: thanks, that works.    (3P04)
	[9:48] PeterYim: == AdamPease presenting ... see: the [1-Pease] slides    (3P05)
	[9:53] SimonSpero: What about horses that only have three legs    (3P06)
	[9:54] SimonSpero: "A horse like that, you don't eat all at once"    (3P07)
	[10:00] TillMossakowski: is the translation from SUO-KIF to OWL described somewhere?    (3P08)
	[10:00] AmandaVizedom: Question for @AdamPease: Any plans for support of Common Logic?    (3P09)
	[10:11] MichaelGruninger: There is a long history behind the relationship of KIF and CL. KIF 3.0 
	(which is the last available document for KIF) evolved into Common Logic, but KIF also included a 
	lot of other notions (such as set theory and metalanguage) that are not contained in Common Logic 
	(which is the standard ISO 24707). Common Logic without sequence variables is first-order 
	expressive. I'm not sure about the precise formal relationship of SUO-KIF and Common Logic -- this 
	would require a closer look at the relationship of SUO-KIF and second-order logic.    (3P0A)
	[10:13] TorstenHahmann: There is a CL to KIF, Prover9, and TPTP translator available from 
	https://github.com/cmungall/cltools; it is in Prolog. It translates the first-order portion of CL. 
	I have developed wrappers to translate (and evaluate) large sets of CL ontologies, I'm currently 
	moving all the source to a Github project: https://github.com/thahmann/macleod    (3P0B)
	[10:08] AmandaVizedom: Comment: I strongly agree with this point: having enough content to perform 
	interesting inferences, and having ready/easy inference capability ready to hand, makes certain kinds
	of error detection and debugging much easier and more naturally integrated into development process.    (3P0C)
	[10:09] anonymous1 morphed into PavithraKenjige    (3P0D)
	[10:41] AdamPease: just reading the chat logs    (3P0E)
	[10:42] AdamPease: KIF and SUO-KIF are quite different, so unless it's specifically 
	a SUO-KIF translator, it won't work for SUMO    (3P0F)
	[10:42] AdamPease: translating CL to the first-order subset of SUO-KIF should be easy    (3P0G)
	[10:42] TillMossakowski: I think in Hets we have some translation of SUO-KIF into Common Logic    (3P0H)
	[10:43] AdamPease: I included a presentation of that in Sigma at one point I think, mainly just 
	renaming some of the logical symbols as I recall    (3P0I)
	[10:44] AdamPease: that sounds great Till    (3P0J)
	[10:45] PeterYim: @AdamPease: re your xx:43where can one find that presentation ... can you supply a 
	link to it?    (3P0K)
	[10:45] AdamPease: ?    (3P0L)
	[10:45] AdamPease: oh, not a slide presentation    (3P0M)
	[10:46] AdamPease: a presentation of logic    (3P0N)
	[10:46] PeterYim: @Adam: this is in reference you your earlier statement "I included a presentation 
	of that in Sigma at one point ..." ... can you supply a link to that presentation?    (3P0O)
	[10:46] AdamPease: use the "traditionalLogic" menu option in the menu title "Formal Language" in the Sigma browser    (3P0P)
	[10:47] PeterYim: Ah ... thanks, Adam    (3P0Q)
	[11:01] TorstenHahmann: @AdamPease: Adam, you mentioned that the KIF version of SUMO can be 
	automatically translated to OWL. can you expand a little on how that works (or give me a pointer)? 
	Which OWL variant (OWL-DL?) do you translate to? In particular, I would think you would need to 
	consider all entailments of the first-order ontology to ensure that the translation is complete 
	(with respect to the expressivity of OWL). How do get around this?    (3P0R)
	[11:02] TillMossakowski: interesting question!    (3P0S)
	[11:03] AdamPease: I make no claims that it's a complete translation    (3P0T)
	[11:03] AdamPease: especially since OWL is so limited    (3P0U)
	[11:03] AdamPease: but even within its limitations, we don't do anything extraordinary    (3P0V)
	[11:03] AdamPease: all binary relations get translated    (3P0W)
	[11:04] AdamPease: we translate all the builtin terms in OWL to their equivalents in SUMO    (3P0X)
	[11:04] AdamPease: all rules get turned into comments    (3P0Y)
	[11:04] AdamPease: that's about it    (3P0Z)
	[11:06] TorstenHahmann: thanks, I was just wondering    (3P10)
	[10:09] PeterYim: == TillMossakowski presenting ... see: the [2-Mossakowski] slides    (3P11)
	[10:09] anonymous1 morphed into SteveRay    (3P12)
	[10:10] anonymous1 morphed into RobertYao    (3P13)
	[10:11] anonymous2 morphed into JorgeCruanes    (3P14)
	[10:11] anonymous1 morphed into DanielCoutoVale    (3P15)
	[10:11] anonymous1 morphed into LamarHenderson    (3P16)
	[10:20] anonymous1 morphed into MarcelloBax    (3P17)
	[10:06] JoelBender: Isn't there a CL to KIF translator?    (3P18)
	[10:06] SimonSpero: Hets sort of does that    (3P19)
	[10:07] SimonSpero: for certain senses of KIF    (3P1A)
	[10:08] SimonSpero: Hets module CommonLogic.Parse_KIF    (3P1B)
	[10:07] MatthewWest: KIF is a dialect of CL    (3P1C)
	[10:19] PeterYim: TillMossakowski - see: http://ontohub-new.informatik.uni-bremen.de/    (3P1D)
	[10:25] PeterYim: == TaniaTudorache presenting ... see: the [3-Tudorache] slides    (3P1E)
	[10:30] DougFoxvog: Tania's Slide 4 mentions "Contextual notes and discussions attached to any 
	entity in the ontology". Can such notes be attached to statements, or merely to the terms?    (3P1F)
	[10:37] AmandaVizedom: Question for @TaniaTudorache: Is it correct to say that Web Protege supports 
	collaborative manual evaluation, and potentially documentation of automated or semi-automated 
	evaluation and/or metrics gathering that others may have done independently, but does not support 
	such evaluation itself?    (3P1G)
	[10:39] PeterYim: @TaniaTudorache: can one access the collected user review input, and 
	slice-and-dice that data ... I guess its, sort of, the same question Amanda is asking    (3P1H)
	[10:41] DougFoxvog: Ref Slide 12. What about an ontology for encoding the reviews using relations & 
	terms defined in the ontology?    (3P1I)
	[10:42] AmandaVizedom: Comment: Tania's slide 12 regarding issues with review suggests, to me, that 
	it would be helpful to have a common understanding of ontology characteristics and evaluation 
	metrics/techniques, such that these could be explicitly addressed in reviews. One would like to see 
	reviews evolve to focus on such shared questions.    (3P1J)
	[10:41] PeterYim: == MichelDumontier presenting ... see: the [4-Dumontier] slides    (3P1K)
	[10:51] AmandaVizedom: Comment: Just noting, per @MichelDumontier's slide 3, that GO is a controlled 
	vocabulary (common in domain, I think) -- i.e., the "nodes" are NL expressions. I wonder whether the 
	significance of annotation change patterns, for example, would be different for a concept-centric ontology, 
	with (potentially multiple) NL expressions associated, that for a controlled vocabulary like this.    (3P1L)
	[10:53] AmandaVizedom: (I would think so; the concept-centric ontology with lexical mappings should 
	be less brittle to language change than the controlled vocab. In any case, it's a good example of 
	the importance of being clear what kind of KR artifact you are dealing with before you can 
	understand what to evaluate and how to interpret it.    (3P1M)
	[10:57] AmandaVizedom: @Michel - re slide 11: Great examples of how different kinds of evaluation 
	can be more or less relevant depending on how the ontology is going to be used (or reused).    (3P1N)
	[11:15] MichelDumontier: @Amanda - The GO has made strides to be a more conformant "ontology", but I 
	find it confusing/lacking in many respects, but doesn't diminish it's current utility in scientific 
	research. Thanks for the positive comment on slide 11 :)    (3P1O)
	[11:03] TerryLongstreth: @Michel - Slide 13 - did you subdivide the volunteer community by 
	expertise? Did expertise/experience make any difference?    (3P1P)
	[11:10] MichelDumontier: @Terry - the volunteers were experts at a conference on the topic    (3P1Q)
	[11:11] TerryLongstreth: Thanks, Michel.    (3P1R)
	[11:11] PeterYim: == KingsleyIdehen presenting ... see: the [5-Idehen] slides    (3P1S)
	[11:16] PeterYim: @ALL: note that Kingsley is working off a slightly updated deck of slides. If you 
	downloaded the slide before he started talking, you might wanted to download the (the updated) slides again    (3P1T)
	[11:24] MatthewWest: Afraid I need to go now.    (3P1U)
	[11:25] MichelDumontier: ugh oh ... Kingsley we lost you!    (3P1V)
	[11:25] MariaPovedaVillalon: yes    (3P1W)
	[11:25] DougFoxvog: I hear Peter, but not Kingsley    (3P1X)
	[11:26] SamirTartir: @Peter: I can hear you    (3P1Y)
	[11:26] PeterYim: Kingsley we lost you, on the voice line, can you dial in again, please    (3P1Z)
	[11:26] MichaelDenny: @Kingsley We've lost you!    (3P20)
	[11:28] PeterYim: === KingsleyIdehen's talk resumes ... (slide#13)    (3P21)
	[11:30] PeterYim: KingsleyIdehen request that AdamPease supply a URI for SUMO that he can link to    (3P22)
	[11:30] AdamPease: maybe I'm misunderstanding but we've had that for years    (3P23)
	[11:31] AdamPease: http://sigma-01.cim3.net:8080/sigma/OWL.jsp?kb=SUMO&term=Object    (3P24)
	[11:31] AdamPease: for example    (3P25)
	[11:30] DougFoxvog: URI for terms that denote creators seems too much of a hack. Assertions about 
	who is responsible for a terms set (which would have the same name space) seem to me to be more 
	worthwhile.    (3P26)
	[11:33] DougFoxvog: Individual assertion, e.g., (termCreatedBy <TERM> <AGENT>) can be used for 
	attribution.    (3P27)
	[11:30] AmandaVizedom: Must drop off. Thanks all!    (3P28)
	[11:33] PeterYim: == Q&A and Open Discussion ...    (3P29)
	[11:33] PeterYim: Again, soliciting help from everyone here: 
	-- for software environment stewards and tool developers, please make sure you participate in the OntologySummit2013 software survey 
	-- and help us get these colleagues of yours to respond to the survey too: http://ontolog-02.cim3.net/wiki/OntologySummit2013_Survey 
	-- ... or provide us with pointers to stewards of relevant software tools/systems/environments so we can reach out to them    (3P2A)
	[11:37] PeterYim: join us again, same time next week, for OntologySummit2013 session-11: "Getting 
	the "hackathon," "clinics" and related activities going" - Co-chairs: MikeDean, KenBaclawski & 
	PeterYim - developing session details at: 
	http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_03_28    (3P2B)
	[11:37] PeterYim: great session ... thanks everyone!    (3P2C)
	[11:37] JoelBender: Thank you!    (3P2D)
	[11:37] MichelDumontier: Thanks!    (3P2E)
	[11:37] TaniaTudorache: Thank you!    (3P2F)
	[11:38] PeterYim: -- session ended: 11:37 am PDT --    (3P2G)
	[11:38] MichaelDenny: Bye all!    (3P2H)
	[11:51] PeterYim: the raw chat-transcript is online now; cleaned up version of the chat-transcript, 
	the audio recording, attendee roster, etc. will be available in a day or two. Look under: 
	http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_03_21#nid3NSZ    (3P2I)
	[11:52] PeterYim: bye everyone!    (3P2J)
 -- end of in-session chat-transcript --    (3NVR)

Additional Resources:    (3NVY)

For the record ...    (3NW5)

How To Join (while the session is in progress)    (3NW6)

Conference Call Details    (3NT8)

Attention: Please take special note on the start time of the event, as the US will be on daylight saving (summer) time on this day, while the EU is still on standard (winter) time!    (3O8A)

Attendees    (3NU5)