ConferenceCall 2011 02 17

For the record ...

[ slides ]
[ slides ]
[ slides ]
[ slides (pdf) ]
[ slides ]
[ slides (pdf) ]
 [ slides ]
 see raw transcript here.    (2NIM)
 (for better clarity, the version below is a re-organized and lightly edited chat-transcript.)
 Participants are welcome to make light edits to their own contributions as they see fit.    (2NIN)
 -- begin in-session chat-transcript --    (2NIO)
	[09:17] PeterYim: .    (2O96)
	Welcome to the OntologySummit2011: Panel Session-4 
	(Track-3) "Value Metrics, Value Models and the Value Proposition" - Thu 2011_02_17    (2O97)
	Summit Theme: OntologySummit2011: Making the Case for Ontology    (2O98)
	Session Title: Value Metrics, Value Models and the Value Proposition - Take I    (2O99)
	Session Co-chairs: Dr. ToddSchneider (Raytheon) & Mr. RexBrooks (Starbourne)    (2O9A)
	Panelists:    (2O9B)
	* Mr. RexBrooks (Starbourne) - "Introduction" 
	* Mr. KurtConrad (Sagebrush) - "Business Value Alignment to Support Ontology Development"
	* Mr. RexBrooks / Mr. ChristianFillies (Semtation GmbH) - "Ontology Integration"
	* Ms. MaryBalboni (Raytheon) - "Ontology Performance"
	* Dr. JohnYanosy (Rockwell Collins) - "Ontology and Business Value"
	* Dr. ToddSchneider (Raytheon) - "Ontology Use Maintenance"    (2O9C)
	.    (2O9D)
	[09:22] PeterYim: please refer to session details at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2011_02_17    (2O9E)
	[09:26] SteveRay: All Hail Watson!    (2O9F)
	[09:28] NicolaGuarino: Hi everybody    (2O9G)
	[09:29] RexBrooks: Hi All    (2O9H)
	[09:30] anonymous morphed into BruceBray    (2O9I)
	[09:32] PatBarkman: hi    (2O9J)
	[09:34] LeoObrst: Hi, all. My old bearded collie was named Watson, and so I was rooting against the 
	humans. ;)    (2O9K)
	[09:34] BobSmith: Hello, and Rex, your voice is still fuzzy    (2O9L)
	[09:35] anonymous morphed into AlexMirzaoff    (2O9M)
	[09:39] SteveRay: Cheat sheet: *3 to un-mute; *2 to mute    (2O9N)
	[09:40] vnc2: session starts    (2O9O)
	[09:41] vnc2: == RexBrooks - introduction ==    (2O9P)
	[09:46] anonymous morphed into PavithraKenjige    (2O9Q)
	[09:48] anonymous1 morphed into KurtConrad    (2O9R)
	[09:48] KurtConrad: I'm here    (2O9S)
	[09:48] SteveRay: Cheat sheet: *3 to un-mute; *2 to mute    (2O9T)
	[09:49] SteveRay: == KurtConrad presents ==    (2O9U)
	[09:51] anonymous morphed into RamGouripeddi    (2O9V)
	[09:54] MichaelGruninger: @Kurt: What are the agents in "agent-specific alignments"?    (2O9W)
	[09:56] PavithraKenjige: Actually I could hear you, but a little low voice    (2O9X)
	[09:56] MichaelGruninger: thanks!    (2O9Y)
	[09:56] anonymous morphed into JohnYanosy    (2O9Z)
	[09:56] SteveRay: We're on slide 4    (2OA0)
	[09:59] SteveRay: Maybe it's just me, but if I were a business person trying to decide whether to 
	invest in an ontological approach, I would be very lost by now.    (2OA1)
	[10:01] JohnYanosy: This slide is very informative with respect to understanding knolwedge sources, 
	not sure how to incorporate tacit knowledge into ontologies    (2OA2)
	[10:06] LeoObrst: @Kurt: there is also a technical notion of "dynamic semantics" for natural 
	language semantics that goes back to Kamp, Heim in the early 1980s and Groenendijk and Stokhof in 
	1990, 1991, that has subsequently been developed by others.    (2OA3)
	[10:04] SteveRay: == RexBrooks presents ==    (2OA4)
	[10:04] SteveRay: Go ahead. We can use the downloaded version as well.    (2OA5)
	[10:06] anonymous morphed into Sarah Goldman    (2OA6)
	[10:08] PatBarkman: novice here ... anyone want to state what "common tool" he's talking about? 
	Semaphore/smartlogic?    (2OA7)
	[10:09] SteveRay: Not sure why the name wasn't mentioned. It could have been a UML tool like 
	Enterprise Architect, MagicDraw, or Rational Rose.    (2OA8)
	[10:09] PatBarkman: ahhh ... thanks Steve    (2OA9)
	[10:24] RexBrooks: @Pat, as far as modeling tools, SemTalk makes Visio into a real modeling tool, at 
	least for BPMN and several other specific kinds of specialized analytics. It can output XMI and that 
	allows me to import it into Enterprise Architect that then gives me all the UML I need and then 
	some. Both SemTalk and EA output images of diagrams and generate code and have dedicated back end 
	databases. i'm still working on getting some of the conceptualizing into Compendium.    (2OAA)
	[10:25] PatBarkman: @Rex ... thanks that helps.    (2OAB)
	[10:13] SteveRay: Still anxious to start hearing suggested metrics...cost? development time? 
	capability? system performance?    (2OAC)
	[10:15] PatBarkman: I have the same questions, Steve ... cost/benefit breakdowns    (2OAD)
	[10:19] RexBrooks: @Steve, don't worry, its coming.    (2OAE)
	[10:19] RexBrooks: @Steve, Since I knew there was all that coming, I felt free to deal with 
	expectations and focusing on the Value Proposition more than the Value Metrics and Models.    (2OAF)
	[10:19] SteveRay: OK    (2OAG)
	[10:24] ToddSchneider: Steve, are we getting closer?    (2OAH)
	[10:24] RexBrooks: hehe    (2OAI)
	[10:15] SteveRay: By the way, Todd was very clear just now on the phone.    (2OAJ)
	[10:17] RexBrooks: @Pat Sorry, SemTalk uses Visio but adds semantic capabilities to it. I happen to 
	be involved with SemTalk USA and wanted to avoid the appearance of attempting to sell a product.    (2OAK)
	[10:17] PatBarkman: thanks Rex    (2OAL)
	[10:25] PeterYim: @MaryBalboni - ref. your slide#5, is there a maturity model for ontology that you 
	are using currently?    (2OAM)
	[10:25] ToddSchneider: Peter, I don't think so.    (2OAN)
	[10:25] PatBarkman: ... and yes I'm luvin' the MA here great job Mary!    (2OAO)
	[10:27] RexBrooks: @Peter I am pretty sure there isn't one that has much support yet.    (2OAP)
	[10:29] PeterYim: ref. maturity model, it may be nice to compare what is out there ... e.g. ref. 
	LeoObrst's OMM which he posted to the [ontology-summit] list yesterday 
	(http://ontolog.cim3.net/forum/ontology-summit/2011-02/msg00061.html ) - which is at: 
	http://ontolog.cim3.net/file/work/OntologySummit2011/reference/ontologyMaturityModel-obrst-2009.pdf    (2OAQ)
	[10:29] SteveRay: @Todd: Yes, I'm breathing a bit easier. So far I'm hearing Performance, and Cost, 
	with some good expansion of each. The reason I'm asking is because I'm thinking of how this will 
	integrate into the other tracks - Use cases, and Application framework.    (2OAR)
	[10:29] ToddSchneider: Steve, understood.    (2OAS)
	[10:31] SteveRay: Not sure I would conflate risk and performance...    (2OAT)
	[10:32] JohnYanosy: great job    (2OAU)
	[10:32] SteveRay: Great talk by Mary - brings out some good metrics.    (2OAV)
	[10:33] MaryBalboni: thank you    (2OAW)
	[10:33] PatBarkman: yes ... I think that's a jump as well Steve, but given the maturity/prevalance 
	of ontology implementations ... it's a fair stepping stone in MA    (2OAX)
	[10:33] Ramdsriram: Are there any case studies (with data) out there which describe how ontologies 
	improved a system performance?    (2OAY)
	[10:33] JohnYanosy: Todd, I will have to be leaving soon for a customer meeting, approx 15 minutes    (2OAZ)
	[10:33] ToddSchneider: All, we have to save questions. JohnYanosy has a hard stop time.    (2OB0)
	[10:34] RexBrooks: It's almost time to introduce John.    (2OB1)
	[10:34] PeterYim: @Todd ... are you doing Q&A after all the panelists have finished their 
	presentations?    (2OB2)
	[10:37] ToddSchneider: Peter, yes.    (2OB3)
	[10:34] PatBarkman: thanks Mary ... great job ... will be referencing your work in mine, so thanks    (2OB4)
	[10:35] PavithraKenjige: thank you Mary    (2OB5)
	[10:35] MaryBalboni: thanks everyone - glad to be part of this group    (2OB6)
	[10:38] LeoObrst: Concerning ONTOCOM, which is basically an ontology cost model, it identifies as 
	cost drivers: Building, Reuse, Personnel, and Project, with sub-categories for each of these. 
	Example: Building: Domain Analysis Complexity, Conceptualization Complexity, Implementation 
	Complexity, Instantiation Complexity, Required Reusability, Documentation Needs. Etc. It also has 
	developed a spreadsheet with these factors, and one can compute the estimated cost based on their 
	data and factor weights - http://ontocom.sti-innsbruck.at/.    (2OB7)
	[10:38] SteveRay: I interpret John's presentation as addressing Capability as the metric in 
	question.    (2OB8)
	[10:40] SteveRay: It's very hard to quantify Capability in the abstract, but in specific examples 
	they could be enumerated.    (2OB9)
	[10:40] ToddSchneider: John is providing a operational view.    (2OBA)
	[10:41] RexBrooks: John's work is particularly useful in the SOA context, especially the emerging 
	ecosystem view.    (2OBB)
	[10:42] RexBrooks: But the evaluation metrics need to be fleshed out a bit.    (2OBC)
	[10:42] SteveRay: Slide 4 has some good raw material for metrics.    (2OBD)
	[10:43] RexBrooks: It's difficult to measure inferencing except for accuracy, e.g. internal 
	consistency with its own definitions.    (2OBE)
	[10:44] ToddSchneider: Rex, the performance of inferencing should be measurable.    (2OBF)
	[10:44] PatBarkman: Thanks John    (2OBG)
	[10:46] JohnYanosy: You are welcome and sorry for not being able to provide more detail, but will be 
	posting more material on the wiki and think the previous metrics would be interesting to apply to 
	these business areas. Great job and will be more active in future. Good by thanks    (2OBH)
	[10:44] RexBrooks: We'll get there eventually Todd,    (2OBI)
	[10:45] BobbinTeegarden: Is part of the value of inferencing in code NOT written (and associated 
	costs')?    (2OBJ)
	[10:46] MaryBalboni: Thanks Todd - :)    (2OBK)
	[10:47] PeterYim: == ToddSchneider presenting ==    (2OBL)
	[10:50] SteveRay: @Todd: Just because the consequence of error is different, it doesn't follow the 
	METRIC is different, just that the value and weighting of that metric may be different.    (2OBM)
	[10:52] RexBrooks: There are some real misunderstandings about the differences between qualitative 
	and quantitative metric and the relationship between them, but that's almost a topic of its own. I 
	come from an Advertising background and we had to provide quant. to satisfy the customer's need to 
	rationalize while appealing to unstated qual. that are often the actual driving motivation to 
	purchase or not in the marketplace.    (2OBN)
	[10:52] LeoObrst: As part of the DARPA HPKB (High Performance Knowledge Bases) and RKF (Rapid 
	Knowledge Formation) programs, large ontology integration efforts to solve a command and 
	control/situational awareness problem, the Program Manager Murray Burke (and before him, Dave 
	Gunning) tried to capture ontology axiom reuse metrics.    (2OBO)
	[10:54] BobbinTeegarden: @Leo Results of that reuse work? Any references online?    (2OBP)
	[11:15] LeoObrst: @Bobbin: yes, some of it is online. See the paper Schrag, Robert, Mike Pool, Vinay 
	Chaudhri, Robert C. Kahlert, Joshua Powers, Paul Cohen, Julie Fitzgerald, and Sunil Mishra. 
	Experimental Evaluation of Subject Matter Expert-oriented Knowledge Base Authoring Tools.    (2OBQ)
	[10:56] MikeBennett: Does this mean that it would be possible to define a quantitative difference 
	between ontologies with losts of "Equivalent class" links versus ontologies which make use of high 
	level patterns or archetypes? Could this make the case for better use of sharing and integration 
	ontologies? Just a thought.    (2OBR)
	[10:56] SteveRay: OK. So far I have heard the following classes of metrics: Cost, Capability, 
	Performance, Quality, System Complexity. Any others?    (2OBS)
	[10:58] SteveRay: People can un-mute themselves. *3 to un-mute; *2 to mute    (2OBT)
	[10:58] MaryBalboni: Maturity related to the depth of model and breadth of validation ...    (2OBU)
	[11:00] SteveRay: @Mary: Wouldn't maturity manifest itself to a customer in terms of capability? In 
	other words, I would put the maturity of a model as a property that feeds in to the capability 
	metric.    (2OBV)
	[11:01] SteveRay: @Mary: Put another way, I would imagine that a customer isn't as interested in the 
	maturity of an embedded ontology itself, but rather in how that might affect performance of their 
	system.    (2OBW)
	[11:01] MaryBalboni: @SteveRay - more capability may be part of maturity - or could be more 
	correctness while operational --- also had thoughts on Complexity can be measured like the old 
	Halsteads measures perhaps...    (2OBX)
	[11:02] SteveRay: @Mary: OK. I accept that model maturity can manifest itself through several 
	classes of metrics, including correctness (which think of as a specialization of the Quality 
	metric).    (2OBY)
	[11:03] MaryBalboni: Operators and operands were counted and an assumption was made on how complex 
	code was based on those numbers - not sure if Ontology could be discected in such a way    (2OBZ)
	[11:03] PeterYim: unlike other maturity models, the Capability Maturity Model for Software 
	Engineering (SEI CMM) for example, in Ontology, "more mature" may correlate with "more 
	sophisticated, and better grasp of semantics (stronger semantics) in the system" and may not 
	correlate directly with performance, much less effectiveness, or even whether it is appropriate. ... 
	Therefore: (a) mapping metrics to the application framework may be necessary, (b) different metrics 
	should be expected at different level of Ontology Maturity (as how the the applications are 
	implemented would be radically different.)    (2OC0)
	[11:03] SteveRay: I'm trying to separate in my own mind the distinction between intrinsic measures 
	of an ontology, versus extrinsic metrics of business value which will be the ones that a customer or 
	decision maker will be evaluating when being pitched.    (2OC1)
	[11:06] RexBrooks: I'd be interested to hear how people think we can measure inferencing.    (2OC2)
	[11:06] AlexMirzaoff: by success of the inference?    (2OC3)
	[11:07] SteveRay: @Rex: Also by maximum compute time    (2OC4)
	[11:07] RexBrooks: What happens if the inference extends over different systems that may offer 
	different definitions in different cases?    (2OC5)
	[11:08] BobbinTeegarden: Measure value of inference in terms of equivalent code it would take to do 
	the same thing in code (as a 'savings' or negative cost)?    (2OC6)
	[11:09] MaryBalboni: cost avoidance    (2OC7)
	[11:09] BobbinTeegarden: No, genuine 'savings'?    (2OC8)
	[11:09] RexBrooks: @Bobbin: Excellent. Never thought of that. Point is, we need lots of use cases to 
	check against.    (2OC9)
	[11:11] MikeBennett: On CMM, there is also a "Data Management Maturity Model" in development by the 
	EDM Council, in collaboration with Carnegie Mellon as owners of the CMM model. This is still in very 
	early stages of development, so there is potential to input to this with metrics for ontology 
	maturity if / when these are defined.    (2OCA)
	[11:11] PatBarkman: but I think there are intrinsic benefits in using CMMi (MA/PPQA) in a 
	before/after comparision ... an existing implementation that gets an ontology added to the 
	architecture    (2OCB)
	[11:14] RexBrooks: @Pat: Yes, seeing the difference in those before and after results would lead to 
	new insights, I'm sure.    (2OCC)
	[11:15] MichaelGruninger: Since we want to demonstrate the benefits of ontologies, let's first 
	consider the benefits in a software application. We can leverage the existing approaches to software 
	engineering by considering functional and nonfunctional requirements. For functional requirements, 
	we need to demonstrate that ontologies can be used to deliver new functionalities. For nonfunctional 
	requirements, we can use existing software metrics such as performance, cost, quality, maintenance. 
	In each case, we want to compare an application without an ontology and an application with an 
	ontology, and show that there is an improvement.    (2OCD)
	[11:11] SteveRay: I completely concur with the points just made by MichaelGruninger.    (2OCE)
	[11:12] PeterYim: +1    (2OCF)
	[11:13] RexBrooks: +2    (2OCG)
	[11:17] BobbinTeegarden: There must be some way to capture the wider comprehension, collaborative 
	common interactions, group understanding, ease of extension, ... some of the things that make use of 
	ontologies truly a step forward.    (2OCH)
	[11:17] RexBrooks: == Leo presenting his maturity model ==    (2OCI)
	[11:17] RexBrooks: Peter just put it up on the vnc!    (2OCJ)
	[11:19] TerryLongstreth: Leo's is a very valuable start, but it doesn't address cost/value    (2OCK)
	[11:17] MaryBalboni: DoDAF also is expanding their framework to concentrate on data such as the 
	CMMI-DM - have not explored if DoDAF new version is addressing Ontology/semantics in detail    (2OCL)
	[11:18] RexBrooks: @Mary: Slowly pulling teeth along the way--I participate in DoDAF Metamodel 2 WG.    (2OCM)
	[11:20] MaryBalboni: @Rex DoDAF is a late bloomer in Data Modeling .. :)    (2OCN)
	[11:20] RexBrooks: @Mary: Yup!    (2OCO)
	[11:07] PavithraKenjige: How Are these different than system development life cycle?    (2OCP)
	[11:18] MichaelGruninger: On the other side, there are costs associated with using ontologies within 
	an application, and perhaps these are not completely covered by the analogy to software engineering    (2OCQ)
	[11:20] ToddSchneider: Michael, one of those out-of-band costs/risks is the availability of 
	experienced people to do the work    (2OCR)
	[11:20] AlanRector: For us, the notions of sustainability and persistence are more relevant than 
	maturity - or perhaps are some of the relevant metrics for maturity for ontologies.    (2OCS)
	[11:21] MichaelGruninger: Did the software engineering community ever do an analysis of the benefits 
	of using object-oriented approaches to software design? We are perhaps facing an analogous problem 
	...    (2OCT)
	[11:21] SteveRay: @Todd: Your point raises the additional metric of Risk (in this case, risk of not 
	being able to maintain the system over time, for example).    (2OCU)
	[11:22] JeffZhuk: Mary, did you start using ontology or in the development stage?    (2OCV)
	[11:23] AlanRector: One key step for our cases is when the primary identifiers ontology moves from 
	text identifiers that inevitably change and are language specific to "nonsemantic IDs", and when 
	there is a sensible ID management scheme in place with the display names in annotations (usually 
	rdf:label or one of the skos:label family.    (2OCW)
	[11:24] MaryBalboni: @MichaelGruningerThere have been OO studies in SW Development - Rationale may 
	have study papers .. probably biased .. but the OO giants in industry are at Rationale    (2OCX)
	[11:25] MaryBalboni: @Yefim Jeff Not at the moment implementing an Ontology but interested in its 
	usefulness in an IA domain    (2OCY)
	[11:26] AlanRector: Maturity of "ontology technology" rather than of a specific ontology?    (2OCZ)
	[11:28] MikeBennett: Mills did mention one or two case studies where there were measured costs of 
	doing it the hard way, and then using ontologies. However most of our Case Studies to date have not 
	included explicit metrics.    (2OD0)
	[11:33] PeterYim: folks from Canada did have some hard numbers - ref. 
	http://ontolog.cim3.net/forum/ontology-summit/2011-02/msg00017.html & 2nd presentation at: 
	http://ontolog.cim3.net/forum/ontology-summit/2010-12/msg00002.html    (2OD1)
	[11:30] SteveRay: Probably about time to wrap things up...    (2OD2)
	[11:31] JeffZhuk: Thanks!    (2OD3)
	[11:31] PatBarkman: Productive and informative ... great session, thanks all    (2OD4)
	[11:31] SteveRay: Thanks for a good session!    (2OD5)
	[11:31] MaryBalboni: thanks to all!    (2OD6)
	[11:33] PeterYim: Great session ... thank you!    (2OD7)
	[11:33] PeterYim: -- session ended: 11:32am PST --    (2OD8)
 -- end of in-session chat-transcript --    (2NIP)

ConferenceCall 2011 02 17

OntologySummit2011: Panel Session-4 - (Track-3) "Value Metrics, Value Models and the Value Proposition" - Thu 2011_02_17 (2NGL)

Abstract: (2NI4)

Agenda: (2NIC)

Proceedings: (2NIJ)

IM Chat Transcript captured during the session: (2NQV)

Audio Recording of this Session (2NIV)

Additional Resources: (2NJ4)

How To Join (while the session is in progress) (2NJE)

Conference Call Details (2NH4)

Attendees: (2M8H)

Your Visited Pages

View Backlinks

Wiki Search