ppy/OntologySummit2013_s05_chat-transcript_unedited_20130214a.txt


------

Chat transcript from room: summit_20130214
2013-02-14 GMT-08:00

------


[08:59] PeterYim: Welcome to the

 = OntologySummit2013: Virtual Panel Session-05 - Thu 2013-02-14 =

Summit Theme: Ontology Evaluation Across the Ontology Lifecycle

* Summit Track Title: Track-D: Software Environments for Evaluating Ontologies

Session Topic: Software Environments for Evaluating Ontologies - I

* Session Co-chairs: Mr. PeterYim (Ontolog; CIM3) and Dr. MichaelDenny (MITRE)

Panelists / Briefings:

* Dr. MichaelDenny (MITRE) - "Ontology Quality and Fitness: A Survey of Software Support"

* Professor MichaelGruninger (U of Toronto) - "Ontology Evaluation Workflow in COLORE"

* Ms. JeanneHolm (Data.gov; NASA/JPL) - "Evaluating US Open Data for Discovery, Interoperability, and Innovation"

* Mr. GavinMatthews (Vertical Search Works) - "Assuring broad quality in large-scale ontologies"

Logistics:

* Refer to details on session page at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_02_14

* (if you haven't already done so) please click on "settings" (top center) and morph from "anonymous" to your RealName (in WikiWord format)

* Mute control: *7 to un-mute ... *6 to mute

* Can't find Skype Dial pad?
** for Windows Skype users: it's under the "Call" dropdown menu as "Show Dial pad"
** for Linux Skype users: please note that the dial-pad is only available on v4.1 (or later or the earlier Skype versions 2.x,)
   if the dialpad button is not shown in the call window you need to press the "d" hotkey to enable it.

.
 == Proceedings: == 

.

[08:51] marcelaVegetti morphed into MarcelaVegetti

[09:19] MatthewWest1 morphed into MatthewWest

[09:19] anonymous1 morphed into Max Petrenko

[09:21] anonymous morphed into KevinSimkins

[09:23] anonymous morphed into Gavin Matthews

[09:24] anonymous morphed into MichaelDenny

[09:27] anonymous morphed into CarmenChui

[09:27] anonymous morphed into Marcos

[09:28] Marcos morphed into MarcosMartinezRomero

[09:29] anonymous morphed into Jeanne Holm

[09:32] anonymous morphed into Lamar henderson

[09:33] anonymous morphed into doug foxvog

[09:39] anonymous morphed into PaulPope

[09:40] SimonSpero morphed into SimonSpero2

[09:40] SimonSpero1 morphed into SimonSpero

[09:42] anonymous morphed into Lamar Henderson

[09:42] Clare Paul morphed into ClarePaul

[09:43] Joanne Luciano: Call for help with software development- is there anyone in the community who knows SADI services? (if not, ok check out sadiframework.org) I have a generalized framework approach that can be used for development of ontologies and evaluation, providing context for intrinsic and extrinsic evaluation. If you can help, or know anyone who can, please get in touch with me.

[09:36] PeterYim: == PeterYim opens the session on behalf of the co-chairs ... see: the [0-Chair] slides

[09:43] PeterYim: == MichaelDenny presenting ... see: the [1-Denny] slides

[09:44] List of members: AlanRector, AmandaVizedom, AnatolyLevenchuk, anonymous, BobbinTeegarden, CarmenChui, ClarePaul, DaliaVaranka, DmitryBorisoglebsky, doug foxvog, ElieAbiLahoud, FabianNeuhaus, Gavin Matthews, Jeanne Holm1, JimDisbrow, Joanne Luciano, JoaoPauloAlmeida, JohnBilmanis, KenBaclawski, KevinSimkins, Lamar Henderson, LeoObrst, MarcelaVegetti, MarcosMartinezRomero, MatthewWest, Max Petrenko, MichaelDenny, MichaelGruninger, MikeDean, OliverKutz2, PaulPope, PeterYim, Ram D. Sriram, Richard Martin, SimonSpero, SteveRay, TerryLongstreth, TillMossakowski, ToddSchneider, VictorAgroskin, vnc2

[09:44] anonymous morphed into DanCerys

[09:44] anonymous morphed into Akram

[09:50] AmandaVizedom: @MichaelDenny - cutting out all human-required evaluation factors might miss much opportunity. Why not include all factors, noting that there may be much that the evaluation environment can do to *assist* with non-automated evaluation and with the tracking and management of evaluation of all factors?

[09:51] anonymous morphed into WillBurns

[09:54] Joanne Luciano: +1 to Amanda's comment -- use computers to assist humans

[09:56] TerryLongstreth: @MichaelDenny -

[09:57] SimonSpero: Not perfect, but not bad: http://www.howto.gov/customer-service/collecting-feedback/basics-of-survey-and-question-design

[09:57] doug foxvog: @MichaelDenny: you suggest characterizing breadth, depth, use considerations, etc. of ontologies. Do you have an ontology which can be used to express these features of ontologies?

[09:58] BobSchloss: [Apologies - last minute unavoidable conflict came up. I will look over the 3 sets of slides from the rest of today tomorrow or next week]

[09:58] TerryLongstreth: Perhaps an element of the Maintenance phase, but I prefer adding 'Retirement' to include archiving, sequestering, or erasing any component

[09:58] PeterYim: @BobSchloss: thanks for coming (albeit briefly)

[10:01] AmandaVizedom: Suggestion for additional evaluation factors (very important for some common applications): Interpreting slide 8 #10 to mean core logical inference: inference type, along with power (e.g., integrated support for probabilistic reasoning/ uncertainty); integrated lexical coverage (languages, support for NLP); integrated topic-relation coverage (facets, subtopic, support for document classification).

[10:01] Joanne Luciano: What are the research questions that the survey seeks to obtain (I didn't see that -maybe I missed it).

[10:01] SimonSpero: More academic: Tourangeau, R., Rips, L.J., and Rasinski, K. (2000), The Psychology of Survey Response, Cambridge: Cambridge University Press. http://www.amstat.org/sections/srms/ThePsychologyofSurveyResponse.pdf

[10:02] SimonSpero: http://www.amazon.com/Psychology-Survey-Response-Roger-Tourangeau/dp/0521576296

[10:02] doug foxvog: MichaelDenny brings up "assess[ing] query performance" for an ontology. Isn't this to a large extent a property of the inference engine used, not necessarily of the ontology?

[10:03] Joanne Luciano: suggest adding an "open question" -- to ask what survey respondents think is important but was not asked on survey.

[10:03] PeterYim: MichaelDenny: ideas and feedback solicited on the survey design between now and Feb-22

[10:04] MarcelaVegetti: +1 to Joanne's comment

[10:04] AmandaVizedom: @doug I'd say both (that is, it is a characteristic of a specific ontology in a specific reasoning environment).

[10:05] PeterYim: == MichaelGruninger presenting ... see: the [2-Gruninger] slides

[10:05] TerryLongstreth: @doug: For the extrinsics - Track B - Todd and I have discussed (argued) this. He won. Basically, if we can't see the ontology except as it presents behaviors, the interpreting mechanism (whether a person, inference engine or generated software) is part of the ontology.

[10:05] anonymous morphed into MikeBennett

[10:06] SimonSpero: Can you count how many different ontologies you have?

[10:08] anonymous morphed into PaulPope

[10:09] doug foxvog: re: Amanda's comment regarding bullet point "Assess the inferencing power of an ontology". The inferencing power of an *ontology* would be a feature of the kinds of expressions used: inheritance, argument constraints, properties of relations, existence and types of rules, expression of uncertainty, etc. Support for actually inferencing using such statements would depend upon the inference engine.

[10:16] Joanne Luciano: @SimonSpero Can you define "different"?

[10:18] AlanRector: There is work in Manchester on various similar metrics - contact Bijan Parsia - bparsia [at] cs.man.ac.uk

[10:18] Joanne Luciano: @PeterKim (and other organizers) - do we have any slots in the summit for "discussion" or any time in the face-to-face set aside for "discussion"

[10:20] MichaelDenny: re: all commenters, thank you -- the current set of software capabilities under each ontology development phase are concluded by a solicitation to the software provider to add their software capabilities that are not already covered. The objective of this survey is simply to compile an inventory of software resources that help users evaluate or promote ontology quality and fitness. There is no present intent to develop these findings into an ontology of evaluation/fitness factors or capabilities.

[10:21] PeterYim: @AlanRector - thank you, Alan, I believe BijanParsia is being contacted and invited to present at another track session ... right, LeoObrst / SteveRay (Track-A)?

[10:22] SimonSpero: Joanne: no.

[10:23] AmandaVizedom: @MichaelDenny: Is there a notion of modularity in the survey, as @MichaelGruninger just discussed I didn't see it, but could have missed)? If not, it seems well worth adding, since it can be important to suitability and there is so much recent and ongoing work on it. Yes?

[10:23] PeterYim: @JoanneLuciano - we will have "discussion" time at the end of each and every virtual session. I am sure there will be plenty of discussion time during the face-to-face Symposium too (although MikeDean & RamSriram, our Symposium co-chairs will be in the best position to provide specifics ... probably closer to the time.)

[10:27] AmandaVizedom: @MichaelGruninger: Can you say more about "Intended Models" as used in the COLORE sense, and how they might relate to things found "in the wild" - that is, in non-research projects?

[10:27] PeterYim: == JeanneHolm presenting ... see: the [3-Holm] slides

[10:31] SimonSpero: @MichaelGruniger: is there anyway to deal with conservative/definitional extensions without making CLIF's quantification of predicates unhelpful in the cases where they be most useful? Would making CL sorted be enough?

[10:31] AlanRector: On modules - there is an OWL module extractor as a web app on the owl.manchester.ac.uk site. It is also available by request to download but new features are being added regularly.

[10:32] FabianNeuhaus: @TIll. Michael mentioned "Mossakowski, T., Codescu, M., Kutz, O., Lange, C., and Gru ninger, M. (2013) Proof Support for Common Logic." in his presentation. Is the paper available?

[10:33] SimonSpero: @AlanRector: http://owl.cs.manchester.ac.uk/

[10:34] ElieAbiLahoud: @AlanRector, owl.manchester.ac.uk seems offline

[10:35] SimonSpero: @ElieAbiLahoud: see url I posted (there was missing cs)

[10:35] ElieAbiLahoud: @SimonSpero, THX

[10:35] MichaelGruninger: @SimonSpero: all of the work we have done so far has used essentially good-old fashioned FOL, and really hasn't exercised the full quantification of predicates in CL. We translated CLIF to Prover9 and do the theorem proving there. Once there is a full CL theorem prover, we can use it.

[10:36] MichaelDenny: @AmandaVizedom: Modules are covered under Management phase (as #4), but should probably appear in Build phase as well.

[10:38] MichaelGruninger: @AmandaVizedom: Suppose you are looking for an organization ontology with relations such as supervises, authorizes, manages. You could use an org chart from your company as an example of your intended semantics of the relations. We would translate such a chart into a set of relations with their extensions

[10:38] MichaelGruninger: (i.e. a model) and then search the repository.

[10:39] Joanne Luciano: Would like to bring everyone's attention to some tools from our lab at RPI (Tetherless World Constellation) that to take gov't data and turn it to RDF - there's also data quality tools available. This is the RPI data.gov site: http://data-gov.tw.rpi.edu/wiki Linked Data Quality Reports: Tim Lebo https://github.com/timrdf/DataFAQs/wiki

[10:43] FabianNeuhaus: @Michael: I think we need to scope the notion of "evaluation". Much of your presentation covered an analysis of the relationships between ontology (e.g., conservative extension, non-conservative extension) etc. This is interesting, but this does not tell you anything about the quality of an ontology or a set of ontologies. .

[10:43] SteveRay: Looks like the open government github platform shouldn't have a "t". Instead, it is https://github.com/opengovplatform

[10:47] MichaelGruninger: @Fabian: First of all, one way in which we evaluate the quality of an ontology is wrt intended models, and we use the relationships between the ontologies in the repository to do this verification. In other words, consistency alone is not enough. Second, in past few weeks, people have raised the idea of ontology comparison as being part of evaluation; it does not address quality per se, but it does evaluate two ontologies against each other e.g. what are their similarities and differences?

[10:48] Joanne Luciano: @jholm Semantic Ecology and Environmental Portal uses gov't data: http://tw.rpi.edu/web/project/SemantEco

[10:49] Jeanne Holm1: Thanks Joanne! Great pointer.

[10:51] PeterYim: == GavinMatthews presenting ... see: the [4-Matthews] slides

[10:52] FabianNeuhaus: @Michael, second point. Yes, this is why I think we need to scope it, in particular for the sake of the communique. If we use the word "evaluation" to mean "analyze ontologies or their use in systems under any conceivable aspect, including their structure, community use, and intellectual property rights " then we will have trouble to get to a reasonable communique.

[10:55] FabianNeuhaus: @Michael, first point. I understood the point, and agree that one very good way to evaluate ontologies is to use intended models.

[10:55] WillBurns: Intellectual property rights as an issue makes the assumption that the data itself has a context which uniquely identifies it and grants it the ability to be copyrighted. In this manner, I suppose we're still addressing the ontology aspect as a 1:1 storage and retrieval methodology whereby the files themselves are married to context. Unfortunately that doesn't *quite* work out in a situation where the volume of data is no longer inherent with context.

[10:57] SimonSpero: @willburns: In US, Australia, and possibly Canada, "mere data" cannot be copyrighted (beyond selection, coordination, and arrangement).

[10:58] doug foxvog: @Gavin: you show a "RT:" (related term?) link. Does the ontology specify the type of relation? Or is the number of relations very limited?

[10:58] WillBurns: @SimonSpero Correct. Specific an unique arrangements can be copyrighted, but the question becomes what happens when we remove all three up front and still manage to store it all?

[10:59] SimonSpero: @doug RT is usually defined as a residual category of all relations that aren't equivalence or hierachical.

[11:00] TillMossakowski: slide 6: what does "BT" mean? subconcept? why is "woman musician" a subconcept of "famous musician"?

[11:01] SimonSpero: @Till: That looks like an incorrect BT

[11:02] MichaelDenny: @FabianNeuhaus: By and large I was being driven by the full range of evaluation factors mentioned in the three tracks which includes, for example, "intellectual property rights". Such usage requirements that go into a user's choice to adopt particular ontologies as components may determine the overall quality or fitness of their ontology product. That said, I also hesitated to include evaluation of this factor as a candidate software capability. I chose to include it to be faithful to the tracks' identifications of factors.

[11:02] SimonSpero: @Till: ?N BT ?B means that everything about ?N is also about ?B

[11:03] doug foxvog: The ontology on slide 6 shows both "woman musician" and "musical performer" as subtypes of "famous musician". Yet "famous musical performer" is a subtype of "musical performer". The class "famous musician" probably is not mis-named since it is a subtype of "celebrity". There seems to be a problem here.

[11:04] SimonSpero: @Till: Diagnostic sentence frame - " It's about a woman musician but it's not about a famous musician" should be unacceptable

[11:05] MatthewWest: Sorry need to drop off now. Considerable food for thought here.

[11:05] SimonSpero: See: http://dc2008.de/wp-content/uploads/2008/10/04_spero_poster.pdf

[11:06] SimonSpero: What's needed is D. Allan Cruse control..

[11:08] doug foxvog: @Simon: yes, "related term" might be all that is needed for generic web searching. But it would be useful if a search for owners of a sports team would yield different links than for members of the team, or for supporters of the team.

[11:09] SimonSpero: @doug: Yes - RT can be specialized (he described a few, I think).

[11:09] AmandaVizedom: I would call everyone's attention to Gavin's slide 9. This level of testing is something almost no one imagines applying to ontologies, even in contexts where the rest of the software environment is so tested. Yet it's entirely possible, and once established, extremely valuable.

[11:11] PeterYim: == Q&A and Open Discussion ...

[11:13] MichaelGruninger: @Fabian: A similar comment can be made about ontology metrics i.e. what do they have to do with ontology quality? I get more insight from knowing that an ontology I have is inconsistent with another ontology than I get from knowing that there are 40 axioms with 5 relations, 12 classes with 3 siblings.

[11:13] ScottHills: @Jeanne Molm, on your slide 16, can you offer a link to more info about "ADMS-based metadata?"

[11:13] List of members: Akram, AlanRector, AmandaVizedom, AnatolyLevenchuk, BobbinTeegarden, CarmenChui, ClarePaul, DaliaVaranka, DanCerys, DmitryBorisoglebsky, doug foxvog, ElieAbiLahoud, FabianNeuhaus, Gavin Matthews, Jeanne Holm1, JimDisbrow1, Joanne Luciano, JoaoPauloAlmeida, JohnBilmanis, KenBaclawski, KevinSimkins, Lamar Henderson, LeoObrst, MarcelaVegetti, Max Petrenko, MichaelDenny, MichaelGruninger, MikeBennett, MikeDean, MikeRiben, OliverKutz2, PeterYim, Ram D. Sriram, ScottHills, SimonSpero, SteveRay, TerryLongstreth, TillMossakowski, ToddSchneider, VictorAgroskin, vnc2

[11:14] TillMossakowski: @Michael: what does "inconsist with" mean? The union being inconsistent?

[11:14] FabianNeuhaus: @Michael. I would (and have) made the same comment about these metrics.

[11:15] Jeanne Holm1: @ScottHills You can find out more about ADMS (a European Union initiative) go to http://joinup.ec.europa.eu/asset/adms/home

[11:15] Gavin Matthews: @TillMossakowski: BT (Broader Term) means sub-collection/sub-type. I'm not claiming our ontology is perfect. I wanted to show how we detected errors. :)

[11:16] ScottHills: @Jeanne Holm, thanks.

[11:16] SimonSpero: @Gavin: Is it restricted to ~genls?

[11:18] SimonSpero: @Gavin: In controlled vocabularies, BT can apply to genls, isa, or intentionally necessary parts (where the absence of a part requires explanation)

[11:19] SimonSpero: @Gavin: there are BTG, BTP, and BTI which are spec preds of BT

[11:20] MichaelGruninger: @Till: Yes, I meant that two ontologies are mutually inconsistent. For example, one ontology entails that time is discrete and the other ontology entails that time is dense.

[11:20] SimonSpero: @Gavin: but undifferentiated broader does not allow you to infer a genls

[11:21] Gavin Matthews: @SimonSpero: We use BT to mean roughly "genls". We distinguish ISA meaning "isa". For practical reasons, we permit a certain underspecification.

[11:24] Gavin Matthews: @doug foxvog: Yes, "RT" means related term. It is our vaguest relation, and we constantly work to specialise it. Relations like "plays role in", "located in", and "has theme" are considered to be sub-relations of RT, but in practice all of those horizontal links have approximately the same effect on disambiguation.

[11:24] AmandaVizedom: During last week's discussion of development methodology, we really talked very little about how evaluation can guide development. The evaluation Gavin discusses is used by ontology developers strongly; we might want to talk about that.

[11:25] MichaelDenny: @SimonSpero: Thanks for the pointers to survey methodology. I am hoping you will be able to help field this survey. I would like to come up with the simplest of implementations that will allow continuing contributions and visibility of results.

[11:26] Joanne Luciano: One simple way evaluation can be used to guide development is with the use of reasoners. Whenever we add a class or an assertion, we run the reasoner and check if we get what we expect. Of course this is by hand, and scaling this is another story.

[11:27] JoaoPauloAlmeida: yes

[11:30] SimonSpero: @Gavin: Underspecification is necessary- there have been attempts to create more specific coverings of associative relationships ( RT), but they all ran in to the problem that they were incomplete, and not useful for IR purposes

[11:32] SteveRay: Must run. Thanks.

[11:33] AlanRector: On tracking envirtonments: We have used Redmine successfully and found it easier than Bugzilla because of the good Wiki and other facilities. So many of t he issues that come up are more complex issues than simple "bugs".

[11:35] AmandaVizedom: Thanks Gavin!

[11:35] SimonSpero: @AlanRector: bug report - running the owltoskos web tool on the skos ontology crashed.

[11:35] Joanne Luciano: Please email me if you'd like to collaborate on a generalized ontology evaluation and development framework. jluciano @ rpi.edu Thanks everyone

[11:37] PeterYim: join us again, same time next week, for OntologySummit2013 session-06: "Ontology Summit 2013: Synthesis-I" - Co-chairs: MichaelGruninger & MatthewWest - http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_02_21

[11:38] Gavin Matthews: Thanks very much everyone. I've found it very interesting.

[11:38] AlanRector: @SimonSpero - please email Bijan Parsia - bparsia [at] cs.man.ac.uk - and Sean Becchofer - sean.bechhofer [at] manchester.ac.uk - they will redirect it to whomever is dealing with it now.

[11:40] PeterYim: -- session ended: 11:39 am PST --

[11:41] List of attendees: Akram, AlanRector, AmandaVizedom, AnatolyLevenchuk, BobSchloss, BobbinTeegarden, CarmenChui, Clare Paul, ClarePaul, DaliaVaranka, DanCerys, DmitryBorisoglebsky, ElieAbiLahoud, FabianNeuhaus, Gavin Matthews, Jeanne Holm, Jeanne Holm1, Jeanne Holm10, Jeanne Holm11, Jeanne Holm12, Jeanne Holm13, Jeanne Holm14, Jeanne Holm15, Jeanne Holm16, Jeanne Holm17, Jeanne Holm18, Jeanne Holm19, Jeanne Holm2, Jeanne Holm20, Jeanne Holm21, Jeanne Holm22, Jeanne Holm23, Jeanne Holm24, Jeanne Holm25, Jeanne Holm3, Jeanne Holm4, Jeanne Holm5, Jeanne Holm6, Jeanne Holm7, Jeanne Holm8, Jeanne Holm9, JimDisbrow, JimDisbrow1, Joanne Luciano, JoaoPauloAlmeida, JohnBilmanis, KenBaclawski, KevinSimkins, KevinSimkins1, KevinSimkins2, KevinSimkins3, Lamar Henderson, Lamar henderson, LeoObrst, MarcelaVegetti, Marcos, MarcosMartinezRomero, MatthewWest, MatthewWest1, Max Petrenko, MichaelDenny, MichaelGruninger, MikeBennett, MikeDean, MikeRiben, OliverKutz, OliverKutz1, O! liverKutz2, PaulPope, PeterYim, Ram D. Sriram, Richard Martin, ScottHills, SimonSpero, SimonSpero1, SimonSpero2, SteveRay, TerryLongstreth, TillMossakowski, ToddSchneider, VictorAgroskin, WillBurns, anonymous, anonymous1, doug foxvog, marcelaVegetti, vnc2

------