This is one of the "OntologySummit2013_Hackathon_Clinics" Projects ...    (3PCL)

(HC-03) Evaluation of OOPS!, OQuaRE and OntoQA for FIBO Ontologies    (3PCM)

Project Champions: MikeBennett, MariCarmenSuarezFigueroa, MariaPovedaVillalon, JesualdoTomasFernandezBreis, AstridDuqueRamos, SamirTartir    (3PET)

Event Date / Time: (Day-3) Sat 2013.04.13 ... see: [ details]    (3PGX)

Archived work output can be found here    (3R5B)

Logistical Details    (3R27)

Details (Saturday and Sunday)    (3QXY)

We have a one and three quarter hour kickoff session using the slides to present an overview of what we wanted to achieve. After this we moved onto Go To Meeting for three sessions of a little under two hours each.    (3R28)

1. Dial-in for the "Open Webcast" segment (Sat 9am - 10:30 Eastern US Time; 2pm - 3:30 London 3pm - 4:30 Europe)    (3QWE)

Details (Sunday only - we will use the chat room and /GoToMeeting only for this)    (3QXZ)

We reconvened on Sunday 14 April at 8am Pacific / 11am Eastern US / 4pm London / 5pm Europe to review results and GoogleDoc table. We ued the /GoToMeeting facility below but not the Conference Bridge.    (3QXX)

2. We will use the chat room facility throughout this session - this is at http://webconf.soaphub.org/conf/room/hc-03    (3QWH)

3. We will use /GoToMeeting (click through for phone and web link) for this session - details below.    (3QWI)

Working Document (Google Docs)    (3QXF)

Throughout the two days and beyond, we are working on the following document:    (3QXG)

https://docs.google.com/document/d/1ErbZV0IFj890lHFcnygsw6n93dxub1AamOu9oBnHdOo/edit    (3QXH)

Outcomes    (3R29)

Saturday 13 April    (3R2R)

In the initial kick-off segment we went through the [ slides ] slide presentation.    (3R2A)

Following this, we took a set of ontologies provided by the EDM Council from the FIBO Foundations proposed specification and applied these to the three sets of tools OOPS! OQuaRE and OntoQA. These are "Business Conceptual Ontology" models.    (3R2B)

Here is a link to the (unedited) in-session chat room:    (3R2S)

http://ontolog.cim3.net/file/work/OntologySummit2013/Hackathon-Clinics/HC-03_FIBO-OOPS-OQuaRE-Evaluation/OntologySummit2013_hc-03_chat-transcript_unedited_20130413a.txt    (3R2T)

Attendees Day 1    (3R2U)

Summary We had practical demonstrations of all 3 tools – OOPS! OQuaRE and OntoQA on real FIBO OWL ontologies, and looked at what measures the tools were showing us. We explored a couple of the metrics in depth. We looked at the OQuaRE table of quality measures and considered some changes and additions. I've tried to capture some of these in edits and additions to the table. Others will also be working asynchronously on this table.    (3R2C)

We had a couple of strong new ideas for quality measures: Jacobus Geluk suggested having a suite of SPARQL queries that can be used as regression tests or for test-driven agile development, along with example instance data. The OntoQA tool has some tests that can be applied separately to that test data. Simon Spero suggested that the ACE plug-in for Protégé can be used not only to provide business descriptions, but as a good quality measure, with a human in the loop, to test whether some of the assertions in the ontology really mean what we meant them to mean.    (3R2D)

On OOPS!, we imported the whole set of the FIBO ontologies we had, into a single test ontology and ran the pitfall scanner on this, analyzing all the imported ontologies. We established that most of the metrics were ones we would want to apply to the FIBO Business Conceptual Ontologies, not just operational ontologies. We can determine the required values for some of these – so for instance in Foundations we want a higher level of confidence in reusability and changeability.    (3R2E)

On OQuaRE we saw the full set of tests run against several of the FIBO Foundations ontologies. We looked at some of these in depth. These implement specific mathematical algorithms which are documented in the OQuaRE document, with cross reference to the business definitions of quality measures which they support. Some of these measures may have higher priorities in FIBO-Foundations than other ontologies, e.g. reusability.    (3R2F)

We had a demonstration of the OntoQA tool, on the FIBO Foundations ontologies. This has metrics for individual knowledge bases as well as for the conceptual model (these may also be applied to test data). These are presented in a spreadsheet, and some of them correspond to items in the OQuaRE table.    (3R2G)

Future versions of the tools may allow more automatic configuration of these settings. In OOPS! we would want to be able to replace their annotation measure (which looks at rdfs:label), with measures that look at SKOS definitions and the like. There are other configuration changes and new developments which would benefit FIBO, and / or the results can be filtered according to the specific requirements with what there is now.    (3R2H)

We had an in depth discussion on some FIBO modeling parameters, including the "Archetype" concept (similar to ontology design patterns) and how this helps with ontology quality. It should be possible to create code which would validate the application of archtypes and related ontology patterns. We also learned more about how the FIBO Archetype concept can be implemented in OWL more formally than it is at present.    (3R2I)

At the end of the session we agreed to work offline on the Google Docs document in preparation for the Day 2 session.    (3R2J)

Sunday 14 April    (3R2W)

On the Sunday we worked through the Google Docs document - all our workings are on that document:    (3R2K)

https://docs.google.com/document/d/1ErbZV0IFj890lHFcnygsw6n93dxub1AamOu9oBnHdOo/edit    (3QXH)

Attendees    (3R2X)

Summary    (3R2Z)

We went through the table one row at a time. For each entry we looked at the quality requirement defined for that entry, expanded this into specific measures to look at in the ontology. Then we reviewed the OQuaRE metrics in the table and added any other OQuaRE metrics that would apply.Then we added metrics from OOPS! and OntoQA which would measure some aspect of the ontology which would be relevant to the same overall quality requirement.    (3R2L)

We got to the end of the "Functional Adequacy" section. The next section will be Maintainability.    (3R2M)

We expect to have a follow-up call to continue where we left off with this document.    (3R2N)

Abstract    (3PCN)

This ontology clinic aims to explore the application of ontology quality measures to ontologies produced under the Financial Industry Business Ontology (FIBO) umbrella.    (3PCO)

In this clinic we will explore the application of the OOPS! and OQuare methodologies and tools to two styles of ontology developed under the FIBO umbrella: Business Conceptual Ontologies (BCOs) which are the FIBO standards themselves; and example "Operational Ontologies" derived from these for deployment in semantic technology applications.    (3PCP)

We would look to establish which types of measure should be applied to each type of ontology and apply the relevant tools and techniques to these. In the case of OQuaRE, these measures will be applied in two ways: 1) application of the complete quality model; 2) application of the OQuaRE subcharacteristics and metrics relevant for FIBO evaluation, with the possibility of modifying the existing associations subcharacteristics-metrics.    (3PCQ)

From this activity we hope to make the first steps towards defining a formal quality process for the future development of formal standards under the FIBO umbrella, a set of quality assurance parameters for users who need to extend the FIBO BCO locally for their own conceptual semantic modeling, and a set of guidance notes, validation and verification techniques etc. for developers of semantic technology applications based on the FIBO standards. We will evaluate to what extent OQuaRE could be a start point for this quality process.    (3PCR)

see: [ Project document ] . [ slides ]    (3PGY)

Collaborators    (3PCS)

OOPS!    (3PCT)

OQuaRE    (3PCV)

OntoQA    (3PCX)

Others    (3PCZ)

Ontologies Involved    (3PD1)

There are two distinct kinds of FIBO ontology:    (3QW0)

Of these, will specifically bring the Business Conceptual Ontologies to this Clinic, as the following:    (3PD5)

Objectives / goals    (3PDH)

FIBO Background    (3PDI)

FIBO is being developed as a series of "Business Conceptual Ontologies" (BCO) for concepts in the financial industry, that is, ontologies which represent industry terms, definitions and relationships at the level of conceptual models. Conceptual models, by definition, should not reflect application constraints. From these, we anticipate that users would derive operational ontologies for specific use cases, which would of course be subject to the relevant application constraints.    (3PDJ)

An open question in the development of FIBO is what ontology quality measures should be applied to the "Conceptual" ontologies, and which of the established OWL modeling best practices are applicable to such an ontology. That is, which requirements of semantic technology applications should be applied to the conceptual ontologies without compromising their requirements as conceptual models.    (3PDK)

To complicate this question further, the BCOs are intended to be presented to business domain subject matter experts for validation, and local extensions of the BCO are intended to be understood and maintained as a business domain asset. In order to support business-friendly presentation in the currently available modeling tools, some compromises have been made in the way that the OWL language is used. Some of those compromises could be reversed once there are better ways of presenting these ontologies to a business audience.    (3PDL)

Meanwhile, we expect potential users of the standards to derive "operational ontologies" from the conceptual ontologies, just as a conventional application developer would develop logical designs from conceptual models such as requirements catalogs. These operational ontologies must of course be subject to the quality requirements of any application (validation and/or verification of the delivered item against the stated business requirements), and since they are OWL ontologies, must also be subject to the quality constraints that are applicable to operational OWL ontologies.    (3PDM)

OQuaRE Background    (3PDN)

OQuaRE is a framework for Ontology Quality Requirements and Evaluation based on ISO/IEC 25000:2005, the standard for Software Quality Requirements and Evaluation. OQuaRE defines intrinsic and extrinsic quality criteria in terms of quality sub-characteristics. OQuaRE aims to define all the elements required for ontology evaluation: evaluation support, evaluation process and metrics. The current version of OQuaRE includes, so far, the quality model and the quality metrics:    (3PDO)

1. The quality model is composed of a set of quality characteristics such as structural, functional adequacy, maintainability etc. and its associated sub-characteristics such as reliability, reusability, availability, redundancy, consistency, etc.    (3PDP)

2. The quality metrics have been taken from the state of the art in ontology, such as Depth of subsumption hierarchy, Class Richness, Tangledness etc.    (3PDQ)

Complete definition of OQuaRE is available at:    (3PDR)

Objectives    (3PDU)

The objectives of this clinic are as follows:    (3PDV)

A: Evaluation of FIBO Business Conceptual ontologies    (3PDW)

B: Consider criteria for future Operational Ontologies    (3PE1)

Deliverables    (3PE7)

Remarks    (3PED)

We see this clinic as a vital first step in our development both of the formal methodologies for FIBO standards development and of the conformance points and developer guidance necessary for end users to make practical use of FIBO in semantic technology-based applications. The tools and techniques which are applied in this clinic (OOPS!, OQuaRE and any others which come to light) will likely form a part of those formal processes going forward.    (3PEE)

Resources / References:    (3PEF)

OOPS!    (3PEG)

OQuaRE    (3PEL)

OntoQA    (3PEP)