ontology-summit
[Top] [All Lists]

[ontology-summit] [quality] Some Terms and Definitions

To: Ontology Summit 2012 discussion <ontology-summit@xxxxxxxxxxxxxxxx>
From: Mike Bennett <mbennett@xxxxxxxxxxxxxxx>
Date: Sun, 12 Feb 2012 16:07:49 +0000
Message-id: <4F37E3D5.4050201@xxxxxxxxxxxxxxx>
The term "quality" may mean different things to different people. 
I want to draw out two of those possible senses and kick off some 
discussion about the possible connections between them.    (01)

Here I am not trying to introduce anything new, I'm trying to 
describe what's in the relevant literature. So these definitions 
are not really discussion points, but people are welcome to 
clarify and correct any vagueness with reference to the literature.    (02)

The two senses I would characterize as:    (03)

1. Quality in the dictionary sense of the word: how good 
something is;
2. Quality in the sense used in the industrial term "Quality 
Assurance".    (04)

In the second sense, Quality Assurance (QA) simply describes an 
approach whereby one can ensure (and demonstrate that one has 
ensured) what the qualities of some deliverable are. QA is not 
about making better things, it is about better making things. 
It's about metrics and measures.    (05)

The basic definitions and parameters of QA are defined in QA 
standards such as ISO 9000. They are also described in national 
standards such as BS5750 or the German SUV system.    (06)

Quality Assurance is about being able to "demonstrate control". 
That is, a firm has formal processes in place by which they 
manage their deliverables. These processes are designed and 
optimized to ensure consistency in the process of creating 
whatever their deliverables are.    (07)

We could refer to (1) and (2) as "Qualitative Quality" and 
"Quantitative Quality" (you see why the choice of the word 
Quality wasn't a good one!)    (08)

To illustrate the difference: Macdonald's Golden Arches has 
arguably the best QA system of any restaurant chain. However, few 
would argue that they make the best burgers. Qualitative quality 
doesn't necessarily follow from quantitive quality. What does 
follow is the business value in consistency. Anyone who finds 
themselves in a strange city with hungry children in tow will not 
take them to an unknown burger chain where they food may or may 
not be excellent; they will take them to the place where they 
know exactly what they will get. This is the business value of a 
good QA system.    (09)

An accurate if tongue in cheek characterization of QA is given by 
Scott Adams in the Dilbert cartoons: "Say you're going to produce 
crap; produce crap; prove you produced crap" (please excuse the 
language). In fact, if you substitute anything at all (any 
quality at all) where Adams has the word 'crap', you have a 
reasonable summary of how QA works. You formally specify the 
things you are going to produce; you produce those things 
according to a well defined (and auditable) set of formal 
processes, and you end up with an auditable set of records 
demonstrating exactly how those things were produced. This is why 
the phrase "Demonstrate control" is the heart of QA.    (010)

Now you could of course specify that you are going to produce 
excellent things. Rolls Royce for example has a reputation for 
doing this. Once the firm knows where its market is, and what 
sort of things it wants to and is able to produce for that 
market, then it simply formalizes the qualities that it wants to 
produce in its products or services.    (011)

How might we apply this to ontologies?    (012)

To the extent that different qualities of an ontology can be 
formally specified, these can be input to a formal, industrial QA 
process.    (013)

To the extent that one can formalize Quality (1), that is "what 
makes a good ontology?", one can apply Quality (2) QA to ensure 
that ontologies are produced which comply with those requirements.    (014)

There are other things one might want to ensure about the 
deliverable ontologies. For example if one is extending an 
ontology which is built according to certain microtheories, one 
would want to ensure that those microtheories are consistently 
applied in the new material. Sometimes this can be detected by 
simple measures such as consistency checking; sometimes perhaps 
it may not.    (015)

Validation and Verification:    (016)

These terms also have specific meanings when applied to formal, 
industrial QA systems, which are narrower than their dictionary 
sense. In software deliverables these translate to:    (017)

Verification: Testing the deliverable against its formal 
specification. Test cases verify, against each functional 
requirement in the specification, that that requirement is met.
Validation: Ensuring that the system as specified and delivered, 
actually meets the customer's expectations (typically User 
Acceptance Test). Here we find out whether the specify-build-test 
set of activities actually resulted in what the customer wanted.    (018)

How these are implemented depends very much on the technology and 
the architecture of course. There may be interesting challenges 
in applying both of these for ontologies.    (019)

Design review / peer review    (020)

The word "Peer review" has a slightly different meaning in 
industry, to that which it has in academia. A QA process does not 
just consist of tests; frequently one designs into the process 
some kind of review activity (typically a meeting), often called 
a design review, also referred to as a form of peer review. This 
is where some deliverable item in the QA process (usually a 
design specification) is presented to a group of people who would 
be in a position to critique it and verify that it is fit for 
purpose. Part of the design of the QA process is to consider the 
knowledge requirements for various forms of design review (e.g. 
system architects, the QA department, coders in the appropriate 
language), and ensure that at the relevant point in the process, 
the right knowledge is gathered around the table for that 
deliverable to be signed off. Another form of this is the code 
walk through.    (021)

Again, there would be similar but different applications of this 
thinking in the various stages in development of an ontology. 
Perhaps you might review the taxonomy (or apply some of the 
structural tests described in the literature), before developing 
the whole ontology. Or not. A QA process has to be designed, it 
doesn't just happen.    (022)

People often think that the imposition of a formal QA regime on a 
technical development implies that some "waterfall" development 
model must be applied. This is not the case. If you look at other 
formal processes such as the Rational Unified Process (RUP), you 
will see the same linkages between deliverables, applied such 
that the deliverables can be changed and delivered to a faster 
time scale. Similarly the "Agile" approach, when examined in 
detail, shows that those linkages are still in place.    (023)

For ontologies, if these are to be updated freuquently and in 
real time, even more imaginative ways of applying the basic 
parameters of QA might be required. For example, one might choose 
to use a "gardening" type of approach, whereby the QA is applied 
after the event by making or proposing updates or corrections. 
Here the QA process followed by Wikipedia is a good example of 
how this may be done.    (024)

Anyway I realise that not everyone has been thinking about 
Quality (2) / quantitive quality, and this is just one part of 
the bigger picture. But I hope that, for those who have spent 
less time in an industrial environment, this is a useful guide to 
what those of us from such an environment usually mean when we 
talk about Quality Assurance.    (025)

The key point is that the more things about quality in the 
natural language, qualitative sense, can be formalized, the more 
these can be made use of in formal, industrial QA.    (026)

Hope this is helpful,    (027)

Mike    (028)




-- 
Mike Bennett
Director
Hypercube Ltd.
89 Worship Street
London EC2A 2BF
Tel: +44 (0) 20 7917 9522
Mob: +44 (0) 7721 420 730
www.hypercube.co.uk
Registered in England and Wales No. 2461068    (029)


_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/   
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/  
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2012/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2012  
Community Portal: http://ontolog.cim3.net/wiki/     (030)
<Prev in Thread] Current Thread [Next in Thread>