Re: [ontolog-forum] SME (subject matter experts) and Ontology developeme

[Chris Partridge <partridge.csj@xxxxxxxxx>]

Hi,

If someone has already mentioned this, then apologies for repetition.

There is quite a bit of philosophical literature on expertise, some of which I have found useful, which mostly backs up the points already made.

One extremely useful result quoted (understandably) by Andy Clarke is the flyball effect (see below: Shaffer DM, McBeath MK (2005) Naive beliefs in baseball) - which (perhaps) gives an example of how experts construct inaccurate models of their expertise.

Another pertinent, though maybe controversial, point is Searle's that one becomes an expert only when one can no longer introspect the expertise. In other words, lack of awareness is a mark of expertise.

The general thrust is that experts are often not good at articulating their expertise (Searle is quite scathing about this).

In case anyone is interested in this literature, I have copied in the references to texts I have found useful below. This is an active area, so the references may be out of date, but I am sure that if you start with these, then a bibliography search will get you up to date.

I'd be interested in other references, if people have time to share, as there does not seem to be sufficient cross-fertilisation in this area.

Best,

Chris

A Novel Ontological Approach to Semantic Interoperability between Legacy Air Defense Command and Control Systems - Published in 'Int. J. Intelligent Defence Support Systems, Vol. 4, No. 3, 2011'.

http://www.borosolutions.co.uk/research/content/files/IJIDSS040302%20PARTRIDGE.pdf/view?

Section 10.4 -  10.4 The nature of SME expertise 

2012-08 - Guidelines for Developing Ontological Architectures in Modelling and Simulation (Book Chapter)

Authors: Chris Partridge, Andy Mitchell, and Sergio de Cesare. Chapter 2 in Ontology, Epistemology, and Teleology for Modeling and Simulation: Philosophical Foundations for Intelligent M&S Applications. Series: Intelligent Systems Reference Library, Vol. 44. Andreas Tolk, (Editor). ISBN 978-3-642-31139-0. Publisher: Springer. Book available at http://www.springer.com/engineering/computational+intelligence+and+complexity/book/978-3-642-31139-0 

http://www.borosolutions.co.uk/research/content/files/2012%20-%20Partridge%20C%20et%20al.%20-%20GUIDELINES%20FOR%20DEVELOPING%20ONTOLOGICAL%20ARCHITECTURES%20IN%20MODELLING%20AND%20SIMULATION.pdf/view?

74. Ryle G (1949) The concept of mind. Hutchinson, London

75. Habermas J (1998) On the pragmatics of communication (trans: Cooke M). Studies in contemporary German social thought. MIT Press, Cambridge, MA

76. Stanley J, Williamson T (2001) Knowing How. Journal of Philosophy 98 (8):411-444. doi:10.2307/2678403

77. Polanyi M (1966) The tacit dimension. Terry Lectures 1962, 1st edn. Doubleday, Garden City, NY

78. Searle JR (1983) Intentionality, an essay in the philosophy of mind. Cambridge University Press, Cambridge Cambridgeshire ; New York

79. Collins HM, Evans R (2007) Rethinking expertise. University of Chicago Press, Chicago

80. Collins HM (2010) Tacit and explicit knowledge. The University of Chicago Press, Chicago

84. Shaffer DM, McBeath MK (2005) Naive beliefs in baseball: systematic distortion in perceived time of apex for fly balls. J of Experimental Psychology Learning Memory Cognition 31 (6):1492-1501. doi:10.1037/0278-7393.31.6.1492

On 15 March 2015 at 09:01, Matthew West <dr.matthew.west@xxxxxxxxx> wrote:

Dear John,
It looks like we are pretty much agreed. Some comments and responses below.
Matthew West


Dear Matthew,

Your answers to Robert's questions are a very good summary of today's state
of the art.  They reflect your many years of work on the theory and practice
of ontology development and application.

I agree with them.  But I'd also like to comment on issues that show the
need for further developments in theory, practice, and supporting
methodology, resources, and tools.

> Q1: Are there any sources that discuss the role of SMEs in ontology
> development?
>
> */[MW>] This is something I should have written about, but find I
> haven't. So my experience is that it takes a team to develop a good
> ontology. I would choose the team to have 3-4 members. You preferably
> need two people who are subject matter experts, and two who are expert
> at ontology development (some team members may combine these skills).
> The reason is that it takes critical review and questioning of an
> ontology as it develops to achieve quality, and tricky problems
> benefit from multiple inputs./*

I agree.  But I would emphasize that really good SMEs who can articulate the
implicit assumptions in their expertise are rare.  The few who have those
abilities are in high demand and can command high fees for their work.
Since your employment at Shell gave you access to experts in the industry,
you had a luxury that is rarely available in most projects.
[MW>] I agree. That quality of people in Shell is something I am very
grateful for in my career. However, even there the SME's were generally
unable to articulate the implicit assumptions in their expertise.

The task of a knowledge engineer is closely related to Socrates' method of
asking pointed questions that expose the critical issues.  That is another
rare skill.
[MW>] Exactly so. And this is what the ontologist has to bring to the task,
and what (in my view) determines the quality of the ontologist. However,
there is no magic in this process. I can teach someone (and have taught
many) how to do it if they have a basic ability and aptitude.

Problem:  How can we design tools and methodologies that could help lesser
mortals with smaller budgets achieve good results?
[MW>] I think the methodologies exist. The key is to take what I would call
an evidence based approach. This means not simply taking an SME's word as
law, but seeking evidence (in documentation, data, the real world) that this
is the case. At least Chris Partridge and myself have methodologies (not
necessarily well documented) that use this approach.

> Q2: If a domain ontology, say of some medical of biological subject,
> does not have one or more practicing SMEs, do you consider that to be
> a problem?
>
> (I do)
>
> */[MW>] Agreed. If a domain is exhaustively documented and is wholely
> unremarkable, you might be able to produce a good result without an
> SME, but I don't recall a project when I did not have some questions
> that required SME input. If nothing else you need to know what the
> objectives/purpose of the ontology is./*

Yes.  But this raises the same issues I mentioned for Q1.

> Q3: To what degree do you believe non-SME ontology developers
> negatively have or do affect the representations of the domain, either
> via (a) (un)intentionally trying to represent domain
> content/entities/phenomena using their own metaphysical world view or
> that of a particular stripe in philosophy despite it clashing with
> domain science, or (b) limited understanding of the domain affecting
> the faithfullness/accuracy of the representation and models?
>
> */[MW>] a) I would always recommend and use an upper level ontology as
> an analysis framework. Done properly it means you ask questions that
> reveal assumptions and context that significantly improves the ontology.
> It is one of the main contributions that an ontologist makes. It
> matters less which upper ontology you use (as long as it is competent
> and you are expert in it). I would be surprised if one clashed with
> the domain science. That would raise alarm bells for me, not least
> because an upper ontology makes ontological, not scientific
> commitments. What you really need to do is understand your ontological
> commitments. These answer questions like:/*

Doug Lenat has also said that the upper ontology is less important than the
middle and lower levels.  I agree.  But that raises some serious questions
about the quality, relevance, and usability of the ontologies.

In particular, I disagree with the (often repeated) claim that ontology is
independent of science.  The best ontologies were developed by
scientists:

  1. Aristotle's father was a physician, and he wrote more voluminous
     and detailed writings on biology than on metaphysics.  Galen was
     inspired by Aristotle's combination of theory and observation.
     In the 17th c, William Harvey said that Aristotle's writings
     on embryology were unsurpassed.

  2. Throughout his long academic career, Kant taught courses on
     Newtonian mechanics as well as logic and metaphysics.  In fact,
     Kant proposed the hypothesis that the planets of our solar system
     developed from a cloud of dust surrounding the sun.  He recognized
     that a rotating cloud was unstable, and it would collapse into
     a disk and later into lumps.  The primary motivation of his
     metaphysics was the need to reconcile science and common sense.

  3. Charles Sanders Peirce had published research papers in chemistry,
     physics, astronomy, and mathematics.  His paper on "Logic Machines",
     published in the American Journal of Psychology in 1887, was cited
     by Marvin Minsky as a pioneering article on artificial intelligence.

  4. Alfred North Whitehead was a mathematician who also taught courses
     on theoretical physics.  He developed a "process" ontology that made
     processes more fundamental than objects.  He defined an object as
     a slowly changing process.  His ontology is more compatible with
     modern physics than any ontology based on physical objects.

For further discussion, see http://www.jfsowa.com/pubs/signproc.htm

[MW>] I agree. You are using "science" in a broader sense than I was
(physics, chemistry and biology). I would consider ontology to be that
broader term, but it's an "Alice in Wonderland" world.

> -Can a particular (something like you and me) also be a class, or are
> classes and particulars disjoint?
>
> -Do things that do not exist now exist at all? So can I talk about
> historical (or future) figures directly, or do I need to talk about
> what is written about them only.
>
> -How is change over time dealt with? (states of things and temporal
> relationships are two possibilities)
>
> -Can some objects have gaps in their existence? (things like
> organizational positions that can have vacancies and systems
> components that can be replaced)

These are very general *scientific* questions that require a precise
notation for stating definitions and reasoning about them:  logic.
[MW>] See above. I would not call these *science* since they are not
physics, chemistry or biology, but ontology.

> */[MW>] /*There are plenty more of these kinds of questions, but at
> the very least you will want to address them consistently if you are
> to produce a quality product. What you can't do is not make
> ontological commitments, the only question is whether you are aware of
> them or not, and whether they make a sensible system of commitments.
> An upper ontology is a bundle of answers to these (and other)
> questions. If domain experts are claiming particular choices are
> necessary in their domain, they are probably mistaken, and it is worth
> taking them through the ontological commitments they are implicitly
> making, and exposing them to some alternatives.
>
> b) Limited understanding of the domain is probably an advantage, since
> it means you will not be making the assumptions that SMEs will make
> from being immersed in the subject. The purpose of an ontologist in
> the team is that they are expert in building ontologies, the purpose
> of the SME in the team is that they are expert in the area for which
> the ontology is being created. You are not going to get a good result
> without both of these present.

I agree that many people who are deeply immersed in a subject can't "see the
forest for the trees".  That is partly the result of an educational and
professional system that rewards specialization.

But many people in every profession have a broader understanding of the
issues.  Unfortunately, there aren't enough of them to work on every project
in ontology, and their consulting fees tend to be too high for most
projects.
[MW>] It is not just what you know, it is who you know. I have often had
SME's on projects that might not have been deep experts themselves, but knew
( and had access to) those who were, or had a research mentality that meant
they could seek out the evidence necessary to support the analysis process.
This is much less demanding.

Conclusion:  We need better methodologies and resources -- supported by
tools that enable ordinary humans to learn and use them.
[MW>] I agree. I've outlined the methodology approach I take (there is much
more detail of course). On tools, for development, I have yet to see
something improve on the flexibility of a whiteboard and digital camera.
However, this is no use for documentation. What you need there is a
diagramming tool that:
- supports the diagramming of instances as well as classes (including
classes as instances of other classes)
- supports different views of the overall ontology, plus having an overall
view you can refer to
- supports detail hiding and expansion (to aid explanation/presentation)
- supports different shaped boxes and line styles associated with different
categories (e.g. subtype/supertype relationship vs general relationship,
class vs particular)
- supports documentation of textual definitions
- supports documentation of rules using logic up to FOL (CL/IKL equivalent)
I'm sure there are some other things I have missed, and as usual I have not
tried to include the things we take for granted in such tools, such as being
able to place and arrange objects.

Regards

Matthew West
Information  Junction
Mobile: +44 750 3385279
Skype: dr.matthew.west
matthew.west@xxxxxxxxxxxxxxxxxxxxxxxxx
http://www.informationjunction.co.uk/
https://www.matthew-west.org.uk/
This email originates from Information Junction Ltd. Registered in England
and Wales No. 6632177.
Registered office: 8 Ennismore Close, Letchworth Garden City, Hertfordshire,
SG6 2SU.

John

_________________________________________________________________
Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/
Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/
Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
Shared Files: http://ontolog.cim3.net/file/ Community Wiki:
http://ontolog.cim3.net/wiki/ To join:
http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J



_________________________________________________________________
Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/
Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/
Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
Shared Files: http://ontolog.cim3.net/file/
Community Wiki: http://ontolog.cim3.net/wiki/
To join: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J

_________________________________________________________________
Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/  
Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/  
Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
Shared Files: http://ontolog.cim3.net/file/
Community Wiki: http://ontolog.cim3.net/wiki/ 
To join: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J    (01)