Yes, the so-called "Red Team" is at minimum the small group of SMEs who have
typically helped you go through the process. But again, you need to describe to
them the resulting semantics you have captured, not just show them logical
expressions. If you can't state what you think their semantics is, and state it
accurately, they won't trust you or the ontology. The same team helps gauge
the results of the competency questions. (01)
I would also add a second "Red Team", if you have that luxury (or glom this
part onto the first Red Team, let's call it Red Team*) and that is another
group of ontologists, who can understand your current ontology, and can
understand from your (and your SMEs') statement of the SME semantics whether or
not the ontology accurately represents that. (02)
It's hard enough to get a Red Team, let alone a Red Team*. Often you have to
substitute ontologist internal review to get the ontologist aspect of Red
Team*, meaning that if you and I develop ontology O, then other ontologists not
involved in developing O formally review it. Think "code review". We used to
have many more code reviews in the past then we do today. Ontology engineering
does not throw software engineering out the door, but instead extends it. (03)
Thanks,
Leo (04)
-----Original Message-----
From: ontology-summit-bounces@xxxxxxxxxxxxxxxx
[mailto:ontology-summit-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Jack Ring
Sent: Thursday, March 03, 2011 3:19 PM
To: Ontology Summit 2011 discussion
Subject: Re: [ontology-summit] An example of the worth of ontology development (05)
Leo,
Makes sense to me. Pls say a few words about the Red Team that vets the work of
the "...small and ontology savvy..." ontology team.
Jack
On Mar 3, 2011, at 1:10 PM, Obrst, Leo J. wrote: (06)
> It is usually the case that you don't want to show the SMEs the details of
>the ontology, since they will not know how to interpret what they see. We had
>this issue in spades at VerticalNet, e.g., where the application actually
>exposed the ontology as categories in the user interface, rather than present
>a better GUI in terms of the audience. We dubbed this the representation vs.
>presentation issue. Applications ALWAYS need to provide their own application
>view to the user, not just expose the underlying technical model, especially
>for ontologies, where the plumbing could be axioms and logical
>assertions/rules.
>
> Now if your SME is indeed also an ontologist, or your ontology tools enable a
>SME to do the right thing without showing the hairy details (and no tool I've
>found does that), then you can show the representation. Otherwise, always show
>the presentation view.
>
> You'll scare the hell out of SMEs because they will interpret your ontology
>class/property "label" as a word that they and their comparable users don't
>use. And what's all these backward Es and upside down As and arrows? In much
>the same way, you don't allow end users to develop crucial enterprise
>databases themselves, you don't allow them to develop ontologies unless they
>are also ontologists. Term \= Concept. Application category \= ontological
>axiom.
>
> However, you do need to show SMEs that the results of the application of the
>ontology are good, accurate, and granular. E.g., for database integration
>efforts, you need them to confirm that the competency questions are answered
>well, accurately, and at the right level of granularity. Results, they can
>judge well. Plumbing, they can't.
>
> Furthermore, no ontology develop by committees! The committee is the
>stakeholder association, not the technical ontology development team, the
>latter of which should be small and ontology savvy. The stakeholder committee
>needs to know that their semantics are captured, and this is best done by
>application results and a strong data/ontology dictionary/lexicon/glossary, in
>addition to audience-tempered briefings.
>
> Thanks,
> Leo
>
> _____________________________________________
> Dr. Leo Obrst The MITRE Corporation, Information Semantics
> lobrst@xxxxxxxxx Information Discovery & Understanding, Command & Control
>Center
> Voice: 703-983-6770 7515 Colshire Drive, M/S H305
> Fax: 703-983-1379 McLean, VA 22102-7508, USA
>
>
>
>
> -----Original Message-----
> From: ontology-summit-bounces@xxxxxxxxxxxxxxxx
>[mailto:ontology-summit-bounces@xxxxxxxxxxxxxxxx] On Behalf Of John F. Sowa
> Sent: Thursday, March 03, 2011 2:33 PM
> To: Amanda Vizedom
> Cc: Ontology Summit 2011 discussion
> Subject: Re: [ontology-summit] An example of the worth of ontology development
>
> On 3/3/2011 12:09 PM, Amanda Vizedom wrote:
>> If we want SME validation of an ontology, we need to produce a
>> validation test that uses the formalized ontology to do something, and
>> that something needs to have results within the SMEs area of
>> understanding and practice. If we want the ontology to be multipurpose
>> and reusable, we should have a variety of such tests. Rather than
>> relying on yet another semi-intelligible visualization of the ontology,
>> and SME feedback on whether this seems right, we should incorporate
>> functional testing into the ontology development and validation workflow.
>
> I very strongly agree with that point.
>
> For an example of a tool that we are currently developing at VivoMind,
> see slides 26 to 30 of the following presentation:
>
> http://www.jfsowa.com/talks/futures.pdf
>
> Slides 26 to 28 describe the proto-ontology extractor, which works with
> a SME to develop an ontology from a set of documents about some subject.
>
> The SMEs get immediate feedback from it by seeing that their changes
> to the ontology produce improved results: more accurate answers to
> their questions about the documents, and improved precision and recall
> of the desired data.
>
>> Part of the SME validation should be search & retrieval tests on the corpus,
>> to see whether the ontology creates results that reflect the SME knowledge.
>
> We are still at an early stage of development. But we demonstrated
> better results in comparison with other tools that the SMEs had used.
>
>> And a bit a time and effort need to go into developing these up front.
>> IME that up-front time and effort is not often done, and budgets don't
>include it.
>
> Speaking as a partner in a small business, we can only do what
> our clients pay us to do. They usually pay for results, not for
> studies that show how or whether we could produce results.
>
> John
>
>
>
>
> _________________________________________________________________
> Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
> Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
> Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
> Community Files: http://ontolog.cim3.net/file/work/OntologySummit2011/
> Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2011
> Community Portal: http://ontolog.cim3.net/wiki/
>
> _________________________________________________________________
> Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
> Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
> Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
> Community Files: http://ontolog.cim3.net/file/work/OntologySummit2011/
> Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2011
> Community Portal: http://ontolog.cim3.net/wiki/ (07)
_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2011/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2011
Community Portal: http://ontolog.cim3.net/wiki/ (08)
_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2011/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2011
Community Portal: http://ontolog.cim3.net/wiki/ (09)
|