Pat,
I would like to thank you again for your comments. We tried to
address the issues that you raised. E.g., we cut down on the
"management speak" and clarified that the reuse of an upper
ontology is only an example how one can specify certain ontology
design decisions.
Most importantly, we tried to make the document as "dogma-proof"
as possible. We stress even in stronger terms that for the sake of
having a comprehensible presentation the "life cycle model"
abstracts of all the real-world differences of how ontologies are
developed. And in several places (including the executive summary)
we assert that because of the lack of maturity of ontology
engineering all recommendations in the communique must be
considered preliminary.
I agree that this might not prevent graduates from the
Dilbert-school of management to try to apply some of the
recommendations in the document blindly and make a mess of it. But
bad managers will mismanage their projects with or without our
communique.
At the same time, for many other people the communique will,
hopefully, provide some useful guidance that is based on the input
from large and diverse group of people with tons of experience in
ontology development.
Best
Fabian
PS: The final version of the communique is available here:
http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013_Communique
On 4/24/13 12:09 AM, Pat Hayes wrote:
On Apr 23, 2013, at 10:42 AM, Fabian Neuhaus wrote:
Pat,
It is always a joy to read your emails. Even if one is the target of your scathing remarks. :-)
Since I came up with the initial outline of the communique, I guess I am mainly to blame for the development of the "managerial fantasy". So, let me try to defend it. You call the document a phantasy, because it "is not based on observations about how the work is ACTUALLY done." And you are right.
Well, apparently it IS partly based on experiences with Cyc, so I have egg on my face for using that as my snarky example :-).
Here is my draft for a communique that is based on how the work is actually done:
"In spite of a lot of literature on the subject, ontology evaluation plays no important role in the development of ontologies today (except for checking for logical consistency). The main reason for this situation is that the development of ontologies is currently not practiced as engineering activity, but as a form of art, where design decisions are more driven by schools of taste than by empirically validated methodologies. Since ontologies seem to be pieces of art, we suggest a metric that has been proven successful in the movie industry for a very long time: so give 2 thumps up for your favorite ontology!"
I am not sure whether you would have endorsed this version of the communique, but at least it would be short. :- )
I would have endorsed this, yes. Such honesty is refreshing, and rare :-) But more seriously, I would prefer to avoid this entire topic altogether. In my opinion it is the social equivalent of a no-op: it seems to be about something but in fact there isn't any actual topic for there to be a communique about.
Instead of describing the current situation of ontology evaluation, we are trying to do something more productive. Namely, to produce a consensus on how ontology development should be done, and which forms of evaluation should happen during the life cycle.
And what does "should" mean, here? Should be done, or should happen, in order to achieve or improve what, exactly?
So its a fantasy, all right. But some peoples fantasies are other peoples visions, and this vision is an attempt to merge the recommendations by the participants in the Ontology Summit on how to organize the development of an ontology. Hence, the document might not be an empirical description of what is happening currently, but it is grounded in the experience of many people who have developed ontologies.
It would be *much* more convincing if it actually cited some of this experience. Particularly along the lines of: y'all should do (/not do) this ... because when we didn't do it (/did it) in this example... this terrible thing happened.... . One or two such anecdotes would be worth volumes of managerspeak.
You argue that developing such a document can cause harm, because it might be used by managers to impose work habits on other people for no good reason. I agree that this is a problem. Any well-intentioned recommendation might be turned into a dogma that is imposed blindly.
It will (either be ignored, or) become dogma, this I will guarantee. It has all the Dilbertian signs: long-winded paragraphs of content-free platitudes written in managerspeak; complicated but utterly meaningless diagrams of "phases" and "cycles", vague but rather threatening promises of how not paying attention to something unspecified will bring about unstated but clearly bad things. (Every book or paper about managing work has some diagram of arrows going in a circle, called a "cycle". This is complete nonsense. Where in the actual world of development of anything, did anyone find a real example of a "cycle"? This whole idea of "cycles" is there only to give managers something to write a report about to other managers about how many times their people have gone "around" it.)
However, you ignore the potential benefits of the document. It is intended to provide help to people who are trying to build (or use) ontologies and are looking for recommendations on how to organize themselves by more experienced ontologists.
So, I guess, it boils down to the question: Do we know enough about the ontology life cycle
You are presuming that ontologies have "life cycles". To me, this sounds already like someone who has drunk the Kool-aid (as John Sowa is fond of saying.)
to make recommendations about development and evaluation that are of any value? You seem to believe that we don't, and, thus, that "to go on record with a detailed, confidently stated account which claims to be normative, is both inappropriate and harmful." I agree that there are many, many open questions
However, this does not mean that we have no idea about how to develop ontologies.
I am sure many people have ideas about how to develop ontologies. I am far less convinced that their collective experience can be summed up in this way, and still less that this document sums it up. I am virtually certain that there is no single best approach to such development, but that it depends upon a host of factors which are hardly mentioned here, such as whether the ontology is Web-based or not, and how large and disparate its intended user base is. An ontology developed by a technical consortium for use in a highly controlled environment to facilitate technical interoperation is one thing; an ontology published as a guide to creating linked Web data relevant to some broad topic is quite another thing.
Thus, the open questions should not prevent us from synthesizing the collective experience that is available right now. We will not get everything right, but it will enable other people to avoid some mistakes.
Perhaps. It will however also cause other people to make mistakes, and it will give power to generations of managers who will use it make the lives of productive people miserable and seriously less productive. Overall, its value is almost certainly negative. (BTW, the very idea of "synthesizing the collective" presumes that there is a common, universally accepted and universally applicable, core of best practices. Again, I doubt this is true: and at the very least, I would like to see an argument made for it, rather than it simply being presumed.)
That's said, I see you point about the potential drawbacks. To minimize the risk of turning the Communique into management dogma, we should put some language into the executive summary and/or the introduction that state the limits of our existing experience explicitly.
That would certainly be a good idea, but in my experience, such cautions and qualifications will be immediately ignored by management theorists and the managers they inspire. Management is inherently unscientific: like politics, it is looking for simple slogans, for the elevator speech rather than something complicated, and therefore not conducive to making a quick, clear decision.
I should say that I still bear the scars from have been at the receiving end of such managementspeak disasters as Total Quality Control, which claimed to be based on empirical and tested work, which it even cited, and when we actually checked the citations, the needed qualifications and cautions were there; but they had been (of course) ignored by the TQC which had hardened into dogma. By the time we (employees of Xerox) were exposed to it, it had become exactly like a religion, with rituals that had to be performed for reasons that were now lost in time, but which to question was heresy. I predict that "conforming to the upper ontology" will soon become such a meaningless ritual, and people will be obliged to invent vacuous "upper" ontologies in order to be able to show their managers how they conform to them. Or, worse, their managers will get irrelevant standard upper ontologies from standard-upper-ontology providers, at enormous cost, and require conformity from their developers,
who will then spend considerable time and energy finding ways of conforming to the letter in order to keep the idiotic managers happy, while managing to do the work they actually need to get done in spite of this timewasting nonsense. One can see signs of this already in the email archives surrounding the OBO foundry, where long discussions take place over what things should be classified as continuants, a question which is only meaningful inside the warped philosophy that has been imposed on this corpus by its management.
Pat
Best
Fabian
On 4/21/13 12:32 PM, Pat Hayes wrote:
On Apr 21, 2013, at 5:11 AM, Amanda Vizedom wrote:
Pat,
Will you share with us the reasons for your non-support of this communique?
Hi Amanda
Sure. (I didn't want to pester you unless you asked :-) Several reasons. First, it is way too long and detailed, more of an essay than a communique. It is hard, I would suggest, for anyone to agree with all of it unequivocally. I feel like I want to nit-pick with the text all over the place. For example, this claim that ontologies must be "transparent to all intended users" (who might not be ontologists) seems to me to be ridiculously optimistic (or perhaps ridiculously restrictive, if it is interpreted as a design constraint) and to be rooted in a naive idea that correctness of an ontology should be obvious to a domain expert. But this is simply false. Just as one would not expect users of a program to find the code source obvious, one cannot expect users of an ontology to find the details of the ontology obvious. There may well be decisions taken in the design, the reasons for which are only apparent to professional ontologists (such as whether to use continuants in talkin
g about time and change.)
I could go through the document nitpicking like this, but it seems pointless, because the entire enterprise is flawed. The fact is, this whole document is a mangerial fantasy. We do not have enough experience with ontology design and deployment to know what the objective standards of "quality" are, still less how to manage teams to achieve this nonexistent standard. We don't know what are the "activities that need to occur during the phases of a life cycle of an ontology", so to go on record with a detailed, confidently stated account which claims to be normative, is both inappropriate and harmful. As I say, this is pure fantasy, but it will be read by some as having an authority and will be used by managers (most of whom know absolutely nothing about ontologies) to impose work habits on other people for no good reason. For example, "
Does the
ontology follow best practices; in particular does it
implement the upper ontology...." Whoa. Is it "best
practice" to even HAVE an upper ontology? That is not
clear. Most Web ontologies, for example, are not subsumed
under any particular upper ontology. If out communique
starts being used to justify managers asking ontologists
to conform to an upper ontology, we will have done far
more harm than good.
I found it very telling that after pages of vacuous managerial-theory babble about life cycles and "phases", most of it content-free (such as "The requirements development and analysis phase involves extending and clarifying initial information until the intended usage is sufficiently captured and understood to effectively guide technical decisions. This process involves an interplay of technical, business, and project-sponsor understanding. Adequate requirements development and analysis is critical to the success of any ontology development or usage."), and an absurd schematic diagram showing tangles of arrows connecting meaningless boxes, we read the almost plaintive remark "
Generally,
appreciation of the full life cycle of an ontology is not
well established within the ontology community." Damn
right. In other words, none of this is based in actual
reality. It is written as though it comprised observations
about the right way to work, but in fact, it is not based
on observations about how the work is ACTUALLY done.
Another
nit-pick, to end. Your second final observation begins: "
Ontology
development shares strong similarities with information
systems development..." Does it, in fact? Is this based on
actual observations? (Of which projects?) Or is this just
an idea which the authors of this document feel *should*
be true?
Again: "
Although
there is much research on ontology evaluation and many
organizations use sophisticated ontology evaluation and
quality management practices, awareness of this research,
these practices, and their importance to successful use of
ontologies is neither widespread nor sufficiently pooled
to constituted an accessible body of knowledge." The claim
that there is "much research" seems to me to be overly
optimistic, to put it mildly; but the main point is, there
is NOT widespread adoption of these practices. There may
be very good reasons for this lack of uptake: the
practices may be of limited utility, or of no real utility
at all. In my experience, that is the most likely
explanation.
My
advice would be to completely toss this document
aside, and start over not with some ready-made
management-science theory about phases and work
cycles, but try observing, if possible with a
somewhat more humble attitude, how some large-scale
ontologies were actually built. You might start with
CYC, the granddaddy of all large-scale ontologies.
You will find that the process bears almost no
relationship to the fantasy you describe here.
Sorry, but you did ask.
Pat
I'm asking not to try to argue, but because we haven't had any input or feedback from you, and I value your insights generally. We are still working to follow through on many of the suggestion and critiques offered so far. This follow-through may happen by substantal change to the communique or by clarification of its scope and, if possible and with summit community support, links to better and detailed references on issues that are out of scope, including other summit products. So, the request is not empty; if you will let us know the reason(s) for your discontent, we may be able to improve the communique by understanding them.
Best,
Amanda
On Apr 20, 2013 11:25 PM, "Pat Hayes" <phayes@xxxxxxx> wrote:
Peter and Mike, greetings.
On Apr 19, 2013, at 7:31 PM, Peter Yim wrote:
Dear Ontology Summit Advisors,
...
p.s. additionally, two very important reminders: *** Please Note ***
...
2. We are expecting to have the Communique ready by the time of the
Symposium (no more wordsmithing of that document at the face-to-face
this time) and, as advised earlier, we are expecting all Advisors to
endorse the Communique (on an opt-out basis.)
Please opt me out of endorsing this Communique. If you would prefer, you may remove me from the Advisory Committee, in order to maintain an appearance of solidarity.
Best wishes
Pat Hayes
------------------------------------------------------------
IHMC (850)434 8903 or (650)494 3973
40 South Alcaniz St. (850)202 4416 office
Pensacola (850)202 4440 fax
FL 32502 (850)291 0667 mobile
phayesAT-SIGNihmc.us http://www.ihmc.us/users/phayes
_________________________________________________________________
Committee Msg Archives: http://ontolog.cim3.net/forum/ontology-summit-advisors/
Subscriber Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit-advisors/
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2013/
Community Discussion: : http://ontolog.cim3.net/forum/ontology-summit/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013
Community Portal: http://ontolog.cim3.net/wiki/
_________________________________________________________________
Committee Msg Archives: http://ontolog.cim3.net/forum/ontology-summit-advisors/
Subscriber Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit-advisors/
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2013/
Community Discussion: : http://ontolog.cim3.net/forum/ontology-summit/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013
Community Portal: http://ontolog.cim3.net/wiki/
------------------------------------------------------------
IHMC (850)434 8903 or (650)494 3973
40 South Alcaniz St. (850)202 4416 office
Pensacola (850)202 4440 fax
FL 32502 (850)291 0667 mobile
phayesAT-SIGNihmc.us http://www.ihmc.us/users/phayes
_________________________________________________________________
Committee Msg Archives: http://ontolog.cim3.net/forum/ontology-summit-advisors/
Subscriber Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit-advisors/
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2013/
Community Discussion: : http://ontolog.cim3.net/forum/ontology-summit/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013
Community Portal: http://ontolog.cim3.net/wiki/
------------------------------------------------------------
IHMC (850)434 8903 or (650)494 3973
40 South Alcaniz St. (850)202 4416 office
Pensacola (850)202 4440 fax
FL 32502 (850)291 0667 mobile
phayesAT-SIGNihmc.us http://www.ihmc.us/users/phayes
|