Dear Pat, John, and Barry, (01)
I think the problem that many have with academic review is that it
is open to abuse and personal prejudice. (02)
An approach that is aimed at being more structured is Document Inspection.
This at least tries to be objective, and is designed to make being subjective
harder. (03)
The approach is to measure a document against its purpose and target audience.
It uses a team of trained inspectors (training is simple and straightforward)
- Divide document so that (for total inspection) each part of the document is
reviewed by 3 inspectors (diminishing returns after 3 in terms of
identifying
new issues). Author is one of the inspectors.
- Identify issues:
- Statements that are untrue, or unclear and/or ambiguous to target audience
- Super-major - show stopper
- Major - subverts the purpose of the document
- Minor - incorrect but no major impact
- Editorial - grammar and spelling, badly laid out diagrams
Review the issues, determine whether document is fit for purpose (no Super
Major,
low count of majors). (04)
This gives a rationale for rejection, and provides the basis for improvement
so that inclusion becomes possible. The issue list is publicly available so that
people can see where the deliverable is, and whether the issues raised are a
concern for them. (05)
This is of course the essence of reaching consensus in a standardization
process,
but if you are getting into any level of approval, you ARE doing
standardization,
however you choose to dress it up. (06)
Regards (07)
Matthew West
Reference Data Architecture and Standards Manager
Shell International Petroleum Company Limited
Registered in England and Wales
Registered number: 621148
Registered office: Shell Centre, London SE1 7NA, United Kingdom (08)
Tel: +44 20 7934 4490 Mobile: +44 7796 336538
Email: matthew.west@xxxxxxxxx
http://www.shell.com
http://www.matthew-west.org.uk/ (09)
> -----Original Message-----
> From: ontology-summit-bounces@xxxxxxxxxxxxxxxx
> [mailto:ontology-summit-bounces@xxxxxxxxxxxxxxxx]On Behalf Of Patrick
> Cassidy
> Sent: 21 March 2008 04:08
> To: 'Ontology Summit 2008'
> Subject: Re: [ontology-summit] [Quality] What means
>
>
> John,
> Among the 'reviewers' is there any reason not to have an
> expert committee
> that can create a binary distinction of, e.g.
> "well-structured" and "not
> well-structured"? The imprimatur can be an alternative to absolute
> exclusion, and still serve the legitimate concerns that Barry
> has about
> poorly constructed ontologies.
>
> Pat
>
> Patrick Cassidy
> MICRA, Inc.
> 908-561-3416
> cell: 908-565-4053
> cassidy@xxxxxxxxx
>
>
> > -----Original Message-----
> > From: ontology-summit-bounces@xxxxxxxxxxxxxxxx
> [mailto:ontology-summit-
> > bounces@xxxxxxxxxxxxxxxx] On Behalf Of John F. Sowa
> > Sent: Thursday, March 20, 2008 11:56 PM
> > To: Ontology Summit 2008
> > Subject: Re: [ontology-summit] [Quality] What means
> >
> > Pat, Barry, Deborah, and Ed,
> >
> > Barry asked an important question that gets to the heart of
> > the issues we have been discussing:
> >
> > BS> What are scientific journals for? Why do they employ a peer
> > > review process?
> >
> > There are two independent issues here: reviewing and publishing.
> > Everybody would agree that reviewing is important, but ideally,
> > the readers/users should have the option of making their own
> > choices based on the reviews. When publication was expensive,
> > the publishers became gatekeepers because it was economically
> > impractical to publish everything.
> >
> > But with the WWW, new options are available. Publication is
> > almost free, and we have the luxury of decoupling the reviewing
> > process from the gatekeeping process. Metadata enables that
> > decoupling:
> >
> > 1. All submissions to the OOR can be made available as soon
> > as they are submitted.
> >
> > 2. The metadata associated with each submission can indicate
> > what tests were made, what the reviewers said, and what
> > results the users, if any, obtained.
> >
> > 3. Users can choose to see ontologies sorted by any criteria
> > they want: in the order of best reviews, most thorough
> > testing, greatest usage, greatest relevance to a particular
> > domain, or any weighted combination.
> >
> > PH> This is where I part company with Barry, and indeed where I
> > > believe that the very idea of controlling the contents of an OOR
> > > (noting that the first O means 'open') needs to be examined very,
> > > very carefully. Of course we would not argue that majority voting
> > > should be used to choose scientific theories; but ontologies,
> > > even those used by scientists, are not themselves scientific
> > > theories.
> >
> > Ontologies overlap philosophy, engineering, science, and
> mathematics.
> > The closest model we have is the metadata registry, but new policies
> > can and should be explored.
> >
> > BS>> While refrigerator manufacturers may allow democratic ranking
> > >> to influence e.g. size and color, they would use other strategies
> > >> e.g. in matters of thermodynamics.
> >
> > PH> Perhaps so: but we are here discussing matters of ontology, and
> > > in the current state of the art, this may have more in common
> > > with consumer product choice than with thermodynamics.
> >
> > That is the point I was trying to emphasize. The application
> > developers have deeper understanding of their specific needs and
> > problems than any general gatekeeper or committee of gatekeepers.
> >
> > DM> CSI, the specification writing organization for building
> > > architecture, says quality is "a mirror of the requirements."
> >
> > That's a good point, which implies that different set of
> > requirements might lead to a different ranking of the same
> > ontologies. No gatekeeper can anticipate the requirements
> > of all possible users.
> >
> > DM> Do you think the gatekeepers can help define the OOR
> requirements
> > > and set up the dynamic tests?
> >
> > I'd prefer to keep the reviewers and replace the gatekeepers with
> > caretakers who have a broader role along the lines you suggested.
> >
> > EB> I'm thinking about bureaucrats. I think that many ontologies
> > > (and more broadly, concept systems including thesauri,
> taxonomies,
> > > etc.) have been and will be developed for use within the mission
> > > areas of government agencies. There can be a vetting process to
> > > "approve" a concept system/ontology for use within a community
> > > of interest.
> >
> > That suggests a further refinement of the roles of reviewers and
> > gatekeepers/caretakers. At the source, there are individuals and/or
> > organizations, who develop ontologies and make them available.
> > Among the users, there may be organizations, coalitions, or
> > bureaucracies that evaluate the ontologies and determine which
> > of them are best suited to their groups of users.
> >
> > That is another reason for replacing the gatekeepers in the OOR
> > with caretakers. Any gatekeeping that might be useful would be
> > better done by user groups at a level close to the applications
> > than by any gatekeeper that is close to the ontology providers.
> >
> > John
> >
> >
> >
> > _________________________________________________________________
> > Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
> > Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-
> > summit/
> > Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
> > Community Files:
> http://ontolog.cim3.net/file/work/OntologySummit2008/
> > Community Wiki: http://ontolog.cim3.net/cgi-
> > bin/wiki.pl?OntologySummit2008
> > Community Portal: http://ontolog.cim3.net/
>
>
> _________________________________________________________________
> Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
> Subscribe/Config:
> http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
> Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
> Community Files: http://ontolog.cim3.net/file/work/OntologySummit2008/
> Community Wiki:
> http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2008
> Community Portal: http://ontolog.cim3.net/
>
> (010)
_________________________________________________________________
Msg Archives: http://ontolog.cim3.net/forum/ontology-summit/
Subscribe/Config: http://ontolog.cim3.net/mailman/listinfo/ontology-summit/
Unsubscribe: mailto:ontology-summit-leave@xxxxxxxxxxxxxxxx
Community Files: http://ontolog.cim3.net/file/work/OntologySummit2008/
Community Wiki: http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2008
Community Portal: http://ontolog.cim3.net/ (011)
|