[Top] [All Lists]

[ontolog-forum] Science, Statistics and Ontology

To: "[ontolog-forum]" <ontolog-forum@xxxxxxxxxxxxxxxx>
From: Ali SH <asaegyn+out@xxxxxxxxx>
Date: Wed, 9 Nov 2011 10:07:58 -0500
Message-id: <CADr70E3RVzBichSAp23j1QQsjW3OQf7b+F303csPFP4-4RRx8A@xxxxxxxxxxxxxx>
Hello all,

I recently came across this article:

Beyond the broader scientific issues it discusses, I think it is of particular note for ontologists.  

The basic thesis of the post is that many scientists have fundamentally misunderstood the meaning of statistical significance, and make several erroneous assumptions and conclusions while committing several logic errors (i.e. transposing the conditional). The article further highlights how an unexamined ontology can lead to a faulty meta-theory which fatally wounds the intended statistical interpretation of the results. 

It’s science’s dirtiest secret: The “scientific method” of testing hypotheses by statistical analysis stands on a flimsy foundation. Statistical tests are supposed to guide scientists in judging whether an experimental result reflects some real effect or is merely a random fluke, but the standard methods mix mutually inconsistent philosophies and offer no meaningful basis for making such decisions. Even when performed correctly, statistical tests are widely misunderstood and frequently misinterpreted. As a result, countless conclusions in the scientific literature are erroneous, and tests of medical dangers or treatments are often contradictory and confusing. 

If there were ever a blatant, inadvertent call for the deployment of ontology throughout the sciences, this might be it. Without actually stating it, this article is largely about ontology and to some degree, collaborative belief revision. 

For one, it highlights the need of making implicit semantics explicit. The basic problem of misapplying inconsistent philosophies arises from a lack of understanding and due to the fact that many of the deployed assumptions remain implicit and their interactions are not properly evaluated or verified. Ontology would help by first explicating these assumptions and in so doing, clarify the intended meanings while highlighting potential inconsistencies. Mind you, I think this is applied ontology in a broader sense than the popular SemWeb implementations; I don't think a lightweight ontology would necessarily help, nor would a computational ontology be necessarily required. But ontological analysis, the process of making implicit assumptions explicit, would yield tangible benefits.

The article further touches on the familiar issue of similar vocabularies but slight varieties in meaning; and how perhaps altogether different conceptualizations of procedures (methodologies) lead to problems when trying to combine results / hypotheses, as illustrated in the following passage:

Another concern is the common strategy of combining results from many trials into a single “meta-analysis,” a study of studies. In a single trial with relatively few participants, statistical tests may not detect small but real and possibly important effects. In principle, combining smaller studies to create a larger sample would allow the tests to detect such small effects. But statistical techniques for doing so are valid only if certain criteria are met. For one thing, all the studies conducted on the drug must be included — published and unpublished. And all the studies should have been performed in a similar way, using the same protocols, definitions, types of patients and doses [emphasis mine]. When combining studies with differences, it is necessary first to show that those differences would not affect the analysis, Goodman notes, but that seldom happens. “That’s not a formal part of most meta-analyses,” he says.
In principle, a proper statistical analysis can suggest an actual risk even though the raw numbers show a benefit. But in this case the criteria justifying such statistical manipulations were not met. In some of the trials, Avandia was given along with other drugs. Sometimes the non-Avandia group got placebo pills, while in other trials that group received another drug. And there were no common definitions.
“Across the trials, there was no standard method for identifying or validating outcomes; events ... may have been missed or misclassified,” Bruce Psaty and Curt Furberg wrote in an editorial accompanying the New England Journal report. “A few events either way might have changed the findings.” 

Lastly, it seems to advocate for more of a Bayesian style belief revision when trying to combine results from multiple experiments, as opposed to the more AGM-like one that is implicitly being performed by many scientists (as described in the article). Namely it taps into the running debate as to whether the principle of "priority to the incoming information" (from the AGM postulates) should be dropped in ontology revision, i.e. [1]. Depending on how you read it, the article has important implications for ontology management, especially in collaborative socio-technical systems.

[1] Mauro Mazzieri and Aldo Franco Dragoni. "Ontology Revision as Non-Prioritized Belief Revision." In Proceedings of International Workshop on Emergent Semantics and Ontology Evolution, 2007


Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/  
Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/  
Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
Shared Files: http://ontolog.cim3.net/file/
Community Wiki: http://ontolog.cim3.net/wiki/ 
To join: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J    (01)

<Prev in Thread] Current Thread [Next in Thread>