[Top] [All Lists]

Re: [ontolog-forum] independent semantic software evaluation frameworks?

To: Raúl García Castro <rgcmme@xxxxxxxxx>
Cc: "[ontolog-forum]" <ontolog-forum@xxxxxxxxxxxxxxxx>, semantic-web at W3C <semantic-web@xxxxxxx>
From: Paola Di Maio <paola.dimaio@xxxxxxxxx>
Date: Thu, 3 Sep 2009 04:47:02 -0700
Message-id: <4a4804720909030447m1a3dcd2cjd291ae197b137938@xxxxxxxxxxxxxx>

just following up with some thoughts (and to make my point inequivocable)
its critical to the validity of an evaluation to capture all the evaluation criteria objectively.

No  single individual or consortium can be trusted to produce an impartial evaluation
mechanism especially if the same researcher/consortium are heavily invovled in the developmet of semantic technologies.

So the SEALS effort looks  great, and it must an 'open' one to maximise its validity and usefulness to the community at large (other than becoming an instrument to self validate the stuff that one has produced)

I look forward to be making a contribution where possible



On 9/3/09, Raúl García Castro <rgcmme@xxxxxxxxx> wrote:
Dear Paola,

If you are interested in the evaluation of semantic technologies, take a look at the future outcomes of the SEALS European project (http://www.seals-project.eu/).

In this project we aim to build an extensible infrastructure for the evaluation of different types of semantic technologies:
.- ontology engineering tools
.- storage and reasoning systems
.- matching tools
.- semantic search tools
.- semantic web service tools

Kind regards,

On Sun, Aug 9, 2009 at 7:03 PM, Paola Di Maio <paola.dimaio@xxxxxxxxx> wrote:
Greetings good people!

I am attempting to systematically aggregate as many independent
evaluations of semantic software tools as possible

(Note: by independent it is intended not carried out by people who are
paid to do so, or have other vested interests  - such as the
devleopers or consultants themselves  or their friends and familieis -
 but by unaffiliated users who are more suitable to appreciate its
benefits and shorcomings)

We are particularly interested initially in the software developed
with EU funding (FP6 and FP7 )

The main frameworks of reference are standard software project
evaluation methods, including

references included there)
Md. Ahsan-ul Murshed and Ramanjit Singh

I have already contacted offlist some of the individuals who have
offered knowledge and views, and I am now  seeking additional wider
general  input on

1.  developing and customizing the  review criteria  and methodology
(so that we can format the distributed evaluations using a common template)

2.  any volunteered inputs in terms of reviews of software and other
deliverables (priority is to assess  cost/benefit ratio, measured in
terms of
functionality and usefulness)

The final goal of this exercise is to contribute  to improve the
effectiveness of EU funding process in semantic web, as well as to
software output at large, as well as to come up with  recommendations.

Your contribution can be submitted authored or anonymous (subject to

Please contact me offlist should you wish to collaborate on this project,


Paola Di Maio


Dr. Raúl García Castro

Ontology Engineering Group (http://www.oeg-upm.net/)
Universidad Politécnica de Madrid
Campus de Montegancedo, s/n - Boadilla del Monte - 28660 Madrid
Phone: +34 91 336 36 70 - Fax: +34 91 352 48 19

Paola Di Maio
Networked Research Lab, UK


Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/  
Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/  
Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
Shared Files: http://ontolog.cim3.net/file/
Community Wiki: http://ontolog.cim3.net/wiki/ 
To join: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J
To Post: mailto:ontolog-forum@xxxxxxxxxxxxxxxx    (01)

<Prev in Thread] Current Thread [Next in Thread>