Re: [ontolog-forum] independent semantic software evaluation frameworks?

[Paola Di Maio <paola.dimaio@xxxxxxxxx>]

Raul


just following up with some thoughts (and to make my point inequivocable)
its critical to the validity of an evaluation to capture all the evaluation criteria objectively.

No  single individual or consortium can be trusted to produce an impartial evaluation
mechanism especially if the same researcher/consortium are heavily invovled in the developmet of semantic technologies.

So the SEALS effort looks  great, and it must an 'open' one to
maximise its validity and usefulness to the community at large (other
than becoming an instrument to self validate the stuff that one has
produced)

I look forward to be making a contribution where possible

cheers

PDM

On 9/3/09, Raúl García Castro <rgcmme@xxxxxxxxx> wrote:

Dear Paola,

If you are interested in the evaluation of semantic technologies, take a look at the future outcomes of the SEALS European project (http://www.seals-project.eu/).

In this project we aim to build an extensible infrastructure for the evaluation of different types of semantic technologies:
.- ontology engineering tools
.- storage and reasoning systems
.- matching tools
.- semantic search tools
.- semantic web service tools

Kind regards,

On Sun, Aug 9, 2009 at 7:03 PM, Paola Di Maio <paola.dimaio@xxxxxxxxx> wrote:

Greetings good people!

I am attempting to systematically aggregate as many independent
evaluations of semantic software tools as possible

(Note: by independent it is intended not carried out by people who are
paid to do so, or have other vested interests  - such as the
devleopers or consultants themselves  or their friends and familieis -
 but by unaffiliated users who are more suitable to appreciate its
benefits and shorcomings)

We are particularly interested initially in the software developed
with EU funding (FP6 and FP7 )

The main frameworks of reference are standard software project
evaluation methods, including


EVALUATION AND RANKING OF ONTOLOGY CONSTRUCTION TOOLS (various
references included there)
Md. Ahsan-ul Murshed and Ramanjit Singh
http://eprints.biblio.unitn.it/archive/00000747/01/013.pdf


I have already contacted offlist some of the individuals who have
offered knowledge and views, and I am now  seeking additional wider
general  input on

1.  developing and customizing the  review criteria  and methodology
(so that we can format the distributed evaluations using a common template)

2.  any volunteered inputs in terms of reviews of software and other
deliverables (priority is to assess  cost/benefit ratio, measured in
terms of
functionality and usefulness)

The final goal of this exercise is to contribute  to improve the
effectiveness of EU funding process in semantic web, as well as to
general
software output at large, as well as to come up with  recommendations.


Your contribution can be submitted authored or anonymous (subject to
verification)


Please contact me offlist should you wish to collaborate on this project,

cheers

Paola Di Maio

--

Dr. Raúl García Castro
http://delicias.dia.fi.upm.es/~rgarcia/

Ontology Engineering Group (http://www.oeg-upm.net/)
Universidad Politécnica de Madrid
Campus de Montegancedo, s/n - Boadilla del Monte - 28660 Madrid
Phone: +34 91 336 36 70 - Fax: +34 91 352 48 19

-- 
Paola Di Maio
**************************************************
Networked Research Lab, UK

***************************************************
_________________________________________________________________
Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/  
Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/  
Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
Shared Files: http://ontolog.cim3.net/file/
Community Wiki: http://ontolog.cim3.net/wiki/ 
To join: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J
To Post: mailto:ontolog-forum@xxxxxxxxxxxxxxxx    (01)