On 4/2/13 9:13 AM, Rodriguez-Muro Mariano (A) wrote:
> Dear Kingsley,
>
> I am the main developer of ontop and quest, we met during ISWC in Boston last
> year. Regarding your comments about our benchmarks of ontop and virtuoso.
>During
> ISWC you made similar remarks regarding our benchmarks and offered direct
> assistance through Virtuoso's support team. We did take your offer and started
> working with Virtuoso's support the team. The support case is #20228, Garry
>is our
> contact.
>
> During the last 4 months he helped us setup our Virtuoso instances and verify
> our results. We did get
> better performance from Virutoso compared to the one you saw in Boston,
> however, the SQL queries produced by Quest still came up ontop of Virtuoso by
>a very
> big margin in the
> tests that we did using BSBM and MySQL 5.5 as a backend. If the rewriting
>process
> is also taken into account, the difference is smaller, but still strongly
> in favour of ontop and Quest plus MySQL. (01)
I can assure you with a 100% certainty that your benchmark numbers for
Virtuoso are inaccurate simply based on your configuration.
>
> The results where also confirmed by Garry using the machines at open link.
>
> In the process we also found and reported several bugs that produced wrong
> answers or crashes in Virtuoso RDF Views. Garry contacted the development
> team of virtuoso and said they were working to fix the problem.
>
> Not all the new results are online, however, some of them and
> the changes we did to the INI file as recommended by Garry are located at
>
> http://ontop.inf.unibz.it/?page_id=74
>
> and in the wiki page that is linked there.
>
> In sum, we are surprised with your comment about our benchmarks and 'the way
> we advertise' our work. (02)
Because you shouldn't publish benchmarks without configuration
information. All you needed to do was include the Virtuoso INI settings
(as you did the MySQL settings). (03)
Our best numbers for BSBM performance and scalability come from the use
of RDF Views over SQL. Typically, BSBM benchmarks have only published
our RDF store numbers [1][2], and I don't see your claims exceeding
those. Thus, you are basically claiming that your product makes a
mockery of Virtuoso's best performance configuration for BSBM. (04)
Also note, there are some new auditable BSBM benchmark results coming
out from the Linked Data Benchmarking Council [1], and I encourage you
to digest the numbers and showcase to everyone how your product can
exceed the numbers that will be unveiled. Again, the numbers we produce
will still be a factor of 5 less than what we can currently achieve via
RDF Views or SQL Data. (05)
Links: (06)
1.
http://wifo5-03.informatik.uni-mannheim.de/bizer/berlinsparqlbenchmark/results/V6/#exploreVirtuoso (07)
-- BSBM results (note: Virtuoso INI is part of the publication)
2.
http://wifo5-03.informatik.uni-mannheim.de/bizer/berlinsparqlbenchmark/results/V6/index.html#comparison
3. http://ldbc.eu/ -- EU funded benchmark project for RDF and Graph
databases . (08)
Kingsley
>
> Greetings,
> Mariano
>
>
>> On 4/1/13 12:12 PM, Michel Dumontier wrote:
>>> ok, how about ontop developed by Mariano Rodriguez and colleagues:
>>> http://ontop.inf.unibz.it/
>>> it enables one to map OWL-QL ontologies to SQL database and answer
>>> queries. no conversion required.
>> If you know these folks well enough, please have them publish their
>> Virtuoso INI settings. Publishing benchmarks against a poorly configured
>> Virtuoso instance e.g., using its default settings (which assume limited
>> memory in the < 2GB range) isn't the way to advertize their work. The
>> claimed disparity speaks volumes :-)
>> (02)
>>
> (09)
-- (010)
Regards, (011)
Kingsley Idehen
Founder & CEO
OpenLink Software
Company Web: http://www.openlinksw.com
Personal Weblog: http://www.openlinksw.com/blog/~kidehen
Twitter/Identi.ca handle: @kidehen
Google+ Profile: https://plus.google.com/112399767740508618350/about
LinkedIn Profile: http://www.linkedin.com/in/kidehen (012)
smime.p7s
Description: S/MIME Cryptographic Signature
_________________________________________________________________
Message Archives: http://ontolog.cim3.net/forum/ontolog-forum/
Config Subscr: http://ontolog.cim3.net/mailman/listinfo/ontolog-forum/
Unsubscribe: mailto:ontolog-forum-leave@xxxxxxxxxxxxxxxx
Shared Files: http://ontolog.cim3.net/file/
Community Wiki: http://ontolog.cim3.net/wiki/
To join: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J (01)
|