A number of users have written to me suggesting that I respond to Altova's latest customer newsletter in which they make the rather suprising claim that their XSLT processor is three times faster than Saxon. At least they had the decency to describe how they measured it - by running a thousand or so conformance tests from the W3C test suite using a batch script in which each transformation was run individually from the command line. Any reasonably competent user knows that that's a hopelessly inefficient way of running Java programs, because the initialisation costs for loading the Java VM a thousand times far exceeds the time spent actually doing useful work. But of course, Altova know that a lot of people will only read the headlines.
Well, I did respond: I've put a counter-claim on my web site (see the last item in the Questions and Answers section) that Saxon is ten times faster than Altova. I freely admit that my methodology is no more scientific than theirs is, though I think my numbers are probably closer to the truth. It would be nice to see some objective numbers produced carefully by an unbiased third party. Vendor figures, including my own, are not very useful, because a vendor always knows how to make their own product run at maximum speed, and they rarely have either the knowledge or the motivation to do the same for their competitors.
Meanwhile, I'm actually quite proud of the fact that Saxonica is now so highly regarded that Altova should feel it necessary to throw mud in this way. When I started the company, nearly three years ago now, I could almost have set that as a business objective: "in three years time, Saxonica will have developed such a strong reputation as a technology leader that competitors will resort to knocking us in their advertising". In marketing circles it's well known that knocking your competition is a strategy that can easily backfire, because it actually raises the profile of the competitor among your own customers, and acknowledges them as a leader. Let's hope that turns out to be true in this case. Certainly, it seems a strategic error to make competitive claims that are so easily disproved.
It's worth remembering also that performance is only one dimension. I've worked hard on Saxon performance over the years, and I think it's pretty good (and certainly very competitive), but performance has never been the number one objective. Correctness comes first, in the sense of 100% conformance to the W3C specifications, closely followed by usability, of which the most important ingredient in my view is the quality of the error messages seen by users during the development cycle. At the moment I think Saxon is well ahead of Altova on all three fronts. It's nice to see that Altova are running the W3C conformance tests, but slightly worrying that they are only running a subset; and if I were a user, I would certainly be asking to see their results.
I rather enjoy watching the robust way in which my friends at Stylus Studio attack Altova. But I prefer watching a good fight from the ringside to taking part in it! It's tempting to join in when people starting throwing things at you, but it's not really my style to do business in that way. I'll try and stand back and leave that to others. I prefer to think (perhaps it's an old-fashioned British attitude) that the way to achieve success is to focus on producing the best product.
|
|
||||||||
|
Login
|
Altova Mudslinging
Comments
Re: Altova Mudslinging
When I saw Altova's latest newsletter I had a good chuckle. I deal with very large XML datasets, between 100MB to 1GB. I use a variety of XSL transformation engines: Altova, MSXML, Saxon, and Xalan. Each has their strengths and weaknesses. None of them handle the processing of large XML datasets well.
What I found extremely funny about Altova's claim was my personal experience in running large XML datasets, on a regular basis, through the various XSL transformation engines. Most of the XSL transforms I run take hours to complete. Xalan by far is the slowest, but it handles large XML datasets as well as Saxon does. MSXML v6 is faster than Xalan and Altova has similar completion times as MSXML. Both Xalan and Saxon handle larger XML datasets, approximately 50MB more of data, than either MSXML or Altova. Saxon by far has the fastest completion times. Many times Saxon completes the same XSL transform a full 30 to 40 minutes before any other XSL transformation engine. From my perspective, while running conformance tests is nice, in my "real" world scenarios, what I see is that Saxon appears to make more efficient use of memory and runs my XSL transforms much quicker than Altova does. I really had to chuckle at their claims. As you indicated, any vendor can game performance testing since they know their products strengths. Keep up the good work on Saxon, Andy. Re: Re: Altova Mudslinging
Hey Michael,
Your Blog rocks!! Just wanted to share something with ya… one blogger to another… There is this amazing site that I came across where u can make money by sharing information…check it out here’s the link http://www.myndnet.com/login.jsp?referral=alpa83&channel=al424 The coolest part is…every time ur information gets sold u get paid for it!! I signed it for it.. very cool stuff… u can also mail me at barot.alpa@gmail.com Cheers! Alpa Re: Altova Mudslinging
by
Dimitre Novatchev
on Sun 29 Oct 2006 19:18 GMT | Permanent Link
I have made my own performance comparisons and posted them in my blog:
http://dnovatchev.spaces.live.com/blog/cns!44B0A32C2CCF7488!325.entry Cheers, Dimitre Novatchev Re: Altova Mudslinging
by
Charles Fold
on Tue 31 Oct 2006 05:16 GMT | Permanent Link
Conformance is key. I consider Saxon the standard to which everyone else aspires.
Re: Altova Mudslinging
by
Bryan Rasmussen
on Tue 31 Oct 2006 10:08 GMT | Permanent Link
sometimes I feel really bad about telling people who call up with validation and processing problems how much Altova sucks.
No, I'm just kidding, this is actually the highlight of my day. Re: Altova Mudslinging
by
Anonymous
on Tue 31 Oct 2006 21:19 GMT | Permanent Link
Hi, Mike lean me on to post some of my personal benchmarks. In one project I generate pretty large set of HTML pages (more then 5000, total size 40 MB) from single 16 MB XML file. During processing a lot of key lookups and calculation is done (you can see output at http://studijniplany.amu.cz). Saxon finishes this stylesheet in less then 3 minutes and consumes ~130 MB of RAM. For the most of the time Saxon utilizes only 70% of CPU -- disc is probably slower then Saxon ;-)
I tried the same stylesheet and data with Altova2007. After 30 minutes only 400 files from 5000 vere generated and 950 MB of RAM was vasted, CPU utilization for Altova process was over 90%. I decided to interrupt test here. For this stylesheet Altova was at least 100 times slower then Saxon. Moreover Altova eat more and more memory with each outputed file, so there is probably some memory leak in this product. OTOH, Saxon memory consuption was constant after initial loading and preparation of documents. Re: Altova Mudslinging
by
Mark Baker
on Fri 03 Nov 2006 19:37 GMT | Permanent Link
It really doesn't matter whether Altova's XSLT processor is fast or slow. Until they fix their parser so that whitespace-only nodes in mixed content are not thrown away, Altova's product is useless for documentation work. As it is, if you have two adjacent words in a paragraph that are each individually tagged, Altova throws away the space between them. I sent them a message about this over a year ago. They incorporated the example I gave them into their docs to explain the problem -- but they have not fixed it in two major releases.
|
Search
Recent Comments
Month Archive
|
||||||