Benchmarks

One of the main goals for the Orion Application Server has been to outperform everything else on the market. On this page we will investigate how we have succeesed and we will add the results we find in our own testing. We will also be happy to include independent benchmarks here, so if you have a servlet, jsp, ejb application, rmi application or anything else that seems suitable for benchmarking, please let us know. Send a mail to info@orionserver.com and we will be happy to include your benchmark here.

JSP vs. ASP, part I (1999-09-26)

The two main competing technologies in the field of server side includes are Java Server Pages and ASP, Active Server Pages. JSP is implemented by a large flora of vendors and ASP is supported by Microsoft. We were convinced that an ASP page could not be as fast as a JSP page run in Orion so we decided to try it out. This test was made on a P-II 266 Mhz with 96 Mb RAM using JDK1.3b1 on Windows NT 4.0 and a simple jsp/asp file with a loop printing the numbers 1-100. Look at the source of the JSP here and the ASP here. The Orion version used was 0.7.2c. The PWS version used was 4.0. As usual we ran out httpbench application to see how they would perfom (as always we rebooted the machine before testing and closed down all applications, even explorer.exe to get as fair results as possible). These were the results: (the X-axis denotes the total number of requests done divided by 100). The first graph shows the number of pages being served per second:

As usual we also look at the latency for requests when load gets heavier:

What's interesting is not just that the JSP solution serves more pages per second and responds alot faster, but also that the ASP solution, for some reason shows a zig-zaw pattern with pauses when it doesn't serve anything We can also see that it seems that the ASP solution never quite reaches a stable state but continues to perform worse as time goes on. Why this happens is so far beyond our understanding, but we decided to look closer at this letting the ASP take some more beating for a longer time to see if performance keeps getting worse. This is what we found: (Latecy and Pages per second put in same graph):

We see that as time goes on the responses are getting slower and slower. We still don't know if this is due to memory leaks or something else, but one thing is certain, next week we'll see how IIS performs doing dynamic content and it will be interesting to see if it manages to handle this better.

JSP performance comparison, part I (1999-09-26)

We are making our first JSP performance comparisons. First out is a test of Orion vs. ServletExec, an engine we had heard performed great. This test was made on a P-II 266 Mhz with 96 Mb RAM using JDK1.3b1 on Windows NT 4.0 and a simple jsp file with a loop printing the numbers 1-100. Look at the source of the JSP here. The Orion version used was 0.7.2c. The PWS version used was 4.0 and the ServletExec version used was 2.1

We deployed this JSP in Orion and in the well-known ServletExec engine (ServletExec is a plugin engine and as a web server we used Microsoft Personal Web Server). The difference was staggering. First, let's look at the number of pages per second Orion and ServletExec managed to serve (the X-axis denotes the total number of requests done divided by 100). This number was, as usual gotten by using out httpbench program (see bottom for details).

We were truly amazed to see Orion outperforming ServletExex by 1145 %. (Yes, that means Orion served 12.45 times more pages in the same time). Now this has to be too good to be true? We figure a big part of this difference is that ServletExec is a plugin engine and not fully integrated with the web server. For that reason, part II of the JSP performance test will also include JRun (using the JRun Web Server), X and the Java Web Server. Stay tuned for these results to get amazed again.

Our httpbench program also measures the latency for the requests, and needless to say, the result qually amazing:

Web server comparison with Apache

This benchmark is a comparison of Orion vs Apache (one of the most well known web servers). It was conducted on a Pentium 133Mhz computer with 32Mb memory running Red Hat Linux 5.1. Orion was running under Blackdowns Java 2 port (thanks guys for the great performance it provided by the way). mod_jrun was installed in the Apache server to provide servlet support. All tests were performed by a remote host using five concurrent sockets to connect to the server and retrieve a single location, time after time. What was measured was general throughput and latency (time taken from the socket creation, during the writes and reads, to the sockets closing. The benchmarking application was custom made, it is a native NT executable, you can get it here (or the source here) to perform your own benchmarks.

Static content (small file, ~0.5k)
Apache: Pages/sec: 114.5. Latency: 43.6ms
Orion: Pages/sec: 219.2. Latency: 19,7ms

Dynamic content: (SimpleServlet)
Apache: Pages/sec: 24.7. Latency: 201.8ms
Orion: Pages/sec: 159.8. Latency: 32.2ms