Benchmarks
One of the main goals for the Orion Application Server has been to outperform everything else on the market. On this page we will investigate how we have succeeded and we will add the results we find in our own testing. We will also be happy to include independent benchmarks here, so if you have a servlet, jsp, ejb application, rmi application or anything else that seems suitable for benchmarking, please let us know. Send a mail to info@orionserver.com and we will be happy to include your benchmark here.
Disclaimer:
Before reading any further, remember that benchmarks are just benchmarks. No matter how much we try to make our benchmarks fair there is much room for factors that can make the benchmarks give the wrong picture of how things perform in real life situations. We publish these results because we often hear from our customers that they want to see us publish the results of the tests we perform. Also, it is about the only way we have to show that our server is performing as well as we say. To understand the performance of your environment, please do your own testing, don't just trust our numbers.
JSP engine performance (2000-07-17)
Update:
Allaire has kindly offered to help perform the JRun tests again at their test center to get as accurate numbers as possible. Until then, please do not read too much into their numbers since they are not yet validated by them.
WebLogic on the other hand has asked to be removed and do not wish to assist in making sure that we have accurate benchmark numbers, so we have no choice other than removing their numbers totally from this test. We hope that they will change their mind and help us make this survey as accurate and complete as possible, as far as benchmarks go, but we must respect their wish to not take part. Their numbers have been completely deleted from this test.
In our latest benchmark we tried to compare JSP performance between different JSP engines in a broader fashion than ever before. Instead of just using one JSP file for the test we used no less than 62 different JSP files testing different aspects of JSPs. All test JSPs were taken from the Jakarta Watchdog project. Watchdog is an open-source test suite for specification compatibility, which means it tests most aspects of the specification. This also makes it interesting for use in benchmarking since you can get this broad comparison, including scripting, bean-access, tag libraries, etc.
We wanted to test the most popular JSP implementations and chose the following five (we wanted to do more but we had to draw the line somewhere):
-
Orion Application Server (1.1.24)
- Apache Tomcat (3.1)
- Allaire JRun (3.0)
- Caucho Resin (1.2b1)
- X
The platform we ran the test on was the following:
- Hardware: AMD Athlon 850 Mb with 256 MB PC-133 SDRAM
- Supporting software: Windows 2000 Professional, JDK 1.3 Final
- Benchmarking software: Microsoft Web Application Stress Tool 1.1.293.1 (available at http://webtool.rte.microsoft.com)
The list of JSPs we used in our test can be found here.
We set the test up like this: We started the server we were testing on a clean computer just after a reboot. When we had installed the servers we deployed the watchdog web application and made sure that all JSPs got compiled. We started the webtool and let it run a first est on the server. Every test had 1 minute of warmup and 2 minutes of testing. On all servers we made four tests, First we let one thread hit the client sequentially, after that we used 4 threads to do the same, in a third test we let 8 threads open 2 sockets each to the server and in a fourth test we let 16 threads open 4 sockets each to the server. This pattern was used to test how well the server could scale under a higher stress level, when more connections was being made to the server.
The results were these: (In requests per second for all five servers, under the four different stress levels)
|
Stress level (Threads x Sockets/Thread)
|
||||
| Server |
1 x 1
|
4 x 1
|
8 x 2 (16)
|
16 x 4(64)
|
| Orion |
615.00
|
622.36
|
597.1
|
579.89
|
| Resin |
606.01
|
598.77
|
588.74
|
575.49
|
| JRun |
(386.17, to be revised)
|
(387.62, to be revised)
|
(370.1, to be revised)
|
(306.34, to be revised)
|
| X |
**
|
**
|
**
|
**
|
| Tomcat |
84.19
|
86.59
|
74.48
|
68.24
|
As a reminder, please note that JRun will be retested, until then, please do not read too much into their numbers.
The numbers expressed in a chart (with JRun and X taken out of the test, JRun being temporarily gone while we retest and X being gone on their request):
This data tells us two things.
First, there's a huge difference in performance between different JSP implementations. Two servers, Orion and Resin easily beats the rest by a big margin.
Second, there is also a difference in how well the servers scale under stress. Orion and Resin goes down to about 95 % of their single-thread capacity under the highest stress level, while the rest go down to about 80 %.
JSP vs ASP, part II (2000-05-21)
Our eariler JSP vs ASP benchmark got alot of attention in many different places. That test is now very old and is not very accurate anymore. Therefore we wanted to do a new benchmark using all the latest technology to see what the resuls would be. We used the following system to perform our tests:
-
Hardware: Intel Pentium III 500mhz with 256 MB RAM
-
Supporting software: Windows 2000 Advanced Server with JDK 1.3 final and Hotspot 2.0 Server VM.
-
Servers: Microsoft IIS 5.0 and Orion Application Server 1.0 RC2.
-
Benchmarking software: Microsoft Web Application Stress Tool 1.1.293.1 (available at http://webtool.rte.microsoft.com)
To get a test that is somewhat more real than a "Hello, World!" page but still simple enough to be able to write it exactly the same in ASP and JSP we made these two pages: bench.jsp and bench.asp. In it we test some dynamic output and static output as well as Session and Application variables. To test how it behaves both in terms of raw speed and how it scales we did 3 tests, scaling up the stress level in every test.
Now to the results. The Stress Tool reports much data and these raw data can be found here. The most important measurements are probably these:
-
Requests per second. The number of requests the server can serve.
- Time to Last Byte. The time an average user will have to wait until the whole page has loaded.
We have collected those numbers and made a table and a graph to show the results:
Raw performance (Requests per second):
|
Stress level (Threads x Sockets/Thread)
|
|||
| Server |
1 x 1
|
4 x 1
|
8 x 2 (16)
|
| Orion |
405.68
|
396.13
|
400.69
|
| IIS 5.0 |
145.82
|
103.59
|
115.65
|
Time to Last Byte in milliseconds (time the user has to wait until the whole page has loaded):
|
Stress level (Threads x Sockets/Thread)
|
|||
| Server | 1 x 1
|
4 x 1
|
8 x 2 (16)
|
| Orion |
0.04
|
0.8
|
37.12
|
| IIS 5.0 |
4.89
|
36.66 |
135.55
|
Our conclusion is that Orion and JSP easily beats ASP served by the most advanced Microsoft platform available (IIS 5.0 running on Windows 2000 Advanced Server).