Update: There is now a part III to this series.

We received great feedback from our previous comparison of Java virtual machines. We acted on all of the comments we received from the Cacao community by tweaking our build configuration. Additionally our upcoming 1.3 release includes a newer Linux kernel (2.6.24). This alleviates the problems we had with phoneME’s and JIT. Here are the same benchmarks we used previously, reflecting the new build configurations.

PhoneME really came into its own now that JIT can be enabled without causing the system to crash. It will be great to pass on serious speed improvements to our users. those who upgrade their software should be able to notice the difference in speed (along with many other great improvements).

Unfortunately, we had little success improving Cacao’s performance. We significantly delayed posting this so that we could try tweaking the configuration as much as possible, but there was no pay off. On most hardware Cacao will benchmark faster than JamVM, and should come close to phoneME. For some unknown reason the combination of OpenEmbedded’s Cacao and our specific hardware does not perform as expected. You should test Cacao on your own hardware for a more accurate portrayal of its performance.

However, phoneME is not perfect. There is a major difference between using Cacao with the GNU Classpath libraries and using phoneME with its library. GNU Classpath is aiming at a J2SE sized set of libraries, while phoneME is J2ME-like so this is to be expected. But phoneME is significantly more short of J2ME than Classpath is of J2SE.

The biggest of detriment of phoneME is the graphics toolkit. With PhoneME we are limited to the AWT toolkit, the original Java GUI library. It has many limitations – from its simple aesthetic appearance to feature shortcomings like the deficient graphics library. PhoneME doesn’t have JSR 209 – Swing for J2ME, since it has not been open sourced. It is also missing several other libraries you need for Mobile Processing. With Cacao we can definately use the Swing library, and most likely can use all of Processing.

Compare the two screen shots below. First up is the TipCalc application using AWT. The second is the same application converted to use Swing. The difference is quite striking.

For now we will continue to use phoneME, but there are some other options for a better interface. Perhaps the most interesting is the new Lightweight UI Toolkit (LWUIT). It feels much like Swing but is significantly easier to port to our system. Another option is the Clutter toolkit, currently used by the Poky applications in our distribution. It currently has no Java bindings but does them for Python, C/C++, and Ruby (among others). A Java binding would certainly be possible. With back end support provided by OpenGL, it would allow us to do more graphical applications than are currently possible.

While new GUI libraries are not going to be coming in the near future, you will be seeing our 1.3 release of the file system very soon. This will bring significant speed boosts to your BUG, so stay tuned for an announcement on a final production tested version.

If you want to jump in and play with the development version now, code is always available and the associated documentation has been posted. And don’t forget to join the Bug-Dev mailing list to get involved with our development discussions on topics like these.

For those who are not BUG users and are just interested in the benchmarks of VMs, I hope these help guide you in your decisions. However we must caution that the VMs perform quite differently depending on your processor. Several people responded to the previous post by running these same benchmarks on different hardware and came up with a completely different result. Our benchmarks are quite repeatable on our hardware so we are confident in these results for our case, but your case might be entirely different. The benchmarks are available here and here, and the time benchmarking takes compared to the value of the results makes it definately worth doing.

Ken announced earlier this week that the next software release for the BUG includes some big changes. Many will be immediately visible to the user, but some of the biggest changes are coming under the hood. The new system is significantly more flexible allowing us to change the packages in our Linux distribution easily.

Our current stack is based on phoneME Advanced, a configuration not available in Open Embedded. That forced us to look at other options as we started working with Poky. The original prototype of the BUG used JamVM and and GNU Classpath, but we have been using phoneME for quite a while. Now was a perfect opportunity to look at other VMs. The obvious alternative is JamVM, but we also tested the Cacao virtual machine due to its support by the Jalimo and OpenMoko projects.

It seems not many people have been comparing VMs on ARM recently so some benchmarks were in order. We based our tests on this article on benchmarking. Our three benchmarks were Scimark2, Linpack, and startup time comparison.

First up was Scimark2, the most comprehensive of the benchmarks. The suite runs various algorithms and reports these scores in mega-flops per second. It also makes a composite score of these tests. JamVM is the clear winner. While Cacao has solid performance on the Monte Carlo prime number finding algorithm, it falls behind in all of the rest of the tests. PhoneME seems to be about 10-20% slower than JamVM in most tests.

Next up was the Linpack benchmark. The first graph is a measure of the speed of the processor in mega flops per second. The second graph is the overall time for running the benchmark so lower is better. Again here we see the same order as the Scimark2 tests. JamVM runs on top for both tests followed by phoneME and Cacao.

Finally, our startup time test. Starting up the OSGi stack currently takes a few seconds, so startup time of the VM has a significant impact on the overall boot time. For this benchmark we started up the Concierge OSGi framework – though it did not load any bundles since we had not ported all of them to the new build system at the time.

Here JamVM performed as expected with an impressive startup time. Note that this is not a constant time, these gains will increase with a larger load. When starting up our full OSGi stack you will probably gain a few seconds with JamVM.

All of this makes it look like the best choice is JamVM, but there are a lot of variables that are swept under the rug above. Most importantly, a 20% difference on a synthetic benchmark does not indicate a 20% improvement in end user experience.

Second, there are some differences in the configurations. We ran these tests using GNU Classpath on JamVM and Cacao. Keep in mind that GNU Classpath is J2SE-like in size, while phoneME Advanced is significantly lighter. This will make a big difference on a benchmark that measures full system performance. These benchmarks are designed for CPU throughput so the additional overhead of Classpath does not show up. However, both Cacao and JamVM can run with phoneME rather than Classpath, it is just not the default behavior in OpenEmbedded.

Perhaps the biggest red herring stems from us not building phoneME with JIT right now. This is due to a problem with older versions of the Linux kernel. With the new build system we will build phoneME with JIT which will gain significant performance. These benchmarks are comparing the current build of phoneME to the Cacao and JamVM default setups in OpenEmbedded, we are not comparing the maximum performance that can be squeezed out of each VM.

Finally, on Thursday Matthias Klose was able to get OpenJDK built for ARM. We will have to take a look at its performance as soon as we have it running on the BUG.

We still are very excited to see the community based VMs performing so well. While there are more factors to consider when choosing a JVM implementation (such as TCK validation, branding concerns, and community support) JamVM definitely warrants a look for anyone interested in embedded Java development on ARM based systems, it certainly impressed us. Look here for a follow up post sometime in the coming months.