Java VMs Compared

Ken announced earlier this week that the next software release for the BUG includes some big changes. Many will be immediately visible to the user, but some of the biggest changes are coming under the hood. The new system is significantly more flexible allowing us to change the packages in our Linux distribution easily.

Our current stack is based on phoneME Advanced, a configuration not available in Open Embedded. That forced us to look at other options as we started working with Poky. The original prototype of the BUG used JamVM and and GNU Classpath, but we have been using phoneME for quite a while. Now was a perfect opportunity to look at other VMs. The obvious alternative is JamVM, but we also tested the Cacao virtual machine due to its support by the Jalimo and OpenMoko projects.

It seems not many people have been comparing VMs on ARM recently so some benchmarks were in order. We based our tests on this article on benchmarking. Our three benchmarks were Scimark2, Linpack, and startup time comparison.

First up was Scimark2, the most comprehensive of the benchmarks. The suite runs various algorithms and reports these scores in mega-flops per second. It also makes a composite score of these tests. JamVM is the clear winner. While Cacao has solid performance on the Monte Carlo prime number finding algorithm, it falls behind in all of the rest of the tests. PhoneME seems to be about 10-20% slower than JamVM in most tests.

Next up was the Linpack benchmark. The first graph is a measure of the speed of the processor in mega flops per second. The second graph is the overall time for running the benchmark so lower is better. Again here we see the same order as the Scimark2 tests. JamVM runs on top for both tests followed by phoneME and Cacao.


Finally, our startup time test. Starting up the OSGi stack currently takes a few seconds, so startup time of the VM has a significant impact on the overall boot time. For this benchmark we started up the Concierge OSGi framework – though it did not load any bundles since we had not ported all of them to the new build system at the time.

Here JamVM performed as expected with an impressive startup time. Note that this is not a constant time, these gains will increase with a larger load. When starting up our full OSGi stack you will probably gain a few seconds with JamVM.

All of this makes it look like the best choice is JamVM, but there are a lot of variables that are swept under the rug above. Most importantly, a 20% difference on a synthetic benchmark does not indicate a 20% improvement in end user experience.

Second, there are some differences in the configurations. We ran these tests using GNU Classpath on JamVM and Cacao. Keep in mind that GNU Classpath is J2SE-like in size, while phoneME Advanced is significantly lighter. This will make a big difference on a benchmark that measures full system performance. These benchmarks are designed for CPU throughput so the additional overhead of Classpath does not show up. However, both Cacao and JamVM can run with phoneME rather than Classpath, it is just not the default behavior in OpenEmbedded.

Perhaps the biggest red herring stems from us not building phoneME with JIT right now. This is due to a problem with older versions of the Linux kernel. With the new build system we will build phoneME with JIT which will gain significant performance. These benchmarks are comparing the current build of phoneME to the Cacao and JamVM default setups in OpenEmbedded, we are not comparing the maximum performance that can be squeezed out of each VM.

Finally, on Thursday Matthias Klose was able to get OpenJDK built for ARM. We will have to take a look at its performance as soon as we have it running on the BUG.

We still are very excited to see the community based VMs performing so well. While there are more factors to consider when choosing a JVM implementation (such as TCK validation, branding concerns, and community support) JamVM definitely warrants a look for anyone interested in embedded Java development on ARM based systems, it certainly impressed us. Look here for a follow up post sometime in the coming months.

Share and Enjoy:
  • Digg
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • E-mail this story to a friend!
  • Live
  • Ma.gnolia
  • Reddit
  • Slashdot
  • SphereIt
  • StumbleUpon
  • TwitThis
  • YahooMyWeb

Tags:

13 Responses to “Java VMs Compared”

  1. Angel Roman Says:

    Kevin,

    Informative blog post.

    Thanks,
    Angel

  2. James Says:

    Great comparison. I’d like to repost on Javalobby with your permission. If you’re interested please contact me at the above address.
    Regards
    James

  3. Christian Thalinger Says:

    What version of CACAO did you use and what were the configure flags (in case try: -XX:+PrintConfig)? Does you ARM CPU have a FPU/VFP or does the Linux Kernel emulate the floating point instructions?

  4. Robert Lougher Says:

    From the documentation the bug uses a ARM1136JF-S core, which has a VFP coprocessor… I guess CACAO is the OpenEmbedded version. Ask Mr. Schuster. He’s just packaged PhoneME on OpenEmbedded so I guess it’s a moot point anyway…

  5. Kevin Schultz Says:

    Christian,

    We are using Cacao 0.98 with software floating point enabled. Our CPU is a Freescale mx31 (ARM11), which does have a hardware fpu. We would get a boost out of using that instead. The build does not have -XX:+PrintConfig as a config option, but I will try that, thanks for the tip.

    We are using the default OpenEmbedded build scripts rather than tuning them to our hardware, and they must be setup for the most general case. We were quite surprised by Cacao’s performance since we had expected it to be on top so this might explain that discrepancy. Additionally we have gotten big boosts out of phoneME lately now that we have JIT enabled, so I guess a second comparison will be in order.

    Thanks,
    Kevin Schultz

  6. Christian Thalinger Says:

    OK, that explains why CACAO is so slow at the floating point benchmarks. But just to be sure, is your toolchain softfloat configured?

  7. Chris Plummer Says:

    I’m not sure why you had issues with the JIT, since the ARM JIT has always been a stable part of the phoneME Linux/ARM port. However, glad to see you got it working now.

    If you have hardware VFP, you should also enable it in the JIT by building with CVM_JIT_USE_FP_HARDWARE=true. Otherwise the JIT will only use GPRs and call out to C functions for all floating point operations, or worse yet, call out to the assembler soft float library.

  8. Christian Thalinger Says:

    I just had some spare minutes and did a Scimark2 run on our hard-float ARM box, because I couldn’t get JamVM working on our soft-float board. Have to talk to Robert about this issue. The ARM box does not have a FPA, so the kernel emulates them. Here are the numbers:

    JamVM:

    SciMark 2.0a

    Composite Score: 0.4885741242527907
    FFT (1024): 0.29904493861307696
    SOR (100×100): 0.6581993778741051
    Monte Carlo : 0.29967875915390363
    Sparse matmult (N=1000, nz=5000): 0.5820827961957977
    LU (100×100): 0.6038647494270701

    java.vendor: GNU Classpath
    java.version: 1.5.0
    os.arch: arm
    os.name: Linux
    os.version: 2.4.22-iyonix

    CACAO:

    SciMark 2.0a

    Composite Score: 1.0139034019666382
    FFT (1024): 0.5376767705902252
    SOR (100×100): 1.581337897500241
    Monte Carlo : 0.6617709243687945
    Sparse matmult (N=1000, nz=5000): 1.273631846838955
    LU (100×100): 1.015099570534974

    java.vendor: GNU Classpath
    java.version: 1.5.0
    os.arch: arm
    os.name: Linux
    os.version: 2.4.22-iyonix

  9. gbenson Says:

    What did you use for your graphs by the way? I like them :)

  10. Bug Blogger » Blog Archive » Java VMs Compared II Says:

    [...] received great feedback from our previous comparison of Java virtual machines. We acted on all of the comments we received from the Cacao community by tweaking our build [...]

  11. Robert Schusters Blog » Blog Archive » PhoneME Advanced Foundation (with JIT) at Jalimo Says:

    [...] VM. See, this code is still does usefull things for us. :-) Buglabs recently did a comparison of Cacao, JamVM and PhoneME Advanced (interpreted only) on ARM systems. Surprisingly (or not :-) ) [...]

  12. Lizing Says:

    Hi,

    May I ask about what is your ram size that your use in this comparison?

    Thank your,
    Lizing

  13. Ed Holdsworth Says:

    did you manage to get comparison figures for openjdk?

Leave a Reply