Ken announced earlier this week that the next software release for the BUG includes some big changes. Many will be immediately visible to the user, but some of the biggest changes are coming under the hood. The new system is significantly more flexible allowing us to change the packages in our Linux distribution easily.

Our current stack is based on phoneME Advanced, a configuration not available in Open Embedded. That forced us to look at other options as we started working with Poky. The original prototype of the BUG used JamVM and and GNU Classpath, but we have been using phoneME for quite a while. Now was a perfect opportunity to look at other VMs. The obvious alternative is JamVM, but we also tested the Cacao virtual machine due to its support by the Jalimo and OpenMoko projects.

It seems not many people have been comparing VMs on ARM recently so some benchmarks were in order. We based our tests on this article on benchmarking. Our three benchmarks were Scimark2, Linpack, and startup time comparison.

First up was Scimark2, the most comprehensive of the benchmarks. The suite runs various algorithms and reports these scores in mega-flops per second. It also makes a composite score of these tests. JamVM is the clear winner. While Cacao has solid performance on the Monte Carlo prime number finding algorithm, it falls behind in all of the rest of the tests. PhoneME seems to be about 10-20% slower than JamVM in most tests.

Next up was the Linpack benchmark. The first graph is a measure of the speed of the processor in mega flops per second. The second graph is the overall time for running the benchmark so lower is better. Again here we see the same order as the Scimark2 tests. JamVM runs on top for both tests followed by phoneME and Cacao.

Finally, our startup time test. Starting up the OSGi stack currently takes a few seconds, so startup time of the VM has a significant impact on the overall boot time. For this benchmark we started up the Concierge OSGi framework – though it did not load any bundles since we had not ported all of them to the new build system at the time.

Here JamVM performed as expected with an impressive startup time. Note that this is not a constant time, these gains will increase with a larger load. When starting up our full OSGi stack you will probably gain a few seconds with JamVM.

All of this makes it look like the best choice is JamVM, but there are a lot of variables that are swept under the rug above. Most importantly, a 20% difference on a synthetic benchmark does not indicate a 20% improvement in end user experience.

Second, there are some differences in the configurations. We ran these tests using GNU Classpath on JamVM and Cacao. Keep in mind that GNU Classpath is J2SE-like in size, while phoneME Advanced is significantly lighter. This will make a big difference on a benchmark that measures full system performance. These benchmarks are designed for CPU throughput so the additional overhead of Classpath does not show up. However, both Cacao and JamVM can run with phoneME rather than Classpath, it is just not the default behavior in OpenEmbedded.

Perhaps the biggest red herring stems from us not building phoneME with JIT right now. This is due to a problem with older versions of the Linux kernel. With the new build system we will build phoneME with JIT which will gain significant performance. These benchmarks are comparing the current build of phoneME to the Cacao and JamVM default setups in OpenEmbedded, we are not comparing the maximum performance that can be squeezed out of each VM.

Finally, on Thursday Matthias Klose was able to get OpenJDK built for ARM. We will have to take a look at its performance as soon as we have it running on the BUG.

We still are very excited to see the community based VMs performing so well. While there are more factors to consider when choosing a JVM implementation (such as TCK validation, branding concerns, and community support) JamVM definitely warrants a look for anyone interested in embedded Java development on ARM based systems, it certainly impressed us. Look here for a follow up post sometime in the coming months.

I love language and have always been fascinated by its mechanics – how
the assembly of words in a sequence can either convey meaning or sow
confusion. Given my interest in the value of splitting structure
(grammar/syntax) and content (words), language has been a potent
catalyst. When I was younger, one of the questions I turned over in my
head all the time was – why can’t there be a language in which it was
impossible to speak nonsense – meaning a language where ANY combination
of words made sense somehow. It’s a wacky question but points, I think,
to an important strength of every successful language – it’s incredible
flexibility. Without the ability to write nonsense, nothing new could
be done. Our idea of nonsense is subjective. The poetry of e.e.
cummings would have likely been viewed as complete garbage 200 years
ago. Or how about a computer program?? But the other thing that
fascinates (inspires) me is that even though the interface between
words is formally defined as grammar, it is also totally open, meaning
those rules are really only a guide. Poetry routinely flouts the rules
to great effect. If I know the words of more than one language the
options for sentence construction increase, as long as the reader
understands the languages used and doesn’t mind dealing with no formal
grammatical structure. Words, like atoms, seem capable of infinite
types of combinations, but gain particular value when used with a
logical grammar (e.g. Shakespeare or Hemingway). All of the above, by
the way, is equally true of music.

Lately I have
been puzzling over creating what in essence is a grammar for the
construction of not word sentences but "object sentences". Is there a
formal way to define how objects should/could go together to create a
useful thing or "sentence"? Sounds crazy because maybe it is. But
consider electronics, which for the purposes of this argument I will
equate to a language – meaning it is contained and its moving pieces
defined. At a very high level every device, gadget and gizmo is
identical – power supply, processor(s), i/o devices. Mix them up in
lots of different ways and you get everything from a digital watch, to
an iPod, to an IBM mainframe; like the words of a sentence. Take vehicles of any kind – the same high level analysis produces the
same result – all vehicles are identical. So what? My point is that, at
least in some cases, there is a starting baseline of components (words)
for the construction of certain categories of product. If you could
elegantly define the interfaces between these basic categories perhaps
you could start to generalize and identify an interface "grammar", at
least for that "language". Would love to see that someday.