This further increases throughput of the lexer. On MRI this seems to save
around one second or so. It now sits at ~6,8 seconds in the big XML benchmark.
On JRuby, combined with some JIT options and invoke dynamic enabled, this can
reduce the average lexing time to around 3,5 seconds. Rubinius, also with a
few aggressive JIT options, seems to stick around 9 seocnds.
When using Git the resulting Gem will contain far too many useless files. For
example, the profile/ and spec/ directories are not needed when building Gems.
To make running benchmarks easier we'll track the XML file in Git in its
compressed form. I also decreased the size of the XML file from ~50 MB to
~10MB.
Yorick Peterse