Go to file

Yorick Peterse 2689d3f65a Initial setup using a C extension. While I've tried to keep Oga pure Ruby for as long as possible the performance of Ragel's Ruby output was not worth the trouble. For example, lexing 10MB of XML would take 5 to 6 seconds at least. Nokogiri on the other hand can parse that same XML into a DOM document in about 300 miliseconds. Such a big performance difference is not acceptable. To work around this the XML/HTML lexer will be implemented in C for MRI/Rubinius and Java for JRuby. For now there's only a C extension as I haven't read up yet on the JRuby API. The end goal is to provide some sort of Ragel "template" that can be used to generate the corresponding C/Java extension code. This would remove the need of duplicating the grammar and associated code. The native extension setup is a hybrid between native and Ruby. The raw Ragel stuff happens in C/Java while the actual logic of actions happens in Ruby. This adds a small amount of overhead but makes it much easier to maintain the lexer. Even with this extra overhead the performance is much better than pure Ruby. The 10MB of XML mentioned above is lexed in about 600 miliseconds. In other words, it's 10 times faster.		2014-05-05 00:31:28 +02:00
benchmark	Cleaned up the average timing benchmark.	2014-05-01 13:43:33 +02:00
checksum	Basic project layout.	2014-02-26 19:50:16 +01:00
doc	Basic project layout.	2014-02-26 19:50:16 +01:00
ext/liboga	Initial setup using a C extension.	2014-05-05 00:31:28 +02:00
lib	Initial setup using a C extension.	2014-05-05 00:31:28 +02:00
profile	Run memory profiling for 60 seconds.	2014-05-01 21:47:51 +02:00
spec	Contextual pull parsing.	2014-04-29 23:05:49 +02:00
task	Initial setup using a C extension.	2014-05-05 00:31:28 +02:00
.editorconfig	Initial setup using a C extension.	2014-05-05 00:31:28 +02:00
.gitignore	Initial setup using a C extension.	2014-05-05 00:31:28 +02:00
.ruby-version	Added a .ruby-version file.	2014-03-18 18:08:25 +01:00
.travis.yml	Allow JRuby to fail for now.	2014-03-18 00:13:33 +01:00
.yardopts	Basic project layout.	2014-02-26 19:50:16 +01:00
Gemfile	Basic project layout.	2014-02-26 19:50:16 +01:00
LICENSE	Added a license.	2014-02-26 22:20:47 +01:00
MANIFEST	Initial setup using a C extension.	2014-05-05 00:31:28 +02:00
README.md	Removed the donations section.	2014-03-25 23:55:10 +01:00
Rakefile	Initial setup using a C extension.	2014-05-05 00:31:28 +02:00
oga.gemspec	Initial setup using a C extension.	2014-05-05 00:31:28 +02:00

README.md

Oga

Oga is (or will be) a pure Ruby, thread-safe HTML (and XML in the future) parser that doesn't trigger segmentation faults on Ruby implementations other than MRI. Oga will initially not focus on performance but instead will focus on proper handling of encodings, stability and a sane API. More importantly it will be pure Ruby only. No C extensions, no Java, no x86 64 assembly, just Ruby.

From Wikipedia:

Oga: A large two-person saw used for ripping large boards in the days before power saws. One person stood on a raised platform, with the board below him, and the other person stood underneath them.

Planned Features

Full support for HTML(5)
Full support for XML, DTDs will probably be ignored.
Support for xpath and CSS selector based queries
SAX/pull parsing APIs that don't make you want to cut yourself

Features

A README

Requirements

Ruby

Development requirements:

Ragel
Racc
Other stuff

Usage

Basic DOM parsing example:

require 'oga'

parser   = Oga::Parser::DOM.new
document = parser.parse('<p>Hello</p>')

puts document.css('p').first.text # => "Hello"

Pull parsing:

require 'oga'

parser = Oga::Parser::Pull.new('<p>Hello</p>')

parser.each do |node|
  puts node.text
end

These examples will probably change once I actually start writing some code.

Why Another HTML/XML parser?

Currently there are a few existing parser out there, the most famous one being Nokogiri. Another parser that's becoming more popular these days is Ox. Ruby's standard library also comes with REXML.

The sad truth is that these existing libraries are problematic in their own ways. Nokogiri for example is extremely unstable on Rubinius. On MRI it works because of the non conccurent nature of MRI, on Jruby it works because it's implemented as Java. Nokogiri also uses libxml2 which is a massive beast of a library, is not thread-safe and problematic to install on certain platforms (apparently). I don't want to compile libxml2 every time I install Nokogiri either.

To give an example about the issues with Nokogiri on Rubinius (or any other Ruby implementation that is not MRI or JRuby), take a look at these issues:

Some of these have been fixed, some have not. The core problem remains: Nokogiri acts in a way that there can be a large number of places where it might break due to throwing around void pointers and what not and expecting that things magically work. Note that I have nothing against the people running these projects, I just heavily, heavily dislike the resulting codebase one has to deal with today.

Ox looks very promising but it lacks a rather crucial feature: parsing HTML (without using a SAX API). It's also again a C extension making debugging more of a pain (at least for me).

I just want an HTML parser that I can rely on stability wise and that is written in Ruby so I can actually debug it. In theory it should also make it easier for other Ruby developers to contribute.

Oga is an attempt at solving this problem. By writing it in pure Ruby the initial performance will probably not be as great. However, I feel this is a problem with individual Ruby implementations, not the language itself. Also, by writing it in Ruby we don't have to deal with all the crazy things of C/C++ or even Java.

In theory it should also allow it to run on every Ruby implementation, be it JRuby, Rubinius, Topaz or even mruby.

License

All source code in this repository is licensed under the MIT license unless specified otherwise. A copy of this license can be found in the file "LICENSE" in the root directory of this repository.