Previously the on_path method itself would specify a block to use when a
node was matched. This makes it impossible to customize this behaviour
which is needed when a path is used in an operator or predicate. By
letting the #compile() method specify the block other handlers can
overwrite it.
This also comes with some changes to the specs as the old behaviour of
the Evaluator was incorrect. The Evaluator would bail after matching a
single node but instead it's meant to continue until it runs out of
parent nodes.
This currently supports index predicates (e.g. "foo[10]") and predicates
using paths (e.g. foo[bar/baz]). The usage of boolean operators and more
complex expressions has not yet been tested as these are not yet
supported in the first place.
This currently comes with exactly 0% test coverage. Once I've
implemented all required handler methods I'll be updating the current
evaluator tests to use the compiler instead. This removes the need for
writing an entirely new set of tests.
Currently the compiler is only capable of compiling basic expressions
such as "foo", "foo/bar" and "foo[@x="y"]/bar".
This change is broken up in to two parts:
1. Using a Hash to track if a node is already in a NodeSet
2. Only calling take_ownership when an owner is set
== Using a Hash
Previously various methods such as NodeSet#push and NodeSet#unshift
would call Array#include? (on the internal "nodes" Array) to see if a
node is already present in the set. This is quite problematic
performance wise, especially for large NodeSets. In fact, for the
attached benchmark the vast majority of the time was spent in
Array#include? calls.
Because a NodeSet demands ordering of nodes and must be able to access
them by index (something Set can't do without relying on Enumerable), a
Hash is used to separately keep track of what nodes are in a NodeSet.
This means that checking the presence of a node is simply a matter of
checking a Hash key's presence.
== Calling take_ownership
The if-check for the "owner" variable has been moved out of the
"take_ownership" method and into the methods that call "take_ownership".
This ensures the method isn't called in the first place if no owner is
present, at the cost of slightly more code repetition. The same applies
to the "remove_ownership" method.
== Conclusion
The combined result is a speedup of about 50x when running the attached
concurrent_time_bench.rb benchmark.
While the performance difference between the old and new approach is
pretty much negligible, it's simply not needed to use #shift/#unshift
here.
Thanks to Mon_Ouie from the #ruby IRC channel for suggesting this.
Without this the following could happen:
1. Thread A acquires the lock and sets the ownership to A.
2. Thread A yields and returns
3. Thread B tries to acquire the lock
4. At this exact moment Thread A calls the "synchronize" method again
and sees that the "owner" variable is still set to Thread A
5. Both thread A and B can now access the underlying data in parallel,
possibly leading to corrupted objects
This can be demonstrated using the following script:
require 'oga'
lru = Oga::LRU.new(64)
threads = 50.times.map do
Thread.new do
loop do
number = rand(100)
lru[number] = number
end
end
end
threads.each(&:join)
Run this for a while on either JRuby or Rubinius and you'll end up with
errors such as "ConcurrencyError: Detected invalid array contents due to
unsynchronized modifications with concurrent users" on JRuby or
"ArgumentError: negative array size" on Rubinius.
Resetting the owner variable ensures the above can never happen. Thanks
to @chrisseaton for bringing this up earlier today.
Prevents a superfluous end tag of a self-closing HTML tag from
closing its parent element prematurely, for example:
```html
<object><param></param><param></param></object>
```
(note <param> is self closing) being turned into:
```html
<object><param/></object><param/>
```
This is a Nokogiri extension (as far as I'm aware) but it's useful
enough to also include in Oga. Selectors such as "foo:nth(2)" are simply
compiled to XPath "descendant::foo[position() = 2]".
Fixes#123
```
element = Oga::XML::Element.new(:name => 'div')
some_node.replace(element)
```
You can also pass a `String` to `replace` and it will be replaced with
a `Oga::XML::Text` node
```
some_node.replace('this will replace the current node with a text node')
```
closes#115
Currently this only disabled the automatic insertion of closing tags, in
the future this may also disable other features if deemed worth the
effort.
Fixes#107
This ensures that entities such as "½" are decoded properly.
Previously this would be ignored as the regular expression used for this
only matched [a-zA-Z].
This was adapted from PR #111.
Previous HTML such as this would be lexed incorrectly:
<div>
<ul>
<li>foo
</ul>
inside div
</div>
outside div
The lexer would see this as the following instead:
<div>
<ul>
<li>foo</li>
inside div
</ul>
outside div
</div>
This commit exposes the name of the closing tag to
XML::Lexer#on_element_end (omitted for self closing tags). This can be
used to automatically close nested tags that were left open, ensuring
the above HTML is lexer correctly.
The new setup ignores namespace prefixes as these are not used in HTML,
XML in turn won't even run the code to begin with since it doesn't allow
one to leave out closing tags.
By encoding single/double quotes we can potentially break input, so lets
stop doing this. This now ensures that this:
<foo>a"b</foo>
Is actually serialized back into the exact same instead of being
serialized into:
<foo>a"b</foo>
This allows for more fine grained control over when to close certain
elements. For example, an unclosed <tr> element should be closed first
when bumping into any element other than <td> or <th>. Using the old
NodeNameSet this would mean having to list every possible HTML element
out there. Using this new setup one can just create a whitelist of the
<td> and <th> elements.
When closing certain HTML elements the lexer should also close whatever
parent elements remain. For example, consider the following HTML:
<table>
<thead>
<tr>
<th>Foo
<th>Bar
<tbody>
...
</tbody>
</table>
Here the "<tbody>" element shouldn't only close the "<th>Bar" element
but also the parent "<tr>" and "<thead>" elements. This ensures we'd end
up with the following HTML:
<table>
<thead>
<tr>
<th>Foo</th>
<th>Bar</th>
</tr>
</thead>
<tbody>
...
</tbody>
</table>
Instead of garbage along the lines of this:
<table>
<thead>
<tr>
<th>Foo</th>
<th>Bar</th>
<tbody>
...
</tbody>
</table></tr></thead>
Fixes#99 (hopefully for good this time)
By using AST::Node#children directly with a splat we save ourselves an
extra method call. This in turn speeds up both the
xpath/evaluator/big_xml_average_bench.rb and
xpath/evaluator/node_matches_bench.rb benchmarks a little bit.
Using the benchmark xpath/evaluator/node_matches_bench.rb the results
prior to this commit were as following for 3 cases:
name only: 737633 i/s
namespace wildcard: 612196 i/s
name wildcard: 516030 i/s
With this commit said numbers have changed to the following:
name only: 746086 i/s
namespace wildcard: 1097168 i/s
name wildcard: 1151255 i/s
This results in the following increase of performance for each case:
name only: 1,011x (insignificant)
namespace wildcard: 1,79x
name wildcard: 2,23x
In the benchmark xpath/evaluator/big_xml_average_bench.rb the difference
isn't really noticable as said benchmark only queries elements by names,
of which the performance hasn't really improved.
This ensures that HTML such as this:
<li>foo
<li>bar
is parsed as this:
<li>foo</li>
<li>bar</li>
and not as this:
<li>
foo
<li>bar</li>
</li>
Fixes#97
This makes it easier to automatically insert preceding tokens when
starting a new element as we now have access to the name. Previously
on_element_start would be invoked first which doesn't receive an
argument.
The XML/HTML lexer is now capable of processing most invalid XML/HTML
(that I can think of at least). This is achieved by inserting missing
closing tags (where needed) and/or ignoring excessive closing tags. For
example, HTML such as this:
<a></a></p>
Results in the following tokens:
[:T_ELEM_START, nil, 1]
[:T_ELEM_NAME, 'a', 1]
[:T_ELEM_CLOSE, nil, 1]
In turn this HTML:
<a>
Results in these tokens:
[:T_ELEM_START, nil, 1]
[:T_ELEM_NAME, 'a', 1]
[:T_ELEM_CLOSE, nil, 1]
Fixes#84
Previously a single Ragel machine was used for processing HTML
script and style tags. This had the unfortunate side-effect that the
following was not parsed correctly (while being valid HTML):
<script>
var foo = "</style>";
</script>
The same applied to style tags:
<style>
/* </script> */
</style>
By using separate machines we can work around the above issue. The
downside is that this can produce multiple T_TEXT nodes, which have to
be stitched back together in the parser.
Similar to comments (ea8b4aa92f) and CDATA
tags (8acc7fc743) processing instructions
are now lexed in separate chunks _with_ proper support for streaming
input.
Related issue: #93
Instead of using a single token (T_CDATA) for a CDATA tag the lexer now
uses 3 tokens:
1. T_CDATA_START
2. T_CDATA_BODY
3. T_CDATA_END
The T_CDATA_BODY token can occur multiple times and is turned into a
single value in the XML parser. This is similar to the way strings are
lexed.
By changing the way CDATA tags are lexed Oga can now lex CDATA tags
containing newlines when using an IO as input. For example, this would
previously fail:
Oga.parse_xml(StringIO.new("<![CDATA[\nfoo]]>"))
Because IO input reads input per line the input for the lexer would be
as following:
"<![CDATA[\n"
"foo]]>"
Related issues: #93
This cache is flushed whenever Element#register_namespace is called.
When this cache is flushed it's also recursively flushed for all child
elements. This makes calls to Element#register_namespace a bit more
expensive but in turn calls to Element#available_namespaces will be a
lot faster.
The results of these methods is now cached until a Node is moved into
another NodeSet. This reduces the time spent in the
xpath/evaluator/big_xml_average_bench.rb benchmark from roughly 10
seconds to roughly 5 seconds per iteration.
In HTML the text of a script/style tag should be left untouched, no
entities must be converted. Doing so would break Javascript such as the
following:
foo&&bar;
Such code is often the result of minifiers doing their dirty business.
This was broken by introducing the process of lazy decoding of XML/HTML
entities. The new setup works similar to how XML::Text#text decodes any
entities that may be present.
Fixes#91
When querying an XML document that explicitly defines the default XML
namespace the XPath evaluator now correctly matches all nodes within
that namespace if no namespace prefix is given in the query. Previously
this would always return an empty set.
Instead of trying to make this class thread-safe I'm going with the
option of simply declaring it unsafe to mutate instances of XML::Text
while reading it in parallel. This removes the need for Mutex
allocations and keeps the code simple.
Fixes#82
Currently all operators are left-associative with no particular precedence. This
causes a few specs to fail for now. Outside of that the new parser should be
able to parse the same input as the Racc based parser.
Removing this makes the process of parsing larger XML documents a bit faster.
The downside is that NodeSet#initialize will no longer filter out duplicate
nodes, though this is not something Oga itself relies upon.
Methods such as NodeSet#push still do ignore elements already present.
This was utterly broken, mainly due to me overlooking it. There are now 2 new
callbacks to handle this properly:
* on_attribute: to handle a single attribute/value pair
* on_attributes: to handle a collection of attributes (as returned by
on_attribute)
By default on_attribut returns a Hash, on_attributes in turn merges all
attribute hashes into a single one. This ensures that on_element _actually_
receives the attributes as a Hash, instead of an Array with random
nil/XML::Attribute values.
Using Array#+ for large sets (e.g. in the benchmarks) is _really_ slow.
Interesting enough Array#unshift uses as much memory as the Racc parser and is
about as fast, even though it has to move memory around.
Yorick Peterse
Dan Fockler
Jakub Pawlowicz
Tero Tasanen