Commit Graph

752 Commits

Author SHA1 Message Date
Yorick Peterse 4f03bf19c1 XPath compiler support for "ancestor" 2015-08-19 20:14:21 +02:00
Yorick Peterse f96b30fb1f Tag the XPath compiler as private 2015-08-19 20:14:21 +02:00
Yorick Peterse f7484f1c8d Use each_ancestor for ancestor-or-self in XPath
The XPath compiler now uses XML::Node#each_ancestor for compiling the
"ancestor-or-self" axis.
2015-08-19 20:14:20 +02:00
Yorick Peterse 52741a3b78 Added XML::Node#each_ancestor
This method can be used to walk through the ancestor tree of a Node.
2015-08-19 20:14:20 +02:00
Yorick Peterse db39b25546 XPath compiler support for ancestor-or-self
This also comes with some changes to the specs as the old behaviour of
the Evaluator was incorrect. The Evaluator would bail after matching a
single node but instead it's meant to continue until it runs out of
parent nodes.
2015-08-19 20:14:20 +02:00
Yorick Peterse d8fbaf75d8 Ruby generator support for while loops 2015-08-19 20:14:20 +02:00
Yorick Peterse 9925b2a9c9 XPath compiler support for predicates
This currently supports index predicates (e.g. "foo[10]") and predicates
using paths (e.g. foo[bar/baz]). The usage of boolean operators and more
complex expressions has not yet been tested as these are not yet
supported in the first place.
2015-08-19 20:14:20 +02:00
Yorick Peterse 6f6151fd52 Added Ruby generator support for Symbols 2015-08-19 20:14:20 +02:00
Yorick Peterse 05a57a807e XPath compiler support for variable references 2015-08-19 20:14:20 +02:00
Yorick Peterse cf2405998b Ruby::Generator support for #[] methods 2015-08-19 20:14:20 +02:00
Yorick Peterse 17f6b3a3bb Added simple method docs for the XPath compiler 2015-08-19 20:14:20 +02:00
Yorick Peterse 2c1b4e7cbc Support for generating "else" statements 2015-08-19 20:14:20 +02:00
Yorick Peterse ebcadd1cb8 XPath compiler support for absolute paths/integers 2015-08-19 20:14:20 +02:00
Yorick Peterse 94f7f85dc3 Added XML::Document#root_node 2015-08-19 20:14:20 +02:00
Yorick Peterse 3300a6df49 Added XPath::Compiler.compile_with_cache 2015-08-19 20:14:20 +02:00
Yorick Peterse 6d01adafc7 XPath compiler now actually returns a Proc 2015-08-19 20:14:20 +02:00
Yorick Peterse d11a754946 Basic support for compiling XPath to Ruby
This currently comes with exactly 0% test coverage. Once I've
implemented all required handler methods I'll be updating the current
evaluator tests to use the compiler instead. This removes the need for
writing an entirely new set of tests.

Currently the compiler is only capable of compiling basic expressions
such as "foo", "foo/bar" and "foo[@x="y"]/bar".
2015-08-19 20:14:20 +02:00
Yorick Peterse 337d126264 Added Ruby::Generator class
This class will be used to serialize a Ruby AST back to valid Ruby
source code (as a String).
2015-08-19 20:14:20 +02:00
Yorick Peterse 6673f176d8 Added Oga::Ruby::Node
This class will be used for building Ruby ASTs that will be generated
based on XPath expressions.
2015-08-19 20:14:20 +02:00
Yorick Peterse c25879f18e Removed some explicit returns in .rll files 2015-08-19 20:14:19 +02:00
Yorick Peterse 08c965bfbc Basic specs for the XPath compiler 2015-08-19 20:14:19 +02:00
Yorick Peterse f10b7aa065 Super basic boilerplate for the XPath compiler 2015-08-19 20:14:19 +02:00
Yorick Peterse 399403c290 Release 1.2.3 2015-08-19 12:09:14 +02:00
Yorick Peterse 4f94d03a85 Improved NodeSet performance by a factor of 50
This change is broken up in to two parts:

1. Using a Hash to track if a node is already in a NodeSet
2. Only calling take_ownership when an owner is set

== Using a Hash

Previously various methods such as NodeSet#push and NodeSet#unshift
would call Array#include? (on the internal "nodes" Array) to see if a
node is already present in the set. This is quite problematic
performance wise, especially for large NodeSets. In fact, for the
attached benchmark the vast majority of the time was spent in
Array#include? calls.

Because a NodeSet demands ordering of nodes and must be able to access
them by index (something Set can't do without relying on Enumerable), a
Hash is used to separately keep track of what nodes are in a NodeSet.
This means that checking the presence of a node is simply a matter of
checking a Hash key's presence.

== Calling take_ownership

The if-check for the "owner" variable has been moved out of the
"take_ownership" method and into the methods that call "take_ownership".
This ensures the method isn't called in the first place if no owner is
present, at the cost of slightly more code repetition. The same applies
to the "remove_ownership" method.

== Conclusion

The combined result is a speedup of about 50x when running the attached
concurrent_time_bench.rb benchmark.
2015-08-19 01:36:37 +02:00
Dan Fockler be7bc8f423 make string comparison faster 2015-08-15 06:36:27 -07:00
Dan Fockler fc38b39aa3 make string comparison a bit faster 2015-08-15 06:34:09 -07:00
Daniel Fockler 496811a23f Fixes #127 2015-08-14 16:15:49 -07:00
Yorick Peterse 0977be81bb Release 1.2.2 2015-08-14 14:49:26 +02:00
Yorick Peterse 9b98e75115 Use pop/push vs shift/unshift in each_node
While the performance difference between the old and new approach is
pretty much negligible, it's simply not needed to use #shift/#unshift
here.

Thanks to Mon_Ouie from the #ruby IRC channel for suggesting this.
2015-08-07 14:08:54 +02:00
Yorick Peterse 32b75bf62c Reset the owner ivar in LRU#synchronize
Without this the following could happen:

1. Thread A acquires the lock and sets the ownership to A.
2. Thread A yields and returns
3. Thread B tries to acquire the lock
4. At this exact moment Thread A calls the "synchronize" method again
   and sees that the "owner" variable is still set to Thread A
5. Both thread A and B can now access the underlying data in parallel,
   possibly leading to corrupted objects

This can be demonstrated using the following script:

    require 'oga'

    lru = Oga::LRU.new(64)

    threads = 50.times.map do
      Thread.new do
        loop do
          number = rand(100)

          lru[number] = number
        end
      end
    end

    threads.each(&:join)

Run this for a while on either JRuby or Rubinius and you'll end up with
errors such as "ConcurrencyError: Detected invalid array contents due to
unsynchronized modifications with concurrent users" on JRuby or
"ArgumentError: negative array size" on Rubinius.

Resetting the owner variable ensures the above can never happen. Thanks
to @chrisseaton for bringing this up earlier today.
2015-08-03 21:47:40 +02:00
Jakub Pawlowicz ed3cbe7975 Fixes #129 - lexing superfluous end tags.
Prevents a superfluous end tag of a self-closing HTML tag from
closing its parent element prematurely, for example:

```html
<object><param></param><param></param></object>
```

(note <param> is self closing) being turned into:

```html
<object><param/></object><param/>
```
2015-07-23 13:16:11 +01:00
Yorick Peterse 5d0e8c99af Release 1.2.1 2015-07-01 00:24:07 +02:00
Jakub Pawlowicz 6786dde3f0 Splits entity decoding into two steps.
Doing decimal & hex decoding separately results in a nicer code
which does not refer to string matching and slicing.
2015-06-30 17:56:26 +02:00
Jakub Pawlowicz 6fc3ef425b Fixes #118 - decoding invalid entities.
Previous regular expression was too greedy in terms of matching
letters from outside of A-F hex scope, and matching letters when
not in hex mode.
2015-06-30 17:56:26 +02:00
Yorick Peterse 8990a62224 Release 1.2.0 2015-06-30 10:58:05 +02:00
Yorick Peterse 71960fff87 Added CSS :nth() pseudo class
This is a Nokogiri extension (as far as I'm aware) but it's useful
enough to also include in Oga. Selectors such as "foo:nth(2)" are simply
compiled to XPath "descendant::foo[position() = 2]".

Fixes #123
2015-06-29 20:51:38 +02:00
Yorick Peterse d26b48feb4 CSS parsing support for commas
The lexer already had the basic plumbing in place but apparently I
completely forgot to also implement the required bits in the parser.

Fixes #121
2015-06-29 18:59:01 +02:00
Yorick Peterse dccf7a7e06 Release 1.1.0 2015-06-29 16:45:21 +02:00
Tero Tasanen 0b4791b277 Ability to replace a node with another node or string
```
element = Oga::XML::Element.new(:name => 'div')
some_node.replace(element)
```

You can also pass a `String` to  `replace` and it will be replaced with
a `Oga::XML::Text` node

```
some_node.replace('this will replace the current node with a text node')
```

closes #115
2015-06-17 21:27:50 +03:00
Yorick Peterse fe7e2e4f74 Use ternary "if" when encoding attribute values 2015-06-16 22:48:42 +02:00
Yorick Peterse 074b53c18c Fix entity encoding of attribute values
This ensures that single and double quotes are also encoded, previously
they would be left as is.

Fixes #113
2015-06-16 22:47:10 +02:00
Yorick Peterse e0837fd44e Fixed YARD parameter name 2015-06-16 11:14:22 +02:00
Yorick Peterse b701a9bdd4 Release 1.0.3 2015-06-16 11:13:06 +02:00
Yorick Peterse b6d34a406d Removed redundant returns 2015-06-16 00:51:51 +02:00
Yorick Peterse bcffd86c50 Removed usage of YARD @!attribute macros 2015-06-16 00:04:10 +02:00
Yorick Peterse 2c18a51ba9 Support for strict parsing of XML documents
Currently this only disabled the automatic insertion of closing tags, in
the future this may also disable other features if deemed worth the
effort.

Fixes #107
2015-06-15 23:53:11 +02:00
Yorick Peterse 4031c4f843 Nuked Oga::XML::Lexer#html
This method was rather pointless since there's already a "html?" method.
2015-06-15 23:45:20 +02:00
Yorick Peterse 020cd34083 Mark the XML lexer class as private
I was supposed to mark this as private for 1.0 but completely forgot.
2015-06-15 23:18:11 +02:00
Yorick Peterse af7f2674af Decoding of entities with numbers
This ensures that entities such as "&frac12;" are decoded properly.
Previously this would be ignored as the regular expression used for this
only matched [a-zA-Z].

This was adapted from PR #111.
2015-06-07 17:42:24 +02:00
Yorick Peterse 5951a6f187 Release 1.0.2 2015-06-03 06:53:31 +02:00
Yorick Peterse 4bfeea2590 Use require vs require_relative
See ruby-ll commit b27fe7cc109a39184ac984405a1e452868f3fac9 for a more
in-depth explanation of this.
2015-06-03 06:42:30 +02:00
Yorick Peterse d0d597e2d9 Allow script/template in various table elements
Fixes #105
2015-05-23 10:46:49 +02:00
Yorick Peterse 5182d0c488 Correct closing of unclosed, nested HTML elements
Previous HTML such as this would be lexed incorrectly:

    <div>
        <ul>
            <li>foo
        </ul>
        inside div
    </div>
    outside div

The lexer would see this as the following instead:

    <div>
        <ul>
            <li>foo</li>
            inside div
        </ul>
    outside div
    </div>

This commit exposes the name of the closing tag to
XML::Lexer#on_element_end (omitted for self closing tags). This can be
used to automatically close nested tags that were left open, ensuring
the above HTML is lexer correctly.

The new setup ignores namespace prefixes as these are not used in HTML,
XML in turn won't even run the code to begin with since it doesn't allow
one to leave out closing tags.
2015-05-23 09:59:50 +02:00
Yorick Peterse 3c6263d8de Updated list of elements that close <p> tags 2015-05-21 21:11:41 +02:00
Yorick Peterse 04948eb211 Release 1.0.1 2015-05-21 11:42:06 +02:00
Yorick Peterse 2766d5f27f Pack HTML entities using "U*"
See https://github.com/YorickPeterse/oga/issues/90#issuecomment-89859273
for more details, apparently I didn't fix this before.
2015-05-21 11:26:01 +02:00
Yorick Peterse c97c1b6899 Do not encode single/double quotes as entities
By encoding single/double quotes we can potentially break input, so lets
stop doing this. This now ensures that this:

    <foo>a"b</foo>

Is actually serialized back into the exact same instead of being
serialized into:

    <foo>a&quot;b</foo>
2015-05-21 11:23:44 +02:00
Yorick Peterse cab47f155c Release 1.0.0 2015-05-20 22:44:59 +02:00
Yorick Peterse 688a1fff0e Use blacklists/whitelists for HTML closing rules
This allows for more fine grained control over when to close certain
elements. For example, an unclosed <tr> element should be closed first
when bumping into any element other than <td> or <th>. Using the old
NodeNameSet this would mean having to list every possible HTML element
out there. Using this new setup one can just create a whitelist of the
<td> and <th> elements.
2015-05-18 00:32:29 +02:00
Yorick Peterse 132d112f5f Removed NodeNameSet class 2015-05-17 21:59:43 +02:00
Yorick Peterse cec8798694 Mark HTML_VOID_ELEMENTS as private 2015-05-17 21:59:14 +02:00
Yorick Peterse bcc101b819 Use Whitelist for HTML_VOID_ELEMENTS 2015-05-17 21:59:00 +02:00
Yorick Peterse 596a9b18d6 Use Whitelist for LITERAL_HTML_ELEMENTS 2015-05-17 21:56:38 +02:00
Yorick Peterse ca16a2976e Added Blacklist/Whitelist classes
These will be used in favour of the NodeNameSet class.
2015-05-17 21:55:06 +02:00
Yorick Peterse 1c095ddaff Added more HTML closing rules for colgroup/caption 2015-05-12 23:14:48 +02:00
Yorick Peterse 1e0b7feb02 Recursively closing of parent HTML elements
When closing certain HTML elements the lexer should also close whatever
parent elements remain. For example, consider the following HTML:

    <table>
        <thead>
            <tr>
                <th>Foo
                <th>Bar
        <tbody>
            ...
        </tbody>
    </table>

Here the "<tbody>" element shouldn't only close the "<th>Bar" element
but also the parent "<tr>" and "<thead>" elements. This ensures we'd end
up with the following HTML:

    <table>
        <thead>
            <tr>
                <th>Foo</th>
                <th>Bar</th>
            </tr>
        </thead>
        <tbody>
            ...
        </tbody>
    </table>

Instead of garbage along the lines of this:

    <table>
        <thead>
            <tr>
                <th>Foo</th>
                <th>Bar</th>
        <tbody>
            ...
        </tbody>
    </table></tr></thead>

Fixes #99 (hopefully for good this time)
2015-05-12 00:35:00 +02:00
Yorick Peterse df96b3d3bb Define the public API using YARD/semver 2015-05-11 21:34:34 +02:00
Yorick Peterse ecdeeacd76 Use Symbol#equal? instead of Symbol#==
At least on JRuby and Rubinius this can be quite a bit faster. On MRI
the difference is not really significant.
2015-05-07 23:03:03 +02:00
Yorick Peterse 5c7c4a6110 Don't use a splat with AST::Node#to_a
By using AST::Node#children directly with a splat we save ourselves an
extra method call. This in turn speeds up both the
xpath/evaluator/big_xml_average_bench.rb and
xpath/evaluator/node_matches_bench.rb benchmarks a little bit.
2015-05-07 01:23:27 +02:00
Yorick Peterse 0298e7068c Don't use Namespace#to_s when matching namespaces
This is a waste of time as it allocates a new String on every call.
2015-05-07 01:04:03 +02:00
Yorick Peterse b9145d83f8 Less html? calls in Element#available_namespaces
Previously it would always call the "html?" method, even if the
available namespaces were already set.
2015-05-07 01:02:59 +02:00
Yorick Peterse b5e63dc50e Improved perf of XPath::Evaluator#node_matches?
Using the benchmark xpath/evaluator/node_matches_bench.rb the results
prior to this commit were as following for 3 cases:

    name only:          737633 i/s
    namespace wildcard: 612196 i/s
    name wildcard:      516030 i/s

With this commit said numbers have changed to the following:

    name only:          746086  i/s
    namespace wildcard: 1097168 i/s
    name wildcard:      1151255 i/s

This results in the following increase of performance for each case:

    name only:          1,011x (insignificant)
    namespace wildcard: 1,79x
    name wildcard:      2,23x

In the benchmark xpath/evaluator/big_xml_average_bench.rb the difference
isn't really noticable as said benchmark only queries elements by names,
of which the performance hasn't really improved.
2015-05-07 00:05:25 +02:00
Yorick Peterse 69180ff686 Extra closing rules for caption/colgroup/head/body
Fixes #99
2015-05-03 01:09:07 +02:00
Yorick Peterse 4b1c296936 Automatically closing of certain HTML tags
This ensures that HTML such as this:

    <li>foo
    <li>bar

is parsed as this:

    <li>foo</li>
    <li>bar</li>

and not as this:

    <li>
        foo
        <li>bar</li>
    </li>

Fixes #97
2015-04-27 18:43:26 +02:00
Yorick Peterse 4b21a2fadc Added NodeNameSet class
This class can be used to more easily create a Set containing both
lowercase and uppercase element names.
2015-04-22 00:54:29 +02:00
Yorick Peterse 8135074a62 Merged on_element_start with on_element_name
This makes it easier to automatically insert preceding tokens when
starting a new element as we now have access to the name. Previously
on_element_start would be invoked first which doesn't receive an
argument.
2015-04-21 23:38:06 +02:00
Yorick Peterse 853d804f34 Decoding of zero padded XML entities
This would previously fail due to the lack of an explicit base to use
for Integer().
2015-04-20 00:13:15 +02:00
Yorick Peterse 13e2c3d82f Better handling of incorrect XML/HTML tags
The XML/HTML lexer is now capable of processing most invalid XML/HTML
(that I can think of at least). This is achieved by inserting missing
closing tags (where needed) and/or ignoring excessive closing tags. For
example, HTML such as this:

    <a></a></p>

Results in the following tokens:

    [:T_ELEM_START, nil, 1]
    [:T_ELEM_NAME, 'a', 1]
    [:T_ELEM_CLOSE, nil, 1]

In turn this HTML:

    <a>

Results in these tokens:

    [:T_ELEM_START, nil, 1]
    [:T_ELEM_NAME, 'a', 1]
    [:T_ELEM_CLOSE, nil, 1]

Fixes #84
2015-04-19 23:19:02 +02:00
Yorick Peterse 84e1bfc955 Release 0.3.4 2015-04-19 22:19:02 +02:00
Yorick Peterse da62fcd75d Decode XML/HTML entities in the SAX parser
This was broken when decoding was moved out of the Lexer class into
XML::Text and XML::Attribute.

Fixes #92
2015-04-18 22:03:44 +02:00
Yorick Peterse 611beb78c7 Release 0.3.3 2015-04-18 20:49:40 +02:00
Yorick Peterse 73fbbfbdbd Use separate Ragel machines for script/style tags
Previously a single Ragel machine was used for processing HTML
script and style tags. This had the unfortunate side-effect that the
following was not parsed correctly (while being valid HTML):

    <script>
    var foo = "</style>";
    </script>

The same applied to style tags:

    <style>
    /* </script> */
    </style>

By using separate machines we can work around the above issue. The
downside is that this can produce multiple T_TEXT nodes, which have to
be stitched back together in the parser.
2015-04-16 01:45:39 +02:00
Yorick Peterse a08829add5 Release 0.3.2 2015-04-15 11:38:09 +02:00
Yorick Peterse 23a441933a Use recursion for parsing string bodies
This is a little bit faster than using the * operator combined with
Array#inject.
2015-04-15 00:49:40 +02:00
Yorick Peterse b2ea20ba61 Lex processing instructions in chunks
Similar to comments (ea8b4aa92f) and CDATA
tags (8acc7fc743) processing instructions
are now lexed in separate chunks _with_ proper support for streaming
input.

Related issue: #93
2015-04-15 00:11:57 +02:00
Yorick Peterse ea8b4aa92f Lex comments in chunks
Similar to this being added for CDATA tags in
8acc7fc743 comments are now also lexed in
chunks.

Related issue: #93
2015-04-14 23:11:22 +02:00
Yorick Peterse 8acc7fc743 Lex CDATA tags in chunks
Instead of using a single token (T_CDATA) for a CDATA tag the lexer now
uses 3 tokens:

1. T_CDATA_START
2. T_CDATA_BODY
3. T_CDATA_END

The T_CDATA_BODY token can occur multiple times and is turned into a
single value in the XML parser. This is similar to the way strings are
lexed.

By changing the way CDATA tags are lexed Oga can now lex CDATA tags
containing newlines when using an IO as input. For example, this would
previously fail:

    Oga.parse_xml(StringIO.new("<![CDATA[\nfoo]]>"))

Because IO input reads input per line the input for the lexer would be
as following:

    "<![CDATA[\n"
    "foo]]>"

Related issues: #93
2015-04-14 22:45:55 +02:00
Yorick Peterse 739e3b474c Optimize Traversal#each_node allocations
This removes the need for allocating an Array for every set of child
nodes.
2015-04-12 22:45:42 +02:00
Yorick Peterse b42f9aaf32 Cache output of Element#available_namespaces
This cache is flushed whenever Element#register_namespace is called.
When this cache is flushed it's also recursively flushed for all child
elements. This makes calls to Element#register_namespace a bit more
expensive but in turn calls to Element#available_namespaces will be a
lot faster.
2015-04-12 20:22:33 +02:00
Yorick Peterse fa838154fc Flush Element#namespace cache
When setting a new namespace name using Element#namespace_name= the
cache used by Element#namespace is flushed properly.
2015-04-11 19:20:50 +02:00
Yorick Peterse b0359b37e5 Cache Node#html? and Node#root_node
The results of these methods is now cached until a Node is moved into
another NodeSet. This reduces the time spent in the
xpath/evaluator/big_xml_average_bench.rb benchmark from roughly 10
seconds to roughly 5 seconds per iteration.
2015-04-11 19:12:26 +02:00
Yorick Peterse 421e6e910b Release 0.3.1 2015-04-08 14:58:53 +02:00
Yorick Peterse 4bdc8a3fdc Don't convert entities in script/style elements
In HTML the text of a script/style tag should be left untouched, no
entities must be converted. Doing so would break Javascript such as the
following:

    foo&&bar;

Such code is often the result of minifiers doing their dirty business.
2015-04-08 14:32:09 +02:00
Yorick Peterse 6a1010c287 Fixed decoding entities in attribute values
This was broken by introducing the process of lazy decoding of XML/HTML
entities. The new setup works similar to how XML::Text#text decodes any
entities that may be present.

Fixes #91
2015-04-07 21:18:22 +02:00
Yorick Peterse ef7f50137a Added Oga::EntityDecoder
This module removes some of the code duplication needed to determine
what entity decoder to use.
2015-04-07 21:18:15 +02:00
Yorick Peterse 3f6aa04e91 Release 0.3.0 2015-04-03 21:11:15 +02:00
Yorick Peterse 3176459307 Ignore declared namespaces in HTML documents
The HTML spec states that any declared namespaces, including the default
namespace are to be ignored.

This fixes #85
2015-03-26 22:38:39 +01:00
Yorick Peterse 5adeae18d0 XPath queries match nodes in the default namespace
When querying an XML document that explicitly defines the default XML
namespace the XPath evaluator now correctly matches all nodes within
that namespace if no namespace prefix is given in the query. Previously
this would always return an empty set.
2015-03-26 01:13:55 +01:00
Yorick Peterse f175414917 Added XML::Element#default_namespace? 2015-03-26 01:10:20 +01:00
Yorick Peterse b6fcd326ef Added XML::Node#html? and XML::Node#xml?
The former has been moved over from XML::Text, the latter just inverts
html?.
2015-03-26 01:02:32 +01:00
Yorick Peterse 4ad502958d Added XML::Attribute#==
Overwriting this method makes it easier to check if a given namespace
equals the default XML (and soon HTML) namespace.
2015-03-26 00:53:16 +01:00
Yorick Peterse f2d69af33b Distinguish default attribute/element namespaces
The previous commit messed this up because I wasn't fully awake.
2015-03-26 00:43:50 +01:00
Yorick Peterse 68ada997a8 Moved default namespace into Oga::XML
The default namespace is now located at Oga::XML::DEFAULT_NAMESPACE
instead of Oga::XML::Attribute::DEFAULT_NAMESPACE.
2015-03-26 00:35:28 +01:00
Yorick Peterse 3cdcdf6daa Corrected YARD formatting 2015-03-23 00:31:56 +01:00
Yorick Peterse 66fa9f62ef Added LRU#maximum=/maximum
This allows one to change the maximum amount of keys stored in the
XPath/CSS caches, for example:

    Oga::XPath::Parser::CACHE.maximum = 2056
2015-03-23 00:26:48 +01:00
Yorick Peterse 12aa21fb50 Use parse_with_cache when querying xpath/css 2015-03-23 00:23:46 +01:00
Yorick Peterse 2c4e490614 Added CSS/XPath Parser.parse_with_cache
This method parses and caches ASTs using Oga::LRU. Currently the default
of 1024 keys is used.

See #71 for more information.
2015-03-23 00:22:59 +01:00
Yorick Peterse 67d7d9af88 Added thread-safe LRU class
This class will be used for storing parser XPath/CSS ASTs.

See #71 for more information.
2015-03-23 00:21:52 +01:00
Yorick Peterse 31e93e54f9 Removed Mutex usage from XML::Text
Instead of trying to make this class thread-safe I'm going with the
option of simply declaring it unsafe to mutate instances of XML::Text
while reading it in parallel. This removes the need for Mutex
allocations and keeps the code simple.

Fixes #82
2015-03-21 01:27:00 +01:00
Yorick Peterse c647f064b5 Remove remaining Racc parsing bits 2015-03-21 01:23:00 +01:00
Yorick Peterse ed14981044 Ported the CSS parser to ruby-ll 2015-03-21 01:23:00 +01:00
Yorick Peterse 2714dbe419 Use the ? operator in the XPath parser 2015-03-21 01:23:00 +01:00
Yorick Peterse 3b74a55d73 Use the ? operator in the XML parser 2015-03-21 01:23:00 +01:00
Yorick Peterse 2bbb7d2b10 Use new operators in the XML parser
This allows the removal of quite a bit of recursion based code.
2015-03-21 01:23:00 +01:00
Yorick Peterse 02da47c1f0 Replaced some XPath parser recursion with * 2015-03-21 01:23:00 +01:00
Yorick Peterse 3b06780802 Removed Racc based XPath parser 2015-03-21 01:23:00 +01:00
Yorick Peterse 588c225c53 Proper XPath operator parsing precedence 2015-03-21 01:23:00 +01:00
Yorick Peterse 0fa9d4df88 Ported remaining XPath parsing bits to ruby-ll.
Currently all operators are left-associative with no particular precedence. This
causes a few specs to fail for now. Outside of that the new parser should be
able to parse the same input as the Racc based parser.
2015-03-21 01:22:59 +01:00
Yorick Peterse 4ebfc849a4 Start porting the XPath parser to ruby-ll.
There are still a few bits left to do such as supporting parenthesis and
assigning the correct precedence to the others.
2015-03-21 01:22:59 +01:00
Yorick Peterse cbdaeb21f4 Unwrap a few lines in the XML parser. 2015-03-21 01:22:59 +01:00
Yorick Peterse cfc6749556 Use splat instead of Array#unshift for attributes. 2015-03-21 01:22:59 +01:00
Yorick Peterse d210c9fb57 Compacted a few XML parser rules. 2015-03-21 01:22:59 +01:00
Yorick Peterse a5cd75cb7e Removed useless string allocs from the XML parser. 2015-03-21 01:22:59 +01:00
Yorick Peterse fdcd712ffe Don't use Array#uniq in NodeSet#initialize.
Removing this makes the process of parsing larger XML documents a bit faster.
The downside is that NodeSet#initialize will no longer filter out duplicate
nodes, though this is not something Oga itself relies upon.

Methods such as NodeSet#push still do ignore elements already present.
2015-03-21 01:22:59 +01:00
Yorick Peterse c36b35ac0f Skip ownership iteration when there's no owner.
There's no point in iterating over all the nodes and assigning ownership if
there's no owner to begin with.
2015-03-21 01:22:59 +01:00
Yorick Peterse 006ef4d51a Port over most of the old XML error handling.
Some messages are a bit different due to ruby-ll's error handling, other than
that it's largely the same stuff as before.
2015-03-21 01:22:59 +01:00
Yorick Peterse 1a326fc516 Remove Racc based XML parser. 2015-03-21 01:22:59 +01:00
Yorick Peterse d8b9725b82 Fixed SAX parsing of XML attributes.
This was utterly broken, mainly due to me overlooking it. There are now 2 new
callbacks to handle this properly:

* on_attribute: to handle a single attribute/value pair
* on_attributes: to handle a collection of attributes (as returned by
  on_attribute)

By default on_attribut returns a Hash, on_attributes in turn merges all
attribute hashes into a single one. This ensures that on_element _actually_
receives the attributes as a Hash, instead of an Array with random
nil/XML::Attribute values.
2015-03-21 01:22:59 +01:00
Yorick Peterse dd626c10d3 Use Array#unshift in the LL XML grammar.
Using Array#+ for large sets (e.g. in the benchmarks) is _really_ slow.
Interesting enough Array#unshift uses as much memory as the Racc parser and is
about as fast, even though it has to move memory around.
2015-03-21 01:22:59 +01:00
Yorick Peterse f94407ee9d Parser callback for XML attributes. 2015-03-21 01:22:59 +01:00
Yorick Peterse a023b35e78 Fixed the pull parser for the XML LL parser. 2015-03-21 01:22:59 +01:00
Yorick Peterse 5eed0d31d6 Ported over most of the XML parser to ruby-ll.
This is still missing the error handling previously present.
2015-03-21 01:22:59 +01:00
Yorick Peterse 15a3ab9ba5 ruby-ll: full support for parsing doctypes. 2015-03-21 01:22:59 +01:00
Yorick Peterse 71aefb53cc Started porting the XML parser to ruby-ll
This is far from done.
2015-03-21 01:22:59 +01:00
Yorick Peterse 2ec91f130f Lazy decoding of XML/HTML entities.
Instead of decoding entities in the lexer we'll do this whenever XML::Text#text
is called. This removes the overhead from the parsing phase and ensures the
process is only triggered when actually needed. Note that calling #to_xml and/or
the #inspect methods on a Text (or parent) instance will also trigger the entity
conversion process.

The new entity decoding API supports both regular entities (e.g. &amp;) as well
as codepoint based entities (both regular and hexadecimal codepoints).

To allow safe read-only access to Text instances from multiple threads a mutex
is used. This mutex ensures that only 1 thread can trigger the conversion
process.

Fixes #68
2015-03-05 23:00:43 +01:00
Yorick Peterse 3e05593536 Release 0.2.3 2015-03-04 11:56:23 +01:00
Yorick Peterse 78e40b55c0 Handle parsing of HTML <style> tags.
This basically re-applies the technique used for HTML <script> tags. With this
extra addition I decided to rename/normalize a few things so it's easier to add
any extra tags in the future. One downside of this setup is that the following
will not be parsed by Oga:

    <style>
        </script>
    </style>

The same applies to script tags containing a literal </style> tag. Since this
particular case is rather unlikely to occur I'm OK with not supporting it as it
_does_ simplify the lexer quite a bit.

Fixes #80
2015-03-03 16:28:05 +01:00
Yorick Peterse 73534375d5 Release 0.2.2 2015-03-03 13:36:32 +01:00
Yorick Peterse 142b467277 Set parent of nodes set using Element#inner_text=
This ensures that any text nodes created using Element#inner_text= have their
parent node set correctly.
2015-03-03 13:13:05 +01:00
Yorick Peterse 503efc32cd Release 0.2.1 2015-03-02 22:12:49 +01:00
Yorick Peterse 874d7124af Don't convert <script> text to XML entities.
Fixes #79.
2015-03-02 17:32:19 +01:00
Yorick Peterse 9a586363e9 Added XML::Document#html? 2015-03-02 16:39:40 +01:00
Yorick Peterse ba2177e2cf Lex contents of <script> tags as plain text.
When lexing input in HTML mode the lexer has to treat _all_ content of a
<script> tag as plain text. This ensures that the lexer can process input such
as "x <y" and "// <foo>" correctly.

Fixes #70.
2015-03-02 16:22:09 +01:00
Yorick Peterse e138aa15ac Removed stray comment in the XPath parser. 2014-12-28 23:55:33 +01:00
Yorick Peterse 746c8052dd Remove all nodes when calling Element#inner_text=
This fixes #64.
2014-12-14 23:32:43 +01:00
Dmitry Krasnoukhov 26baf89440 Add missing entities to the decode/encode lists 2014-11-21 01:53:11 +02:00
Yorick Peterse cbb2815146 Support for inline doctype rules plus newlines.
This adds support for lexing/parsing XML documents that use an IO as input _and_
contain doctype rules with newlines in them.

This fixes #63.
2014-11-18 20:02:55 +01:00
Yorick Peterse 922cee913d Release 0.2.0 2014-11-17 23:26:19 +01:00
Yorick Peterse ad4f650c5d Fixed XML entity encoding/decoding ordering.
Thanks to @krasnoukhov for providing the initial patch, which this commit is
largely based on.

This fixes #49.
2014-11-17 22:39:43 +01:00
Yorick Peterse cd86d5d294 Allow removal of element attributes. 2014-11-17 09:00:40 +01:00
Yorick Peterse 804646cc5e Don't modify raw namespaces.
When calling Element#available_namespaces the list of namespaces returned by
Element#namespaces must not be modified.
2014-11-17 00:01:16 +01:00
Yorick Peterse 6753d6a26d Slightly better docs for the XPath/CSS parsers. 2014-11-16 23:40:19 +01:00
Yorick Peterse 57adabc068 Ensure SAX after_element receives meaningful args
This changes the behaviour of after_element when parsing documents using the SAX
parsing API. Previously it would always receive a nil argument, which is kinda
pointless. This commit changes that by making sure it receives a namespace name
(if any) and the element name.

This fixes #54.
2014-11-16 23:32:32 +01:00
Yorick Peterse 23b408fe4f Cleaned up CSS parser code for counting siblings. 2014-11-15 18:31:08 +01:00
Yorick Peterse b464815577 Fixed AST generation for nth-(first|last)-of-type. 2014-11-15 18:27:15 +01:00
Yorick Peterse 9eead81a7c Fixed AST for :only-of-type 2014-11-15 18:08:26 +01:00
Yorick Peterse 1c301d40e2 Properly fixed AST for first-of-type/last-of-type
This requires keeping track of the current element being processed. This in turn
allows the usage of count() + preceding-sibling/following-sibling.
2014-11-15 17:58:56 +01:00
Yorick Peterse f1d574f342 Evaluate XPath predicates for every context node.
Instead of evaluating a predicate once for all context nodes, they should
instead be evaluated separately per context node.
2014-11-15 00:31:44 +01:00
Yorick Peterse 6daa3e7a00 Reverted AST changes for first-of-type
Functions can't be used in combination with axes, so I'll just need to fix the
position() function to work properly.
2014-11-14 23:51:46 +01:00
Yorick Peterse 2d6a2be2e8 Revert "Fixed XPath AST for :last-of-type"
Axes can't be used in combination with functions.

This reverts commit b0b572a584.
2014-11-14 23:49:49 +01:00
Yorick Peterse b0b572a584 Fixed XPath AST for :last-of-type
This should count following nodes, not merely the position.
2014-11-14 23:27:15 +01:00
Yorick Peterse 0128dc50ae Fixed CSS evaluation of :first-of-type
The old XPath "position() = 1" would work in Nokogiri due to the way they
retrieve descendants. In Oga however this would simply always return the first
node.

To fix this Oga now counts the amount of preceding siblings that match the same
full name.
2014-11-14 01:25:03 +01:00
Yorick Peterse e3a26c5d15 Allow querying of nodes using CSS. 2014-11-14 01:05:29 +01:00
Yorick Peterse 01b88d8c68 Use correct root for preceding/following(-sibling)
This ensures these axes work correctly when scoped to a node instead of a
document.
2014-11-13 01:11:29 +01:00
Yorick Peterse b0f6409d1e Wrap predicate nodes around others in CSS ASTs. 2014-11-13 00:47:26 +01:00
Yorick Peterse 817a5e075b Wrap predicate AST nodes _around_ other nodes.
This means that "foo[1]" uses this AST:

    (predicate (test nil "foo") (int 1))

Instead of this AST:

    (test nil "foo" (int 1))

This makes it easier for the XPath evaluator to process predicates correctly.
2014-11-12 22:59:38 +01:00
Yorick Peterse 857ac517d5 Added XML::NodeSet#==
This method can be used to compare two NodeSet instances. By using
XML::NodeSet#equal_nodes?() the need for exposing the "nodes" instance variable
is also removed.
2014-11-09 18:33:16 +01:00
Yorick Peterse a586a512b8 Removed support for CSS such as "|X"
This expression could be used to get all elements that _don't_ have any
namespace. The problem is that this can't be expressed as just a node test,
instead the resulting XPath would have to look something like the following:

    X[local-name() = name()]

However, since the XPath predicates are already created for pseudo classes and
such, also injecting the above into it would be a real big pain. As such I've
decided not to support it.
2014-11-05 00:45:40 +01:00
Yorick Peterse 1c20ef52ae XPath idents can't start with a star.
That is, names such as "*foo" are not valid. This was most likely a typo in the
first place.
2014-11-05 00:38:17 +01:00
Yorick Peterse 61801fe562 Allow CSS identifiers to start with an underscore. 2014-11-05 00:37:40 +01:00
Yorick Peterse f1316c50fb Allow XPath idents to start with an underscore.
This is valid in XML so XPath should support it as well.
2014-11-05 00:35:47 +01:00
Yorick Peterse eb3a3e7630 Use the descendant axis for CSS selectors.
Instead of using "descendant-or-self" Oga will use "descendant". This ensures
that expressions such as "foo *" don't return a set also including the "foo"
element.

Nokogiri solves this problem in a somewhat different way by using //foo//* for
the CSS expression "foo *". While this works in Nokogiri the expression
"descendant-or-self::node()" is slow as a snail in Oga (due to its nature of
retrieving _all_ nodes first). By using "descendant" we can work around this
problem.
2014-11-05 00:22:09 +01:00
Yorick Peterse 602f2fe8bb Apply node() type tests to the document too.
When running XPath queries such as "self::node()" the result should be a set
containing the document itself. This in turn fixes expressions such as
descendant-or-self::node()/a.
2014-11-04 23:41:45 +01:00
Yorick Peterse de04c61df9 Parsing support for the :empty pseudo. 2014-11-04 00:00:23 +01:00
Yorick Peterse 1b870406de Parsing support for the :only-of-type pseudo. 2014-11-03 23:47:25 +01:00
Yorick Peterse 2cf058457d Parsing support for the :only-child pseudo. 2014-11-03 23:27:56 +01:00
Yorick Peterse 759bb71d17 Parsing support for the :last-of-type pseudo. 2014-11-03 23:01:44 +01:00
Yorick Peterse 10632eafd4 Parsing support for the :first-of-type pseudo. 2014-11-03 23:00:15 +01:00
Yorick Peterse 7a606a11d3 Parsing support for the :last-child pseudo class. 2014-11-03 22:58:08 +01:00
Yorick Peterse b4fec3cc8c Parsing support for the :first-child pseudo class. 2014-11-03 22:45:00 +01:00
Yorick Peterse 020d979fba Parsing support for :nth-last-of-type(). 2014-11-03 22:15:21 +01:00
Yorick Peterse bc8be9f725 Fixed various incorrect YARD tags. 2014-11-02 21:23:29 +01:00
Yorick Peterse 2e1320c2dc Explicit return in step_modulo_value. 2014-11-02 19:32:02 +01:00
Yorick Peterse ab8b451dc3 Parsing support for :nth-of-type() 2014-11-02 19:29:39 +01:00
Yorick Peterse 0faceffacb Parsing support for nth-child(n+X) 2014-11-02 19:29:09 +01:00
Yorick Peterse 8d8d74ec41 Drop nth-child support of all negative sequences
This removes parsing support for selectors such as :nth-child(-n-6). According
to the CSS spec this isn't valid anyway (confirmed by testing it in Chromium).
As a result there's no point in supporting it in any way.
2014-11-02 19:19:05 +01:00
Yorick Peterse b31288b7d2 Use correct modulo for nth-child and negatives. 2014-11-02 18:53:23 +01:00
Yorick Peterse 9cce93fc4a Parsing support for :nth-last-child. 2014-11-01 20:58:28 +01:00
Yorick Peterse 03f897c2b7 Support for all possible nth-child arguments.
That is, as far as I can tell based on Nokogiri's behaviour (which Oga now
matches).
2014-10-30 23:03:46 +01:00
Yorick Peterse 0e6aefb727 Fixed parsing of nth-child(n) and nth-child(-n) 2014-10-30 00:24:31 +01:00
Yorick Peterse 87f6b9c723 Basic support for :nth-child()
This already includes support for formulas such as 2n, odd, even and 2n+1.
Negative formulas, just "n" and others are not yet supported.
2014-10-28 00:21:11 +01:00
Yorick Peterse 46646e2ace Support for custom grouping of XPath expressions.
This allows the use of expressions such as "(A or B) and C".

This fixes #59.
2014-10-26 22:38:05 +01:00
Yorick Peterse 39c0f7147c Support for parsing the :root pseudo class. 2014-10-26 22:20:23 +01:00
Yorick Peterse 32764c9a14 Proper parsing support for all CSS operators. 2014-10-26 12:45:43 +01:00
Yorick Peterse 24ae791f00 Better support for lexing multi-line strings.
When lexing multi-line strings everything used to work fine as long as the input
were to be read as a whole. However, when using an IO instance all hell would
break loose. Due to the lexer reading IO instances on a per line basis,
sometimes Ragel would end up setting "ts" to NULL. For example, the following
input would break the lexer:

    <foo class="\nbar" />

Due to the input being read per line, the following data would be sent to the
lexer:

    <foo class="\n
    bar" />

This would result in different (or NULL) pointers being used for building a
string, in turn resulting in memory allocation errors.

To work around this the string lexing setup has been broken into separate
machines for single and double quoted strings. The tokens used have also been
changed so that instead of just "T_STRING" there are now the following tokens:

* T_STRING_SQUOTE
* T_STRING_DQUOTE
* T_STRING_BODY

A string can have multiple T_STRING_BODY tokens (= multi-line strings, only the
case for IO inputs). These strings are stitched back together by the parser.

This fixes #58.
2014-10-26 11:39:56 +01:00
Yorick Peterse b304b8b077 Fixed descendant-or-self with a predicate.
Processing of this axis along with a predicate wouldn't quite work out. Even if
the predicate returned false the node would still be matched (which should not
be the case).
2014-10-23 01:12:10 +02:00
Yorick Peterse 7ee7f25239 Support for parsing CSS axes. 2014-10-23 00:42:45 +02:00
Yorick Peterse 9955f61bcb Renamed CSS axis tokens.
These have been renamed as following:

T_CHILD => T_GREATER
T_FOLLOWING => T_TILDE
T_FOLLOWING_DIRECT => T_PLUS
2014-10-21 23:25:11 +02:00
Yorick Peterse 851e7d6d0b First pass of rewriting the CSS parser.
The new parser uses way less confusing rule names, is a bit more strict and in
general much less of a pain to deal with.
2014-10-21 23:19:31 +02:00
Yorick Peterse e3de65a258 Lex whitespace preceding CSS axes separately.
Previously input such as "x > y" would result in the following token sequences:

    T_IDENT, T_CHILD, T_IDENT

This commit changes this to the following:

    T_IDENT, T_SPACE, T_CHILD, T_IDENT

This allows the parser to use T_SPACE as a terminal token, this in turn prevents
around 16 shift/reduce conflicts from arising.

This does mean that input such as " > y" or " x > y" is now invalid. This
however can be solved by simply _not_ adding leading/trailing whitespace to CSS
queries.
2014-10-21 23:18:46 +02:00