Commit Graph

675 Commits

Author SHA1 Message Date
Yorick Peterse 13e2c3d82f Better handling of incorrect XML/HTML tags
The XML/HTML lexer is now capable of processing most invalid XML/HTML
(that I can think of at least). This is achieved by inserting missing
closing tags (where needed) and/or ignoring excessive closing tags. For
example, HTML such as this:

    <a></a></p>

Results in the following tokens:

    [:T_ELEM_START, nil, 1]
    [:T_ELEM_NAME, 'a', 1]
    [:T_ELEM_CLOSE, nil, 1]

In turn this HTML:

    <a>

Results in these tokens:

    [:T_ELEM_START, nil, 1]
    [:T_ELEM_NAME, 'a', 1]
    [:T_ELEM_CLOSE, nil, 1]

Fixes #84
2015-04-19 23:19:02 +02:00
Yorick Peterse 84e1bfc955 Release 0.3.4 2015-04-19 22:19:02 +02:00
Yorick Peterse da62fcd75d Decode XML/HTML entities in the SAX parser
This was broken when decoding was moved out of the Lexer class into
XML::Text and XML::Attribute.

Fixes #92
2015-04-18 22:03:44 +02:00
Yorick Peterse 611beb78c7 Release 0.3.3 2015-04-18 20:49:40 +02:00
Yorick Peterse 73fbbfbdbd Use separate Ragel machines for script/style tags
Previously a single Ragel machine was used for processing HTML
script and style tags. This had the unfortunate side-effect that the
following was not parsed correctly (while being valid HTML):

    <script>
    var foo = "</style>";
    </script>

The same applied to style tags:

    <style>
    /* </script> */
    </style>

By using separate machines we can work around the above issue. The
downside is that this can produce multiple T_TEXT nodes, which have to
be stitched back together in the parser.
2015-04-16 01:45:39 +02:00
Yorick Peterse a08829add5 Release 0.3.2 2015-04-15 11:38:09 +02:00
Yorick Peterse 23a441933a Use recursion for parsing string bodies
This is a little bit faster than using the * operator combined with
Array#inject.
2015-04-15 00:49:40 +02:00
Yorick Peterse b2ea20ba61 Lex processing instructions in chunks
Similar to comments (ea8b4aa92f) and CDATA
tags (8acc7fc743) processing instructions
are now lexed in separate chunks _with_ proper support for streaming
input.

Related issue: #93
2015-04-15 00:11:57 +02:00
Yorick Peterse ea8b4aa92f Lex comments in chunks
Similar to this being added for CDATA tags in
8acc7fc743 comments are now also lexed in
chunks.

Related issue: #93
2015-04-14 23:11:22 +02:00
Yorick Peterse 8acc7fc743 Lex CDATA tags in chunks
Instead of using a single token (T_CDATA) for a CDATA tag the lexer now
uses 3 tokens:

1. T_CDATA_START
2. T_CDATA_BODY
3. T_CDATA_END

The T_CDATA_BODY token can occur multiple times and is turned into a
single value in the XML parser. This is similar to the way strings are
lexed.

By changing the way CDATA tags are lexed Oga can now lex CDATA tags
containing newlines when using an IO as input. For example, this would
previously fail:

    Oga.parse_xml(StringIO.new("<![CDATA[\nfoo]]>"))

Because IO input reads input per line the input for the lexer would be
as following:

    "<![CDATA[\n"
    "foo]]>"

Related issues: #93
2015-04-14 22:45:55 +02:00
Yorick Peterse 739e3b474c Optimize Traversal#each_node allocations
This removes the need for allocating an Array for every set of child
nodes.
2015-04-12 22:45:42 +02:00
Yorick Peterse b42f9aaf32 Cache output of Element#available_namespaces
This cache is flushed whenever Element#register_namespace is called.
When this cache is flushed it's also recursively flushed for all child
elements. This makes calls to Element#register_namespace a bit more
expensive but in turn calls to Element#available_namespaces will be a
lot faster.
2015-04-12 20:22:33 +02:00
Yorick Peterse fa838154fc Flush Element#namespace cache
When setting a new namespace name using Element#namespace_name= the
cache used by Element#namespace is flushed properly.
2015-04-11 19:20:50 +02:00
Yorick Peterse b0359b37e5 Cache Node#html? and Node#root_node
The results of these methods is now cached until a Node is moved into
another NodeSet. This reduces the time spent in the
xpath/evaluator/big_xml_average_bench.rb benchmark from roughly 10
seconds to roughly 5 seconds per iteration.
2015-04-11 19:12:26 +02:00
Yorick Peterse 421e6e910b Release 0.3.1 2015-04-08 14:58:53 +02:00
Yorick Peterse 4bdc8a3fdc Don't convert entities in script/style elements
In HTML the text of a script/style tag should be left untouched, no
entities must be converted. Doing so would break Javascript such as the
following:

    foo&&bar;

Such code is often the result of minifiers doing their dirty business.
2015-04-08 14:32:09 +02:00
Yorick Peterse 6a1010c287 Fixed decoding entities in attribute values
This was broken by introducing the process of lazy decoding of XML/HTML
entities. The new setup works similar to how XML::Text#text decodes any
entities that may be present.

Fixes #91
2015-04-07 21:18:22 +02:00
Yorick Peterse ef7f50137a Added Oga::EntityDecoder
This module removes some of the code duplication needed to determine
what entity decoder to use.
2015-04-07 21:18:15 +02:00
Yorick Peterse 3f6aa04e91 Release 0.3.0 2015-04-03 21:11:15 +02:00
Yorick Peterse 3176459307 Ignore declared namespaces in HTML documents
The HTML spec states that any declared namespaces, including the default
namespace are to be ignored.

This fixes #85
2015-03-26 22:38:39 +01:00
Yorick Peterse 5adeae18d0 XPath queries match nodes in the default namespace
When querying an XML document that explicitly defines the default XML
namespace the XPath evaluator now correctly matches all nodes within
that namespace if no namespace prefix is given in the query. Previously
this would always return an empty set.
2015-03-26 01:13:55 +01:00
Yorick Peterse f175414917 Added XML::Element#default_namespace? 2015-03-26 01:10:20 +01:00
Yorick Peterse b6fcd326ef Added XML::Node#html? and XML::Node#xml?
The former has been moved over from XML::Text, the latter just inverts
html?.
2015-03-26 01:02:32 +01:00
Yorick Peterse 4ad502958d Added XML::Attribute#==
Overwriting this method makes it easier to check if a given namespace
equals the default XML (and soon HTML) namespace.
2015-03-26 00:53:16 +01:00
Yorick Peterse f2d69af33b Distinguish default attribute/element namespaces
The previous commit messed this up because I wasn't fully awake.
2015-03-26 00:43:50 +01:00
Yorick Peterse 68ada997a8 Moved default namespace into Oga::XML
The default namespace is now located at Oga::XML::DEFAULT_NAMESPACE
instead of Oga::XML::Attribute::DEFAULT_NAMESPACE.
2015-03-26 00:35:28 +01:00
Yorick Peterse 3cdcdf6daa Corrected YARD formatting 2015-03-23 00:31:56 +01:00
Yorick Peterse 66fa9f62ef Added LRU#maximum=/maximum
This allows one to change the maximum amount of keys stored in the
XPath/CSS caches, for example:

    Oga::XPath::Parser::CACHE.maximum = 2056
2015-03-23 00:26:48 +01:00
Yorick Peterse 12aa21fb50 Use parse_with_cache when querying xpath/css 2015-03-23 00:23:46 +01:00
Yorick Peterse 2c4e490614 Added CSS/XPath Parser.parse_with_cache
This method parses and caches ASTs using Oga::LRU. Currently the default
of 1024 keys is used.

See #71 for more information.
2015-03-23 00:22:59 +01:00
Yorick Peterse 67d7d9af88 Added thread-safe LRU class
This class will be used for storing parser XPath/CSS ASTs.

See #71 for more information.
2015-03-23 00:21:52 +01:00
Yorick Peterse 31e93e54f9 Removed Mutex usage from XML::Text
Instead of trying to make this class thread-safe I'm going with the
option of simply declaring it unsafe to mutate instances of XML::Text
while reading it in parallel. This removes the need for Mutex
allocations and keeps the code simple.

Fixes #82
2015-03-21 01:27:00 +01:00
Yorick Peterse c647f064b5 Remove remaining Racc parsing bits 2015-03-21 01:23:00 +01:00
Yorick Peterse ed14981044 Ported the CSS parser to ruby-ll 2015-03-21 01:23:00 +01:00
Yorick Peterse 2714dbe419 Use the ? operator in the XPath parser 2015-03-21 01:23:00 +01:00
Yorick Peterse 3b74a55d73 Use the ? operator in the XML parser 2015-03-21 01:23:00 +01:00
Yorick Peterse 2bbb7d2b10 Use new operators in the XML parser
This allows the removal of quite a bit of recursion based code.
2015-03-21 01:23:00 +01:00
Yorick Peterse 02da47c1f0 Replaced some XPath parser recursion with * 2015-03-21 01:23:00 +01:00
Yorick Peterse 3b06780802 Removed Racc based XPath parser 2015-03-21 01:23:00 +01:00
Yorick Peterse 588c225c53 Proper XPath operator parsing precedence 2015-03-21 01:23:00 +01:00
Yorick Peterse 0fa9d4df88 Ported remaining XPath parsing bits to ruby-ll.
Currently all operators are left-associative with no particular precedence. This
causes a few specs to fail for now. Outside of that the new parser should be
able to parse the same input as the Racc based parser.
2015-03-21 01:22:59 +01:00
Yorick Peterse 4ebfc849a4 Start porting the XPath parser to ruby-ll.
There are still a few bits left to do such as supporting parenthesis and
assigning the correct precedence to the others.
2015-03-21 01:22:59 +01:00
Yorick Peterse cbdaeb21f4 Unwrap a few lines in the XML parser. 2015-03-21 01:22:59 +01:00
Yorick Peterse cfc6749556 Use splat instead of Array#unshift for attributes. 2015-03-21 01:22:59 +01:00
Yorick Peterse d210c9fb57 Compacted a few XML parser rules. 2015-03-21 01:22:59 +01:00
Yorick Peterse a5cd75cb7e Removed useless string allocs from the XML parser. 2015-03-21 01:22:59 +01:00
Yorick Peterse fdcd712ffe Don't use Array#uniq in NodeSet#initialize.
Removing this makes the process of parsing larger XML documents a bit faster.
The downside is that NodeSet#initialize will no longer filter out duplicate
nodes, though this is not something Oga itself relies upon.

Methods such as NodeSet#push still do ignore elements already present.
2015-03-21 01:22:59 +01:00
Yorick Peterse c36b35ac0f Skip ownership iteration when there's no owner.
There's no point in iterating over all the nodes and assigning ownership if
there's no owner to begin with.
2015-03-21 01:22:59 +01:00
Yorick Peterse 006ef4d51a Port over most of the old XML error handling.
Some messages are a bit different due to ruby-ll's error handling, other than
that it's largely the same stuff as before.
2015-03-21 01:22:59 +01:00
Yorick Peterse 1a326fc516 Remove Racc based XML parser. 2015-03-21 01:22:59 +01:00
Yorick Peterse d8b9725b82 Fixed SAX parsing of XML attributes.
This was utterly broken, mainly due to me overlooking it. There are now 2 new
callbacks to handle this properly:

* on_attribute: to handle a single attribute/value pair
* on_attributes: to handle a collection of attributes (as returned by
  on_attribute)

By default on_attribut returns a Hash, on_attributes in turn merges all
attribute hashes into a single one. This ensures that on_element _actually_
receives the attributes as a Hash, instead of an Array with random
nil/XML::Attribute values.
2015-03-21 01:22:59 +01:00
Yorick Peterse dd626c10d3 Use Array#unshift in the LL XML grammar.
Using Array#+ for large sets (e.g. in the benchmarks) is _really_ slow.
Interesting enough Array#unshift uses as much memory as the Racc parser and is
about as fast, even though it has to move memory around.
2015-03-21 01:22:59 +01:00
Yorick Peterse f94407ee9d Parser callback for XML attributes. 2015-03-21 01:22:59 +01:00
Yorick Peterse a023b35e78 Fixed the pull parser for the XML LL parser. 2015-03-21 01:22:59 +01:00
Yorick Peterse 5eed0d31d6 Ported over most of the XML parser to ruby-ll.
This is still missing the error handling previously present.
2015-03-21 01:22:59 +01:00
Yorick Peterse 15a3ab9ba5 ruby-ll: full support for parsing doctypes. 2015-03-21 01:22:59 +01:00
Yorick Peterse 71aefb53cc Started porting the XML parser to ruby-ll
This is far from done.
2015-03-21 01:22:59 +01:00
Yorick Peterse 2ec91f130f Lazy decoding of XML/HTML entities.
Instead of decoding entities in the lexer we'll do this whenever XML::Text#text
is called. This removes the overhead from the parsing phase and ensures the
process is only triggered when actually needed. Note that calling #to_xml and/or
the #inspect methods on a Text (or parent) instance will also trigger the entity
conversion process.

The new entity decoding API supports both regular entities (e.g. &amp;) as well
as codepoint based entities (both regular and hexadecimal codepoints).

To allow safe read-only access to Text instances from multiple threads a mutex
is used. This mutex ensures that only 1 thread can trigger the conversion
process.

Fixes #68
2015-03-05 23:00:43 +01:00
Yorick Peterse 3e05593536 Release 0.2.3 2015-03-04 11:56:23 +01:00
Yorick Peterse 78e40b55c0 Handle parsing of HTML <style> tags.
This basically re-applies the technique used for HTML <script> tags. With this
extra addition I decided to rename/normalize a few things so it's easier to add
any extra tags in the future. One downside of this setup is that the following
will not be parsed by Oga:

    <style>
        </script>
    </style>

The same applies to script tags containing a literal </style> tag. Since this
particular case is rather unlikely to occur I'm OK with not supporting it as it
_does_ simplify the lexer quite a bit.

Fixes #80
2015-03-03 16:28:05 +01:00
Yorick Peterse 73534375d5 Release 0.2.2 2015-03-03 13:36:32 +01:00
Yorick Peterse 142b467277 Set parent of nodes set using Element#inner_text=
This ensures that any text nodes created using Element#inner_text= have their
parent node set correctly.
2015-03-03 13:13:05 +01:00
Yorick Peterse 503efc32cd Release 0.2.1 2015-03-02 22:12:49 +01:00
Yorick Peterse 874d7124af Don't convert <script> text to XML entities.
Fixes #79.
2015-03-02 17:32:19 +01:00
Yorick Peterse 9a586363e9 Added XML::Document#html? 2015-03-02 16:39:40 +01:00
Yorick Peterse ba2177e2cf Lex contents of <script> tags as plain text.
When lexing input in HTML mode the lexer has to treat _all_ content of a
<script> tag as plain text. This ensures that the lexer can process input such
as "x <y" and "// <foo>" correctly.

Fixes #70.
2015-03-02 16:22:09 +01:00
Yorick Peterse e138aa15ac Removed stray comment in the XPath parser. 2014-12-28 23:55:33 +01:00
Yorick Peterse 746c8052dd Remove all nodes when calling Element#inner_text=
This fixes #64.
2014-12-14 23:32:43 +01:00
Dmitry Krasnoukhov 26baf89440 Add missing entities to the decode/encode lists 2014-11-21 01:53:11 +02:00
Yorick Peterse cbb2815146 Support for inline doctype rules plus newlines.
This adds support for lexing/parsing XML documents that use an IO as input _and_
contain doctype rules with newlines in them.

This fixes #63.
2014-11-18 20:02:55 +01:00
Yorick Peterse 922cee913d Release 0.2.0 2014-11-17 23:26:19 +01:00
Yorick Peterse ad4f650c5d Fixed XML entity encoding/decoding ordering.
Thanks to @krasnoukhov for providing the initial patch, which this commit is
largely based on.

This fixes #49.
2014-11-17 22:39:43 +01:00
Yorick Peterse cd86d5d294 Allow removal of element attributes. 2014-11-17 09:00:40 +01:00
Yorick Peterse 804646cc5e Don't modify raw namespaces.
When calling Element#available_namespaces the list of namespaces returned by
Element#namespaces must not be modified.
2014-11-17 00:01:16 +01:00
Yorick Peterse 6753d6a26d Slightly better docs for the XPath/CSS parsers. 2014-11-16 23:40:19 +01:00
Yorick Peterse 57adabc068 Ensure SAX after_element receives meaningful args
This changes the behaviour of after_element when parsing documents using the SAX
parsing API. Previously it would always receive a nil argument, which is kinda
pointless. This commit changes that by making sure it receives a namespace name
(if any) and the element name.

This fixes #54.
2014-11-16 23:32:32 +01:00
Yorick Peterse 23b408fe4f Cleaned up CSS parser code for counting siblings. 2014-11-15 18:31:08 +01:00
Yorick Peterse b464815577 Fixed AST generation for nth-(first|last)-of-type. 2014-11-15 18:27:15 +01:00
Yorick Peterse 9eead81a7c Fixed AST for :only-of-type 2014-11-15 18:08:26 +01:00
Yorick Peterse 1c301d40e2 Properly fixed AST for first-of-type/last-of-type
This requires keeping track of the current element being processed. This in turn
allows the usage of count() + preceding-sibling/following-sibling.
2014-11-15 17:58:56 +01:00
Yorick Peterse f1d574f342 Evaluate XPath predicates for every context node.
Instead of evaluating a predicate once for all context nodes, they should
instead be evaluated separately per context node.
2014-11-15 00:31:44 +01:00
Yorick Peterse 6daa3e7a00 Reverted AST changes for first-of-type
Functions can't be used in combination with axes, so I'll just need to fix the
position() function to work properly.
2014-11-14 23:51:46 +01:00
Yorick Peterse 2d6a2be2e8 Revert "Fixed XPath AST for :last-of-type"
Axes can't be used in combination with functions.

This reverts commit b0b572a584.
2014-11-14 23:49:49 +01:00
Yorick Peterse b0b572a584 Fixed XPath AST for :last-of-type
This should count following nodes, not merely the position.
2014-11-14 23:27:15 +01:00
Yorick Peterse 0128dc50ae Fixed CSS evaluation of :first-of-type
The old XPath "position() = 1" would work in Nokogiri due to the way they
retrieve descendants. In Oga however this would simply always return the first
node.

To fix this Oga now counts the amount of preceding siblings that match the same
full name.
2014-11-14 01:25:03 +01:00
Yorick Peterse e3a26c5d15 Allow querying of nodes using CSS. 2014-11-14 01:05:29 +01:00
Yorick Peterse 01b88d8c68 Use correct root for preceding/following(-sibling)
This ensures these axes work correctly when scoped to a node instead of a
document.
2014-11-13 01:11:29 +01:00
Yorick Peterse b0f6409d1e Wrap predicate nodes around others in CSS ASTs. 2014-11-13 00:47:26 +01:00
Yorick Peterse 817a5e075b Wrap predicate AST nodes _around_ other nodes.
This means that "foo[1]" uses this AST:

    (predicate (test nil "foo") (int 1))

Instead of this AST:

    (test nil "foo" (int 1))

This makes it easier for the XPath evaluator to process predicates correctly.
2014-11-12 22:59:38 +01:00
Yorick Peterse 857ac517d5 Added XML::NodeSet#==
This method can be used to compare two NodeSet instances. By using
XML::NodeSet#equal_nodes?() the need for exposing the "nodes" instance variable
is also removed.
2014-11-09 18:33:16 +01:00
Yorick Peterse a586a512b8 Removed support for CSS such as "|X"
This expression could be used to get all elements that _don't_ have any
namespace. The problem is that this can't be expressed as just a node test,
instead the resulting XPath would have to look something like the following:

    X[local-name() = name()]

However, since the XPath predicates are already created for pseudo classes and
such, also injecting the above into it would be a real big pain. As such I've
decided not to support it.
2014-11-05 00:45:40 +01:00
Yorick Peterse 1c20ef52ae XPath idents can't start with a star.
That is, names such as "*foo" are not valid. This was most likely a typo in the
first place.
2014-11-05 00:38:17 +01:00
Yorick Peterse 61801fe562 Allow CSS identifiers to start with an underscore. 2014-11-05 00:37:40 +01:00
Yorick Peterse f1316c50fb Allow XPath idents to start with an underscore.
This is valid in XML so XPath should support it as well.
2014-11-05 00:35:47 +01:00
Yorick Peterse eb3a3e7630 Use the descendant axis for CSS selectors.
Instead of using "descendant-or-self" Oga will use "descendant". This ensures
that expressions such as "foo *" don't return a set also including the "foo"
element.

Nokogiri solves this problem in a somewhat different way by using //foo//* for
the CSS expression "foo *". While this works in Nokogiri the expression
"descendant-or-self::node()" is slow as a snail in Oga (due to its nature of
retrieving _all_ nodes first). By using "descendant" we can work around this
problem.
2014-11-05 00:22:09 +01:00
Yorick Peterse 602f2fe8bb Apply node() type tests to the document too.
When running XPath queries such as "self::node()" the result should be a set
containing the document itself. This in turn fixes expressions such as
descendant-or-self::node()/a.
2014-11-04 23:41:45 +01:00
Yorick Peterse de04c61df9 Parsing support for the :empty pseudo. 2014-11-04 00:00:23 +01:00
Yorick Peterse 1b870406de Parsing support for the :only-of-type pseudo. 2014-11-03 23:47:25 +01:00
Yorick Peterse 2cf058457d Parsing support for the :only-child pseudo. 2014-11-03 23:27:56 +01:00
Yorick Peterse 759bb71d17 Parsing support for the :last-of-type pseudo. 2014-11-03 23:01:44 +01:00
Yorick Peterse 10632eafd4 Parsing support for the :first-of-type pseudo. 2014-11-03 23:00:15 +01:00
Yorick Peterse 7a606a11d3 Parsing support for the :last-child pseudo class. 2014-11-03 22:58:08 +01:00
Yorick Peterse b4fec3cc8c Parsing support for the :first-child pseudo class. 2014-11-03 22:45:00 +01:00
Yorick Peterse 020d979fba Parsing support for :nth-last-of-type(). 2014-11-03 22:15:21 +01:00
Yorick Peterse bc8be9f725 Fixed various incorrect YARD tags. 2014-11-02 21:23:29 +01:00
Yorick Peterse 2e1320c2dc Explicit return in step_modulo_value. 2014-11-02 19:32:02 +01:00
Yorick Peterse ab8b451dc3 Parsing support for :nth-of-type() 2014-11-02 19:29:39 +01:00
Yorick Peterse 0faceffacb Parsing support for nth-child(n+X) 2014-11-02 19:29:09 +01:00
Yorick Peterse 8d8d74ec41 Drop nth-child support of all negative sequences
This removes parsing support for selectors such as :nth-child(-n-6). According
to the CSS spec this isn't valid anyway (confirmed by testing it in Chromium).
As a result there's no point in supporting it in any way.
2014-11-02 19:19:05 +01:00
Yorick Peterse b31288b7d2 Use correct modulo for nth-child and negatives. 2014-11-02 18:53:23 +01:00
Yorick Peterse 9cce93fc4a Parsing support for :nth-last-child. 2014-11-01 20:58:28 +01:00
Yorick Peterse 03f897c2b7 Support for all possible nth-child arguments.
That is, as far as I can tell based on Nokogiri's behaviour (which Oga now
matches).
2014-10-30 23:03:46 +01:00
Yorick Peterse 0e6aefb727 Fixed parsing of nth-child(n) and nth-child(-n) 2014-10-30 00:24:31 +01:00
Yorick Peterse 87f6b9c723 Basic support for :nth-child()
This already includes support for formulas such as 2n, odd, even and 2n+1.
Negative formulas, just "n" and others are not yet supported.
2014-10-28 00:21:11 +01:00
Yorick Peterse 46646e2ace Support for custom grouping of XPath expressions.
This allows the use of expressions such as "(A or B) and C".

This fixes #59.
2014-10-26 22:38:05 +01:00
Yorick Peterse 39c0f7147c Support for parsing the :root pseudo class. 2014-10-26 22:20:23 +01:00
Yorick Peterse 32764c9a14 Proper parsing support for all CSS operators. 2014-10-26 12:45:43 +01:00
Yorick Peterse 24ae791f00 Better support for lexing multi-line strings.
When lexing multi-line strings everything used to work fine as long as the input
were to be read as a whole. However, when using an IO instance all hell would
break loose. Due to the lexer reading IO instances on a per line basis,
sometimes Ragel would end up setting "ts" to NULL. For example, the following
input would break the lexer:

    <foo class="\nbar" />

Due to the input being read per line, the following data would be sent to the
lexer:

    <foo class="\n
    bar" />

This would result in different (or NULL) pointers being used for building a
string, in turn resulting in memory allocation errors.

To work around this the string lexing setup has been broken into separate
machines for single and double quoted strings. The tokens used have also been
changed so that instead of just "T_STRING" there are now the following tokens:

* T_STRING_SQUOTE
* T_STRING_DQUOTE
* T_STRING_BODY

A string can have multiple T_STRING_BODY tokens (= multi-line strings, only the
case for IO inputs). These strings are stitched back together by the parser.

This fixes #58.
2014-10-26 11:39:56 +01:00
Yorick Peterse b304b8b077 Fixed descendant-or-self with a predicate.
Processing of this axis along with a predicate wouldn't quite work out. Even if
the predicate returned false the node would still be matched (which should not
be the case).
2014-10-23 01:12:10 +02:00
Yorick Peterse 7ee7f25239 Support for parsing CSS axes. 2014-10-23 00:42:45 +02:00
Yorick Peterse 9955f61bcb Renamed CSS axis tokens.
These have been renamed as following:

T_CHILD => T_GREATER
T_FOLLOWING => T_TILDE
T_FOLLOWING_DIRECT => T_PLUS
2014-10-21 23:25:11 +02:00
Yorick Peterse 851e7d6d0b First pass of rewriting the CSS parser.
The new parser uses way less confusing rule names, is a bit more strict and in
general much less of a pain to deal with.
2014-10-21 23:19:31 +02:00
Yorick Peterse e3de65a258 Lex whitespace preceding CSS axes separately.
Previously input such as "x > y" would result in the following token sequences:

    T_IDENT, T_CHILD, T_IDENT

This commit changes this to the following:

    T_IDENT, T_SPACE, T_CHILD, T_IDENT

This allows the parser to use T_SPACE as a terminal token, this in turn prevents
around 16 shift/reduce conflicts from arising.

This does mean that input such as " > y" or " x > y" is now invalid. This
however can be solved by simply _not_ adding leading/trailing whitespace to CSS
queries.
2014-10-21 23:18:46 +02:00
Yorick Peterse 21c27bf48e Surround class values with spaces.
When using a CSS class selector the resulting XPath string passed to contains()
should be surrounded by spaces.
2014-10-20 09:29:42 +02:00
Yorick Peterse 15ebdb7de4 Fixed parsing of CSS class selectors.
When a class selector is used it should be checked as one of the possible
values, not as _the_ only value (unlike ID selectors).
2014-10-20 00:45:41 +02:00
Yorick Peterse 174d33c597 Re-enabled parsing of CSS predicates. 2014-10-20 00:39:12 +02:00
Yorick Peterse d4150fd0f5 First step at rewriting the CSS parser.
The new setup will not involve a separate transformation stage, instead the CSS
parser will directly emit an XPath AST. This reduces the overhead needed for
parsing/evaluating CSS selectors while also simplifying the code. The downside
is that I basically have to re-write 80% of the parser.
2014-10-20 00:30:16 +02:00
Yorick Peterse ea2baa2020 Swap child node order for CSS pseudo classes. 2014-10-16 23:18:14 +02:00
Yorick Peterse 63d27fa709 Swap child order of CSS class and id nodes.
This makes it easier to transform the AST at a later stage.
2014-10-16 23:13:54 +02:00
Yorick Peterse 073e8fbe5b Basic boilerplate for converting CSS to XPath. 2014-10-16 00:25:31 +02:00
Yorick Peterse 48eb4f83df Lexing/parsing of CSS pseudos with ident arguments
This allows the lexing/parsing of expressions such as "html:lang(en)".
2014-10-15 09:42:26 +02:00
Yorick Peterse d9a4221a0a Remove :axis CSS node types.
The various axes are now simply their own node types.
2014-10-12 18:08:35 +02:00
Yorick Peterse ed0cd7826e Fixed precedence of ID/class CSS selectors 2014-10-07 23:05:34 +02:00
Yorick Peterse 91f9cc984b Parsing of pseudo classes without node tests. 2014-10-07 23:01:58 +02:00
Yorick Peterse a6b0bd96c8 Support for parsing CSS class/ID selectors. 2014-10-07 22:57:23 +02:00
Yorick Peterse 6792127600 Reworked CSS parser rules.
This includes better rules for parsing separate path members, pseudo class
arguments and some changes to remove all remaining parsing conflicts.
2014-10-07 22:47:47 +02:00
Yorick Peterse b40c0243ce Tighten up lexing of CSS predicates.
Operators can now only occur inside predicates and any whitespcae in these
predicates is ignored.
2014-10-07 22:17:04 +02:00
Yorick Peterse 625b9eeffd Lexing of CSS axes with surrounding whitespace. 2014-10-07 22:06:45 +02:00
Yorick Peterse 619c0bbc14 Emit tokens for whitespace in the CSS lexer. 2014-10-07 21:55:41 +02:00
Yorick Peterse d960eb7cd5 Removed CSS lexer code that was commented out. 2014-10-07 09:29:11 +02:00
Yorick Peterse 16d66a7eb6 Better parsing for the nth-child pseudo class.
This uses stricter (and more correct) rules in both the lexer and the parser.
The resulting AST has also received a small rework to make it more compact and
less confusing.
2014-10-06 23:52:46 +02:00
Yorick Peterse d0a8a3b18c Basic support for parsing CSS pseudo classes.
This currently does not yet allow chained pseudo classes, nor does it allow for
pseudos such as nth-child(2n).
2014-10-05 23:46:41 +02:00
Yorick Peterse e2b36ad9a4 Merge the CSS "expression" and "path" parser rules 2014-10-05 23:36:15 +02:00
Yorick Peterse 50ee66419e Rename CSS "node operators" to "axes". 2014-10-05 23:33:46 +02:00
Yorick Peterse 197cb052be Tighten up CSS predicate member rules.
CSS predicates can't contain full blown expressions, only attribute node tests
and operators.
2014-10-05 23:20:10 +02:00
Yorick Peterse 8fef62fca0 Support for parsing CSS operators. 2014-10-05 10:06:58 +02:00
Yorick Peterse e03cd42735 Stricter lexing rules for XPath wildcards. 2014-10-05 09:57:25 +02:00
Yorick Peterse 2dd148539d Parsing of CSS predicates.
This adds support for parsing expressions such as "foo[class]".
2014-10-05 09:32:21 +02:00
Yorick Peterse 773ff4ce45 Support for parsing multiple CSS node tests. 2014-10-05 01:28:19 +02:00
Yorick Peterse b9a1f914bd Basic CSS parser boilerplate.
This currently only parses single node tests (e.g. just "foo").
2014-10-02 23:32:07 +02:00
Yorick Peterse 4eea6d8359 Removed useless ivar in the XPath parser. 2014-10-02 22:52:39 +02:00
Yorick Peterse cc3e752e1f Removed custom AST::Node class.
Since this class did nothing other than extend AST::Node we might as well use
the latter.
2014-10-02 22:49:29 +02:00
Yorick Peterse 73c5dbe636 Basic setup for lexing CSS pseudo selectors.
This includes support for the crazy 2n+1 syntax you can use with selectors such
as :nth-child().

CSS selectors: doing what XPath already does using an even crazier syntax,
because screw you.
2014-09-28 22:38:25 +02:00
Yorick Peterse ea4a429430 Lexing of various CSS operators. 2014-09-28 22:38:25 +02:00
Yorick Peterse 2ede705f1b Lexing of various primitive CSS tokens.
This includes brackes, commas, pipes (used for namespaces) and more.
2014-09-28 22:38:24 +02:00
Yorick Peterse aa60115c0a Basic boilerplate for lexing CSS selectors. 2014-09-28 22:38:24 +02:00
Yorick Peterse 0299ff1ea4 Support for Enumerator inputs in the XML lexer.
This fixes #48.
2014-09-28 22:27:30 +02:00
Yorick Peterse 5f7256eb8f Encode/decode XML entities.
When lexing XML entities such as &amp; and &lt; these sequences are now
converted into their "actual" forms. In turn, Oga::XML::Text#to_xml ensures they
are encoded when the method is called.

Performance wise this puts some strain on the lexer, for every T_TEXT/T_STRING
node now potentially has to have its content modified. In the benchmark
xml/lexer/string_average_bench.rb the average processing time is now about the
same as before the improvements made in
8db77c0a09. I was hoping that the lexer would
still be a bit faster, but alas this is not the case. Doing this in native code
would be a nightmare as C doesn't have a proper string replacement function. I'm
not old/sadistic enough to write on myself just yet.

This fixes #49
2014-09-28 21:53:25 +02:00
Yorick Peterse 3307e2f4d2 Remove source lines from parser error messages.
This was a gimmick in the first place. It doesn't work well with IO instances
(= requires re-reading of the input), the code was too complex and it wasn't
that useful in the end. Lets just get rid of it.

This fixes #53.
2014-09-25 22:58:06 +02:00
Yorick Peterse 8db77c0a09 Count newlines of text nodes in native code.
Instead of relying on String#count for counting newlines in text nodes, Oga now
does this in C/Java. String#count isn't exactly the fastest way of counting
characters. Performance was measured using
benchmark/xml/lexer/string_average_bench.rb. Before this patch the results were
as following:

    MRI:   0.529s
    Rbx:   4.965s
    JRuby: 0.622s

After this patch:

    MRI:   0.424s
    Rbx:   1.942s
    JRuby: 0.665s => numbers vary a bit, seem roughly the same as before

The commands used for benchmarking:

    $ rake clean # to make sure that C exts aren't shared between MRI/Rbx
    $ rake generate
    $ rake fixtures
    $ ruby benchmark/xml/lexer/string_average_bench.rb

The big difference for Rbx is probably due to the implementation of String#count
not being super fast. Some changes were made
(https://github.com/rubinius/rubinius/pull/3133) to the method, but this hasn't
been released yet.

JRuby seems to perform in a similar way, so either it was already optimizing
things for me or I suck at writing well performing Java code.

This fixes #51.
2014-09-25 22:49:11 +02:00
Yorick Peterse 4469ffc5b1 Improved HTML void element detection performance.
This ensures we only call String#downcase if we can't find an all lowercased
*and* all uppercased version of the element name. This in turn can save as many
object allocations as there are HTML opening tags.

This fixes #52.
2014-09-24 11:07:34 +02:00
Yorick Peterse c3a5ce745c Release 0.1.3 2014-09-24 00:24:00 +02:00
Yorick Peterse 75f4e81533 Use namespace_name in Attribute#to_xml
Instead of using `namespace.name` lets just use `namespace_name`. This fixes the
problem of serializing attributes where the namespace prefix is "xmlns" as the
namespace for this isn't registered by default.

This fixes #47.
2014-09-24 00:19:58 +02:00
Yorick Peterse 4818176ae0 Corrected lying documentation.
Something something I should keep that in sync in the future.
2014-09-23 17:35:18 +02:00
Yorick Peterse ee94f0dbfe Release 0.1.2. 2014-09-23 16:23:49 +02:00
Yorick Peterse 2f5c61b3eb Small cleanup for the XPath lexer. 2014-09-19 01:10:25 +02:00
Yorick Peterse 79d9ed3214 Corrected YARD argument name. 2014-09-16 14:50:17 +02:00
Yorick Peterse 317b49bcf6 Implemented a basic SAX API.
This API is a little bit dodgy (similar to Nokogiri's API) due to the use of
separate parser and handler classes. This is done to ensure that the return
values of callback methods (e.g. on_element) aren't used by Racc for building
AST trees. This also ensures that whatever variables are set by the handler
don't conflict with any variables of the parser.

This fixes #42.
2014-09-16 14:30:46 +02:00
Yorick Peterse 9e935e5d24 Force the usage of the Racc Gem.
This ensures that we're loading the Gem and not the standard library version.
2014-09-16 11:58:56 +02:00
Yorick Peterse cdfeeed85f Provide somewhat more meaningful parser errors.
While still a bit cryptic this is probably as best as we can get it. An example:

    Oga.parse_xml("<namefoo:bar=\"10\"")

    parser.rb:116:in `on_error': Unexpected string on line 1: (Racc::ParseError)

    => 1: <namefoo:bar="10"

This fixes #43.
2014-09-16 01:09:06 +02:00
Yorick Peterse 32b11ef1e2 self-close certain XML/HTML elements.
When an XML element has no child nodes a self-closing tag is used. When parsing
documents/elements in HTML mode this is only done if the element is a so called
"void element" (e.g. <link> tags).

This fixes #46.
2014-09-16 00:44:38 +02:00
Yorick Peterse 6fc7e2e254 Track document types when parsing.
When parsing XML/HTML documents the corresponding document type (:html or :xml)
is stored in Document#type.
2014-09-16 00:25:51 +02:00
Yorick Peterse 9959f5cda4 Don't remove namespace attributes when registering
When registering namespaces from an attributes list the attributes should _not_
be removed.

This fixes #45.
2014-09-15 22:04:03 +02:00
Yorick Peterse 795e669632 Ignore default NS when serializing elements.
When a default namespace is set (using xmlns="...") the method
XML::Element#to_xml should _not_ include the namespace prefix in the output.
2014-09-15 21:42:08 +02:00
Yorick Peterse 1abba1be9c Don't overwrite namespace_name with "xmlns".
This was a leftover from an early prototype.
2014-09-15 21:38:53 +02:00
Yorick Peterse 41526e7013 Whitespace pedantics in Element#namespace. 2014-09-15 21:37:54 +02:00
Yorick Peterse abbd8d6f84 Handle registering of default XML namespaces.
When the default namespace is registered (using xmlns="...") Oga now properly
sets the namespace of the container and all child elements.

This fixes #44.
2014-09-15 21:36:15 +02:00
Yorick Peterse b06eadc812 XPath support for absolute paths without tests.
This allows Oga to parse and evaluate the XPath expression "/". This expression
can be used to select just the root node/document.

This fixes #35.
2014-09-15 17:06:58 +02:00
Yorick Peterse 398aaf68bc Return a correct list of available namespaces.
This ensures that inner namespaces take precedence over outer namespaces.

Fixes #40.
2014-09-14 18:42:02 +02:00
Tero Tasanen 9f71b1ec7b Ignore casing when testing for html void elements
Fixes #36
2014-09-14 12:02:02 +03:00
Yorick Peterse ee538ddcc2 Merge pull request #34 from ttasanen/fix_comment
Fix function name in comment
2014-09-13 16:22:55 +02:00
Tero Tasanen 5979798356 Fix function name in comment 2014-09-13 16:22:06 +03:00
Yorick Peterse d082822cdc Removed the node_type method crap.
The logic this was used for now only resides in the pull parser, instead of
being exposed to the public.

This fixes #30.
2014-09-13 15:09:52 +02:00
Yorick Peterse 9ab5c302f7 Release 0.1.1. 2014-09-13 11:50:30 +02:00
Yorick Peterse b8a82b2094 Separate XML attributes by spaces.
This was originally reported by @jrochkind and partially patched by @billdueber.
My patches are built upon the latter, but without the need of using Array#map,
Array#join, etc. They also contain a few style changes.

This fixes #32 and #33.
2014-09-13 11:47:06 +02:00
Yorick Peterse 019ba8c660 Set the initial version to 0.1.0. 2014-09-11 14:11:24 +02:00
Yorick Peterse 319d622fa5 Include namespaces when converting attrs to XML. 2014-09-11 14:03:04 +02:00
Yorick Peterse c45d32a37e Methods for adding attributes to elements.
The methods XML::Element#add_attribute and XML::Element#set can be used to more
easily add attributes to elements. The first method simply adds an Attribute
instance and links it to the element. This allows for fine grained control over
what data the attribute should contain. The second method ("set") simply sets an
attribute based on a name and value, optionally creating the attribute if it
doesn't already exist.
2014-09-10 23:55:29 +02:00
Yorick Peterse 6cb2d54875 Added XML::Element#inner_text=
This method can be used to more easily set the text of an element, without
having to manually muck around with XML::Text instances.
2014-09-10 23:25:39 +02:00
Yorick Peterse 6d19c9b311 Added XML::Element#get
This method can be used to directly retrieve an attribute value.
2014-09-10 19:03:32 +02:00
Yorick Peterse e2dab952d0 Added XML::Node#before and XML::Node#after
These methods can be used to insert nodes before/after other nodes without
manually having to mess around with node sets.

This fixes #28.
2014-09-09 22:48:31 +02:00
Yorick Peterse fccc6359e1 Corrected YARD formatting. 2014-09-09 00:47:51 +02:00
Yorick Peterse ef03a12f99 Optimize descendant-or-self and child axes.
By using NodeSet#concat we can further reduce the amount of object allocations.
This in turn greatly reduces the time it takes to query large documents using
descendant-or-self.
2014-09-05 20:20:13 +02:00
Yorick Peterse 9f6035e784 Reduce object allocations in on_axis_descendant
By using Traversal#each_node and _not_ calling on_test() (which was only called
for node_matches?) we can save ourselves a few object allocations.
2014-09-05 19:58:37 +02:00
Yorick Peterse 98984de540 Moved Document#each_node into a separate module.
This allows it to be re-used by XML::Node.
2014-09-05 19:42:38 +02:00
Yorick Peterse 8f562c24dd Use NodeSet#concat for descendant-or-self.
In a sample XML file this removes around 3400 NodeSet allocations.
2014-09-05 10:56:49 +02:00
Yorick Peterse bcdce306e5 Added NodeSet#concat.
This method allows two sets to be concatenated together without the need of
creating a 3rd node set (as is the case with NodeSet#+).
2014-09-05 10:07:22 +02:00
Yorick Peterse 1fdf876a93 Tweaked docs of the XPath code a bit. 2014-09-04 10:17:32 +02:00
Yorick Peterse d8e2b97031 Tweaked docs of the XML parsers. 2014-09-04 09:34:59 +02:00
Yorick Peterse fbf3ae6e36 Manually increment XPath indexes.
This ensures that the index is only incremented for elements that match a node
test.
2014-09-03 23:56:13 +02:00