Commit Graph

742 Commits

Author SHA1 Message Date
Yorick Peterse b0b572a584 Fixed XPath AST for :last-of-type
This should count following nodes, not merely the position.
2014-11-14 23:27:15 +01:00
Yorick Peterse 0128dc50ae Fixed CSS evaluation of :first-of-type
The old XPath "position() = 1" would work in Nokogiri due to the way they
retrieve descendants. In Oga however this would simply always return the first
node.

To fix this Oga now counts the amount of preceding siblings that match the same
full name.
2014-11-14 01:25:03 +01:00
Yorick Peterse e3a26c5d15 Allow querying of nodes using CSS. 2014-11-14 01:05:29 +01:00
Yorick Peterse 01b88d8c68 Use correct root for preceding/following(-sibling)
This ensures these axes work correctly when scoped to a node instead of a
document.
2014-11-13 01:11:29 +01:00
Yorick Peterse b0f6409d1e Wrap predicate nodes around others in CSS ASTs. 2014-11-13 00:47:26 +01:00
Yorick Peterse 817a5e075b Wrap predicate AST nodes _around_ other nodes.
This means that "foo[1]" uses this AST:

    (predicate (test nil "foo") (int 1))

Instead of this AST:

    (test nil "foo" (int 1))

This makes it easier for the XPath evaluator to process predicates correctly.
2014-11-12 22:59:38 +01:00
Yorick Peterse 857ac517d5 Added XML::NodeSet#==
This method can be used to compare two NodeSet instances. By using
XML::NodeSet#equal_nodes?() the need for exposing the "nodes" instance variable
is also removed.
2014-11-09 18:33:16 +01:00
Yorick Peterse a586a512b8 Removed support for CSS such as "|X"
This expression could be used to get all elements that _don't_ have any
namespace. The problem is that this can't be expressed as just a node test,
instead the resulting XPath would have to look something like the following:

    X[local-name() = name()]

However, since the XPath predicates are already created for pseudo classes and
such, also injecting the above into it would be a real big pain. As such I've
decided not to support it.
2014-11-05 00:45:40 +01:00
Yorick Peterse 1c20ef52ae XPath idents can't start with a star.
That is, names such as "*foo" are not valid. This was most likely a typo in the
first place.
2014-11-05 00:38:17 +01:00
Yorick Peterse 61801fe562 Allow CSS identifiers to start with an underscore. 2014-11-05 00:37:40 +01:00
Yorick Peterse f1316c50fb Allow XPath idents to start with an underscore.
This is valid in XML so XPath should support it as well.
2014-11-05 00:35:47 +01:00
Yorick Peterse eb3a3e7630 Use the descendant axis for CSS selectors.
Instead of using "descendant-or-self" Oga will use "descendant". This ensures
that expressions such as "foo *" don't return a set also including the "foo"
element.

Nokogiri solves this problem in a somewhat different way by using //foo//* for
the CSS expression "foo *". While this works in Nokogiri the expression
"descendant-or-self::node()" is slow as a snail in Oga (due to its nature of
retrieving _all_ nodes first). By using "descendant" we can work around this
problem.
2014-11-05 00:22:09 +01:00
Yorick Peterse 602f2fe8bb Apply node() type tests to the document too.
When running XPath queries such as "self::node()" the result should be a set
containing the document itself. This in turn fixes expressions such as
descendant-or-self::node()/a.
2014-11-04 23:41:45 +01:00
Yorick Peterse de04c61df9 Parsing support for the :empty pseudo. 2014-11-04 00:00:23 +01:00
Yorick Peterse 1b870406de Parsing support for the :only-of-type pseudo. 2014-11-03 23:47:25 +01:00
Yorick Peterse 2cf058457d Parsing support for the :only-child pseudo. 2014-11-03 23:27:56 +01:00
Yorick Peterse 759bb71d17 Parsing support for the :last-of-type pseudo. 2014-11-03 23:01:44 +01:00
Yorick Peterse 10632eafd4 Parsing support for the :first-of-type pseudo. 2014-11-03 23:00:15 +01:00
Yorick Peterse 7a606a11d3 Parsing support for the :last-child pseudo class. 2014-11-03 22:58:08 +01:00
Yorick Peterse b4fec3cc8c Parsing support for the :first-child pseudo class. 2014-11-03 22:45:00 +01:00
Yorick Peterse 020d979fba Parsing support for :nth-last-of-type(). 2014-11-03 22:15:21 +01:00
Yorick Peterse bc8be9f725 Fixed various incorrect YARD tags. 2014-11-02 21:23:29 +01:00
Yorick Peterse 2e1320c2dc Explicit return in step_modulo_value. 2014-11-02 19:32:02 +01:00
Yorick Peterse ab8b451dc3 Parsing support for :nth-of-type() 2014-11-02 19:29:39 +01:00
Yorick Peterse 0faceffacb Parsing support for nth-child(n+X) 2014-11-02 19:29:09 +01:00
Yorick Peterse 8d8d74ec41 Drop nth-child support of all negative sequences
This removes parsing support for selectors such as :nth-child(-n-6). According
to the CSS spec this isn't valid anyway (confirmed by testing it in Chromium).
As a result there's no point in supporting it in any way.
2014-11-02 19:19:05 +01:00
Yorick Peterse b31288b7d2 Use correct modulo for nth-child and negatives. 2014-11-02 18:53:23 +01:00
Yorick Peterse 9cce93fc4a Parsing support for :nth-last-child. 2014-11-01 20:58:28 +01:00
Yorick Peterse 03f897c2b7 Support for all possible nth-child arguments.
That is, as far as I can tell based on Nokogiri's behaviour (which Oga now
matches).
2014-10-30 23:03:46 +01:00
Yorick Peterse 0e6aefb727 Fixed parsing of nth-child(n) and nth-child(-n) 2014-10-30 00:24:31 +01:00
Yorick Peterse 87f6b9c723 Basic support for :nth-child()
This already includes support for formulas such as 2n, odd, even and 2n+1.
Negative formulas, just "n" and others are not yet supported.
2014-10-28 00:21:11 +01:00
Yorick Peterse 46646e2ace Support for custom grouping of XPath expressions.
This allows the use of expressions such as "(A or B) and C".

This fixes #59.
2014-10-26 22:38:05 +01:00
Yorick Peterse 39c0f7147c Support for parsing the :root pseudo class. 2014-10-26 22:20:23 +01:00
Yorick Peterse 32764c9a14 Proper parsing support for all CSS operators. 2014-10-26 12:45:43 +01:00
Yorick Peterse 24ae791f00 Better support for lexing multi-line strings.
When lexing multi-line strings everything used to work fine as long as the input
were to be read as a whole. However, when using an IO instance all hell would
break loose. Due to the lexer reading IO instances on a per line basis,
sometimes Ragel would end up setting "ts" to NULL. For example, the following
input would break the lexer:

    <foo class="\nbar" />

Due to the input being read per line, the following data would be sent to the
lexer:

    <foo class="\n
    bar" />

This would result in different (or NULL) pointers being used for building a
string, in turn resulting in memory allocation errors.

To work around this the string lexing setup has been broken into separate
machines for single and double quoted strings. The tokens used have also been
changed so that instead of just "T_STRING" there are now the following tokens:

* T_STRING_SQUOTE
* T_STRING_DQUOTE
* T_STRING_BODY

A string can have multiple T_STRING_BODY tokens (= multi-line strings, only the
case for IO inputs). These strings are stitched back together by the parser.

This fixes #58.
2014-10-26 11:39:56 +01:00
Yorick Peterse b304b8b077 Fixed descendant-or-self with a predicate.
Processing of this axis along with a predicate wouldn't quite work out. Even if
the predicate returned false the node would still be matched (which should not
be the case).
2014-10-23 01:12:10 +02:00
Yorick Peterse 7ee7f25239 Support for parsing CSS axes. 2014-10-23 00:42:45 +02:00
Yorick Peterse 9955f61bcb Renamed CSS axis tokens.
These have been renamed as following:

T_CHILD => T_GREATER
T_FOLLOWING => T_TILDE
T_FOLLOWING_DIRECT => T_PLUS
2014-10-21 23:25:11 +02:00
Yorick Peterse 851e7d6d0b First pass of rewriting the CSS parser.
The new parser uses way less confusing rule names, is a bit more strict and in
general much less of a pain to deal with.
2014-10-21 23:19:31 +02:00
Yorick Peterse e3de65a258 Lex whitespace preceding CSS axes separately.
Previously input such as "x > y" would result in the following token sequences:

    T_IDENT, T_CHILD, T_IDENT

This commit changes this to the following:

    T_IDENT, T_SPACE, T_CHILD, T_IDENT

This allows the parser to use T_SPACE as a terminal token, this in turn prevents
around 16 shift/reduce conflicts from arising.

This does mean that input such as " > y" or " x > y" is now invalid. This
however can be solved by simply _not_ adding leading/trailing whitespace to CSS
queries.
2014-10-21 23:18:46 +02:00
Yorick Peterse 21c27bf48e Surround class values with spaces.
When using a CSS class selector the resulting XPath string passed to contains()
should be surrounded by spaces.
2014-10-20 09:29:42 +02:00
Yorick Peterse 15ebdb7de4 Fixed parsing of CSS class selectors.
When a class selector is used it should be checked as one of the possible
values, not as _the_ only value (unlike ID selectors).
2014-10-20 00:45:41 +02:00
Yorick Peterse 174d33c597 Re-enabled parsing of CSS predicates. 2014-10-20 00:39:12 +02:00
Yorick Peterse d4150fd0f5 First step at rewriting the CSS parser.
The new setup will not involve a separate transformation stage, instead the CSS
parser will directly emit an XPath AST. This reduces the overhead needed for
parsing/evaluating CSS selectors while also simplifying the code. The downside
is that I basically have to re-write 80% of the parser.
2014-10-20 00:30:16 +02:00
Yorick Peterse ea2baa2020 Swap child node order for CSS pseudo classes. 2014-10-16 23:18:14 +02:00
Yorick Peterse 63d27fa709 Swap child order of CSS class and id nodes.
This makes it easier to transform the AST at a later stage.
2014-10-16 23:13:54 +02:00
Yorick Peterse 073e8fbe5b Basic boilerplate for converting CSS to XPath. 2014-10-16 00:25:31 +02:00
Yorick Peterse 48eb4f83df Lexing/parsing of CSS pseudos with ident arguments
This allows the lexing/parsing of expressions such as "html:lang(en)".
2014-10-15 09:42:26 +02:00
Yorick Peterse d9a4221a0a Remove :axis CSS node types.
The various axes are now simply their own node types.
2014-10-12 18:08:35 +02:00
Yorick Peterse ed0cd7826e Fixed precedence of ID/class CSS selectors 2014-10-07 23:05:34 +02:00
Yorick Peterse 91f9cc984b Parsing of pseudo classes without node tests. 2014-10-07 23:01:58 +02:00
Yorick Peterse a6b0bd96c8 Support for parsing CSS class/ID selectors. 2014-10-07 22:57:23 +02:00
Yorick Peterse 6792127600 Reworked CSS parser rules.
This includes better rules for parsing separate path members, pseudo class
arguments and some changes to remove all remaining parsing conflicts.
2014-10-07 22:47:47 +02:00
Yorick Peterse b40c0243ce Tighten up lexing of CSS predicates.
Operators can now only occur inside predicates and any whitespcae in these
predicates is ignored.
2014-10-07 22:17:04 +02:00
Yorick Peterse 625b9eeffd Lexing of CSS axes with surrounding whitespace. 2014-10-07 22:06:45 +02:00
Yorick Peterse 619c0bbc14 Emit tokens for whitespace in the CSS lexer. 2014-10-07 21:55:41 +02:00
Yorick Peterse d960eb7cd5 Removed CSS lexer code that was commented out. 2014-10-07 09:29:11 +02:00
Yorick Peterse 16d66a7eb6 Better parsing for the nth-child pseudo class.
This uses stricter (and more correct) rules in both the lexer and the parser.
The resulting AST has also received a small rework to make it more compact and
less confusing.
2014-10-06 23:52:46 +02:00
Yorick Peterse d0a8a3b18c Basic support for parsing CSS pseudo classes.
This currently does not yet allow chained pseudo classes, nor does it allow for
pseudos such as nth-child(2n).
2014-10-05 23:46:41 +02:00
Yorick Peterse e2b36ad9a4 Merge the CSS "expression" and "path" parser rules 2014-10-05 23:36:15 +02:00
Yorick Peterse 50ee66419e Rename CSS "node operators" to "axes". 2014-10-05 23:33:46 +02:00
Yorick Peterse 197cb052be Tighten up CSS predicate member rules.
CSS predicates can't contain full blown expressions, only attribute node tests
and operators.
2014-10-05 23:20:10 +02:00
Yorick Peterse 8fef62fca0 Support for parsing CSS operators. 2014-10-05 10:06:58 +02:00
Yorick Peterse e03cd42735 Stricter lexing rules for XPath wildcards. 2014-10-05 09:57:25 +02:00
Yorick Peterse 2dd148539d Parsing of CSS predicates.
This adds support for parsing expressions such as "foo[class]".
2014-10-05 09:32:21 +02:00
Yorick Peterse 773ff4ce45 Support for parsing multiple CSS node tests. 2014-10-05 01:28:19 +02:00
Yorick Peterse b9a1f914bd Basic CSS parser boilerplate.
This currently only parses single node tests (e.g. just "foo").
2014-10-02 23:32:07 +02:00
Yorick Peterse 4eea6d8359 Removed useless ivar in the XPath parser. 2014-10-02 22:52:39 +02:00
Yorick Peterse cc3e752e1f Removed custom AST::Node class.
Since this class did nothing other than extend AST::Node we might as well use
the latter.
2014-10-02 22:49:29 +02:00
Yorick Peterse 73c5dbe636 Basic setup for lexing CSS pseudo selectors.
This includes support for the crazy 2n+1 syntax you can use with selectors such
as :nth-child().

CSS selectors: doing what XPath already does using an even crazier syntax,
because screw you.
2014-09-28 22:38:25 +02:00
Yorick Peterse ea4a429430 Lexing of various CSS operators. 2014-09-28 22:38:25 +02:00
Yorick Peterse 2ede705f1b Lexing of various primitive CSS tokens.
This includes brackes, commas, pipes (used for namespaces) and more.
2014-09-28 22:38:24 +02:00
Yorick Peterse aa60115c0a Basic boilerplate for lexing CSS selectors. 2014-09-28 22:38:24 +02:00
Yorick Peterse 0299ff1ea4 Support for Enumerator inputs in the XML lexer.
This fixes #48.
2014-09-28 22:27:30 +02:00
Yorick Peterse 5f7256eb8f Encode/decode XML entities.
When lexing XML entities such as &amp; and &lt; these sequences are now
converted into their "actual" forms. In turn, Oga::XML::Text#to_xml ensures they
are encoded when the method is called.

Performance wise this puts some strain on the lexer, for every T_TEXT/T_STRING
node now potentially has to have its content modified. In the benchmark
xml/lexer/string_average_bench.rb the average processing time is now about the
same as before the improvements made in
8db77c0a09. I was hoping that the lexer would
still be a bit faster, but alas this is not the case. Doing this in native code
would be a nightmare as C doesn't have a proper string replacement function. I'm
not old/sadistic enough to write on myself just yet.

This fixes #49
2014-09-28 21:53:25 +02:00
Yorick Peterse 3307e2f4d2 Remove source lines from parser error messages.
This was a gimmick in the first place. It doesn't work well with IO instances
(= requires re-reading of the input), the code was too complex and it wasn't
that useful in the end. Lets just get rid of it.

This fixes #53.
2014-09-25 22:58:06 +02:00
Yorick Peterse 8db77c0a09 Count newlines of text nodes in native code.
Instead of relying on String#count for counting newlines in text nodes, Oga now
does this in C/Java. String#count isn't exactly the fastest way of counting
characters. Performance was measured using
benchmark/xml/lexer/string_average_bench.rb. Before this patch the results were
as following:

    MRI:   0.529s
    Rbx:   4.965s
    JRuby: 0.622s

After this patch:

    MRI:   0.424s
    Rbx:   1.942s
    JRuby: 0.665s => numbers vary a bit, seem roughly the same as before

The commands used for benchmarking:

    $ rake clean # to make sure that C exts aren't shared between MRI/Rbx
    $ rake generate
    $ rake fixtures
    $ ruby benchmark/xml/lexer/string_average_bench.rb

The big difference for Rbx is probably due to the implementation of String#count
not being super fast. Some changes were made
(https://github.com/rubinius/rubinius/pull/3133) to the method, but this hasn't
been released yet.

JRuby seems to perform in a similar way, so either it was already optimizing
things for me or I suck at writing well performing Java code.

This fixes #51.
2014-09-25 22:49:11 +02:00
Yorick Peterse 4469ffc5b1 Improved HTML void element detection performance.
This ensures we only call String#downcase if we can't find an all lowercased
*and* all uppercased version of the element name. This in turn can save as many
object allocations as there are HTML opening tags.

This fixes #52.
2014-09-24 11:07:34 +02:00
Yorick Peterse c3a5ce745c Release 0.1.3 2014-09-24 00:24:00 +02:00
Yorick Peterse 75f4e81533 Use namespace_name in Attribute#to_xml
Instead of using `namespace.name` lets just use `namespace_name`. This fixes the
problem of serializing attributes where the namespace prefix is "xmlns" as the
namespace for this isn't registered by default.

This fixes #47.
2014-09-24 00:19:58 +02:00
Yorick Peterse 4818176ae0 Corrected lying documentation.
Something something I should keep that in sync in the future.
2014-09-23 17:35:18 +02:00
Yorick Peterse ee94f0dbfe Release 0.1.2. 2014-09-23 16:23:49 +02:00
Yorick Peterse 2f5c61b3eb Small cleanup for the XPath lexer. 2014-09-19 01:10:25 +02:00
Yorick Peterse 79d9ed3214 Corrected YARD argument name. 2014-09-16 14:50:17 +02:00
Yorick Peterse 317b49bcf6 Implemented a basic SAX API.
This API is a little bit dodgy (similar to Nokogiri's API) due to the use of
separate parser and handler classes. This is done to ensure that the return
values of callback methods (e.g. on_element) aren't used by Racc for building
AST trees. This also ensures that whatever variables are set by the handler
don't conflict with any variables of the parser.

This fixes #42.
2014-09-16 14:30:46 +02:00
Yorick Peterse 9e935e5d24 Force the usage of the Racc Gem.
This ensures that we're loading the Gem and not the standard library version.
2014-09-16 11:58:56 +02:00
Yorick Peterse cdfeeed85f Provide somewhat more meaningful parser errors.
While still a bit cryptic this is probably as best as we can get it. An example:

    Oga.parse_xml("<namefoo:bar=\"10\"")

    parser.rb:116:in `on_error': Unexpected string on line 1: (Racc::ParseError)

    => 1: <namefoo:bar="10"

This fixes #43.
2014-09-16 01:09:06 +02:00
Yorick Peterse 32b11ef1e2 self-close certain XML/HTML elements.
When an XML element has no child nodes a self-closing tag is used. When parsing
documents/elements in HTML mode this is only done if the element is a so called
"void element" (e.g. <link> tags).

This fixes #46.
2014-09-16 00:44:38 +02:00
Yorick Peterse 6fc7e2e254 Track document types when parsing.
When parsing XML/HTML documents the corresponding document type (:html or :xml)
is stored in Document#type.
2014-09-16 00:25:51 +02:00
Yorick Peterse 9959f5cda4 Don't remove namespace attributes when registering
When registering namespaces from an attributes list the attributes should _not_
be removed.

This fixes #45.
2014-09-15 22:04:03 +02:00
Yorick Peterse 795e669632 Ignore default NS when serializing elements.
When a default namespace is set (using xmlns="...") the method
XML::Element#to_xml should _not_ include the namespace prefix in the output.
2014-09-15 21:42:08 +02:00
Yorick Peterse 1abba1be9c Don't overwrite namespace_name with "xmlns".
This was a leftover from an early prototype.
2014-09-15 21:38:53 +02:00
Yorick Peterse 41526e7013 Whitespace pedantics in Element#namespace. 2014-09-15 21:37:54 +02:00
Yorick Peterse abbd8d6f84 Handle registering of default XML namespaces.
When the default namespace is registered (using xmlns="...") Oga now properly
sets the namespace of the container and all child elements.

This fixes #44.
2014-09-15 21:36:15 +02:00
Yorick Peterse b06eadc812 XPath support for absolute paths without tests.
This allows Oga to parse and evaluate the XPath expression "/". This expression
can be used to select just the root node/document.

This fixes #35.
2014-09-15 17:06:58 +02:00
Yorick Peterse 398aaf68bc Return a correct list of available namespaces.
This ensures that inner namespaces take precedence over outer namespaces.

Fixes #40.
2014-09-14 18:42:02 +02:00
Tero Tasanen 9f71b1ec7b Ignore casing when testing for html void elements
Fixes #36
2014-09-14 12:02:02 +03:00
Yorick Peterse ee538ddcc2 Merge pull request #34 from ttasanen/fix_comment
Fix function name in comment
2014-09-13 16:22:55 +02:00
Tero Tasanen 5979798356 Fix function name in comment 2014-09-13 16:22:06 +03:00
Yorick Peterse d082822cdc Removed the node_type method crap.
The logic this was used for now only resides in the pull parser, instead of
being exposed to the public.

This fixes #30.
2014-09-13 15:09:52 +02:00
Yorick Peterse 9ab5c302f7 Release 0.1.1. 2014-09-13 11:50:30 +02:00
Yorick Peterse b8a82b2094 Separate XML attributes by spaces.
This was originally reported by @jrochkind and partially patched by @billdueber.
My patches are built upon the latter, but without the need of using Array#map,
Array#join, etc. They also contain a few style changes.

This fixes #32 and #33.
2014-09-13 11:47:06 +02:00
Yorick Peterse 019ba8c660 Set the initial version to 0.1.0. 2014-09-11 14:11:24 +02:00
Yorick Peterse 319d622fa5 Include namespaces when converting attrs to XML. 2014-09-11 14:03:04 +02:00
Yorick Peterse c45d32a37e Methods for adding attributes to elements.
The methods XML::Element#add_attribute and XML::Element#set can be used to more
easily add attributes to elements. The first method simply adds an Attribute
instance and links it to the element. This allows for fine grained control over
what data the attribute should contain. The second method ("set") simply sets an
attribute based on a name and value, optionally creating the attribute if it
doesn't already exist.
2014-09-10 23:55:29 +02:00
Yorick Peterse 6cb2d54875 Added XML::Element#inner_text=
This method can be used to more easily set the text of an element, without
having to manually muck around with XML::Text instances.
2014-09-10 23:25:39 +02:00
Yorick Peterse 6d19c9b311 Added XML::Element#get
This method can be used to directly retrieve an attribute value.
2014-09-10 19:03:32 +02:00
Yorick Peterse e2dab952d0 Added XML::Node#before and XML::Node#after
These methods can be used to insert nodes before/after other nodes without
manually having to mess around with node sets.

This fixes #28.
2014-09-09 22:48:31 +02:00
Yorick Peterse fccc6359e1 Corrected YARD formatting. 2014-09-09 00:47:51 +02:00
Yorick Peterse ef03a12f99 Optimize descendant-or-self and child axes.
By using NodeSet#concat we can further reduce the amount of object allocations.
This in turn greatly reduces the time it takes to query large documents using
descendant-or-self.
2014-09-05 20:20:13 +02:00
Yorick Peterse 9f6035e784 Reduce object allocations in on_axis_descendant
By using Traversal#each_node and _not_ calling on_test() (which was only called
for node_matches?) we can save ourselves a few object allocations.
2014-09-05 19:58:37 +02:00
Yorick Peterse 98984de540 Moved Document#each_node into a separate module.
This allows it to be re-used by XML::Node.
2014-09-05 19:42:38 +02:00
Yorick Peterse 8f562c24dd Use NodeSet#concat for descendant-or-self.
In a sample XML file this removes around 3400 NodeSet allocations.
2014-09-05 10:56:49 +02:00
Yorick Peterse bcdce306e5 Added NodeSet#concat.
This method allows two sets to be concatenated together without the need of
creating a 3rd node set (as is the case with NodeSet#+).
2014-09-05 10:07:22 +02:00
Yorick Peterse 1fdf876a93 Tweaked docs of the XPath code a bit. 2014-09-04 10:17:32 +02:00
Yorick Peterse d8e2b97031 Tweaked docs of the XML parsers. 2014-09-04 09:34:59 +02:00
Yorick Peterse fbf3ae6e36 Manually increment XPath indexes.
This ensures that the index is only incremented for elements that match a node
test.
2014-09-03 23:56:13 +02:00
Yorick Peterse 81a62a6f00 Added XML::Attribute#text
This allows attributes to also be used for axes such as "=".
2014-09-03 23:38:00 +02:00
Yorick Peterse dc5874f5aa Show XML parsing errors when using IO as input.
Previously this wouldn't display anything due to the IO object being exhausted.
To fix this the input has to be wound back to the start, which means re-reading
it. Sadly I can't think of a way around this that doesn't require buffering
lines while parsing them (which massively increases memory usage).
2014-09-03 22:52:59 +02:00
Yorick Peterse d67f43508d Use #each_line instead of #lines.
IO#each_line is deprecated so lets not rely on it.
2014-09-03 22:31:54 +02:00
Yorick Peterse bd31379c85 Fixed processing of nested predicates.
This ensures that nested predicates and functions that depend on predicates are
processed correctly.
2014-09-03 20:56:07 +02:00
Yorick Peterse e858b54c58 Use nested node stacks for predicates.
This ensures the current context node is set correctly when using the "self"
axis inside a path that's inside a predicate, e.g.

    foo/bar[baz/. = "something"]

Here the "self" axis should refer to foo/bar/baz, _not_ foo/bar.
2014-09-03 19:54:16 +02:00
Yorick Peterse 2b96d65103 XPath "self" axis inside predicates.
The "self" axis should use the current context node when inside a predicate.
2014-09-03 09:40:17 +02:00
Yorick Peterse 71f2b42074 Convenience methods for parsing XML/HTML. 2014-09-03 09:31:48 +02:00
Yorick Peterse d92133ef43 Benchmark Oga XPath evaluation without the parser.
This gives better insight in the performance of the evaluator itself.
2014-09-02 20:55:42 +02:00
Yorick Peterse 9649b50cc9 XML module for more easily querying using XPath. 2014-09-02 20:16:52 +02:00
Yorick Peterse ad34ab47a0 Support for binding/evaluating XPath variables. 2014-09-02 19:04:02 +02:00
Yorick Peterse 5a0e8c5480 Lexing/parsing of XPath variable references. 2014-09-02 10:52:08 +02:00
Yorick Peterse 5de37bdf81 Support for the XPath ">=" operator. 2014-09-01 22:45:01 +02:00
Yorick Peterse 099227901a Support for the XPath "<=" operator. 2014-09-01 22:40:50 +02:00
Yorick Peterse 6eacf74da4 Fixed comparing node equality in XPath expressions
Previously this would take the text of the entire node set, not of the first
node in the set.
2014-09-01 22:34:18 +02:00
Yorick Peterse 8884db8cb6 Support for the XPath ">" operator. 2014-09-01 22:23:35 +02:00
Yorick Peterse 6b45a03cb4 Support for the XPath "<" operator. 2014-09-01 22:20:32 +02:00
Yorick Peterse e1d9e62b72 Support for the XPath "!=" operator. 2014-09-01 20:48:11 +02:00
Yorick Peterse 9e5f15787d Support for the XPath "=" operator. 2014-09-01 18:57:12 +02:00
Yorick Peterse a70645fb89 Support for the XPath sub/- operator. 2014-08-29 09:41:17 +02:00
Yorick Peterse 89686b6cff Support for the XPath mul/* operator. 2014-08-29 09:36:40 +02:00
Yorick Peterse 034b360d13 Support for the XPath "mod" operator. 2014-08-29 09:31:11 +02:00
Yorick Peterse 78c8cd1323 Support for the XPath "div" operator. 2014-08-28 23:05:12 +02:00
Yorick Peterse ced7f739fc Support for the XPath "add" / "+" operator. 2014-08-28 21:18:09 +02:00
Yorick Peterse 4fa40b58cf Support for the XPath "or" operator. 2014-08-28 21:01:12 +02:00
Yorick Peterse 4f189d9218 Support for the XPath "and" operator. 2014-08-28 09:42:55 +02:00
Yorick Peterse 809ed9bfa6 Handle boolean values in the boolean() function. 2014-08-28 09:36:21 +02:00
Yorick Peterse 8fb8fb17b6 Clarified docs for floor(), ceiling() and round(). 2014-08-28 00:01:33 +02:00
Yorick Peterse 543112dcdc Support for the XPath round() function. 2014-08-28 00:00:25 +02:00
Yorick Peterse a2b8e3c954 Support for the XPath ceiling() function. 2014-08-27 23:56:43 +02:00
Yorick Peterse c8fb1ad202 Support for the XPath floor() function. 2014-08-27 23:52:23 +02:00
Yorick Peterse 7c68f2a49b Raise for non node sets in the sum() function.
According to the XPath spec this function *can only* take node sets, nothing
else. Lets actually enforce that.
2014-08-27 23:47:19 +02:00
Yorick Peterse ac06670c24 Better conversion of types to numbers.
The XPath number() function should also be capable of converting booleans to
numbers, something it previously was not able to do. In order to do this
reliably we can't rely on the string() function as this would make it impossible
to distinguish between literal string values and booleans. This is due to
true(), which returns a TrueClass, being converted to the string "true". This
string in turn can't be converted to a float.
2014-08-27 23:38:47 +02:00
Yorick Peterse fcb28d5ae8 Specs for various XPath evaluator helper methods. 2014-08-27 23:37:12 +02:00
Yorick Peterse 30a5d01ebd Support for the XPath sum() function. 2014-08-27 23:05:04 +02:00
Yorick Peterse 585b3535b2 Support for the XPath lang() function. 2014-08-27 20:26:27 +02:00
Yorick Peterse 10e82de87b Handle boolean values in predicate results. 2014-08-27 20:26:10 +02:00
Yorick Peterse 7c41fa814f Default attribute namespaces.
When an attribute is prefixed with "xml" the default namespace should be used
automatically. This namespace is not registered on element level by default as
this namespace isn't registered manually, instead it's a "magic" namespace. This
also ensures we match the behaviour of libxml more closely, hopefully reducing
confusion.
2014-08-27 20:24:40 +02:00
Yorick Peterse d2f991538d Support for the XPath true()/false() functions. 2014-08-27 09:37:28 +02:00
Yorick Peterse 4ef79bad90 Support for the XPath not() function. 2014-08-27 09:35:02 +02:00
Yorick Peterse 338aeeb514 Work around JRuby issue #1923.
String#start_with?() returns false on JRuby when used with an empty string. See
https://github.com/jruby/jruby/issues/1923 for more information.
2014-08-26 21:08:50 +02:00
Yorick Peterse 29870c21f2 Use char Arrays in on_call_translate().
When running a 1.9 based Ruby Enumerable doesn't have the method #[].
2014-08-26 20:41:16 +02:00
Yorick Peterse e288ab88f5 Support for the XPath boolean() function. 2014-08-26 20:22:28 +02:00
Yorick Peterse bcd138a15a Lexing of explicit negative/positive XPath numbers 2014-08-26 20:21:30 +02:00
Yorick Peterse 8295fa5783 Support for the XPath translate() function. 2014-08-26 18:14:44 +02:00
Yorick Peterse 7f3f626744 Support for the XPath normalize-space() function. 2014-08-26 00:06:58 +02:00
Yorick Peterse 06bed1cfdd Support for the XPath string-length() function. 2014-08-25 23:46:18 +02:00
Yorick Peterse a60057db5c Proper handling of decimals for string() calls.
When calling the string() XPath function floats with zero decimals (10.0, 5.0,
etc) should result in a string without any decimals. Ruby converts 10.0 to
"10.0" whereas XPath expects "10".
2014-08-25 23:21:36 +02:00
Yorick Peterse 6c0c5ab720 Revert "Return XPath integers as actual integers."
The particular case of string(10) having to return "10" instead of "10.0" will
be handled separately. Returning integers breaks behaviour/expectations
elsewhere.

This reverts commit 431a253000.
2014-08-25 23:14:22 +02:00
Yorick Peterse a1a2190fe2 Support for the XPath substring() function. 2014-08-25 23:13:34 +02:00
Yorick Peterse 5382891106 Support for the XPath number() function. 2014-08-25 23:13:03 +02:00
Yorick Peterse ba058627f2 Corrected docs for on_call_string(). 2014-08-25 23:12:45 +02:00
Yorick Peterse b2ca18e127 Support for the XPath substring-after() function. 2014-08-25 22:19:13 +02:00
Yorick Peterse b9cdb4a72b Support for the XPath substring-before() function. 2014-08-25 22:11:05 +02:00
Yorick Peterse 83b873e3c1 Support for the XPath contains() function. 2014-08-25 21:52:33 +02:00
Yorick Peterse 5b65d6c31a Support for the xpath starts-with() function. 2014-08-25 09:43:06 +02:00
Yorick Peterse 276a5ab83b Support for lexing empty XPath strings. 2014-08-25 09:42:51 +02:00
Yorick Peterse b688c6dc1b Support for the XPath concat() function. 2014-08-23 20:24:18 +02:00
Yorick Peterse 431a253000 Return XPath integers as actual integers.
This is to ensure that calls such as string(10) return "10" and not "10.0". It
also saves integer -> float conversions when they're not needed.
2014-08-23 20:22:58 +02:00
Yorick Peterse b316fd3e1c Support for the XPath string() function. 2014-08-22 11:06:10 +02:00
Yorick Peterse ff8a4ad3aa Ignore comment nodes in NodeSet#text. 2014-08-22 11:05:39 +02:00
Yorick Peterse 99be3182ae Support for evaluating XPath floats. 2014-08-22 10:59:03 +02:00
Yorick Peterse 6ac3408a71 Match all node types when using node()
Previously this would only match element and text nodes.
2014-08-22 10:58:36 +02:00
Yorick Peterse da09f1296c Support for using namespace-uri() with attributes. 2014-08-21 22:36:18 +02:00
Yorick Peterse 0d41693bfc Support for using name() with attributes. 2014-08-21 21:05:06 +02:00
Yorick Peterse acc056eea4 Support for using local-name() on attributes. 2014-08-21 21:03:28 +02:00
Yorick Peterse 1dd6416bea Basic support for the XPath name() function. 2014-08-21 20:47:20 +02:00
Yorick Peterse e7019ceb4c Renamed context_node to function_node. 2014-08-21 19:06:14 +02:00
Yorick Peterse 8cc0db2283 Support for the XPath namespace-uri() function. 2014-08-21 19:02:41 +02:00
Yorick Peterse 9e20b5ca3e Support for the XPath local-name() function. 2014-08-21 10:21:02 +02:00
Yorick Peterse 2deb7a6d84 Support for the XPath id() function.
This comes with the limitation that it *always* uses the "id" attribute. This is
due to Oga not supporting DTD parsing/evaluation.
2014-08-20 21:06:35 +02:00
Yorick Peterse d351bc26cc Support for the XPath count() function. 2014-08-20 10:16:06 +02:00
Yorick Peterse 709fa365e0 XPath support for last() + evaluator docs.
I really dislike using a stack as it introduces an internal state. Sadly there
doesn't seem to be an easy way around this.
2014-08-19 22:59:26 +02:00
Yorick Peterse e0895be675 Better setup for XPath predicates.
This allows filtering of nodes by indexes (e.g. using last() or a literal
number) and uses the correct index range (1 to N in XPath). The function
position() is not yet supported as this requires access to the current node,
which isn't passed down the call stack just yet.
2014-08-19 19:56:56 +02:00
Yorick Peterse 202f74a8eb on_call_last() should return an index, not a node. 2014-08-19 19:56:41 +02:00
Yorick Peterse 423af37422 Basic support for the XPath last() function. 2014-08-18 19:00:32 +02:00
Yorick Peterse 2817784e6b Support for the XPath pipe operator. 2014-08-17 22:04:08 +02:00
Yorick Peterse d1735750c1 Only store unique nodes in XML::NodeSet.
It's called a "set" after all.
2014-08-17 22:03:44 +02:00
Yorick Peterse bb503728af XPath support for processing instructions. 2014-08-16 22:37:57 +02:00
Yorick Peterse 0d7609da88 Support for parsing XML processing instructions. 2014-08-15 22:23:26 +02:00
Yorick Peterse 8f4eaf3823 Lexing of XML processing instructions. 2014-08-15 22:04:45 +02:00
Yorick Peterse ccd95d69d8 Support for the XPath comment() test. 2014-08-15 20:49:13 +02:00
Yorick Peterse 4d7f224892 Support for the XPath text() type test. 2014-08-15 10:46:00 +02:00
Yorick Peterse 14aa420091 Use a new base class for XML text nodes.
The classes Text, Cdata and Comment now extend CharacterData instead of Text.
2014-08-15 10:43:16 +02:00
Yorick Peterse 24bc84e15e Added XML::Element#text_nodes.
This method returns all the text nodes directly nested in an element.
2014-08-15 10:07:49 +02:00
Yorick Peterse d0092b434d Removed Document#available_namespaces.
Namespaces aren't scoped per document but instead per element, thus this method
doesn't make that much sense. This also fixes the remaining, failing XPath test.
2014-08-14 23:12:33 +02:00
Yorick Peterse d34e4697de Match node types in node_matches?
The method XPath::Evaluator#node_matches? now has a special case to handle
"type-test" nodes. This in turn fixes a bunch of failing tests such as those for
the XPath query "parent::node()".
2014-08-14 22:54:19 +02:00
Yorick Peterse a437d67573 Renamed node_type to type_test. 2014-08-14 22:35:41 +02:00
Yorick Peterse 05f6fc2f8d Implement node() as a type test, not a function. 2014-08-14 22:30:14 +02:00
Yorick Peterse 6ad5170476 Support for lexing/parsing XPath type tests.
Unlike what I thought before syntax such as "node()" is not a function call.
Instead this is a special node test that tests the *types* of nodes, not their
names.
2014-08-14 21:51:58 +02:00
Yorick Peterse 23441bb5a4 Basic support for the XPath node() function. 2014-08-14 18:17:08 +02:00
Yorick Peterse a133b923a2 Only emit extra T_SLASH tokens for "//". 2014-08-13 01:28:43 +02:00
Yorick Peterse 4d956c9ef0 Support for the XPath "namespace" axis. 2014-08-11 00:58:57 +02:00
Yorick Peterse 873bd82273 Stricted matching of namespaced elements. 2014-08-11 00:47:07 +02:00
Yorick Peterse 33c28f633b Proper namespace support for elements.
This is still a bit rough on the edges but already way better than the broken
setup I had before.
2014-08-11 00:41:36 +02:00
Yorick Peterse 04cbbdcf9e Proper namespace support for attributes.
This separates namespace handling into namespace names and namespace objects.
The namespace objects are retrieved from the element an attribute belongs to.
Once retrieved the namespace is cached, due to the overhead of retrieving
namespaces in large documents.
2014-08-11 00:40:17 +02:00
Yorick Peterse fe8f77cf45 Basic work for supporting namespace URIs. 2014-08-08 19:03:42 +02:00
Yorick Peterse f002061aaa Extra type validation for XML::Element options. 2014-08-07 21:10:01 +02:00
Yorick Peterse b1388ff84a Ripped out inspect fuckery.
The old code used for generating Object#inspect values has been ripped out (for
the most part). The result is a non indented but far more compact #inspect
output. The code for this is also easier and doesn't break the signature of
Object#inspect.
2014-08-07 21:09:10 +02:00
Yorick Peterse 3b2279e410 Don't create empty Namespace nodes. 2014-08-07 20:16:46 +02:00
Yorick Peterse 4e18989972 Remove the uri attribute from Namespace.
Oga won't be handling URIs any time soon. The rationale is that they server zero
purpose when it comes to just parsing XML. Another goal of Oga is to make it
easy to modify and reserialize documents back to XML. If namespaces would also
store the URIs this would make this process more difficult.
2014-08-07 20:11:17 +02:00
Yorick Peterse 97e59fe449 Use the Namespace class for namespaces vs Strings. 2014-08-07 20:03:26 +02:00
Yorick Peterse f653203220 Tests for the Namespace class. 2014-08-07 20:02:56 +02:00
Yorick Peterse 8e8ea64206 Fixed serializing of elements to XML. 2014-08-06 00:04:42 +02:00
Yorick Peterse e0bbc81351 Added a very basic Namespace class. 2014-08-06 00:00:08 +02:00
Yorick Peterse d7df908649 Trimmed XML inspect values. 2014-08-05 23:57:12 +02:00
Yorick Peterse 26d4bdc5b1 Support for the XPath "self" axis. 2014-08-05 21:10:12 +02:00
Yorick Peterse 8a9b26fa73 Basic support for the preceding-sibling xpath axis 2014-08-05 19:28:26 +02:00
Yorick Peterse fc1d9776f3 Basic support for the XPath "preceding" axis. 2014-08-05 10:16:37 +02:00
Yorick Peterse 375f3d7870 Basic support for the XPath "parent" axis.
The usage of `parent::node()` is not yet supported.
2014-08-05 09:34:57 +02:00
Yorick Peterse c0a6610d65 Use has_parent? in on_axis_following_sibling. 2014-08-04 21:57:16 +02:00
Yorick Peterse a1f80b4995 Support for the "following-sibling" axis.
This also comes with some small cleanups regarding
XPath::Evaluator#node_matches?. This change removes the need to, every time,
also use can_match_node?() to prevent NoMethodError errors from popping up.
2014-08-04 21:51:51 +02:00
Yorick Peterse 57c0f4b35e Renamed `node` to `ast_node`.
This should make it a bit easier to understand what kind of data the variable is
holding.
2014-08-04 19:01:27 +02:00
Yorick Peterse 211caf00c6 Proper support for the XPath "following" axis. 2014-08-04 18:57:21 +02:00
Yorick Peterse 57fcbbd0fc Allow Document#each_node to skip child nodes.
Child nodes can be skipped by throwing :skip_children.
2014-08-04 10:00:32 +02:00
Yorick Peterse ef1ad5406a Don't yield indexes in Document#each_node.
These indexes won't be used so there's no point in yielding them.
2014-08-04 09:08:39 +02:00
Yorick Peterse 5c23333f46 Traverse document nodes in document order.
The method Document#each_node now yields the nodes in the correct order.
2014-08-01 23:34:32 +02:00
Yorick Peterse c419d8849b Shift instead of pop nodes when yielding all nodes 2014-08-01 19:00:29 +02:00
Yorick Peterse 34e2d28bbd Document#all_nodes -> Document#each_node
This method has been renamed and now yields nodes and their indexes instead of
buffering them in a node set.
2014-07-31 18:57:05 +02:00
Yorick Peterse 4bbf0c98ae Use breadth-first-search for returning all nodes.
This still uses a stack but at least no longer relies on the call stack. I
decided not to go with the Morris in-order algorithm [1] as it modifies the tree
during a search. This would not work well if a document were to be accessed from
multiple threads at once (which should be possible for read-only operations).

I might change this method to actually perform a search (opposed to just
returning everything). This will require some closer inspection of the
available XPath axes to determine if this is needed.

Tests will also be added once I've taken care of the above.

[1]: http://en.wikipedia.org/wiki/Tree_traversal#Morris_in-order_traversal_using_threading
2014-07-30 22:27:09 +02:00
Yorick Peterse 8fe71f298b Half-assed way of retrieving all document nodes.
This currently only works for documents, is not tested and most likely will leak
memory due to being recursive.
2014-07-30 19:56:56 +02:00
Yorick Peterse 52a4375278 Prepare setup for *actual* following support.
The previous commit was nonsense as I didn't understand XPath's "following" axis
properly. This commit introduces proper tests and a note for future me so that I
can implement it properly.
2014-07-30 00:16:44 +02:00
Yorick Peterse 9a97d936e3 Support for the XPath "following" axis. 2014-07-29 23:09:16 +02:00
Yorick Peterse 55e3388e30 Unfuck XPath axes evaluation.
The evaluation of axes has been fixed by changing the initial context as well as
the behaviour of some of the handler methods.

The initial context has been changed so that it's simply a NodeSet of whatever
the root object is, either a Document or an Element instance. Previously this
would be set to the child nodes of a Document in case the root object was a
Document. This in turn would prevent "child" axes from operating correctly.
2014-07-28 00:44:05 +02:00
Yorick Peterse 23de57a3a0 Parse bare XPath node tests as child axes.
When parsing a bare node test such as "A" this is now parsed as following:

    (axis "child" (test nil "A"))

Instead of this:

    (test nil "A")

According to the XPath specification both are identical and this simplifies some
of the code in the XPath evaluator.
2014-07-28 00:34:26 +02:00
Yorick Peterse 1916799fef Basic boilerplate for descendant-or-self. 2014-07-25 21:24:39 +02:00
Yorick Peterse dd37b028a0 Support for the XPath descendant axis. 2014-07-24 09:49:05 +02:00
Yorick Peterse fd37bcff1f Corrected an XPath example. 2014-07-23 18:48:53 +02:00
Yorick Peterse 54e109bf97 Corrected various YARD tags. 2014-07-22 21:28:44 +02:00
Yorick Peterse a0ecba6321 Support for the XPath child axis. 2014-07-22 21:25:02 +02:00
Yorick Peterse ec08b41737 Specs for various XPath::Evaluator helper methods. 2014-07-22 21:04:09 +02:00
Yorick Peterse 1f9d2ede95 Skip node matching for incompatible nodes. 2014-07-22 20:45:15 +02:00
Yorick Peterse 05cf07755d Basic docs for the XPath evaluator. 2014-07-22 16:12:21 +02:00
Yorick Peterse 4142504101 Revert "Lex XPath axes without a node test."
Upon further investigation this change turned out to be useless. Nokogiri/libxml
does not allow the use of long axes without tests, instead it ends up
lexing/parsing such a value as a simple node test.

This reverts commit f699b0d097.
2014-07-22 15:53:00 +02:00
Yorick Peterse c43c38fab9 Emit node() calls for certain short axes.
An axes such as "." is the same as "self::node()". To simplify things on
parser/evaluator level we'll emit the corresponding tokens for a "node()"
function call for these axes.
2014-07-22 15:50:39 +02:00
Yorick Peterse f699b0d097 Lex XPath axes without a node test. 2014-07-20 07:57:45 +02:00
Yorick Peterse 488000438b Support for querying attributes using XPath. 2014-07-20 07:47:01 +02:00
Yorick Peterse d5569ead0b Use XML::Attribute for element attributes.
Instead of using a raw Hash Oga now uses the XML::Attribute class for storing
information about element attributes.

Attributes are stored as an Array of XML::Attribute instances. This allows the
attributes to be more easily modified. If they were stored as a Hash you'd not
only have to update the attributes themselves but also the Hash that contains
them.

While using an Array has a slight runtime cost in most cases the amount of
attributes is small enough that this doesn't really pose a problem. If webscale
performance is desired at some point in the future Oga could most likely cache
the lookup of an attribute. This however is something for the future.
2014-07-20 07:29:37 +02:00
Yorick Peterse ce86785da6 Added the XML::Attribute class.
This class will replace the use of raw Hash/String values for attributes in
upcoming commits.
2014-07-16 10:08:11 +02:00
Yorick Peterse d09ab26680 Support for the ancestor-or-self axis. 2014-07-15 09:41:12 +02:00
Yorick Peterse 0211f60826 Evaluation of XPath "ancestor" axes. 2014-07-14 09:43:30 +02:00
Yorick Peterse 03f40d4024 Ripped out internal state of XPath::Evaluator.
Instead of keeping track of an internal state in @stack and @context the various
processing methods now take the context as an extra argument and return the
nodes they produced. This makes it easier to recursively call certain methods, a
requirement for processing XPath axes (e.g. the "ancestor" axis).
2014-07-14 00:07:43 +02:00
Yorick Peterse 56982dd543 Fail early when processing (test) nodes.
If a certain segment yields no nodes the on_path() method should not process any
following segments.
2014-07-12 00:22:18 +02:00
Yorick Peterse d6aec6aa16 Compare nodes in on_path instead of indexes.
This saves some code and effectively does the same thing.
2014-07-12 00:20:19 +02:00
Yorick Peterse 5808ffa7a4 Updated XPath evaluator for the new AST.
This is still a bit of a hack. Then again, it already was a hack to begin with.
2014-07-12 00:18:32 +02:00
Yorick Peterse be4f4ad744 Trimmed the XPath AST even further.
The excessive use of (path) nodes has been dropped and all parsing conflicts
have been dealt with.
2014-07-12 00:13:25 +02:00
Yorick Peterse aeb4eba260 Trim the XPath AST.
The AST has been simplified by adjusting the way (path) nodes are nested.
Operators now also use `paths` instead of `expression` to allow for expressions
such as `/A or /B`. Sadly this introduces quite a bunch of conflicts in the
parser but we'll deal with those later if needed.
2014-07-10 20:19:23 +02:00
Yorick Peterse 8fbc582547 XPath evaluation for name/namespace wildcards. 2014-07-09 22:09:20 +02:00
Yorick Peterse ed45058983 Basic support for evaluating XPath wildcards. 2014-07-09 20:06:31 +02:00
Yorick Peterse 74130df40d XPath evaluation support for absolute paths. 2014-07-09 09:28:00 +02:00
Yorick Peterse a398af19fe Use a stack based XPath evaluator.
This evaluator is capable of processing very, very basic XPath expressions. It
does not yet know anything about axes, functions, operators, etc.
2014-07-08 23:57:21 +02:00
Yorick Peterse 9c661e1e60 Added XML::NodeSet#+ and XML::NodeSet#to_a 2014-07-08 23:25:09 +02:00
Yorick Peterse 266c66569e Removed internal state of XPath::Evaluator.
Come to think of it it might actually be easier to implement the evaluator as
an actual VM. That is, instead of directly running on the AST it runs on some
flavour of bytecode. Alternatively it runs directly on the AST but behaves more
like a (stack based) VM. This would most likely be easier than passing a cursor
to every node processing method.
2014-07-08 19:19:59 +02:00
Yorick Peterse 808e1e8c47 Initial, half-assed attempt at an XPath evaluator. 2014-07-07 19:41:09 +02:00
Yorick Peterse 8b381ac970 Added Node#remove.
This method can be used to remove individual nodes without first having to
retrieve the NodeSet they are stored in.
2014-07-04 10:26:41 +02:00
Yorick Peterse e334e50ca6 Added Node#previous_element and Node#next_element.
These methods can be used similar to #previous and #next expect that they only
return Element instances opposed to all Node instances.
2014-07-04 10:18:18 +02:00
Yorick Peterse 94965961ce Added XML::Element#inner_text.
This method can be used to retrieve the text of the given node only. In other
words, unlike Element#text it does not contain the text of any child nodes.
2014-07-04 09:50:47 +02:00
Yorick Peterse a15c19c4af Retrieving root nodes using Node#root_node.
This method uses a loop to traverse upwards the DOM tree in order to find the
root document/element. While this might have an impact on performance I don't
expect Oga itself to call this method very often. The benefit is that Node
instances don't require users to manually pass the top level document as an
argument.
2014-07-03 00:26:15 +02:00
Yorick Peterse e69fbc3ea7 Remove ownership when using NodeSet#delete. 2014-07-01 10:15:39 +02:00
Yorick Peterse 4e2b7f529d Cleared up docs a bit of XML::Node. 2014-07-01 10:03:04 +02:00
Yorick Peterse 30845e3d65 Reliably remove nodes from owned sets.
The combination of iterating over an array and removing values from it results
in not all elements being removed. For example:

    numbers = [10, 20, 30]

    numbers.each do |num|
      numbers.delete(num)
    end

    numbers # => [20]

As a result of this the NodeSet#remove method uses two iterations:

1. One iteration to retrieve all NodeSet instances to remove nodes from.
2. One iteration to actually remove the nodes.

For the second iteration we iterate over the node sets and then over the nodes.
This ensures that we always remove all nodes instead of leaving some behind.
2014-07-01 09:57:43 +02:00
Yorick Peterse 5073056831 Added require() for StringIO.
This is needed since the lexer checks if the input is a StringIO instance.
2014-07-01 09:37:52 +02:00
Yorick Peterse 8314d24435 First pass at using the NodeSet class.
The documentation still leaves a lot to be desired and so does the API. There
also appears to be a problem where NodeSet#remove doesn't properly remove all
nodes from a set. Outside of that we're making slow progress towards a proper
DOM API.
2014-06-30 23:03:48 +02:00
Yorick Peterse e71fe3d6fa Indexing of NodeSet instances.
Similar to arrays the NodeSet class now allows one to retrieve nodes for a
given index.
2014-06-29 23:59:27 +02:00
Yorick Peterse 253575dc37 Basic docs for the NodeSet class. 2014-06-29 21:34:18 +02:00
Yorick Peterse d2e74d8a0b Specs for the NodeSet class. 2014-06-26 19:52:03 +02:00
Yorick Peterse eb9d4fbccc Changed NodeSet to behave more like an Array. 2014-06-26 09:37:54 +02:00
Yorick Peterse a98f50b63b NodeSet#push should not take node ownership. 2014-06-26 09:37:30 +02:00
Yorick Peterse 15fa7a2068 Remove explicit index tracking of NodeSet.
Instead the various nodes can use NodeSet#index (aka Array#index) instead. This
has a slight performance overhead on very large (millions) of nodes but should
be fine in most other cases.
2014-06-25 09:41:58 +02:00
Yorick Peterse 884dbd9563 Rough sketch for a NodeSet class. 2014-06-24 19:06:45 +02:00
Yorick Peterse b8cc6b5031 XPath::Parser#parse returns a XPath::Node instance 2014-06-23 20:23:08 +02:00
Yorick Peterse 03be9f241c Docs on Racc operator precedence. 2014-06-23 10:36:42 +02:00
Yorick Peterse 96a7a40fdc Disable code coverage for JRuby specific code. 2014-06-23 09:42:14 +02:00
Yorick Peterse d2f15e37d0 Corrected XPath operator precedence.
The previous commit didn't fully change the operator precedence according to
the XPath 1.0 specification. Also thanks to @whitequark for clearing up a few
things about Racc's operator precedence system.
2014-06-23 00:30:42 +02:00
Yorick Peterse a440d3f003 Fixed XPath operator precedence.
Apparently using multiple `left` rules with T_AND and T_OR being separate
solves this problem. Riiiiight....
2014-06-23 00:15:43 +02:00
Yorick Peterse 6a2f4fa82d Parsing support for more XPath operators.
This still messes up some tests due to botched token precedence (by the looks
of it).
2014-06-22 21:27:53 +02:00
Yorick Peterse 514c342cab Lex wildcards as T_IDENT instead of T_STAR. 2014-06-20 20:37:34 +02:00
Yorick Peterse 45db337c76 Use Array#unshift for multiple xpath call args. 2014-06-17 20:12:25 +02:00
Yorick Peterse b3ffc28cc7 Removed shift/reduce conflict in the xpath parser. 2014-06-17 20:09:44 +02:00
Yorick Peterse 497f57ccd2 Basic parser setup for XPath function calls. 2014-06-17 19:57:17 +02:00
Yorick Peterse 894de7f909 Lex all XPath expressions in a single machine.
This allows literal values such as strings and numbers to be used as function
arguments.
2014-06-17 19:56:57 +02:00
Yorick Peterse bb7af98257 Updated used ASTs for all XPath parser specs. 2014-06-17 18:51:33 +02:00
Yorick Peterse 2298ef618b Reworked handling of relative vs absolute XPaths. 2014-06-16 20:19:39 +02:00
Yorick Peterse eba2d9954d Support for parsing basic XPath expressions. 2014-06-12 00:20:46 +02:00
Yorick Peterse 70f3b7fa92 Lex XPath operators using individual tokens.
Instead of lexing every operator as T_OP they now use individual tokens such as
T_EQ and T_LT.
2014-06-09 23:35:54 +02:00
Yorick Peterse 7244e28eec Corrected docs of the xpath lexer. 2014-06-04 19:33:46 +02:00
Yorick Peterse 1d2f9e6db6 Added T_STAR as an XPath parser token. 2014-06-02 09:27:25 +02:00
Yorick Peterse e11b9ed32c Basic XPath parser setup. 2014-06-01 23:02:28 +02:00
Yorick Peterse 54de2df0c7 Support for lexing XPath wildcard expressions.
To support this we need to require whitespace around the "*" operator. This is
not ideal but it will do for now.
2014-06-01 23:01:24 +02:00
Yorick Peterse 8dd8d7a519 Basic working XPath lexer.
This doesn't lex everything of the XPath specification just yet and needs more
tests.
2014-06-01 19:24:35 +02:00
Yorick Peterse a50b76a2d8 Cleaned up XPath lexer boilerplate a bit. 2014-05-29 19:25:49 +02:00
Yorick Peterse e0b07332d9 Boilerplate for the XPath lexer. 2014-05-29 19:25:49 +02:00
Yorick Peterse be3f8fb494 Removed the on_newline XML lexer callback. 2014-05-29 14:21:48 +02:00
Yorick Peterse ead5c71fee Cleaned up the XML parser grammar.
This resolves all shift/reduce and reduce/reduce conflicts that were previously
present.
2014-05-29 01:37:19 +02:00
Yorick Peterse 49780e2b04 Fix for useless XML parser rules.
Something tells me that using : and | in your syntax might not be the best
decision.
2014-05-28 21:36:06 +02:00
Yorick Peterse 28edc7726f Rewind IO input upon resetting the lexer. 2014-05-26 00:33:20 +02:00
Yorick Peterse 629dcd3fe6 Support for IO inputs in the lexer.
Using IO/StringIO objects one can parse large XML files without first having to
read the entire file into memory. This can potentially save a lot of memory at
the cost of a slightly slower runtime.

For IO like instances the lexer will consume the input line by line. If a
String is given it's consumed as a whole instead. A small side effect of
reading the input line by line is that text such as "foo\nbar" will be lexed as
two tokens instead of one.

Fixes #19.
2014-05-26 00:30:39 +02:00
Yorick Peterse 6b9d65923a Use a method for getting input in the XML lexer.
Instead of directly accessing the `data` instance variable the C/Java code now
uses the method `read_data`. This is part of one of the various steps required
to allow Oga to read data from IO like instances. It also means I can freely
change the name of the instance variable without also having to change the
C/Java code.
2014-05-21 00:27:23 +02:00
Yorick Peterse cd0f3380c4 Merge multiple CDATA tokens into a single token.
The tokens T_CDATA_START, T_TEXT and T_CDATA_END have been merged together into
T_CDATA.
2014-05-19 09:36:19 +02:00
Yorick Peterse a4fb5c1299 Merge multiple comment tokens into a single one.
The tokens T_COMMENT_START, T_TEXT and T_COMMENT_END have been merged into a
single token: T_COMMENT. This simplifies both the lexer and the parser.
2014-05-19 09:30:30 +02:00
Yorick Peterse c891dd88cb Removed useless code from the XML parser. 2014-05-18 23:30:26 +02:00
Yorick Peterse 81a81f0ab0 Don't create Arrays when not needed. 2014-05-16 17:05:42 +02:00
Yorick Peterse fd2f727183 Only set explicit ivars in the lexer. 2014-05-15 19:48:18 +02:00
Yorick Peterse 44bf1dd1ca Split up handling of element names/namespaces.
This is now split up on Ragel level, simplifying the corresponding Ruby code.
2014-05-15 10:22:05 +02:00
Yorick Peterse 723a273e4f Enforce symbols for element attributes.
This comes with a little bit of memory overhead but this should be minor in
most cases.
2014-05-15 01:04:26 +02:00
Yorick Peterse f4b9bbd4ac Removed lazy way of setting instance variables.
This process is quite a bit slower compared to setting instance variables
directly.
2014-05-15 00:43:13 +02:00
Yorick Peterse 19f04f98f7 Support for lexing/parsing inline doctypes. 2014-05-10 00:28:11 +02:00
Yorick Peterse fe74d60138 Manually bootstrap JRuby after all.
After discussing this with @headius I've decided to do this the manual way
anyway. Apparently the basic load service stuff is deprecated and not very
reliable.
2014-05-07 22:32:34 +02:00
Yorick Peterse b8efed5177 Renamed on_start_doctype to on_doctype_start. 2014-05-06 23:18:44 +02:00
Yorick Peterse 2053018d07 Slap JRuby so that it can load the .jar file. 2014-05-06 20:45:26 +02:00
Yorick Peterse 6e685378e0 Setup Ragel for JRuby and load things the hard way 2014-05-06 19:06:04 +02:00
Yorick Peterse ee756037e7 Removed unused YARD tag. 2014-05-05 09:45:10 +02:00
Yorick Peterse aeab885a7f Docs for the Ruby part of the XML lexer. 2014-05-05 09:44:35 +02:00
Yorick Peterse 2689d3f65a Initial setup using a C extension.
While I've tried to keep Oga pure Ruby for as long as possible the performance
of Ragel's Ruby output was not worth the trouble. For example, lexing 10MB of
XML would take 5 to 6 seconds at least. Nokogiri on the other hand can parse
that same XML into a DOM document in about 300 miliseconds. Such a big
performance difference is not acceptable.

To work around this the XML/HTML lexer will be implemented in C for
MRI/Rubinius and Java for JRuby. For now there's only a C extension as I
haven't read up yet on the JRuby API. The end goal is to provide some sort of
Ragel "template" that can be used to generate the corresponding C/Java
extension code. This would remove the need of duplicating the grammar and
associated code.

The native extension setup is a hybrid between native and Ruby. The raw Ragel
stuff happens in C/Java while the actual logic of actions happens in Ruby. This
adds a small amount of overhead but makes it much easier to maintain the lexer.
Even with this extra overhead the performance is much better than pure Ruby.
The 10MB of XML mentioned above is lexed in about 600 miliseconds. In other
words, it's 10 times faster.
2014-05-05 00:31:28 +02:00
Yorick Peterse baaa24a760 Indentation fix in the lexer. 2014-05-04 18:06:43 +02:00
Yorick Peterse f18e8893de Removed the buffering crap from the lexer. 2014-05-04 17:39:08 +02:00
Yorick Peterse 9dfdefee47 Removed XML::Lexer#buffering?
Instead of wrapping a predicate method around the ivar we'll just access it
directly. This reduces average lexing times in the big XML benchmark from 7,5
to ~7 seconds.
2014-05-01 22:59:56 +02:00
Yorick Peterse f607cf50dc Use local variables for Ragel.
Instead of using instance variables for ts, te, etc we'll use local variables.
Grand wizard overloard @whitequark suggested that this would be quite a bit
faster, which turns out to be true. For example, the big XML lexer benchmark
would, prior to this commit, complete in about 9 - 9,3 seconds. With this
commit that hovers around 8,5 seconds.
2014-05-01 13:00:29 +02:00
Yorick Peterse 83f6d5437e Contextual pull parsing.
This adds the ability to more easily act upon specific node types and nestings
when using the pull parsing API.

A basic example of this API looks like the following (only including relevant
code):

    parser.parse do |node|
      parser.on(:element, %w{people person}) do
        people << {:name => nil, :age => nil}
      end

      parser.on(:text, %w{people person name}) do
        people.last[:name] = node.text
      end

      parser.on(:text, %w{people person age}) do
        people.last[:age] = node.text.to_i
      end
    end

This fixes #6.
2014-04-29 23:05:49 +02:00
Yorick Peterse 1a413998a3 Track the current node in the pull parser.
The current node is tracked in the instance method `node`.
2014-04-29 21:21:05 +02:00
Yorick Peterse 45b0cdf811 Track element name nesting in the pull parser.
Tracking the names of nested elements makes it a lot easier to do contextual
pull parsing. Without this it's impossible to know what context the parser is
in at a given moment.

For memory reasons the parser currently only tracks the element names. In the
future it might perhaps also track extra information to make parsing easier.
2014-04-28 23:40:36 +02:00
Yorick Peterse 030a0068bd Basic pull parsing setup.
This parser extends the regular DOM parser but instead delegates certain nodes
to a block instead of building a DOM tree.

The API is a bit raw in its current form but I'll extend it and make it a bit
more user friendly in the following commits. In particular I want to make it
easier to figure out if a certain node is nested inside another node.
2014-04-28 17:22:17 +02:00
Yorick Peterse fd5bbbc9a2 Move element recursion handling into a method.
This makes it easier to disable later on in the streaming parser.
2014-04-28 10:25:05 +02:00
Yorick Peterse 785ec26fe7 Create Element instances before recursing. 2014-04-28 10:21:34 +02:00
Yorick Peterse 9939cf49eb Move parser callback code into dedicated methods. 2014-04-28 10:18:55 +02:00
Yorick Peterse 5d05aed6ec Corrected docs for XML::Parser. 2014-04-26 12:57:35 +02:00
Yorick Peterse f53fe4ed7c Reset the lexer when resetting the parser.
Also removed the unused @lines instance variable.
2014-04-25 00:15:24 +02:00
Yorick Peterse 83ff0e6656 Various small parser cleanups. 2014-04-25 00:07:53 +02:00
Yorick Peterse ecf6851711 Revert "Move linking of child nodes to a dedicated mixin."
This doesn't actually make things any easier. It also introduces a weirdly
named mixin.

This reverts commit 0968465f0c.
2014-04-24 21:16:31 +02:00
Yorick Peterse 0968465f0c Move linking of child nodes to a dedicated mixin. 2014-04-24 09:43:50 +02:00
Yorick Peterse 08d412da7e First shot at removing the AST layer.
The AST layer is being removed because it doesn't really serve a useful
purpose. In particular when creating a streaming parser the AST nodes would
only introduce extra overhead.

As a result of this the parser will instead emit a DOM tree directly instead of
first emitting an AST.
2014-04-21 23:05:39 +02:00
Yorick Peterse 9ee9ec14cb Lexer: only pop elements when needed. 2014-04-19 01:10:32 +02:00
Yorick Peterse 54e6650338 Don't use define_method in the lexer.
Profiling showed that calls to methods defined using `define_method` are
really, really slow. Before this commit the lexer would process 3000-4000
lines per second. With this commit that has been increased to around 10 000
lines per second.

Thanks to @headius for mentioning the (potential) overhead of define_method.
2014-04-17 19:08:26 +02:00
Yorick Peterse d9fa4b7c45 Lex input as a sequence of bytes.
Instead of lexing the input as a raw String or as a set of codepoints it's
treated as a sequence of bytes. This removes the need of String#[] (replaced by
String#byteslice) which in turn reduces the amount of memory needed and speeds
up the lexing time.

Thanks to @headius and @apeiros for suggesting this and rubber ducking along!
2014-04-17 17:45:05 +02:00