Commit Graph

446 Commits

Author SHA1 Message Date
Yorick Peterse e942086f2d Fixed counting of newlines in XML declarations 2015-04-15 00:22:58 +02:00
Yorick Peterse b2ea20ba61 Lex processing instructions in chunks
Similar to comments (ea8b4aa92f) and CDATA
tags (8acc7fc743) processing instructions
are now lexed in separate chunks _with_ proper support for streaming
input.

Related issue: #93
2015-04-15 00:11:57 +02:00
Yorick Peterse ea8b4aa92f Lex comments in chunks
Similar to this being added for CDATA tags in
8acc7fc743 comments are now also lexed in
chunks.

Related issue: #93
2015-04-14 23:11:22 +02:00
Yorick Peterse 8acc7fc743 Lex CDATA tags in chunks
Instead of using a single token (T_CDATA) for a CDATA tag the lexer now
uses 3 tokens:

1. T_CDATA_START
2. T_CDATA_BODY
3. T_CDATA_END

The T_CDATA_BODY token can occur multiple times and is turned into a
single value in the XML parser. This is similar to the way strings are
lexed.

By changing the way CDATA tags are lexed Oga can now lex CDATA tags
containing newlines when using an IO as input. For example, this would
previously fail:

    Oga.parse_xml(StringIO.new("<![CDATA[\nfoo]]>"))

Because IO input reads input per line the input for the lexer would be
as following:

    "<![CDATA[\n"
    "foo]]>"

Related issues: #93
2015-04-14 22:45:55 +02:00
Yorick Peterse b42f9aaf32 Cache output of Element#available_namespaces
This cache is flushed whenever Element#register_namespace is called.
When this cache is flushed it's also recursively flushed for all child
elements. This makes calls to Element#register_namespace a bit more
expensive but in turn calls to Element#available_namespaces will be a
lot faster.
2015-04-12 20:22:33 +02:00
Yorick Peterse fa838154fc Flush Element#namespace cache
When setting a new namespace name using Element#namespace_name= the
cache used by Element#namespace is flushed properly.
2015-04-11 19:20:50 +02:00
Yorick Peterse b0359b37e5 Cache Node#html? and Node#root_node
The results of these methods is now cached until a Node is moved into
another NodeSet. This reduces the time spent in the
xpath/evaluator/big_xml_average_bench.rb benchmark from roughly 10
seconds to roughly 5 seconds per iteration.
2015-04-11 19:12:26 +02:00
Yorick Peterse 4bdc8a3fdc Don't convert entities in script/style elements
In HTML the text of a script/style tag should be left untouched, no
entities must be converted. Doing so would break Javascript such as the
following:

    foo&&bar;

Such code is often the result of minifiers doing their dirty business.
2015-04-08 14:32:09 +02:00
Yorick Peterse 6a1010c287 Fixed decoding entities in attribute values
This was broken by introducing the process of lazy decoding of XML/HTML
entities. The new setup works similar to how XML::Text#text decodes any
entities that may be present.

Fixes #91
2015-04-07 21:18:22 +02:00
Yorick Peterse ef7f50137a Added Oga::EntityDecoder
This module removes some of the code duplication needed to determine
what entity decoder to use.
2015-04-07 21:18:15 +02:00
Yorick Peterse 0800654c96 Support lexing or carriage returns
Fixes #89.
2015-04-03 00:46:37 +02:00
Yorick Peterse 3176459307 Ignore declared namespaces in HTML documents
The HTML spec states that any declared namespaces, including the default
namespace are to be ignored.

This fixes #85
2015-03-26 22:38:39 +01:00
Yorick Peterse 5adeae18d0 XPath queries match nodes in the default namespace
When querying an XML document that explicitly defines the default XML
namespace the XPath evaluator now correctly matches all nodes within
that namespace if no namespace prefix is given in the query. Previously
this would always return an empty set.
2015-03-26 01:13:55 +01:00
Yorick Peterse f175414917 Added XML::Element#default_namespace? 2015-03-26 01:10:20 +01:00
Yorick Peterse b6fcd326ef Added XML::Node#html? and XML::Node#xml?
The former has been moved over from XML::Text, the latter just inverts
html?.
2015-03-26 01:02:32 +01:00
Yorick Peterse 4ad502958d Added XML::Attribute#==
Overwriting this method makes it easier to check if a given namespace
equals the default XML (and soon HTML) namespace.
2015-03-26 00:53:16 +01:00
Yorick Peterse f2d69af33b Distinguish default attribute/element namespaces
The previous commit messed this up because I wasn't fully awake.
2015-03-26 00:43:50 +01:00
Yorick Peterse 68ada997a8 Moved default namespace into Oga::XML
The default namespace is now located at Oga::XML::DEFAULT_NAMESPACE
instead of Oga::XML::Attribute::DEFAULT_NAMESPACE.
2015-03-26 00:35:28 +01:00
Yorick Peterse 66fa9f62ef Added LRU#maximum=/maximum
This allows one to change the maximum amount of keys stored in the
XPath/CSS caches, for example:

    Oga::XPath::Parser::CACHE.maximum = 2056
2015-03-23 00:26:48 +01:00
Yorick Peterse 2c4e490614 Added CSS/XPath Parser.parse_with_cache
This method parses and caches ASTs using Oga::LRU. Currently the default
of 1024 keys is used.

See #71 for more information.
2015-03-23 00:22:59 +01:00
Yorick Peterse 67d7d9af88 Added thread-safe LRU class
This class will be used for storing parser XPath/CSS ASTs.

See #71 for more information.
2015-03-23 00:21:52 +01:00
Yorick Peterse 45d84d31da Renamed rspec helper files 2015-03-22 22:50:03 +01:00
Yorick Peterse 70e4942d3e CSS parser spec for "+ b" 2015-03-21 01:23:00 +01:00
Yorick Peterse 6039e1dbeb XPath parsing spec for axes with predicates 2015-03-21 01:23:00 +01:00
Yorick Peterse 62fa2a9cc5 Spec for XPath functions inside predicates. 2015-03-21 01:23:00 +01:00
Yorick Peterse 194d981996 XPath specs for paths with multiple members. 2015-03-21 01:22:59 +01:00
Yorick Peterse fdcd712ffe Don't use Array#uniq in NodeSet#initialize.
Removing this makes the process of parsing larger XML documents a bit faster.
The downside is that NodeSet#initialize will no longer filter out duplicate
nodes, though this is not something Oga itself relies upon.

Methods such as NodeSet#push still do ignore elements already present.
2015-03-21 01:22:59 +01:00
Yorick Peterse f83c03aaec Fixed typo in NodeSet spec. 2015-03-21 01:22:59 +01:00
Yorick Peterse 006ef4d51a Port over most of the old XML error handling.
Some messages are a bit different due to ruby-ll's error handling, other than
that it's largely the same stuff as before.
2015-03-21 01:22:59 +01:00
Yorick Peterse d8b9725b82 Fixed SAX parsing of XML attributes.
This was utterly broken, mainly due to me overlooking it. There are now 2 new
callbacks to handle this properly:

* on_attribute: to handle a single attribute/value pair
* on_attributes: to handle a collection of attributes (as returned by
  on_attribute)

By default on_attribut returns a Hash, on_attributes in turn merges all
attribute hashes into a single one. This ensures that on_element _actually_
receives the attributes as a Hash, instead of an Array with random
nil/XML::Attribute values.
2015-03-21 01:22:59 +01:00
Yorick Peterse 605d565104 Use sax_parse_html for HTML documents.
I suspect the only reason this test ever passed due to Racc's error handling.
Either way this was using the wrong method.
2015-03-21 01:22:59 +01:00
Yorick Peterse 2ec91f130f Lazy decoding of XML/HTML entities.
Instead of decoding entities in the lexer we'll do this whenever XML::Text#text
is called. This removes the overhead from the parsing phase and ensures the
process is only triggered when actually needed. Note that calling #to_xml and/or
the #inspect methods on a Text (or parent) instance will also trigger the entity
conversion process.

The new entity decoding API supports both regular entities (e.g. &amp;) as well
as codepoint based entities (both regular and hexadecimal codepoints).

To allow safe read-only access to Text instances from multiple threads a mutex
is used. This mutex ensures that only 1 thread can trigger the conversion
process.

Fixes #68
2015-03-05 23:00:43 +01:00
Yorick Peterse 3b2055a30b Refactored handling of literal HTML elements.
This ensures newlines can appear in <style> / <script> tags when using IOs as
input.
2015-03-04 11:44:31 +01:00
Yorick Peterse 78e40b55c0 Handle parsing of HTML <style> tags.
This basically re-applies the technique used for HTML <script> tags. With this
extra addition I decided to rename/normalize a few things so it's easier to add
any extra tags in the future. One downside of this setup is that the following
will not be parsed by Oga:

    <style>
        </script>
    </style>

The same applies to script tags containing a literal </style> tag. Since this
particular case is rather unlikely to occur I'm OK with not supporting it as it
_does_ simplify the lexer quite a bit.

Fixes #80
2015-03-03 16:28:05 +01:00
Yorick Peterse 142b467277 Set parent of nodes set using Element#inner_text=
This ensures that any text nodes created using Element#inner_text= have their
parent node set correctly.
2015-03-03 13:13:05 +01:00
Yorick Peterse 874d7124af Don't convert <script> text to XML entities.
Fixes #79.
2015-03-02 17:32:19 +01:00
Yorick Peterse 9a586363e9 Added XML::Document#html? 2015-03-02 16:39:40 +01:00
Yorick Peterse 351b5ac004 Added spec for lexing inline HTML script tags.
Related issue: #70
2015-03-02 16:20:06 +01:00
Yorick Peterse 47a3c5e7f8 Use describe/it instead of context/example.
This keeps things consistent with the general testing guidelines in the Ruby
community. This in turn should hopefully make my life easier as I don't have to
tell people to use this rather odd stlye I was using before.
2015-01-08 23:01:53 +01:00
Yorick Peterse 746c8052dd Remove all nodes when calling Element#inner_text=
This fixes #64.
2014-12-14 23:32:43 +01:00
Dmitry Krasnoukhov 26baf89440 Add missing entities to the decode/encode lists 2014-11-21 01:53:11 +02:00
Yorick Peterse cbb2815146 Support for inline doctype rules plus newlines.
This adds support for lexing/parsing XML documents that use an IO as input _and_
contain doctype rules with newlines in them.

This fixes #63.
2014-11-18 20:02:55 +01:00
Yorick Peterse ad4f650c5d Fixed XML entity encoding/decoding ordering.
Thanks to @krasnoukhov for providing the initial patch, which this commit is
largely based on.

This fixes #49.
2014-11-17 22:39:43 +01:00
Yorick Peterse cd86d5d294 Allow removal of element attributes. 2014-11-17 09:00:40 +01:00
Yorick Peterse 804646cc5e Don't modify raw namespaces.
When calling Element#available_namespaces the list of namespaces returned by
Element#namespaces must not be modified.
2014-11-17 00:01:16 +01:00
Yorick Peterse 57adabc068 Ensure SAX after_element receives meaningful args
This changes the behaviour of after_element when parsing documents using the SAX
parsing API. Previously it would always receive a nil argument, which is kinda
pointless. This commit changes that by making sure it receives a namespace name
(if any) and the element name.

This fixes #54.
2014-11-16 23:32:32 +01:00
Yorick Peterse 448ff56e38 Fixed CSS eval specs for nth-(first|last)-of-type. 2014-11-15 18:27:26 +01:00
Yorick Peterse b464815577 Fixed AST generation for nth-(first|last)-of-type. 2014-11-15 18:27:15 +01:00
Yorick Peterse 9eead81a7c Fixed AST for :only-of-type 2014-11-15 18:08:26 +01:00
Yorick Peterse 1c301d40e2 Properly fixed AST for first-of-type/last-of-type
This requires keeping track of the current element being processed. This in turn
allows the usage of count() + preceding-sibling/following-sibling.
2014-11-15 17:58:56 +01:00