Fix many places where malloc failures aren't reported.
Make some API function return an error code. Changing the return type
from void to int is technically an ABI break but should be safe on most
platforms.
- xmlNodeSetContent
- xmlNodeSetContentLen
- xmlNodeAddContent
- xmlNodeAddContentLen
- xmlNodeSetBase
Introduce new API functions that return a separate error code if a
memory allocation fails.
- xmlNodeGetAttrValue
- xmlNodeGetBaseSafe
- xmlGetNsListSafe
Introduce private functions xmlTreeEnsureXMLDecl and xmlSplitQName4.
Don't report low-level errors to the global error handler.
Fix tree
Introduce xmlGetNsListSafe
Fix tree
Introduce a new API function xmlAddIDSafe that returns a separate error
code if a memory allocation fails.
Store a pointer to the ID struct in xmlAttr so attributes can be
freed without allocating memory. It's impossible to report malloc
failures in deallocation code.
Introduce XML_INPUT_HAS_ENCODING flag for xmlParserInput which is set
when xmlSwitchEncoding is called. The parser can use the flag to
reliably detect whether an encoding was already set via user override,
BOM or other auto-detection. In this case, the encoding declaration
won't be used to switch the encoding.
Before, an inscrutable mix of ctxt->charset, ctxt->input->encoding
and ctxt->input->buf->encoder was used.
Introduce private helper functions to switch encodings used by both the
XML and HTML parser:
- xmlDetectEncoding which skips over the BOM, allowing to remove the
BOM checks from other encoding functions.
- xmlSetDeclaredEncoding, replacing htmlCheckEncodingDirect, which warns
about encoding mismatches.
If users override the encoding, store the declared instead of the actual
encoding in xmlDoc. In this case, the actual encoding is known and the
raw value from the doc is more useful.
Also use the input flags to store the ISO-8859-1 fallback state.
Restrict the fallback to cases where no encoding was specified. (The
fallback is only useful in recovery mode and these days broken UTF-8 is
probably more likely than ISO-8859-1, so it might eventually be removed
completely.)
The 'charset' member of xmlParserCtxt is now unused. The 'encoding'
member of xmlParserInput is now unused.
The 'standalone' member of xmlParserInput is renamed to 'flags'.
A new parser state XML_PARSER_XML_DECL is added for the push parser.
This code has been broken and deprecated since version 2.6.0, released
in 2003. Because of a bug in commit 961b535c, DOCBparser.c was never
compiled since 2012. I couldn't find a Debian package using any of its
symbols, so it seems safe to remove this module.
One of the operation on the reader could resolve entities
leading to the classic expansion issue. Make sure the
buffer used for xmlreader operation is bounded.
Introduce a new allocation type for the buffers for this effect.
For https://bugzilla.gnome.org/show_bug.cgi?id=689483
It seems there are functions that do use the const qualifier for some of the
arguments, but it seems that there are a lot of functions that don't use it and
probably should.
So I created a patch against 2.9.0 that makes as much as possible const in
tree.h, and changed other files as needed.
There were a lot of cases like "const xmlNodePtr node". This doesn't actually
do anything, there the *pointer* is constant not the object it points to. So I
changed those to "const xmlNode *node".
I also removed some consts, mostly in the Copy functions, because those
functions can actually modify the doc or node they copy from
Building 2.9.0 on MSVC7.1 was failing
This is because HAVE_CONFIG_H is not #defined
The patch addresses the above, adds testrecurse.exe and the
standard "make check" suite of tests to the MSVC makefile, and also
fixes the following (MSVC7.1) warnings:
buf.c(674) : warning C4028: formal parameter 1 different from
declaration
libxml2\timsort.h(71) : warning C4028: formal parameter 1 different from
declaration
As suggested by Andrew W. Nosenko:
Proposal: expose the new xmlBufShrink() to the "public" API for
compatibility with xmlBufUse().
Reason: the following scenario:
1. Read something into xmlParserInputBuffer (e.g. using
xmlParserInputBufferRead())
2. Extract content through xmlBufContent()
3. Extract content length through xmlBufUse(). Result have type
'size_t'.
4. Use this content
5. Now, you need to shrink the buffer. How to do it? Doing that
through legacy xmlBufferShrink() is unsafe because it uses 'unsigned
int' and the whole point of introducing the new API was handling the
cases, when 'unsigned int' is not enough. Therefore, need to use the
new xmlBufShrink(). But it is "private".
Therefore, I propose to expose the new xmlBufShrink() in the same way,
as xmlBufContent() and xmlBufUse() are exposed.
Various cleanups
* configure.in: force regeneration of APIs in my environment
* buf.c buf.h enc.h encoding.c include/libxml/tree.h
include/libxml/xmlerror.h save.h tree.c: various comment cleanups
pointed by apibuild
* doc/apibuild.py: added the 3 new internal headers in the excludes
* doc/libxml2-api.xml doc/libxml2-refs.xml: regenerated the API
* doc/symbols.xml: listing new entry points for 2.9.0
* doc/devhelp/*: regenerated
Now tree.h exports LIBXML2_NEW_BUFFER macro indicating that the
API uses the new buffers, important to keep code working with
both versions.
* tree.h buf.h: also export xmlBufContent(), xmlBufEnd(), and xmlBufUse()
to help port the old code
* buf.c: make sure the compatibility counters are updated on
buffer usage, to keep proper working of application compiled
against the old structures, but take care of int overflow
* include/libxml/tree.h: adds xmlBufGetNodeContent and xmlBufNodeDump
as xmlBuf based equivalents of xmlNodeGetContent and xmlNodeDump
* tree.c: implements one new routine and converts xmlNodeBufGetContent
to use the xmlBuf equivalent. It should behave better as a result
in case of data larger than 2GB.
This also add converter functions between xmlBuf and xmlBuffer
* buf.c buf.h: the old xmlBuffer routines but modified for size_t
and using xmlBuf instead of xmlBuffer
* Makefile.am: add the 2 new files
* include/libxml/xmlerror.h: add an entry for the new module
* include/libxml/tree.h: expose the xmlBufPtr type but not the
structure which stay private
On Fri, May 11, 2012 at 9:10 AM, Daniel Veillard <veillard@redhat.com> wrote:
> Hi Conrad,
>
> that's interesting ! I was initially afraid of a sudden explosion of
> memory allocations for building a tree since by default buffers tend to
> "waste" memory by using doubling allocations, but that's not the case.
> xmllint --noout doc/libxml2-api.xml
> when compiled with memory debug produce
>
> paphio:~/XML -> cat .memdump
> MEMORY ALLOCATED : 0, MAX was 12756699
>
> and without your patch 12755657, i.e. the increase is minimal.
Heh, I thought that too. Actually you're looking at the result with XML_ALLOC_EXACT! This
is because EXACT adds 10bytes "spare" on each alloc, and that interestingly wastes about the
same amount of space as XML_ALLOC_DOUBLEIT on this example (see below).
So it turns out that the default realloc() on my system actually handles this case really
well — and I guess that all the time in xmlRealloc() was actually in xmlStrlen, not the
underlying realloc() after all (sorry for misleading you). If you replace the realloc()
with a bad one (like valgrind's), then the performance degrades severely.
This patch implements a HYBRID allocator which has the behaviour you describe (it's
like EXACT to start with, though without the spare 10 bytes; and switches to DOUBLEIT
after 4kb) — that gets the memory back down to 12755657, with no noticeable impact on the
performance of the synthetic pathological example under valgrind.
In summary:
max_memory on ./xmllint --noout doc/libxml2-api.xml,
valgrind time on https://gist.github.com/2656940
max_memory valgrind time
before | 12755657 | 29:18.2
EXACT | 12756699 | 2:58.6 <-- this is the state after the first patch.
DOUBLEIT | 12756727 | 0:02.7
HYBRID | 12755754 | 0:02.7 <-- this is the state with both patches.
>
> There is also the cost of creating the buffers all the time.
> I need to read the code and check but I may be interested in an hybrid
> approach where we switch to buffer only when the text node starts to
> become too big (4k would remove nearly all usuall types of "document"
> usage, i.e. not blocks of data)
I tried to avoid too much buffer creation by introducing the xmlBufferDetach function,
which allows re-using one buffer to construct many strings. It's maybe a bit of a "hack"
in API terms though I thought the gains would be worth it.
Conrad
------8<------
To keep memory usage tight in normal conditions it's desirable to only
allocate as much space as is needed. Unfortunately this can lead to
problems when constructing a long string out of small chunks, because
every chunk you add will need to resize the buffer.
To fix this XML_ALLOC_HYBRID will switch (when the buffer is 4kb big)
from using exact allocations to doubling buffer size every time it is
full. This limits the number of buffer resizes to O(log n) (down from
O(n)), and thus greatly increases the performance of constructing very
large strings in this manner.
Hi Veillard and all,
Firstly, thanks for libxml: it's awesome!
I noticed recently that libxml was taking a surprisingly long time to perform some
operations (many minutes instead of milliseconds), and so I did some digging. It turns out
that the problem was caused by the realloc()ing done in xmlNodeAddContentLen() which can
be called many (many) times when assigning some content into a node.
For background, I'm dealing with XML that contains emails, these can have large
attachments (~6MB) which are base-64 encoded, line-wrapped at 78 chars, and each line ends
with . This means that xmlNodeAddContentLen() is being called about 200,000 times,
and so there are 200,000 reallocs of a 6MB string, which takes a while... (I put a synthetic
example of this at https://gist.github.com/2656940)
The attached patch works around that problem by using the existing buffer API to merge the
strings together before even creating the text node, this keeps the number of realloc()s
at a managable level.
I'd love feedback on the patch, and am happy to fix problems with it, or explore other
solutions if you think that this is barking up the wrong tree :).
Thanks,
Conrad
P.S. Should I create a bug for this too?
------8<------
Before this change xmlStringGetNodeList would perform a realloc() of the
entire new content for every XML entity in the assigned text in order to
merge together adjacent text nodes. This had the effect of making
xmlSetNodeContent O(n^2), which led to unexpectedly bad performance on
inputs that contained a large number of XML entities.
After this change the memory management is done by the buffer API,
avoiding the need to continually re-measure and realloc() the string.
For my test data (6MB of 80 character lines, each ending with )
this takes the time to xmlSetNodeContent from about 500 seconds to
around 50ms. I have not profiled smaller cases, though I tried to
minimize the performance impact of my change by avoiding unnecessary
string copying.
Signed-off-by: Conrad Irwin <conrad.irwin@gmail.com>
* include/libxml/tree.h tree.c python/generator.py: adds
element traversal support
* valid.c: avoid a warning
* doc/*: regenerated
daniel
svn path=/trunk/; revision=3804
* include/libxml/tree.h tree.c: make a new kind of buffer where
shrinking and adding in head can avoid reallocation or full
buffer memmoves
* encoding.c xmlIO.c: use the new kind of buffers for output
buffers
Daniel
svn path=/trunk/; revision=3787
* runxmlconf.c: more progresses against the official regression tests
* runsuite.c: small cleanup for non-leak reports
* include/libxml/tree.h: parsing flags and other properties are
now added to the document node, this is generally useful and
allow to make Name and NmToken validations based on the parser
flags, more specifically the 5th edition of XML or not
* HTMLparser.c tree.c: small side effects for the previous changes
* parser.c SAX2.c valid.c: the bulk of teh changes are here,
the parser and validation behaviour can be affected, parsing
flags need to be copied, lot of changes. Also fixing various
validation problems in the regression tests.
Daniel
svn path=/trunk/; revision=3762
* xmllint.c: added --html --memory to test htmlReadMemory to
test #321632
* HTMLparser.c: added various initialization calls which may help
#321632 but not conclusive
* testapi.c tree.c include/libxml/tree.h: fixed compilation with
--with-minimum --with-sax1 and --with-minimum --with-schemas
fixing #326442
Daniel
* tree.c include/libxml/tree.h: Fixed a bug in
xmlDOMWrapAdoptNode(); the tree traversal stopped if the
very first given node had an attribute node :-( This was due
to a missed check in the traversal mechanism.
Expanded the xmlDOMWrapCtxt: it now holds the namespace map
used in xmlDOMWrapAdoptNode() and xmlDOMWrapCloneNode() for
reusal; so the map-items don't need to be created for every
cloning/adoption. Added a callback function to it for
retrieval of xmlNsPtr to be set on node->ns; this is needed
for my custom handling of ns-references in my DOM wrapper.
Substituted code which created the XML namespace decl on
the doc for a call to xmlTreeEnsureXMLDecl(). Removed
those nastly "warnigns" from the docs of the clone/adopt
functions; they work fine on my side.
* valid.c: fixed an uninitialized variable
* xmlregexp.c include/libxml/xmlregexp.h: extended the API to
add the parser, serializer and some debugging
* include/libxml/xmlversion.h.in: made the new support compiled
by default if Schemas is included
* testRegexp.c: cleanup and integration of the first part of the
new code with a special switch
* xmllint.c: show up Expr in --version if compiled in
* include/libxml/tree.h: moved the xmlBuffer definition up
Daniel
* xstc/Makefile.am README README.tests Makefile.tests Makefile.am:
preparing to make testsuite releases along with code source releases
* gentest.py testapi.c: fixed a couple of problem introduced by
the new Schemas support for Readers
* xpath.c: fixed the XPath attribute:: bug #309580, #309864 in a crude
but simple way.
* xmlschemas.c include/libxml/tree.h: fixed a couple of problems
raised by the doc builder.
* doc/*: made rebuild
Daniel
* tree.c include/libxml/tree.h: Added
xmlDOMWrapReconcileNamespaces(), xmlDOMWrapAdoptNode() and
xmlDOMWrapRemoveNode() to the API. These are functions intended
to be used with DOM-wrappers.
Synchronized the header files with the library code in order
to assure that all the various conditionals (LIBXML_xxxx_ENABLED)
were the same in both. Modified the API database content to more
accurately reflect the conditionals. Enhanced the generation
of that database. Although there was no substantial change to
any of the library code's logic, a large number of files were
modified to achieve the above, and the configuration script
was enhanced to do some automatic enabling of features (e.g.
--with-xinclude forces --with-xpath). Additionally, all the format
errors discovered by apibuild.py were corrected.
* configure.in: enhanced cross-checking of options
* doc/apibuild.py, doc/elfgcchack.xsl, doc/libxml2-refs.xml,
doc/libxml2-api.xml, gentest.py: changed the usage of the
<cond> element in module descriptions
* elfgcchack.h, testapi.c: regenerated with proper conditionals
* HTMLparser.c, SAX.c, globals.c, tree.c, xmlschemas.c, xpath.c,
testSAX.c: cleaned up conditionals
* include/libxml/[SAX.h, SAX2.h, debugXML.h, encoding.h, entities.h,
hash.h, parser.h, parserInternals.h, schemasInternals.h, tree.h,
valid.h, xlink.h, xmlIO.h, xmlautomata.h, xmlreader.h, xpath.h]:
synchronized the conditionals with the corresponding module code
* doc/examples/tree2.c, doc/examples/xpath1.c, doc/examples/xpath2.c:
added additional conditions required for compilation
* doc/*.html, doc/html/*.html: rebuilt the docs
* debugXML.c include/libxml/xmlerror.h: added checking for names
values and dictionnaries generates a tons of errors
* SAX2.ccatalog.c parser.c relaxng.c tree.c xinclude.c xmlwriter.c
include/libxml/tree.h: fixing the errors in the regression tests
Daniel
* SAX2.c, encoding.c, error.c, parser.c, tree.c, uri.c, xmlIO.c,
xmlreader.c, include/libxml/tree.h: many further little changes
for OOM problems. Now seems to be getting closer to "ok".
* testOOM.c: added code to intercept more errors, found more
problems with library. Changed method of flagging / counting
errors intercepted.
* tree.c, include/libxml/tree.h: moved serialization of
attribute text data (xmlSerializeContent) into a separate
routine (xmlSerializeTxtContent) so it can be used by xmlwriter.c
* xmlwriter.c: changed handling of attribute string to use the
routine above (fixed bug 131548)
