defusedxml
XML bomb protection for Python stdlib modules
Downloads: 0 (30 days)
Description
===================================================
defusedxml -- defusing XML bombs and other exploits
===================================================
.. image:: https://img.shields.io/pypi/v/defusedxml.svg
:target: https://pypi.org/project/defusedxml/
:alt: Latest Version
.. image:: https://img.shields.io/pypi/pyversions/defusedxml.svg
:target: https://pypi.org/project/defusedxml/
:alt: Supported Python versions
.. image:: https://travis-ci.org/tiran/defusedxml.svg?branch=master
:target: https://travis-ci.org/tiran/defusedxml
:alt: Travis CI
.. image:: https://codecov.io/github/tiran/defusedxml/coverage.svg?branch=master
:target: https://codecov.io/github/tiran/defusedxml?branch=master
:alt: codecov
.. image:: https://img.shields.io/pypi/dm/defusedxml.svg
:target: https://pypistats.org/packages/defusedxml
:alt: PyPI downloads
.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
:target: https://github.com/psf/black
:alt: Code style: black
..
"It's just XML, what could probably go wrong?"
Christian Heimes <christian@python.org>
Synopsis
========
The results of an attack on a vulnerable XML library can be fairly dramatic.
With just a few hundred **Bytes** of XML data an attacker can occupy several
**Gigabytes** of memory within **seconds**. An attacker can also keep
CPUs busy for a long time with a small to medium size request. Under some
circumstances it is even possible to access local files on your
server, to circumvent a firewall, or to abuse services to rebound attacks to
third parties.
The attacks use and abuse less common features of XML and its parsers. The
majority of developers are unacquainted with features such as processing
instructions and entity expansions that XML inherited from SGML. At best
they know about ``<!DOCTYPE>`` from experience with HTML but they are not
aware that a document type definition (DTD) can generate an HTTP request
or load a file from the file system.
None of the issues is new. They have been known for a long time. Billion
laughs was first reported in 2003. Nevertheless some XML libraries and
applications are still vulnerable and even heavy users of XML are
surprised by these features. It's hard to say whom to blame for the
situation. It's too short sighted to shift all blame on XML parsers and
XML libraries for using insecure default settings. After all they
properly implement XML specifications. Application developers must not rely
that a library is always configured for security and potential harmful data
by default.
.. contents:: Table of Contents
:depth: 2
Attack vectors
==============
billion laughs / exponential entity expansion
---------------------------------------------
The `Billion Laughs`_ attack -- also known as exponential entity expansion --
uses multiple levels of nested entities. The original example uses 9 levels
of 10 expansions in each level to expand the string ``lol`` to a string of
3 * 10 :sup:`9` bytes, hence the name "billion laughs". The resulting string
occupies 3 GB (2.79 GiB) of memory; intermediate strings require additional
memory. Because most parsers don't cache the intermediate step for every
expansion it is repeated over and over again. It increases the CPU load even
more.
An XML document of just a few hundred bytes can disrupt all services on a
machine within seconds.
Example XML::
<!DOCTYPE xmlbomb [
<!ENTITY a "1234567890" >
<!ENTITY b "&a;&a;&a;&a;&a;&a;&a;&a;">
<!ENTITY c "&b;&b;&b;&b;&b;&b;&b;&b;">
<!ENTITY d "&c;&c;&c;&c;&c;&c;&c;&c;">
]>
<bomb>&d;</bomb>
quadratic blowup entity expansion
---------------------------------
A quadratic blowup attack is similar to a `Billion Laughs`_ attack; it abuses
entity expansion, too. Instead of nested entities it repeats one large entity
with a couple of thousand chars over and over again. The attack isn't as
efficient as the exponential case but it avoids triggering countermeasures of
parsers against heavily nested entities. Some parsers limit the depth and
breadth of a single entity but not the total amount of expanded text
throughout an entire XML document.
A medium-sized XML document with a couple of hundred kilobytes can require a
couple of hundred MB to several GB of memory. When the attack is combined
with some level of nested expansion an attacker is able to achieve a higher
ratio of success.
::
<!DOCTYPE bomb [
<!ENTITY a "xxxxxxx... a couple of ten thousand chars">
]>
<bomb>&a;&a;&a;... repeat</bomb>
external entity expansion (remote)
----------------------------------
Entity declarations can contain more than just text for replacement. They can
also point to external resources by public identifiers or system identifiers.
System identifiers are standard URIs. When the URI is a URL (e.g. a
``http://`` locator) some parsers download the resource from the remote
location and embed them into the XML document verbatim.
Simple example of a parsed external entity::
<!DOCTYPE external [
<!ENTITY ee SYSTEM "http://www.python.org/some.xml">
]>
<root>ⅇ</root>
The case of parsed external entities works only for valid XML content. The
XML standard also supports unparsed external entities with a
``NData declaration``.
External entity expansion opens the door to plenty of exploits. An attacker
can abuse a vulnerable XML library and application to rebound and forward
network requests with the IP address of the server. It highly depends
on the parser and the application what kind of exploit is possible. For
example:
* An attacker can circumvent firewalls and gain access to restricted
resources as all the requests are made from an internal and trustworthy
IP address, not from the outside.
* An attacker can abuse a service to attack, spy on or DoS your servers but
also third party services. The attack is disguised with the IP address of
the server and the attacker is able to utilize the high bandwidth of a big
machine.
* An attacker can exhaust additional resources on the machine, e.g. with
requests to a service that doesn't respond or responds with very large
files.
* An attacker may gain knowledge, when, how often and from which IP address
an XML document is accessed.
* An attacker could send mail from inside your network if the URL handler
supports ``smtp://`` URIs.
external entity expansion (local file)
--------------------------------------
External entities with references to local files are a sub-case of external
entity expansion. It's listed as an extra attack because it deserves extra
attention. Some XML libraries such as lxml disable network access by default
but still allow entity expansion with local file access by default. Local
files are either referenced with a ``file://`` URL or by a file path (either
relative or absolute).
An attacker may be able to access and download all files that can be read by
the application process. This may include critical configuration files, too.
::
<!DOCTYPE external [
<!ENTITY ee SYSTEM "file:///PATH/TO/simple.xml">
]>
<root>ⅇ</root>
DTD retrieval
-------------
This case is similar to external entity expansion, too. Some XML libraries
like Python's xml.dom.pulldom retrieve document type definitions from remote
or local locations. Several attack scenarios from the external entity case
apply to this issue as well.
::
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head/>
<body>text</body>
</html>
Python XML Libraries
====================
.. csv-table:: vulnerabilities and features
:header: "kind", "sax", "etree", "minidom", "pulldom", "xmlrpc", "lxml", "genshi"
:widths: 24, 7, 8, 8, 7, 8, 8, 8
:stub-columns: 0
"billion laughs", "**True**", "**True**", "**True**", "**True**", "**True**", "False (1)", "False (5)"
"quadratic blowup", "**True**", "**True**", "**True**", "**True**", "**True**", "**True**", "False (5)"
"external entity expansion (remote)", "**True**", "False (3)", "False (4)", "**True**", "false", "False (1)", "False (5)"
"external entity expansion (local file)", "**True**", "False (3)", "False (4)", "**True**", "false", "**True**", "False (5)"
"DTD retrieval", "**True**", "False", "False", "**True**", "false", "False (1)", "False"
"gzip bomb", "False", "False", "False", "False", "**True**", "**partly** (2)", "False"
"xpath support (7)", "False", "False", "False", "False", "False", "**True**", "False"
"xsl(t) support (7)", "False", "False", "False", "False", "False", "**True**", "False"
"xinclude support (7)", "False", "**True** (6)", "False", "False", "False", "**True** (6)", "**True**"
"C library", "expat", "expat", "expat", "expat", "expat", "libxml2", "expat"
1. Lxml is protected against billion laughs attacks and doesn't do network
lookups by default.
2. libxml2 and lxml are not directly vulnerable to gzip decompression bombs
but they don't protect you against them either.
3. xml.etree doesn't expand entities and raises a ParserError when an entity
occurs.
4. minidom doesn't expand entities and simply returns the unexpanded entity
verbatim.
5. genshi.input of genshi 0.6 doesn't support entity expansion and raises a
ParserError when an entity occurs.
6. Library has (limited) XInclude support but requires an additional step to
process inclusion.
7. These are features but they may introduce exploitable holes, see
`Other things to consider`_
Settings in standard library
----------------------------
xml.sax.handler Features
........................
feature_external_ges (http://xml.org/sax/features/external-general-entities)
disables external entity expansion
feature_external_pes (http://xml.org/sax/features/external-parameter-entities)
the option is ignored and doesn't modify any functionality
DOM