Remove docs for HTMLTokenizer and HTMLSanitizer

twm · twm · commit abf622432db7 · 2017-04-15T09:36:43.000-07:00
HTMLTokenizer is now a private API (I cannot find a public export).
HTMLSanitizer no longer exists as a tokenizer, and has been replaced
with a filter.
diff --git a/doc/movingparts.rst b/doc/movingparts.rst
@@ -169,41 +169,3 @@ the following way:
 * If all else fails, the default encoding will be used. This is usually
   `Windows-1252 <http://en.wikipedia.org/wiki/Windows-1252>`_, which is
   a common fallback used by Web browsers.
-
-
-Tokenizers
-----------
-
-The part of the parser responsible for translating a raw input stream
-into meaningful tokens is the tokenizer. Currently html5lib provides
-two.
-
-To set up a tokenizer, simply pass it when instantiating
-a :class:`~html5lib.html5parser.HTMLParser`:
-
-.. code-block:: python
-
-  import html5lib
-  from html5lib import sanitizer
-
-  p = html5lib.HTMLParser(tokenizer=sanitizer.HTMLSanitizer)
-  p.parse("<p>Surprise!<script>alert('Boo!');</script>")
-
-HTMLTokenizer
-~~~~~~~~~~~~~
-
-This is the default tokenizer, the heart of html5lib. The implementation
-can be found in `html5lib/tokenizer.py
-<https://github.com/html5lib/html5lib-python/blob/master/html5lib/tokenizer.py>`_.
-
-HTMLSanitizer
-~~~~~~~~~~~~~
-
-This is a tokenizer that removes unsafe markup and CSS styles from the
-input. Elements that are known to be safe are passed through and the
-rest is converted to visible text. The default configuration of the
-sanitizer follows the `WHATWG Sanitization Rules
-<http://wiki.whatwg.org/wiki/Sanitization_rules>`_.
-
-The implementation can be found in `html5lib/sanitizer.py
-<https://github.com/html5lib/html5lib-python/blob/master/html5lib/sanitizer.py>`_.
