Update documentation for modular typeshed

JukkaL · JukkaL · commit 959b49927cdb · 2020-06-30T15:46:03.000+01:00
diff --git a/docs/source/command_line.rst b/docs/source/command_line.rst
@@ -802,6 +802,26 @@ format into the specified directory.
 Miscellaneous
 *************
 
+.. option:: --install-types
+
+    This flag causes mypy to install known missing stub packages for
+    third-party libraries using pip.  It will display the pip command
+    line to run, and expects a confirmation before installing
+    anything.
+
+    If you use this option without providing any files or modules to
+    type check, mypy will install stub packages suggested during the
+    previous mypy run. If there are files or modules to type check,
+    mypy first type checks those, and proposes to install missing
+    stubs at the end of the run, but only if any missing modules were
+    detected.
+
+    .. note::
+
+        This is new in mypy 0.800. Previous mypy versions included a
+        selection of third-party packages, instead of having them
+        installed separately.
+
 .. option:: --junit-xml JUNIT_XML
 
     Causes mypy to generate a JUnit XML test result document with
diff --git a/docs/source/getting_started.rst b/docs/source/getting_started.rst
@@ -303,10 +303,11 @@ Library stubs and typeshed
 Mypy uses library *stubs* to type check code interacting with library
 modules, including the Python standard library. A library stub defines
 a skeleton of the public interface of the library, including classes,
-variables and functions, and their types. Mypy ships with stubs from
-the `typeshed <https://github.com/python/typeshed>`_ project, which
-contains library stubs for the Python builtins, the standard library,
-and selected third-party packages.
+variables and functions, and their types. Mypy ships with stubs for
+the standard library from the `typeshed
+<https://github.com/python/typeshed>`_ project, which contains library
+stubs for the Python builtins, the standard library, and selected
+third-party packages.
 
 For example, consider this code:
 
@@ -318,11 +319,40 @@ Without a library stub, mypy would have no way of inferring the type of ``x``
 and checking that the argument to :py:func:`chr` has a valid type.
 
 Mypy complains if it can't find a stub (or a real module) for a
-library module that you import. Some modules ship with stubs that mypy
-can automatically find, or you can install a 3rd party module with
-additional stubs (see :ref:`installed-packages` for details).  You can
-also :ref:`create stubs <stub-files>` easily. We discuss ways of
-silencing complaints about missing stubs in :ref:`ignore-missing-imports`.
+library module that you import. Some modules ship with stubs or inline
+annotations that mypy can automatically find, or you can install
+additional stubs using pip (see :ref:`fix-missing-imports` and
+:ref:`installed-packages` for the details). For example, you can install
+the stubs for the ``requests`` package like this:
+
+.. code-block::
+
+  python3 -m pip install types-requests
+
+The stubs are usually packaged in a distribution named
+``types-<distribution>``.  Note that the distribution name may be
+different from the name of the package that you import. For example,
+``types-PyYAML`` contains stubs for the ``yaml`` package. Mypy can
+often suggest the name of the stub distribution:
+
+.. code-block:: text
+
+  prog.py:1: error: Library stubs not installed for "yaml" (or incompatible with Python 3.8)
+  prog.py:1: note: Hint: "python3 -m pip install types-PyYAML"
+  ...
+
+.. note::
+
+   Starting in mypy 0.800, most third-party package stubs must be
+   installed explicitly. This decouples mypy and stub versioning,
+   allowing stubs to updated without updating mypy. This also allows
+   stubs not originally included with mypy to be installed. Earlier
+   mypy versions included a fixed set of stubs for third-party
+   packages.
+
+You can also :ref:`create
+stubs <stub-files>` easily. We discuss ways of silencing complaints
+about missing stubs in :ref:`ignore-missing-imports`.
 
 Configuring mypy
 ****************
diff --git a/docs/source/installed_packages.rst b/docs/source/installed_packages.rst
@@ -3,54 +3,92 @@
 Using installed packages
 ========================
 
-:pep:`561` specifies how to mark a package as supporting type checking.
-Below is a summary of how to create PEP 561 compatible packages and have
-mypy use them in type checking.
-
-Using PEP 561 compatible packages with mypy
-*******************************************
-
-Generally, you do not need to do anything to use installed packages that
-support typing for the Python executable used to run mypy. Note that most
-packages do not support typing. Packages that do support typing should be
-automatically picked up by mypy and used for type checking.
-
-By default, mypy searches for packages installed for the Python executable
-running mypy. It is highly unlikely you want this situation if you have
-installed typed packages in another Python's package directory.
-
-Generally, you can use the :option:`--python-version <mypy --python-version>` flag and mypy will try to find
-the correct package directory. If that fails, you can use the
-:option:`--python-executable <mypy --python-executable>` flag to point to the exact executable, and mypy will
-find packages installed for that Python executable.
-
-Note that mypy does not support some more advanced import features, such as zip
-imports and custom import hooks.
-
-If you do not want to use typed packages, use the :option:`--no-site-packages <mypy --no-site-packages>` flag
-to disable searching.
-
-Note that stub-only packages (defined in :pep:`PEP 561: Stub-only Packages
-<561#stub-only-packages>`) cannot be used with ``MYPYPATH``. If you want mypy
-to find the package, it must be installed. For a package ``foo``, the name of
-the stub-only package (``foo-stubs``) is not a legal package name, so mypy
-will not find it, unless it is installed.
-
-Making PEP 561 compatible packages
-**********************************
-
-:pep:`561` notes three main ways to distribute type information. The first is a
-package that has only inline type annotations in the code itself. The second is
-a package that ships :ref:`stub files <stub-files>` with type information
-alongside the runtime code. The third method, also known as a "stub only
-package" is a package that ships type information for a package separately as
-stub files.
-
-If you would like to publish a library package to a package repository (e.g.
-PyPI) for either internal or external use in type checking, packages that
-supply type information via type comments or annotations in the code should put
-a ``py.typed`` file in their package directory. For example, with a directory
-structure as follows
+Packages installed with pip can declare that they support type
+checking. For example, the `aiohttp
+<https://docs.aiohttp.org/en/stable/>`_ package has built-in support
+for type checking.
+
+Packages can also provide stubs for a library. For example,
+``types-requests`` is a stub-only package that provides stubs for the
+`requests <https://requests.readthedocs.io/en/master/>`_ package.
+Stub packages are usually published from `typeshed
+<https://github.com/python/typeshed>`_, a shared repository for Python
+library stubs, and have a name of form ``types-<library>``. Note that
+many stub packages are not maintained by the original maintainers of
+the package.
+
+The sections below explain how mypy can use these packages, and how
+you can create such packages.
+
+.. note::
+
+   :pep:`561` specifies how a package can declare that it supports
+   type checking.
+
+Using installed packages with mypy (PEP 561)
+********************************************
+
+Typically mypy will automatically find and use installed packages that
+support type checking or provide stubs. This requires that you install
+the packages in the Python environment that you use to run mypy.  As
+many packages don't support type checking yet, you may also have to
+install a separate stub package, usually named
+``types-<library>``. (See :ref:`fix-missing-imports` for how to deal
+with libraries that don't support type checking and are also missing
+stubs.)
+
+If you have installed typed packages in another Python installation or
+environment, mypy won't automatically find them. One option is to
+install another copy of those packages in the environment in which you
+use to run mypy. Alternatively, you can use the
+:option:`--python-executable <mypy --python-executable>` flag to point
+to the target Python executable, and mypy will find packages installed
+for that Python executable.
+
+Note that mypy does not support some more advanced import features,
+such as zip imports and custom import hooks.
+
+If you don't want to use installed packages that provide type
+information at all, use the :option:`--no-site-packages <mypy
+--no-site-packages>` flag to disable searching for installed packages.
+
+Note that stub-only packages cannot be used with ``MYPYPATH``. If you
+want mypy to find the package, it must be installed. For a package
+``foo``, the name of the stub-only package (``foo-stubs``) is not a
+legal package name, so mypy will not find it, unless it is installed
+(see :pep:`PEP 561: Stub-only Packages <561#stub-only-packages>` for
+more information).
+
+Creating PEP 561 compatible packages
+************************************
+
+.. note::
+
+  You can generally ignore this section unless you maintain a package on
+  PyPI, or want to publish type information for an existing PyPI
+  package.
+
+:pep:`561` describes three main ways to distribute type
+information:
+
+1. A package has inline type annotations in the Python implementation.
+
+2. A package ships :ref:`stub files <stub-files>` with type
+   information alongside the Python implementation.
+
+3. A package ships type information for another package separately as
+   stub files (also known as a "stub-only package").
+
+If you want to create a stub-only package for an existing library, the
+simplest way is to contribute stubs to the `typeshed
+<https://github.com/python/typeshed>`_ repository, and a stub package
+will automatically be uploaded to PyPI.
+
+If you would like to publish a library package to a package repository
+yourself (e.g. on PyPI) for either internal or external use in type
+checking, packages that supply type information via type comments or
+annotations in the code should put a ``py.typed`` file in their
+package directory. For example, here is a typical directory structure:
 
 .. code-block:: text
 
@@ -60,7 +98,7 @@ structure as follows
         lib.py
         py.typed
 
-the ``setup.py`` might look like
+The ``setup.py`` file could look like this:
 
 .. code-block:: python
 
@@ -80,7 +118,7 @@ the ``setup.py`` might look like
    ``setup()``, or mypy will not be able to find the installed package.
 
 Some packages have a mix of stub files and runtime files. These packages also
-require a ``py.typed`` file. An example can be seen below
+require a ``py.typed`` file. An example can be seen below:
 
 .. code-block:: text
 
@@ -91,7 +129,7 @@ require a ``py.typed`` file. An example can be seen below
         lib.pyi
         py.typed
 
-the ``setup.py`` might look like:
+The ``setup.py`` file might look like this:
 
 .. code-block:: python
 
@@ -121,7 +159,7 @@ had stubs for ``package_c``, we might do the following:
         __init__.pyi
         lib.pyi
 
-the ``setup.py`` might look like:
+The ``setup.py`` might look like this:
 
 .. code-block:: python
 
@@ -134,3 +172,8 @@ the ``setup.py`` might look like:
         package_data={"package_c-stubs": ["__init__.pyi", "lib.pyi"]},
         packages=["package_c-stubs"]
     )
+
+If you have separate stubs for Python 2 and Python 3, you can place
+the Python 2 stubs in a directory with the suffix ``-python2-stubs``.
+We recommend that Python 2 and Python 3 stubs are bundled together for
+simplicity, instead of distributing them separately.
diff --git a/docs/source/running_mypy.rst b/docs/source/running_mypy.rst
@@ -124,18 +124,22 @@ The third outcome is what mypy will do in the ideal case. The following
 sections will discuss what to do in the other two cases.
 
 .. _ignore-missing-imports:
+.. _fix-missing-imports:
 
 Missing imports
 ***************
 
-When you import a module, mypy may report that it is unable to
-follow the import.
+When you import a module, mypy may report that it is unable to follow
+the import.
 
-This can cause errors that look like the following::
+This can cause errors that look like the following:
 
-    main.py:1: error: No library stub file for standard library module 'antigravity'
-    main.py:2: error: Skipping analyzing 'django': found module but no type hints or library stubs
-    main.py:3: error: Cannot find implementation or library stub for module named 'this_module_does_not_exist'
+.. code-block:: text
+
+    main.py:1: error: Library stubs not installed for "requests" (or incompatible with Python 3.8)
+    main.py:2: error: No library stub file for standard library module "antigravity"
+    main.py:3: error: Skipping analyzing 'django': found module but no type hints or library stubs
+    main.py:4: error: Cannot find implementation or library stub for module named "this_module_does_not_exist"
 
 If you get any of these errors on an import, mypy will assume the type of that
 module is ``Any``, the dynamic type. This means attempting to access any
@@ -149,7 +153,37 @@ attribute of the module will automatically succeed:
     # But this type checks, and x will have type 'Any'
     x = does_not_exist.foobar()
 
-The next three sections describe what each error means and recommended next steps.
+The next four sections describe what each error means and recommended next steps.
+
+Library stubs not installed
+---------------------------
+
+If mypy can't find stubs for a third-party library, and it knows that stubs exist for
+the library, you will get a message like this:
+
+.. code-block:: text
+
+    main.py:1: error: Library stubs not installed for "yaml" (or incompatible with Python 3.8)
+    main.py:1: note: Hint: "python3 -m pip install types-PyYAML"
+    main.py:1: note: (or run "mypy --install-types" to install all missing stub packages)
+
+You can resolve the issue by running the suggested pip command or
+commands. Alternatively, you can use :option:`--install-types <mypy
+--install-types>` to install all known missing stubs:
+
+.. code-block:: text
+
+    mypy --install-types
+
+This installs any stub packages that were suggested in the previous
+mypy run. You can also use your normal mypy command line with the
+extra :option:`--install-types <mypy --install-types>` option to
+install missing stubs at the end of the run (if any were found).
+
+You can also get this message if the stubs only support Python 3 and
+your target Python version is Python 2, or vice versa. In this case
+follow instructions in
+:ref:`missing-type-hints-for-third-party-library`.
 
 Missing type hints for standard library module
 ----------------------------------------------
