Skip to content

PEP 544: Protocols #224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 40 commits into from
Mar 18, 2017
Merged
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
3fa3e48
Collect various ideas
ilevkivskyi Mar 5, 2017
51b0ca0
Some formatting and reordering
ilevkivskyi Mar 5, 2017
993f7b3
Add some examples
ilevkivskyi Mar 5, 2017
e384336
Add planned link templates
ilevkivskyi Mar 6, 2017
7ea5d41
Add links + minor changes
Mar 6, 2017
cdcf62f
Polishing rationale
Mar 6, 2017
1ffed9b
Some more reshuffling and formatting
Mar 6, 2017
72ceae6
Add more examples
Mar 6, 2017
6bea2e8
Add more examples to existing approaches
Mar 6, 2017
d5972c3
Typos, reordering, and few more details (backport)
ilevkivskyi Mar 6, 2017
57d375f
Update list of protocols in typing
ilevkivskyi Mar 6, 2017
9d4d685
Defining protocols plus minor changes and formatting
ilevkivskyi Mar 7, 2017
82258d5
Explicitly declaring implementation and other changes
ilevkivskyi Mar 7, 2017
5d9fb7c
More polishing
ilevkivskyi Mar 7, 2017
a6e6d9e
Edit rejected/postponed ideas
ilevkivskyi Mar 7, 2017
3175013
Runtime things, reorder links
ilevkivskyi Mar 7, 2017
cbff669
Runtime decorator
ilevkivskyi Mar 7, 2017
dfccd06
Backward compatible part and last bits
Mar 8, 2017
60f4d52
Some clarifications
ilevkivskyi Mar 9, 2017
60e7f7f
Add links in text
ilevkivskyi Mar 9, 2017
c90aa1c
Caption style, add cross-refs
Mar 9, 2017
b008de1
Remove redundant links; + minor changes
ilevkivskyi Mar 10, 2017
02cca5c
One more tiny change
ilevkivskyi Mar 10, 2017
7d89b6b
Merge remote-tracking branch 'upstream/master' into protocols
ilevkivskyi Mar 10, 2017
0f3732a
Copyediting changes
JelleZijlstra Mar 10, 2017
95fbf58
Merge pull request #1 from JelleZijlstra/patch-2
ilevkivskyi Mar 10, 2017
cb65bff
Rename PEP with a valid number to get the build running
ilevkivskyi Mar 10, 2017
817bf2f
Reflow to 79 characters
ilevkivskyi Mar 10, 2017
2d89ba9
fix typo
JelleZijlstra Mar 10, 2017
0efcbff
Some grammar tweaks
brettcannon Mar 10, 2017
ebd4b17
Merge pull request #3 from brettcannon/patch-1
ilevkivskyi Mar 10, 2017
0de36be
Implement Guido's idea of EIBTI plus minor comments
ilevkivskyi Mar 11, 2017
767c58b
Fix typo
ilevkivskyi Mar 11, 2017
efc3154
Make implementation enforcement optional; fix order of Protocolbase
ilevkivskyi Mar 12, 2017
7d714c3
Add missing @abstractmethod decorators
ilevkivskyi Mar 13, 2017
d4ab050
Minor clarification
ilevkivskyi Mar 13, 2017
d9d21c2
Implement Jukka's and David's comments; few more minor things
ilevkivskyi Mar 16, 2017
4dfbfb2
Implement most comments by Łukasz; few more to do
ilevkivskyi Mar 17, 2017
d51420e
More changes in response to comments
Mar 18, 2017
f6240c8
Remove one reamining 'All'
ilevkivskyi Mar 18, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Some grammar tweaks
  • Loading branch information
brettcannon authored Mar 10, 2017
commit 0efcbff45e26229082ec20b5d97a84f5819cba3a
42 changes: 21 additions & 21 deletions pep-0544.txt
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ of the library. Moreover, extensive use of ABCs might impose additional
runtime costs.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moreover, extensive use of ABCs might impose additional runtime costs.

Do we have any numbers? We know ABCs aren't free but I don't know if this is worth mentioning unless it's a major factor. The rationale for protocols to be the pythonic, dynamic and idiomatic version of ABCs is strong enough.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

class C(collections.abc.Iterable):
    def __iter__(self):
        return []

is 2x slower than

class C:
    def __iter__(self):
        return []

The numbers for typing.Iterable are even worse, but I have some ideas on how to improve those. The 2x slowdown is still not much, so I use "extensive" and "might". If you think it is not necessary we could remove this altogether.


The intention of this PEP is to solve all these problems
by allowing to write the above code without explicit base classes in
the class definition, and ``Bucket`` would still be implicitly considered
by allowing users to write the above code without explicit base classes in
the class definition, allowing ``Bucket`` to be implicitly considered
a subtype of both ``Sized`` and ``Iterable[int]`` by static type checkers
using structural [wiki-structural]_ subtyping::

Expand All @@ -66,7 +66,7 @@ using structural [wiki-structural]_ subtyping::
result: int = collect(Bucket()) # Passes type check

The same functionality will be provided for user-defined protocols, as
specified below. The code with a protocol class matches common Python
specified below. The above code with a protocol class matches common Python
conventions much better. It is also automatically extensible and works
with additional, unrelated classes that happen to implement
the required protocol.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is also automatically extensible and works with additional, unrelated classes that happen to implement the required protocol.

Yes, this is the killer feature.

Expand Down Expand Up @@ -158,18 +158,18 @@ approaches related to structural subtyping in Python and other languages:
The drawback of this approach is the necessity to either subclass
the abstract class or register an implementation explicitly::

from abc import ABCMeta
from abc import ABC

class MyTuple(metaclass=ABCMeta):
class MyTuple(ABC):
pass

MyTuple.register(tuple)

assert issubclass(tuple, MyTuple)
assert isinstance((), MyTuple)

As mentioned in `rationale`_, we want to avoid such necessity, especially
in static context. However, in runtime context, ABCs are good candidates for
As mentioned in the `rationale`_, we want to avoid such necessity, especially
in static context. However, in a runtime context, ABCs are good candidates for
protocol classes and they are already used extensively in
the ``typing`` module.

Expand Down Expand Up @@ -319,9 +319,9 @@ Protocol members
There are two ways to define methods that are protocol members:

* Use the ``@abstractmethod`` decorator
* Leave the method body empty, i.e. it should be either a single ``pass``
statement, a single ``...`` ellipsis literal, or a single
``raise NotImplementedError`` statement.
* Leave the method body empty, i.e. it should be either docstring,
a single ``pass`` statement, a single ``...`` ellipsis literal,
or a single ``raise NotImplementedError`` statement.

A combination of both may be used if necessary for runtime purposes.
If the method body is not empty but decorated by the ``@abstractmethod``
Expand Down Expand Up @@ -383,7 +383,7 @@ annotation should be used as specified by PEP 526.
Explicitly declaring implementation
-----------------------------------

To explicitly declare that a certain class implements a given protocols, they
To explicitly declare that a certain class implements the given protocols, they
can be used as regular base classes. In this case:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm worried this could be pretty confusing: you could have a class that explicitly implements some protocols and implicitly implements others. I don't know of any other language that allows this.


* A class could use implementations of some methods in protocols, both
Expand Down Expand Up @@ -545,7 +545,7 @@ relationships are subject to the following rules:
if and only if ``X`` implements all protocol members of ``P``. In other
words, subtyping with respect to a protocol is always structural.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If X is a protocol itself, "implements" gets a different meaning, right? I think in that case it just means that it must define all of P's methods right? Maybe it would be less ambiguous if you separated the two cases into separate bullets.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it would be less ambiguous if you separated the two cases into separate bullets.

I agree.

* Edge case: for recursive protocols, structural subtyping is decided
positively for all situations that are type safe. Continuing
positively for all situations that are type safe. Continuing the
previous example::

class Tree(Generic[T]):
Expand Down Expand Up @@ -590,7 +590,7 @@ classes. For example::

In addition, we propose to add another special type construct
``All`` that represents intersection types. Although for normal types
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd consider leaving out All from this proposal, at least initially. This is something we can add later if there is a need, but currently I don't see evidence for this being important. Effectively the same behavior can be achieved through multiple inheritance. Besides, it would be a bit arbitrary if All only worked with protocols, and making All work with arbitrary types would be a much more complex feature, and not as directly related to this PEP.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could include this for three reasons:

  • I could easily imagine a function parameter that requires more than one protocol, especially if the preferred style is to have several simple protocols, rather than few complex. By the way, some classes in collections.abc are simple intersections, for example Collection == All[Sized, Iterable, Container].
  • It could be easily implemented for protocols. I also propose to use All instead of Intersection to emphasize it only works for protocols (mnemonics: variable implements all these protocols). On the contrary, I think that intersections of nominal classes would be very rarely used.
  • If we decide to include this later, then people might simply be not aware of this feature (this is a minor reason, but still we need to take this into account).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with Jukka -- I think it's confusing for All to work only with protocols, extending All to work with all types would add considerable complexity, and I'm not convinced there's a pressing need.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was pushing for Intersection for the original PEP because I saw the need for being able to intersect ABCs. Given that protocols themselves solve this, and we can use multiple inheritance as Jukka is suggesting, I agree that it's wiser to leave this out.

I would still like to have it, but that can be a separate PEP. I agree it's complex to implement for concrete types, and better yet, for combinations of both.

it is not very useful, there are many situation where a variable should
it is not very useful, there are many situations where a variable should
implement more than one protocol. Annotation by ``All[Proto1, Proto2, ...]``
means that a given variable or parameter must implement all protocols
``Proto1``, ``Proto2``, etc. either implicitly or explicitly. Example::
Expand Down Expand Up @@ -630,7 +630,7 @@ The same rule applies to variables::
var = Concrete # OK
var().meth() # OK

Assigning protocol class to a variable is allowed if it is not explicitly
Assigning a protocol class to a variable is allowed if it is not explicitly
typed, and such assignment creates a type alias. For non-protocol classes,
the behavior of ``Type[]`` is not changed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For non-protocol classes, the behavior of Type[] is not changed.

No, we want to make ABCs behave the same way.


Expand Down Expand Up @@ -672,8 +672,8 @@ at runtime.
However, it should be possible for protocol types to implement custom
instance and class checks when this makes sense, similar to how ``Iterable``
and other ABCs in ``collections.abc`` and ``typing`` already do it,
but this is limited to non-generic and to unsubscripted generic protocols
(statically ``Iterable`` is equivalent to ``Iterable[Any]`).
but this is limited to non-generic and unsubscripted generic protocols
(``Iterable`` is statically equivalent to ``Iterable[Any]`).
The ``typing`` module will define a special ``@runtime`` class decorator that
automatically adds a ``__subclasshook__()`` that provides the same semantics
for class and instance checks as for ``collections.abc`` classes::
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't love the automatic isinstance check support, even though that's how the mypy Protocol class used to work. I don't think that they can be made to work reliably for arbitrary ABCs. Consider this example:

class P(Protocol):
    @abstractmethod
    def common_method_name(self, x: int) -> int: ... 

class X:
    <a bunch of methods>
    def common_method_name(self) -> None: ...

def do_stuff(o: Union[P, X]) -> int:
    if isinstance(o, P):
        return o.common_method_name(1)  # oops what if it's an X instance instead?
    else:
        return (do something with an X)

Maybe a type checker can warn about this example, but more generally the isinstance check could be unreliable, except for things where there is a common signature convention such as Iterable.

Another potentially problematic case:

class P(Protocol):
    x: int

class C:
    def initialize(self) -> None:
        self.x = 0

c1 = C()
c2 = C()
c2.initialize()
isinstance(c1, P)  # False
isinstance(c2, P)  # True

def f(x: Union[P, int]) -> None:
    if isinstance(x, P):
        # type of x is P here
        ...
    else:
        # type of x is int here?
        print(x + 1)

f(C())   # oops

Again, maybe we can live with this, but this could be pretty surprising. I'd argue that requiring an explicit class decorator would be better, since we can then attach warnings about problems like this in the documentation. The programmer would be able to evaluate whether the benefits outweigh the potential for confusion for each protocol and explicitly opt in -- but the default behavior would be safer.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd argue that requiring an explicit class decorator would be better, since we can then attach warnings about problems like this in the documentation.

This is how I initially wrote this, since I intuitively felt that this could not be 100% reliable. Guido wanted to make it default, by now we have good examples to make this opt-in, so that I will revert this to class decorator.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand how a class decorator would solve the problem. Wouldn't that mean that some classes that implement the protocol (i.e. the decorated ones) will work properly with isinstance at runtime but others will not?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

classes that implement the protocol (i.e. the decorated ones)

The decorator is applied to protocol, not to implementation. This was in a previous draft. At runtime the decorator will add same __subclasshook__() to a protocol, that collections.abc classes have now. As Jukka pointed out, there are pros and cons of such approach, and a user could decide.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would the __subclasshook__ work?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically, if a class has all necessary attributes, then it is a subclass. This is why I mention below that "type checkers will narrow types after such checks by the type erased Proto (i.e. with all variables having type Any and all methods having type Callable[..., Any])" because we cannot reliably check the types of attributes at runtime.

This is how it is done in collections.abc see https://github.com/python/cpython/blob/master/Lib/_collections_abc.py#L72 and e.g. https://github.com/python/cpython/blob/master/Lib/_collections_abc.py#L93

Expand Down Expand Up @@ -761,7 +761,7 @@ Implementation details
----------------------

The runtime implementation could be done in pure Python without any
effects on the core interpreter and standard library except in
effects on the core interpreter and standard library except in the
``typing`` module:

* Define class ``typing.Protocol`` similar to ``typing.Generic``.
Expand All @@ -779,8 +779,8 @@ effects on the core interpreter and standard library except in
be provided at runtime.


Changes in typing module
------------------------
Changes in the typing module

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do you propose to handle the backwards-incompatibility problem for all currently released versions of Python?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea is that there should be no backward incompatibility. The classes in typing module (as well as in collections.abc) already behave like protocols at runtime:

class Whatever:
    def __iter__(self): return []
assert isinstance(Whatever(), Iterable)

as I mention below "Practically, few changes will be needed in typing since some of these classes already behave the necessary way at runtime", most of these will be changed only in typeshed.

Also, after I implement the changes proposed by Jukka, all code that passes mypy now, will still pass (unless I don't see some subtleties).

----------------------------

The following classes in ``typing`` module will be protocols:

Expand Down Expand Up @@ -896,8 +896,8 @@ Use assignments to check explicitly that a class implements a protocol
----------------------------------------------------------------------

In Go language the explicit checks for implementation are performed
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"In Go language" -> either "In Go" or "In the Go language"

via dummy assignments [golang]_. Such way is also possible with the current
proposal. Example::
via dummy assignments [golang]_. Such a way is also possible with the
current proposal. Example::

class A:
def __len__(self) -> float:
Expand Down
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy