Subject

Thread A starts Thread B to read a chunked HTTP response. At some point, regardless of whether the server has more data, I want to break the connection from the client side. Thread B is blocked trying to read the socket, so I'm trying to call HTTPResponse.close() from Thread A. This call hangs.

I'm not sure if this is a urllib3 issue, or a limitation of the underlying Python synchronous I/O model, or something I am doing wrong.

Environment

• Python 3.7.13, 3.10.6
• platform: Darwin-22.2.0-x86_64-i386-64bit, Linux-5.10.104-linuxkit-x86_64-with-debian-10.13
• urllib3 1.26.13

Steps to Reproduce

Unzip this archive and run main.py.

urllib3-close.zip

This starts a simple HTTP server on a daemon thread; makes a request to it from another thread, which receives a little bit of data and then blocks waiting for more; then, from the main thread, tries to close the response.

Expected Behavior

I hoped that closing the response would cause the underlying socket to be closed, causing the client thread to get a read error and terminate. For this demo app, the output would look like this:

main [MainThread] starting server
  server [Thread-1] listening on port 10000
main [MainThread] starting client
    client [Thread-2] opening connection
main [MainThread] sleeping a while
  server [Thread-1] starting stream
127.0.0.1 - - [05/Jan/2023 13:38:11] "GET / HTTP/1.1" 200 -
  server [Thread-1] wrote 10 bytes
    client [Thread-2] reading data
    client [Thread-2] read 10 bytes
main [MainThread] about to close response
main [MainThread] closed response
    client [Thread-2] failed: <some kind of I/O error>
    client [Thread-2] no longer reading
main [MainThread] about to clear connection pool
main [MainThread] cleared connection pool
main [MainThread] exiting

Actual Behavior

The HTTPResponse.close() call hangs. The output looks like this, and then the process waits indefinitely:

main [MainThread] starting server
  server [Thread-1] listening on port 10000
main [MainThread] starting client
    client [Thread-2] opening connection
main [MainThread] sleeping a while
  server [Thread-1] starting stream
127.0.0.1 - - [05/Jan/2023 13:38:11] "GET / HTTP/1.1" 200 -
  server [Thread-1] wrote 10 bytes
    client [Thread-2] reading data
    client [Thread-2] read 10 bytes
main [MainThread] about to close response

The only way I have found to make it not hang is to make the server send some more data; then, the blocking read on the client read unblocks, and the client is able to notice that the HTTPResponse is no longer in a valid state. But then that no longer corresponds to the use case I'm talking about, where the server does not have more data yet but I still want to break the connection.