hi,

we have a setup where a bunch of compute nodes can are behind a proxy, and there is no other way to connect to the internet.

these compute nodes should be registered to a rancher cluster that is located elsewhere, so to connect to the cluster and run the registration, we have to go through the proxy. this proxy is tinyproxy.

so we run the agent docker container with HTTP_PROXY env variables, and we see some traffic through the entire chain all the way to the rancher server. in tcpdump, we see communication between compute nodes and proxy, we see an initial

 CONNECT $hostname-of-rancher-server

which is followed what looks like a TLS handshake (first a little bit i was able to identify as a SNI header, then we get a bunch of binary things with readable certificate information inside. at the same time, we see an incoming HTTPS connection on the TLS offloader on the opposite side.

all in all, this looks very normal & exactly as i would expect. however, the agent fails with this:

time="2019-12-18T15:56:53Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp ${v4 addr of tls offloader}:443: i/o timeout"
time="2019-12-18T15:56:53Z" level=error msg="Remotedialer proxy error" error="dial tcp ${v4 addr of tls offloader}:443: i/o timeout"

if we use some ssh tunneling trickery and mess with /etc/hosts, thus connecting without the proxy, everything is fine, i.e. node registration with https:// and wss:// works nicely.

if we run the same thing with tinyproxy in between, it fails (see above) and tinyproxy says this:

CONNECT Dec 18 16:49:07 [28094]: Connect (file descriptor 6): $hostname-client [$ip compute]
CONNECT Dec 18 16:49:07 [28094]: Request (file descriptor 6): CONNECT $hostname-server:443 HTTP/1.1
INFO Dec 18 16:49:07 [28094]: No upstream proxy for $hostname-server
CONNECT Dec 18 16:49:07 [28094]: Established connection to host "$hostname-server" using file descriptor 7.
> INFO Dec 18 16:49:07 [28094]: Not sending client headers to remote machine
INFO Dec 18 16:49:13 [28094]: Closed connection between local client (fd:6) and remote client (fd:7)

i wonder what happens here, and how we might be able to debug this further. i am all out of ideas at the moment (short of completely changing the entire approach). the user agent header from the agent is "Go-http-client/1.1"