Downloads during build process can't handle reset connections #434
Description
In the case that a connection is broken, we may want to consider a retry attempt. This really applies to all downloads across all datasets, but this is an example for broad sanger proteomics
Error:
Traceback (most recent call last):
File "/app/02a-broad_sanger_proteomics.py", line 80, in <module>
main()
~~~~^^
File "/app/02a-broad_sanger_proteomics.py", line 58, in main
r = requests.get(sanger_protfile)
File "/opt/venv/lib/python3.13/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/opt/venv/lib/python3.13/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.13/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/opt/venv/lib/python3.13/site-packages/requests/sessions.py", line 746, in send
r.content
File "/opt/venv/lib/python3.13/site-packages/requests/models.py", line 902, in content
self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.13/site-packages/requests/models.py", line 822, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(10190848 bytes read, 102480742 more expected)', IncompleteRead(10190848 bytes read, 102480742 more expected))
Sub-issues
Metadata
Metadata
Assignees
Labels
Type
Projects
Status