Skip to content

bpo-26253: Add compressionlevel to tarfile stream #2962

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 25, 2022
Merged

bpo-26253: Add compressionlevel to tarfile stream #2962

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
89fa205
bpo-26253: Add compressionlevel to tarfile stream
jarondl Jul 31, 2017
31b9835
bpo-26253: don't add compresslevel as an attribute
jarondl Jul 31, 2017
c217360
bpo-26253: tarfile: add compresslevel unit tests
jarondl Aug 1, 2017
f48acbd
Apply suggestions from code review
serhiy-storchaka Jun 17, 2022
2b1b417
Update Misc/NEWS.d/next/Library/2017-07-31-13-35-28.bpo-26253.8v_sCs.rst
serhiy-storchaka Jun 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
bpo-26253: Add compressionlevel to tarfile stream 
`tarfile` already accepts a compressionlevel argument for creating
files. This patch adds the same for stream-based tarfile usage.
The default is 9, the value that was previously hard-coded.
  • Loading branch information
@jarondl
jarondl committed May 8, 2022
commit 89fa205d27819c4d4cd444abbfde76e0abe15cea
4 changes: 2 additions & 2 deletions Doc/library/tarfile.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,8 @@ Some facts and figures:
If *fileobj* is specified, it is used as an alternative to a :term:`file object`
opened in binary mode for *name*. It is supposed to be at position 0.

For modes ``'w:gz'``, ``'r:gz'``, ``'w:bz2'``, ``'r:bz2'``, ``'x:gz'``,
``'x:bz2'``, :func:`tarfile.open` accepts the keyword argument
For modes ``'w:gz'``, ``'x:gz'``, ``'w|gz'``, ``'w:bz2'``, ``'x:bz2'``,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add the versionchanged directive.

Document the change in the What's New document.

Copy link
Contributor Author

@jarondl jarondl Aug 1, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

versionchanged - Done

``'w|bz2'``, :func:`tarfile.open` accepts the keyword argument
*compresslevel* (default ``9``) to specify the compression level of the file.

For modes ``'w:xz'`` and ``'x:xz'``, :func:`tarfile.open` accepts the
Expand Down
33 changes: 19 additions & 14 deletions Lib/tarfile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -336,7 +336,8 @@ class _Stream:
_Stream is intended to be used only internally.
"""

def __init__(self, name, mode, comptype, fileobj, bufsize):
def __init__(self, name, mode, comptype, fileobj, bufsize,
compresslevel=9):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since _Stream is intended to be used only internally, we are free to change the signature of its constructor. The default argument for compresslevel is not needed. This just duplicates its value.

"""Construct a _Stream object.
"""
self._extfileobj = True
Expand All @@ -350,14 +351,15 @@ def __init__(self, name, mode, comptype, fileobj, bufsize):
fileobj = _StreamProxy(fileobj)
comptype = fileobj.getcomptype()

self.name = name or ""
self.mode = mode
self.name = name or ""
self.mode = mode
self.comptype = comptype
self.fileobj = fileobj
self.bufsize = bufsize
self.buf = b""
self.pos = 0
self.closed = False
self.fileobj = fileobj
self.bufsize = bufsize
self.compresslevel = compresslevel
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it needed to save compresslevel as an attribute? Wouldn't be enough to pass compresslevel as an argument to _init_write_gz()?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No it is not needed. My first approach was to pass it as an argument to _init_write_gz(), with a default value of 9 (not to change the signature). But then I got the sense that an attribute goes more naturally with the rest of the class. I can revert to the first approach if you prefer.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other attributes are used in multiple _Stream methods or are needed for simulating a file object. But compresslevel is used only in the constructor (_init_write_gz() is a factored out part of the constructor). It is not needed after creating a _Stream object.

self.buf = b""
self.pos = 0
self.closed = False

try:
if comptype == "gz":
Expand All @@ -383,7 +385,7 @@ def __init__(self, name, mode, comptype, fileobj, bufsize):
self.cmp = bz2.BZ2Decompressor()
self.exception = OSError
else:
self.cmp = bz2.BZ2Compressor()
self.cmp = bz2.BZ2Compressor(self.compresslevel)

elif comptype == "xz":
try:
Expand Down Expand Up @@ -413,10 +415,11 @@ def __del__(self):
def _init_write_gz(self):
"""Initialize for writing with gzip compression.
"""
self.cmp = self.zlib.compressobj(9, self.zlib.DEFLATED,
-self.zlib.MAX_WBITS,
self.zlib.DEF_MEM_LEVEL,
0)
self.cmp = self.zlib.compressobj(self.compresslevel,
self.zlib.DEFLATED,
-self.zlib.MAX_WBITS,
self.zlib.DEF_MEM_LEVEL,
0)
timestamp = struct.pack("<L", int(time.time()))
self.__write(b"\037\213\010\010" + timestamp + b"\002\377")
if self.name.endswith(".gz"):
Expand Down Expand Up @@ -1649,7 +1652,9 @@ def not_compressed(comptype):
if filemode not in ("r", "w"):
raise ValueError("mode must be 'r' or 'w'")

stream = _Stream(name, filemode, comptype, fileobj, bufsize)
compresslevel = kwargs.pop("compresslevel", 9)
stream = _Stream(name, filemode, comptype, fileobj, bufsize,
compresslevel)
try:
t = cls(name, filemode, stream, **kwargs)
except:
Expand Down
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Library/2017-07-31-13-35-28.bpo-26253.8v_sCs.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Adjustable compression level for tarfile streams.
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy