Skip to content

bpo-25872: Add unit tests for linecache and threading #25913

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
May 18, 2021
Merged

bpo-25872: Add unit tests for linecache and threading #25913

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
5143e74
bpo-25872: add test_oserror
uniocto May 5, 2021
fc1f92d
refactor linecache
uniocto May 5, 2021
fc4df5d
add test_multithread_modify_file()
uniocto May 5, 2021
a977922
refactor test_threading
uniocto May 5, 2021
759de2b
refactor test_multithread_modify_file_noerror()
uniocto May 5, 2021
5666a1d
refactor raise_oserror()
uniocto May 5, 2021
4513859
Change black box tests and refactoring
uniocto May 8, 2021
8376d21
refactor LineCacheInvalidationTests
uniocto May 9, 2021
708e348
refactor test_multithread_modify_file_noerror
uniocto May 9, 2021
f021de2
refactor assertions
uniocto May 9, 2021
a396fad
Update Lib/test/test_linecache.py
uniocto May 9, 2021
833c9b9
Update Lib/test/test_linecache.py
uniocto May 9, 2021
bcc20ab
Update Lib/test/test_threading.py
uniocto May 9, 2021
ba62b83
change __file__ and rm traceback.format_stack
uniocto May 13, 2021
420a638
add traceback
uniocto May 13, 2021
2c8269b
fix type in field name
iritkatriel May 14, 2021
8efe040
reduce repetition in setUp
iritkatriel May 14, 2021
240eb9a
check cache keys directly, not count
iritkatriel May 14, 2021
f66a050
Merge pull request #2 from iritkatriel/pr25913
uniocto May 15, 2021
78897e7
addCleanup to outside of the mofidy_file()
uniocto May 15, 2021
ac550ef
Merge remote-tracking branch 'upstream/main' into fix-issue-25872
uniocto May 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions Lib/test/test_linecache.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,13 @@ def f():
def f():
return 3''' # No ending newline

SOURCE_4 = '''
raise OSError
'''

def raise_oserror(*args, **kwargs):
raise OSError


class TempFile:

Expand Down Expand Up @@ -181,6 +188,37 @@ def test_checkcache(self):
self.assertEqual(line, getline(source_name, index + 1))
source_list.append(line)

def test_checkcache_oserror(self):
linecache.clearcache()
_ = linecache.getlines(FILENAME)
self.assertTrue(_)
self.assertEqual(1, len(linecache.cache.keys()))

with support.swap_attr(os, 'stat', raise_oserror):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that os.stat is used in checkcache is an implementation detail, not a part of that's function's external API, so I don't think the unit test should depend on that.

I would write this as a black-box test. And also test that the cache clearing is selective. Note that there are two cases in checkcache() where the cache is popped: OSError from stat, or size/timestamp don't match.

So, create three files f1, f2, f3 and load them into the cache. Then delete f1, modify f2 and call checkcache. Ensure that the cache entries for f1 and f2 were removed from the cache but the entries for f3 are still there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your feedback.
As you said, other tests are written in such a way that they do not depend on the internal implementation.
I will try to write them in a black box as you suggested, so I would appreciate it if you could take look again !!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I write a black box test, and would be happy if you could check it out.

def test_oserror(self):
    def _oserror_helper():
        linecache.clearcache()
        be_deleted_file = os_helper.TESTFN + '.1'
        be_modified_file = os_helper.TESTFN + '.2'
        unchange_file = os_helper.TESTFN + '.3'
        self.addCleanup(os_helper.unlink, be_deleted_file)
        self.addCleanup(os_helper.unlink, be_modified_file)
        self.addCleanup(os_helper.unlink, unchange_file)
        with open(be_deleted_file, 'w', encoding='utf-8') as source:
            source.write('print("will be deleted")')
        with open(be_modified_file, 'w', encoding='utf-8') as source:
            source.write('print("will be modified")')
        with open(unchange_file, 'w', encoding='utf-8') as source:
            source.write('print("unchange")')

        _ = linecache.getlines(be_deleted_file)
        _ = linecache.getlines(be_modified_file)
        _ = linecache.getlines(unchange_file)
        self.assertEqual(3, len(linecache.cache.keys()))

        os.remove(be_deleted_file)
        with open(be_modified_file, 'w', encoding='utf-8') as source:
            source.write('print("was modified")')
        return (be_deleted_file, be_modified_file, unchange_file)

    deleted_file, modified_file, unchange_file = _oserror_helper()
    _ = linecache.checkcache(deleted_file)
    self.assertEqual(2, len(linecache.cache.keys()))
    _ = linecache.checkcache(modified_file)
    self.assertEqual(1, len(linecache.cache.keys()))
    _ = linecache.checkcache(unchange_file)
    self.assertEqual(1, len(linecache.cache.keys()))

    deleted_file, modified_file, unchange_file = _oserror_helper()
    _ = linecache.updatecache(deleted_file)
    self.assertEqual(2, len(linecache.cache.keys()))
    _ = linecache.updatecache(modified_file)
    self.assertEqual(2, len(linecache.cache.keys()))
    _ = linecache.updatecache(unchange_file)
    self.assertEqual(2, len(linecache.cache.keys()))

# pop all cache
_ = linecache.checkcache()
self.assertEqual(0, len(linecache.cache.keys()))

def test_updatecache_oserror(self):
linecache.clearcache()
source_name = os_helper.TESTFN
self.addCleanup(os_helper.unlink, os_helper.TESTFN)

with open(source_name, 'w', encoding='utf-8') as source:
source.write(SOURCE_4)
_ = linecache.getlines(source_name)
self.assertEqual(1, len(linecache.cache.keys()))

with support.swap_attr(os, 'stat', raise_oserror):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above - delete/modify the file instead of mocking os.stat.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As shown above, I write delete/modify test.

# Trace OSError with no pop cache
_ = linecache.updatecache('dummy')
self.assertEqual(1, len(linecache.cache.keys()))

with support.swap_attr(os, 'stat', raise_oserror):
# Trace OSError with pop cache
_ = linecache.updatecache(source_name)
self.assertEqual(0, len(linecache.cache.keys()))

def test_lazycache_no_globals(self):
lines = linecache.getlines(FILENAME)
linecache.clearcache()
Expand Down
19 changes: 19 additions & 0 deletions Lib/test/test_threading.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1338,6 +1338,25 @@ def run(self):
# explicitly break the reference cycle to not leak a dangling thread
thread.exc = None

def test_multithread_modify_file_noerror(self):
import traceback
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe PEP-8 asks for imports to be at module scope

Copy link
Contributor Author

@uniocto uniocto May 8, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @iritkatriel
Thank you for your feedback and I'll rewrite it to support pep8.
Incidentally, pep-8 point out where I have not changed this time.
Should I also refactor the following?

> pep8 Lib/test/test_threading.py 
Lib/test/test_threading.py:45:5: E301 expected 1 blank line, found 0
Lib/test/test_threading.py:47:5: E301 expected 1 blank line, found 0
Lib/test/test_threading.py:49:5: E301 expected 1 blank line, found 0
Lib/test/test_threading.py:52:1: E302 expected 2 blank lines, found 1
Lib/test/test_threading.py:145:41: E228 missing whitespace around modulo operator
Lib/test/test_threading.py:152:80: E501 line too long (88 > 79 characters)
Lib/test/test_threading.py:171:9: E301 expected 1 blank line, found 0
Lib/test/test_threading.py:223:9: E265 block comment should start with '# '
Lib/test/test_threading.py:248:80: E501 line too long (80 > 79 characters)
Lib/test/test_threading.py:259:40: E261 at least two spaces before inline comment
Lib/test/test_threading.py:285:24: E261 at least two spaces before inline comment
Lib/test/test_threading.py:307:36: E261 at least two spaces before inline comment
Lib/test/test_threading.py:408:13: E128 continuation line under-indented for visual indent
Lib/test/test_threading.py:424:21: E128 continuation line under-indented for visual indent
Lib/test/test_threading.py:436:69: E231 missing whitespace after ':'
Lib/test/test_threading.py:450:26: E128 continuation line under-indented for visual indent
Lib/test/test_threading.py:458:26: E128 continuation line under-indented for visual indent
Lib/test/test_threading.py:716:9: E301 expected 1 blank line, found 0
Lib/test/test_threading.py:733:80: E501 line too long (82 > 79 characters)
Lib/test/test_threading.py:751:9: E301 expected 1 blank line, found 0
Lib/test/test_threading.py:769:44: E261 at least two spaces before inline comment
Lib/test/test_threading.py:838:40: E231 missing whitespace after ','
Lib/test/test_threading.py:900:13: E301 expected 1 blank line, found 0
Lib/test/test_threading.py:902:13: E301 expected 1 blank line, found 0
Lib/test/test_threading.py:913:1: E303 too many blank lines (3)
Lib/test/test_threading.py:969:80: E501 line too long (82 > 79 characters)
Lib/test/test_threading.py:1009:80: E501 line too long (87 > 79 characters)
Lib/test/test_threading.py:1066:47: E203 whitespace before ':'
Lib/test/test_threading.py:1193:61: E703 statement ends with a semicolon
Lib/test/test_threading.py:1237:80: E501 line too long (81 > 79 characters)
Lib/test/test_threading.py:1399:80: E501 line too long (80 > 79 characters)
Lib/test/test_threading.py:1449:14: E127 continuation line over-indented for visual indent
Lib/test/test_threading.py:1450:14: E127 continuation line over-indented for visual indent
Lib/test/test_threading.py:1509:1: E302 expected 2 blank lines, found 1
Lib/test/test_threading.py:1512:1: E302 expected 2 blank lines, found 1
Lib/test/test_threading.py:1515:1: E302 expected 2 blank lines, found 1
Lib/test/test_threading.py:1519:1: E302 expected 2 blank lines, found 1
Lib/test/test_threading.py:1522:1: E302 expected 2 blank lines, found 1
Lib/test/test_threading.py:1526:1: E302 expected 2 blank lines, found 1
Lib/test/test_threading.py:1529:1: E302 expected 2 blank lines, found 1
Lib/test/test_threading.py:1532:1: E302 expected 2 blank lines, found 1
Lib/test/test_threading.py:1535:1: E302 expected 2 blank lines, found 1
Lib/test/test_threading.py:1651:17: E128 continuation line under-indented for visual indent

> pep8 Lib/test/test_linecache.py
Lib/test/test_linecache.py:229:9: E301 expected 1 blank line, found 0

def modify_file():
with open(__file__, 'a') as fp:
fp.write(' ')
traceback.format_stack()

threads = [
threading.Thread(target=modify_file)
for i in range(100)
]
try:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which exceptions are you trying to ignore here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing, sorry. I removed try.

for t in threads:
t.start()
for t in threads:
t.join()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why you don't do

for t in threads:
    t.start()
    t.join()

?

Copy link
Contributor Author

@uniocto uniocto May 9, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing, sorry.
So I fixed it to your feedback.

finally:
pass


class ThreadRunFail(threading.Thread):
def run(self):
Expand Down
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy