`rmtree` sometimes fails due to "directory not empty" on Linux #128076

christophfriedrich · 2024-12-18T18:46:04Z

Bug report

Bug description:

I have a temp directory in which I have one zip file per day of a month (along the pattern ./agrisens_YYYYMMDD.zip) and the extracted folders (many files along the pattern ./agrisens_YYYYMMDD/foo.bar). The zip files are listed in a downloaded_files variable. My code to clear the temp directory (without deleting it entirely) is like this:

print('Cleaning up temporary files...')
for filename in downloaded_files:
    foldername = filename[0:-4]  # remove last 4 characters (i.e. the file extension (.zip))
    os.remove('temp/'+filename)  # delete zip file
    shutil.rmtree('temp/'+foldername)  # delete folder where zip file was extracted to (os.rmdir only deletes empty directories, but here we still have content in it (and don't want to go through it ourselves))
print('DONE')

Repeatedly, this loop failed half-way through:

Cleaning up temporary files...
Traceback (most recent call last):
  File "/datacube/Scripts/agrisens/dwd_data-v6.py", line 317, in <module>
    shutil.rmtree('temp/'+foldername)  # delete folder where zip file was extracted to (os.rmdir only deletes empty directories, but here we still have content in it (and don't want to go through it ourselves))
  File "/opt/conda/lib/python3.10/shutil.py", line 731, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/opt/conda/lib/python3.10/shutil.py", line 729, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: 'temp/agrisens_20231217'

Interestingly, when looking at the file system, the folder is actually empty!

(base) datacube@d6f03dd71d64:/datacube/Scripts/agrisens/temp$ ll
total 1222793
drwxrws--- 17 datacube 4010225     4096 Dec 18 17:16 ./
drwxrws---  5 datacube 4010225     4096 Dec 18 17:09 ../
drwxrws---  2 datacube 4010225     4096 Dec 18 17:16 agrisens_20231116/
drwxrws---  2 datacube 4010225     4096 Dec 18 16:39 agrisens_20231117/
-rw-rw----  1 datacube 4010225 86510075 Dec 18 16:38 agrisens_20231117.zip
drwxrws---  2 datacube 4010225     4096 Dec 18 16:39 agrisens_20231118/
-rw-rw----  1 datacube 4010225 89981192 Dec 18 16:38 agrisens_20231118.zip
...

(base) datacube@d6f03dd71d64:/datacube/Scripts/agrisens/temp/agrisens_20231217$ ll
total 1
drwxrws---  2 datacube 4010225 4096 Dec 18 18:06 ./
drwxrws--- 17 datacube 4010225 4096 Dec 18 18:06 ../
(base) datacube@d6f03dd71d64:/datacube/Scripts/agrisens/temp/agrisens_20231217$

So it's quite likely that some race condition happened so that the folder was not empty on the attempt to delete it, but shortly after.

Note also that the folder of the day before the criticised folder didn't get removed either. And neither the zip file of the day in question, despite the call to os.remove being before the call to shutil.rmtree in the loop.

The same issue also happened with the previous month:

Cleaning up temporary files...
Traceback (most recent call last):
  File "/datacube/Scripts/agrisens/dwd_data-v6.py", line 317, in <module>
    shutil.rmtree('temp/'+foldername)  # delete folder where zip file was extracted to (os.rmdir only deletes empty directories, but here we still have content in it (and don't want to go through it ourselves))
  File "/opt/conda/lib/python3.10/shutil.py", line 731, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/opt/conda/lib/python3.10/shutil.py", line 729, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: 'temp/agrisens_20231116'

In both cases, the fail happened quite exactly in the middle of the month -- might be coincidental, might be noteworthy.

Has this been discussed elsewhere?

I looked through open issues with "rmtree" and saw two that might be related:

The bug sounds similar to shutil.rmtree will frequently fail on Windows under heavy load due to racy deletion #84324, but for me it's happening on Linux, not on Windows
The actual bug might be workaroundable through shutil.rmtree and os.listdir cannot recover on error conditions #52769

Other notes
The file system being written to is a network storage that is mounted into a virtual machine, from where it is mounted into the Docker container where my actual code runs, so not the closest setup -> a good environment for race conditions to happen. To my knowledge, no Windows is involved anywhere in the chain.

Version

> python3 --version
Python 3.10.12

CPython versions tested on:

3.10

Operating systems tested on:

Linux

The text was updated successfully, but these errors were encountered:

nesergen · 2025-02-27T09:58:15Z

Hello! I have the same problem. Raises an error about an non-empty folder, but really the folder is empty.
This behavior began after upgrade the system (Linux Mint 20.3 > 21 ) .The Python changed also from 3.8 to 3.10. With Python 3.8 all worked normally.

hoodmane · 2025-03-12T15:50:50Z

I am also having the same problem in Python 3.12.

picnixz · 2025-03-12T16:03:52Z

I think there are some flaky issues due to the OS itself (@barneygale maybe you have some idea?) I can suggest using ignore_errors=True to bypass errors or use an explicitly onexc handler to have a better control on the exceptions being ignored.

hoodmane · 2025-03-13T11:14:01Z

Thanks for the suggestion. What I currently have is:

            try:
                shutil.rmtree(self.build_dir)
            except OSError as e:
                if e.strerror == "Directory not empty":
                    # Not sure why this happens, but trying again seems to fix it usually?
                    shutil.rmtree(self.build_dir)
                else:
                    raise

which seems to reduce the frequency of the problem but not eliminate it entirely. Maybe I'll add a sleep for a few milliseconds before trying again, and do three tries. I bet that will be good enough, but I'll have to see.

hoodmane · 2025-03-13T11:19:02Z

So maybe:

def retrying_rmtree(d):
    for _ in range(3):
        try:
            return shutil.rmtree(d)
        except OSError as e:
            if e.strerror == "Directory not empty":
                # wait a bit and try again up to 3 tries
                time.sleep(0.01)
            else:
                raise
    raise RuntimeError(f"shutil.rmtree('{d}') failed with ENOTEMPTY three times")

picnixz · 2025-03-13T16:00:18Z

It's indeed a possibility to have a sleep/retry but I eventually think there's no real issue with shutil.rmtree. It's just that OSErrors are raised by default unless ignore_errors=True or an exception handler is specified. I'm going to close this one as wont fix unless someone can find a faithful reproducer which would highlight an issue in shutil.rmtree. However, I don't think the stdlib should do the "retry/sleep" (but we could perhaps document this)?

@barneygale WDYT? closing this one as wont fix or document the flakiness of shutil.rmtree in some cases? (or maybe there is an issue?)

picnixz · 2025-06-07T22:53:34Z

Considering the workaround with ignore_errors=True to suppress flaky issues, I'm going to close this issue as "not planned". If users are worried about suppressing errors, they can also pass an error callback to check if the error must be really ignored or not.

christophfriedrich added the type-bug An unexpected behavior, bug, or error label Dec 18, 2024

picnixz added the stdlib Python modules in the Lib dir label Dec 20, 2024

agriyakhetarpal mentioned this issue Mar 12, 2025

Try again if rmtree raises OsError: Directory not empty pyodide/pyodide-build#123

Merged

picnixz added the pending The issue will be closed if no feedback is provided label Mar 13, 2025

picnixz changed the title ~~rmtree fails due to "directory not empty" on Linux~~ rmtree sometimes fails due to "directory not empty" on Linux Mar 13, 2025

picnixz added the OS-linux label Mar 13, 2025

foreveremain mentioned this issue Mar 27, 2025

Occasional error from SlivkaQueueRunner during tests - 'Directory not empty' bartongroup/slivka#155

Open

picnixz closed this as not planned Won't fix, can't repro, duplicate, stale Jun 7, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

`rmtree` sometimes fails due to "directory not empty" on Linux #128076

`rmtree` sometimes fails due to "directory not empty" on Linux #128076

christophfriedrich commented Dec 18, 2024 •

edited by github-actions bot

Loading

nesergen commented Feb 27, 2025

Uh oh!

hoodmane commented Mar 12, 2025

Uh oh!

picnixz commented Mar 12, 2025

Uh oh!

hoodmane commented Mar 13, 2025 •

edited

Loading

Uh oh!

hoodmane commented Mar 13, 2025 •

edited

Loading

Uh oh!

picnixz commented Mar 13, 2025

Uh oh!

picnixz commented Jun 7, 2025

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Uh oh!

rmtree sometimes fails due to "directory not empty" on Linux #128076

rmtree sometimes fails due to "directory not empty" on Linux #128076

Comments

christophfriedrich commented Dec 18, 2024 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

nesergen commented Feb 27, 2025

Uh oh!

hoodmane commented Mar 12, 2025

Uh oh!

picnixz commented Mar 12, 2025

Uh oh!

hoodmane commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hoodmane commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

picnixz commented Mar 13, 2025

Uh oh!

picnixz commented Jun 7, 2025

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

`rmtree` sometimes fails due to "directory not empty" on Linux #128076

`rmtree` sometimes fails due to "directory not empty" on Linux #128076

christophfriedrich commented Dec 18, 2024 •

edited by github-actions bot

Loading

hoodmane commented Mar 13, 2025 •

edited

Loading

hoodmane commented Mar 13, 2025 •

edited

Loading