Suppress FileNotFoundError when deleting keys in the obstore adapter #3140

lukasbindreiter · 2025-06-18T08:23:24Z

Suppress FileNotFoundError in delete operation of the obstore adapter.

Since some underlying stores allow this, while others raise an error, supressing the error results in consistent behaviour across all stores. This also mimics how the same thing is implemented in the fsspec adapter.

Fixes #3136

TODO:

Add unit tests and/or doctests in docstrings
Changes documented as a new file in changes/
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

dstansby · 2025-06-18T08:55:25Z

Thanks for the PR/fix! This is slightly surprising behaviour to me - I would expect trying to delete a non-existent file to raise an error. But I guess if this is existing behaviour/design across other stores, this is good.

To be sure that we're getting consistent behaviour across stores, could you move the test you added to the StoreTests class, so it's run across all stores that are tested?

d-v-b · 2025-06-18T08:57:33Z

I think a better long-term solution would be for the stores to all raise the same exception when attempting to delete a missing key, and for the caller of the store method to handle that exception specifically in the context of attempting to remove an empty chunk. But this fix also works!

kylebarron · 2025-06-18T15:10:17Z

I think a better long-term solution would be for the stores to all raise the same exception when attempting to delete a missing key

From the obstore side I don't have total control over that. The upstream s3 implementation does not raise an error: https://docs.rs/object_store/latest/object_store/trait.ObjectStore.html#method.delete_stream

lukasbindreiter · 2025-06-23T08:36:47Z

Thanks for the PR/fix! This is slightly surprising behaviour to me - I would expect trying to delete a non-existent file to raise an error. But I guess if this is existing behaviour/design across other stores, this is good.

Yes this seems to be the behaviour across stores:

local store uses path.unlink(missing_ok=True)
https://github.com/zarr-developers/zarr-python/blob/main/src/zarr/storage/_local.py#L218
memory store suppresses KeyError
https://github.com/zarr-developers/zarr-python/blob/main/src/zarr/storage/_memory.py#L143
fsspec store also suppresses FileNotFoundError:
https://github.com/zarr-developers/zarr-python/blob/main/src/zarr/storage/_fsspec.py#L359

so I believe it makes sense to do the same for obstore as well.

And when thinking about the main purpose of the function (to clear chunks) this behaviour makes sense - it seems the function isn't really about deleting an object, instead it is to make sure that an object doesn't exist in the store.

To be sure that we're getting consistent behaviour across stores, could you move the test you added to the StoreTests class, so it's run across all stores that are tested?

Good idea, I've moved the test now to StoreTests 👍

Across stores is a bit ambiguous in this context though - there is the zarr adapter stores (fsspec, local, memory, obstore) - the test now runs across all of those.

However the issue / bugfix was only for the obstore adapter, since there is inconsistent behaviour across obstore stores (S3, GCS, Azure, Local, InMemory) - but as @kylebarron has mentioned that seems to be out of control of obstore itself.

src/zarr/storage/_obstore.py

maxrjones

My approval doesn't mean anything really, but I'm submitting it because I ran into this issue and tested your branch locally to confirm that it solved the problem. I also made two code suggestions to apply Kyle's recommendation for convenience.

src/zarr/storage/_obstore.py

kylebarron · 2025-06-23T23:39:42Z

In developmentseed/obstore#487 I propose deprecating the obstore.exceptions.NotFoundError class

lukasbindreiter · 2025-06-24T08:06:44Z

Thanks @maxrjones and @kylebarron for the review - I've now updated the comment and removed the obstore.exceptions.NotFoundError again

lukasbindreiter · 2025-06-25T09:31:37Z

@d-v-b Thanks for enabling the merge.

I checked the failing test for the windows-latest py3.11 run, which is this one.

def test_sync_timeout() -> None:
    duration = 0.02

    async def foo() -> None:
        await asyncio.sleep(duration)

    with pytest.raises(asyncio.TimeoutError):
        sync(foo(), timeout=duration / 10)

It seems completely unrelated to the changes in the PR, probably a flaky failure? Could you potentially retry it?

d-v-b · 2025-06-25T10:55:42Z

@lukasbindreiter the flaky test theory seems valid! thanks for this contribution.

(I accidentally posted this in #3136 (comment))

d-v-b · 2025-06-30T09:01:13Z

@meeseeksdev backport to 3.0.9

…eting keys in the obstore adapter

…he obstore adapter (#3180) Co-authored-by: Lukas Bindreiter <lukas.bindreiter@tilebox.com>

* Enable tests on 3.0.x branch (#3135) * Changelog for 3.0.9 * Backport PR #3149: Add GroupNotFound error to API docs (#3179) Co-authored-by: David Stansby <dstansby@gmail.com> * Backport PR #3140: Suppress FileNotFoundError when deleting keys in the obstore adapter (#3180) Co-authored-by: Lukas Bindreiter <lukas.bindreiter@tilebox.com> * Backport PR #3138: Add with_read_only() convenience method to store (#3181) Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> * Create read only copy if needed when opening a store path (#3156) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening Co-authored-by: Tom Nicholas <tom@earthmover.io> * Improve code coverage --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> (cherry picked from commit 5731c6c) * Create read only copy if needed when opening a store path (#3156) (#3182) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py * Update src/zarr/storage/_common.py * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening * Improve code coverage --------- (cherry picked from commit 5731c6c) Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> * Remove breaking check about `auto_mkdir` for FSSpecStore (#3193) * Remove breaking check from _make_async * Update expected error * Change import structure to protect against AttributeError * changelog * add test to ensure that we can create a read-only copy of the store with auto_mkdir=False * only test if the async wrapper is available --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> (cherry picked from commit 5a24487) * Remove breaking check about `auto_mkdir` for FSSpecStore (#3193) (#3203) * Remove breaking check from _make_async * Update expected error * Change import structure to protect against AttributeError * changelog * add test to ensure that we can create a read-only copy of the store with auto_mkdir=False * only test if the async wrapper is available --------- (cherry picked from commit 5a24487) Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> * Add missing import for AsyncFileSystemWrapper for `_make_async` in `_fsspec.py` (#3195) * Add missing import for AsyncFileSystemWrapper in `_fsspec.py` * Add missing changelog entry for AsyncFileSystemWrapper import fix * Move AsyncFileSystemWrapper import past the version check in `_fsspec.py` * Add newline after AsyncFileSystemWrapper import in `_fsspec.py` * Simplify import statement for AsyncFileSystemWrapper in `_fsspec.py` --------- Co-authored-by: Altay Sansal <altay.sansal@tgs.com> (cherry picked from commit 97aa42f) * Auto backport of pr 3195 on 3.0.10 (#3204) * Remove breaking check about `auto_mkdir` for FSSpecStore (#3193) * Remove breaking check from _make_async * Update expected error * Change import structure to protect against AttributeError * changelog * add test to ensure that we can create a read-only copy of the store with auto_mkdir=False * only test if the async wrapper is available --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> (cherry picked from commit 5a24487) * Add missing import for AsyncFileSystemWrapper for `_make_async` in `_fsspec.py` (#3195) * Add missing import for AsyncFileSystemWrapper in `_fsspec.py` * Add missing changelog entry for AsyncFileSystemWrapper import fix * Move AsyncFileSystemWrapper import past the version check in `_fsspec.py` * Add newline after AsyncFileSystemWrapper import in `_fsspec.py` * Simplify import statement for AsyncFileSystemWrapper in `_fsspec.py` --------- Co-authored-by: Altay Sansal <altay.sansal@tgs.com> (cherry picked from commit 97aa42f) --------- Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: Altay Sansal <tasansal@users.noreply.github.com> Co-authored-by: Altay Sansal <altay.sansal@tgs.com> * 3.0.9 release notes (#3183) * Create read only copy if needed when opening a store path (#3156) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening Co-authored-by: Tom Nicholas <tom@earthmover.io> * Improve code coverage --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> (cherry picked from commit 5731c6c) * release notes --------- Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> * release notes --------- Co-authored-by: Lumberbot (aka Jack) <39504233+meeseeksmachine@users.noreply.github.com> Co-authored-by: Lukas Bindreiter <lukas.bindreiter@tilebox.com> Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> Co-authored-by: Altay Sansal <tasansal@users.noreply.github.com> Co-authored-by: Altay Sansal <altay.sansal@tgs.com>

github-actions bot added needs release notes Automatically applied to PRs which haven't added release notes and removed needs release notes Automatically applied to PRs which haven't added release notes labels Jun 18, 2025

lukasbindreiter mentioned this pull request Jun 18, 2025

FileNotFoundError when writing an array that contains empty chunks #3136

Closed

lukasbindreiter closed this Jun 23, 2025

lukasbindreiter force-pushed the main branch from 2ec553c to 2911be8 Compare June 23, 2025 08:16

lukasbindreiter added 3 commits June 23, 2025 10:17

Suppress FileNotFoundError when deleting keys in the obstore adapter

120a6bc

Add unit test

c6ded01

Document bugfix changes

90c6c02

lukasbindreiter reopened this Jun 23, 2025

Run delete nonexistent key test across all stores

e740027

kylebarron reviewed Jun 23, 2025

View reviewed changes

src/zarr/storage/_obstore.py Outdated Show resolved Hide resolved

maxrjones approved these changes Jun 23, 2025

View reviewed changes

src/zarr/storage/_obstore.py Outdated Show resolved Hide resolved

src/zarr/storage/_obstore.py Outdated Show resolved Hide resolved

Suppress only builtins.FileNotFoundError in obstore.delete adapter

76a0497

d-v-b approved these changes Jun 24, 2025

View reviewed changes

Merge branch 'main' into main

30af240

d-v-b enabled auto-merge (squash) June 24, 2025 19:45

Merge branch 'main' into main

a6fa2c5

d-v-b merged commit 27615fd into zarr-developers:main Jun 25, 2025
30 checks passed

d-v-b added this to the 3.0.9 milestone Jun 30, 2025

meeseeksmachine pushed a commit to meeseeksmachine/zarr-python that referenced this pull request Jun 30, 2025

Backport PR zarr-developers#3140: Suppress FileNotFoundError when del…

ab11eb5

…eting keys in the obstore adapter

meeseeksmachine mentioned this pull request Jun 30, 2025

Backport PR #3140 on branch 3.0.9 (Suppress FileNotFoundError when deleting keys in the obstore adapter) #3180

Merged

d-v-b pushed a commit that referenced this pull request Jun 30, 2025

Backport PR #3140: Suppress FileNotFoundError when deleting keys in t…

e6bb1ae

…he obstore adapter (#3180) Co-authored-by: Lukas Bindreiter <lukas.bindreiter@tilebox.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Suppress FileNotFoundError when deleting keys in the obstore adapter #3140

Suppress FileNotFoundError when deleting keys in the obstore adapter #3140

Uh oh!

lukasbindreiter commented Jun 18, 2025 •

edited

Loading

Uh oh!

dstansby commented Jun 18, 2025

Uh oh!

d-v-b commented Jun 18, 2025

Uh oh!

kylebarron commented Jun 18, 2025

Uh oh!

lukasbindreiter commented Jun 23, 2025

Uh oh!

Uh oh!

maxrjones left a comment

Uh oh!

Uh oh!

Uh oh!

kylebarron commented Jun 23, 2025

Uh oh!

lukasbindreiter commented Jun 24, 2025

Uh oh!

lukasbindreiter commented Jun 25, 2025

Uh oh!

Uh oh!

d-v-b commented Jun 25, 2025

Uh oh!

d-v-b commented Jun 30, 2025

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Uh oh!

Suppress FileNotFoundError when deleting keys in the obstore adapter #3140

Suppress FileNotFoundError when deleting keys in the obstore adapter #3140

Uh oh!

Conversation

lukasbindreiter commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dstansby commented Jun 18, 2025

Uh oh!

d-v-b commented Jun 18, 2025

Uh oh!

kylebarron commented Jun 18, 2025

Uh oh!

lukasbindreiter commented Jun 23, 2025

Uh oh!

Uh oh!

maxrjones left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kylebarron commented Jun 23, 2025

Uh oh!

lukasbindreiter commented Jun 24, 2025

Uh oh!

lukasbindreiter commented Jun 25, 2025

Uh oh!

Uh oh!

d-v-b commented Jun 25, 2025

Uh oh!

d-v-b commented Jun 30, 2025

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

lukasbindreiter commented Jun 18, 2025 •

edited

Loading