ENH: Add madvise for memmap objects #24489
Conversation
Closes numpygh-13172 [skip ci]
|
Using I am not quite sure what happens for Not sure I feel we need to clean up the error for Python (Python's mmap should maybe do this), but OTOH, it doesn't see terrible?
Right, but this is an existing problem that I don't think we can do anything about. |
There was a problem hiding this comment.
Sorry we never got back to this, there was a new issue that may have been related. By now, madvise always exists (but I am not sure if the method exists on all systems, so the error is probably still right).
I don't really remember where we seemed a bit stuck now?! In the end the only slight complexity seems behavior for views (and if the error is a bit annoying, that isn't the end of the world).
I would be fine with views working on their memory range, but even just rejecting them for now seems also OK.
| @@ -322,10 +326,72 @@ def flush(self): | |||
| See Also | |||
| -------- | |||
| memmap | |||
| """ | |||
| if self._mmap is None: | |||
There was a problem hiding this comment.
I think this makes sense, but should double check or _mmap is propagated on views. isinstance(self.base, mmap) may also make sense actually.
| """ | ||
| if self.base is not None and hasattr(self.base, 'flush'): | ||
| self.base.flush() | ||
| if self._mmap is None: |
There was a problem hiding this comment.
I was wondering about views in that comment (if said view is also a memmap, e.g. slice).
I think that is a worry, although we could side-step it by rejecting madvise on views (again via checking if self.base is self._mmap, not 100% sure that always exists).
| "multiple of mmap.PAGESIZE.") | ||
| else: | ||
| extra_info = "" | ||
| raise OSError(e.errno, e.strerror + extra_info) |
There was a problem hiding this comment.
Maybe add from e, although I would also be OK to just put this into the documentation.
Unless this can be triggered by an offset= during mmap creation?
| madvise | ||
| Flush any changes in memory to file on disk. | ||
| When you delete a memmap object, flush is called first to write | ||
| changes to disk. |
| fp._mmap = None | ||
| expected_err_msg = ("madvise cannot be used, because the memory map it " | ||
| "is not backed by any memory.") | ||
| with pytest.raises(TypeError, match=re.escape(expected_err_msg)): |
There was a problem hiding this comment.
Wouldn't worry with escaping for that one . at the end, but fair :).
| if self._mmap is None: | ||
| raise TypeError( | ||
| "flush cannot be used, because the memory map it " | ||
| "is not backed by any memory.") |
There was a problem hiding this comment.
The error seems a bit confusing. Maybe just "not actually backed by a memory map"?
|
(If you are not interested anymore, don't hesitate to just close, maybe someone will pick it up eventually.) |
Adds madvise to memmap. See issue #13172 for more information
@seberg we discussed shortly on the sprint day last week about a
TypeErrorbeing raised when the memmap is not backed by any memory, but I could not find a reasonable case where this can happen. It is always backed when constructed. So I changed the access onbaseto_mmapso I can make test for the error, sincebaseis not writable. Then the function is also more consistent with the__getitem__function that also uses_mmap. But now the error message is misleading, because_mmapcan beNone, whilebaseis notNone, so the memmap it is still backed up. I don't think thisNonecheck actually makes sense in the current version and would remove it and switch usage everywhere tobasein memmap.What can actually happen is that the mmap is closed, and one can get a segfault because of invalid accesses (
my_memmap.base.close()using functions marked as public). Maybe we test for this instead?What do you think?