Skip to content

ENH: add a casting option 'same_value' and use it in np.astype #29129

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 28 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
cb8ea97
start implementing same_value casting
mattip Feb 17, 2025
3859a73
work through more places that check 'cast', add a TODO
Feb 17, 2025
9dc63a5
add a test, percolate casting closer to inner loops
mattip May 15, 2025
7cad6af
use SAME_VALUE_CAST flag for one inner loop variant
mattip May 29, 2025
03e9ceb
aligned test of same_value passes. Need more tests
mattip Jun 1, 2025
93b8bce
handle unaligned casting with 'same_value'
mattip Jun 1, 2025
87e2487
extend tests to use source-is-complex
mattip Jun 1, 2025
9aa1e5a
fix more interfaces to pass casting around, disallow using 'same_valu…
mattip Jun 2, 2025
953714e
raise in places that have a kwarg casting, besides np.astype
mattip Jun 2, 2025
cd3e144
refactor based on review comments
mattip Jun 9, 2025
6d6b045
CHAR_MAX,MIN -> SCHAR_MAX,MIN
mattip Jul 21, 2025
1293657
copy context flags
mattip Jul 22, 2025
d151c91
add 'same_value' to typing stubs
mattip Jul 23, 2025
6254ae5
document new feature
mattip Jul 24, 2025
6922b35
test, check exact float->int casting: refactor same_value check into …
mattip Jul 24, 2025
a323a4b
enable astype same_value casting for scalars
mattip Jul 24, 2025
aec8ea1
typo
mattip Jul 24, 2025
26c0fa1
fix ptr-to-src_value -> value casting errors
mattip Jul 25, 2025
0846081
fix linting and docs, ignore warning better
mattip Jul 25, 2025
76e01c1
gcc warning is different
mattip Jul 25, 2025
963ea05
fixes from review, typos
mattip Jul 26, 2025
4a9a498
fix compile warning ignore and make filter in tests more specific, di…
mattip Jul 27, 2025
10c4493
fix warning filters
mattip Jul 28, 2025
58a0a09
emit PyErr inside the loop
mattip Jul 31, 2025
64b8747
macOS can emit FPEs when touching NAN
mattip Jul 31, 2025
9d55847
Fix can-cast logic everywhere for same-value casts (only allow numeric)
seberg Jun 13, 2025
a800154
reorder and simplify, from review
mattip Aug 5, 2025
70d92bc
revert last commit and remove redundant checks
mattip Aug 5, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions doc/release/upcoming_changes/29129.enhancement.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
``'same_value'`` for casting by value
-------------------------------------
The ``casting`` kwarg now has a ``'same_value'`` option that checks the actual
values can be round-trip cast without changing value. Currently it is only
implemented in `ndarray.astype`. This will raise a ``ValueError`` if any of the
values in the array would change as a result of the cast, including rounding of
floats or overflowing of ints.
5 changes: 5 additions & 0 deletions doc/source/reference/c-api/array.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4452,5 +4452,10 @@ Enumerated Types

Allow any cast, no matter what kind of data loss may occur.

.. c:enumerator:: NPY_SAME_VALUE_CASTING

Allow any cast, but error if any values change during the cast. Currently
supported only in ``ndarray.astype(... casting='same_value')``
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we make this public on the C-API, I think we should be clear that this is only supported by cast safety requests and not to define a cast safety.
Should also add .. versionadded:: ... with a note that on previous numpy version it will behave the same as "unsafe".
(or you guard the define with an NPY_FEATURE_VERSION or both. I would be fine with just a note that it is effectively ignored on older versions also.)


.. index::
pair: ndarray; C-API
1 change: 1 addition & 0 deletions numpy/__init__.cython-30.pxd
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,7 @@ cdef extern from "numpy/arrayobject.h":
NPY_SAFE_CASTING
NPY_SAME_KIND_CASTING
NPY_UNSAFE_CASTING
NPY_SAME_VALUE_CASTING

ctypedef enum NPY_CLIPMODE:
NPY_CLIP
Expand Down
1 change: 1 addition & 0 deletions numpy/__init__.pxd
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,7 @@ cdef extern from "numpy/arrayobject.h":
NPY_SAFE_CASTING
NPY_SAME_KIND_CASTING
NPY_UNSAFE_CASTING
NPY_SAME_VALUE_CASTING

ctypedef enum NPY_CLIPMODE:
NPY_CLIP
Expand Down
2 changes: 1 addition & 1 deletion numpy/__init__.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -965,7 +965,7 @@ _DTypeBuiltinKind: TypeAlias = L[0, 1, 2]

_ArrayAPIVersion: TypeAlias = L["2021.12", "2022.12", "2023.12", "2024.12"]

_CastingKind: TypeAlias = L["no", "equiv", "safe", "same_kind", "unsafe"]
_CastingKind: TypeAlias = L["no", "equiv", "safe", "same_kind", "same_value", "unsafe"]

_OrderKACF: TypeAlias = L["K", "A", "C", "F"] | None
_OrderACF: TypeAlias = L["A", "C", "F"] | None
Expand Down
11 changes: 10 additions & 1 deletion numpy/_core/_add_newdocs.py
Original file line number Diff line number Diff line change
Expand Up @@ -3185,7 +3185,7 @@
'C' order otherwise, and 'K' means as close to the
order the array elements appear in memory as possible.
Default is 'K'.
casting : {'no', 'equiv', 'safe', 'same_kind', 'unsafe'}, optional
casting : {'no', 'equiv', 'safe', 'same_kind', 'same_value', 'unsafe'}, optional
Controls what kind of data casting may occur. Defaults to 'unsafe'
for backwards compatibility.

Expand All @@ -3195,6 +3195,12 @@
* 'same_kind' means only safe casts or casts within a kind,
like float64 to float32, are allowed.
* 'unsafe' means any data conversions may be done.
* 'same_value' means any data conversions may be done, but the values
must not change, including rounding of floats or overflow of ints

.. versionadded:: 2.4
Support for ``'same_value'`` was added.

subok : bool, optional
If True, then sub-classes will be passed-through (default), otherwise
the returned array will be forced to be a base-class array.
Expand All @@ -3217,6 +3223,9 @@
ComplexWarning
When casting from complex to float or int. To avoid this,
one should use ``a.real.astype(t)``.
ValueError
When casting using ``'same_value'`` and the values change or would
overflow

Examples
--------
Expand Down
3 changes: 2 additions & 1 deletion numpy/_core/code_generators/cversions.txt
Original file line number Diff line number Diff line change
Expand Up @@ -79,5 +79,6 @@
# Version 19 (NumPy 2.2.0) No change
0x00000013 = 2b8f1f4da822491ff030b2b37dff07e3
# Version 20 (NumPy 2.3.0)
# Version 20 (NumPy 2.4.0) No change
0x00000014 = e56b74d32a934d085e7c3414cb9999b8,
# Version 21 (NumPy 2.4.0) Add 'same_value' casting, header additions
0x00000015 = e56b74d32a934d085e7c3414cb9999b8,
9 changes: 8 additions & 1 deletion numpy/_core/include/numpy/dtype_api.h
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,15 @@ typedef struct PyArrayMethod_Context_tag {

/* Operand descriptors, filled in by resolve_descriptors */
PyArray_Descr *const *descriptors;
#if NPY_FEATURE_VERSION > NPY_2_3_API_VERSION
void * _reserved;
/*
* Optional flag to pass information into the inner loop
* If set, it will be NPY_CASTING
*/
uint64_t flags;
/* Structure may grow (this is harmless for DType authors) */
#endif
} PyArrayMethod_Context;


Expand Down Expand Up @@ -144,7 +152,6 @@ typedef struct {
#define NPY_METH_contiguous_indexed_loop 9
#define _NPY_METH_static_data 10


/*
* The resolve descriptors function, must be able to handle NULL values for
* all output (but not input) `given_descrs` and fill `loop_descrs`.
Expand Down
2 changes: 2 additions & 0 deletions numpy/_core/include/numpy/ndarraytypes.h
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,8 @@ typedef enum {
NPY_SAME_KIND_CASTING=3,
/* Allow any casts */
NPY_UNSAFE_CASTING=4,
/* Allow any casts, check that no values overflow/change */
NPY_SAME_VALUE_CASTING=5,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that it really works exactly for builtin numeric types only, do you care for making it public on the C-API?

I still expect the only reasonable way to make this public will be to have the two states, which somewhat makes me want to give this a value of 8+4=12.

} NPY_CASTING;

typedef enum {
Expand Down
5 changes: 4 additions & 1 deletion numpy/_core/include/numpy/numpyconfig.h
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@
#define NPY_2_1_API_VERSION 0x00000013
#define NPY_2_2_API_VERSION 0x00000013
#define NPY_2_3_API_VERSION 0x00000014
#define NPY_2_4_API_VERSION 0x00000015


/*
Expand Down Expand Up @@ -172,8 +173,10 @@
#define NPY_FEATURE_VERSION_STRING "2.0"
#elif NPY_FEATURE_VERSION == NPY_2_1_API_VERSION
#define NPY_FEATURE_VERSION_STRING "2.1"
#elif NPY_FEATURE_VERSION == NPY_2_3_API_VERSION /* also 2.4 */
#elif NPY_FEATURE_VERSION == NPY_2_3_API_VERSION
#define NPY_FEATURE_VERSION_STRING "2.3"
#elif NPY_FEATURE_VERSION == NPY_2_4_API_VERSION
#define NPY_FEATURE_VERSION_STRING "2.4"
#else
#error "Missing version string define for new NumPy version."
#endif
Expand Down
3 changes: 2 additions & 1 deletion numpy/_core/meson.build
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,8 @@ C_ABI_VERSION = '0x02000000'
# 0x00000013 - 2.1.x
# 0x00000013 - 2.2.x
# 0x00000014 - 2.3.x
C_API_VERSION = '0x00000014'
# 0x00000015 - 2.4.x
C_API_VERSION = '0x00000015'

# Check whether we have a mismatch between the set C API VERSION and the
# actual C API VERSION. Will raise a MismatchCAPIError if so.
Expand Down
4 changes: 2 additions & 2 deletions numpy/_core/src/common/array_assign.h
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ PyArray_AssignRawScalar(PyArrayObject *dst,
NPY_NO_EXPORT int
raw_array_assign_scalar(int ndim, npy_intp const *shape,
PyArray_Descr *dst_dtype, char *dst_data, npy_intp const *dst_strides,
PyArray_Descr *src_dtype, char *src_data);
PyArray_Descr *src_dtype, char *src_data, NPY_CASTING casting);

/*
* Assigns the scalar value to every element of the destination raw array
Expand All @@ -59,7 +59,7 @@ raw_array_wheremasked_assign_scalar(int ndim, npy_intp const *shape,
PyArray_Descr *dst_dtype, char *dst_data, npy_intp const *dst_strides,
PyArray_Descr *src_dtype, char *src_data,
PyArray_Descr *wheremask_dtype, char *wheremask_data,
npy_intp const *wheremask_strides);
npy_intp const *wheremask_strides, NPY_CASTING casting);

/******** LOW-LEVEL ARRAY MANIPULATION HELPERS ********/

Expand Down
1 change: 1 addition & 0 deletions numpy/_core/src/multiarray/_multiarray_tests.c.src
Original file line number Diff line number Diff line change
Expand Up @@ -2168,6 +2168,7 @@ run_casting_converter(PyObject* NPY_UNUSED(self), PyObject *args)
case NPY_SAFE_CASTING: return PyUnicode_FromString("NPY_SAFE_CASTING");
case NPY_SAME_KIND_CASTING: return PyUnicode_FromString("NPY_SAME_KIND_CASTING");
case NPY_UNSAFE_CASTING: return PyUnicode_FromString("NPY_UNSAFE_CASTING");
case NPY_SAME_VALUE_CASTING: return PyUnicode_FromString("NPY_SAME_VALUE_CASTING");
default: return PyLong_FromLong(casting);
}
}
Expand Down
Loading
Loading
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy