8000 PoC,WIP: Proof of concept for extending buffer protocol string by seberg · Pull Request #50 · seberg/numpy · GitHub
[go: up one dir, main page]
Skip to content

PoC,WIP: Proof of concept for extending buffer protocol string #50

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

PoC,WIP: Proof of concept for extending buffer protocol string #50

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7d80728
BUG: Fix searchsorted and CheckFromAny byte-swapping logic
seberg Mar 3, 2025
6905e2f
PoC,WIP: Proof of concept for extending buffer protocol string
seberg Mar 4, 2025
55dfd59
Add basic PoC for string dtype and typed memoryview
seberg Mar 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
PoC,WIP: Proof of concept for extending buffer protocol string 
This is a proof of concept, that allows roundtripping code such
as:
```
a = np.arange(100000).astype("T")
print(memoryview(a).format)  # just to see it
via_memview_string = np.asarray(memoryview(a))

a = np.arange(1000).astype("m8[ms]")  # or even struct "m8[...],i"
print(memoryview(a).format)  # just to see it
via_memview_timedelta = np.asarray(memoryview(a))

```
I opted to simply store the `descr` pointer directly.  This seems
just as well because the descr owns the side-car buffer.
One could still export it to non-Python (somewhat) since accessing
that side-car buffer _doesn't_ need Python API in the end (just
struct layout information).
  • Loading branch information
@seberg
seberg committed Mar 4, 2025
commit 6905e2f37b7355b6dc4245b7597c293b7ec704a4
29 changes: 29 additions & 0 deletions numpy/_core/_internal.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -627,6 +627,10 @@ def _view_is_safe(oldtype, newtype):
'X': 'function pointers',
}

_time_unit_map = ("Y", "M", "W", "<invalid>", "D", "h", "m", "s", "ms",
"us", "ns", "ps", "fs", "as", "generic")


class _Stream:
def __init__(self, s):
self.s = s
Expand Down Expand Up @@ -721,6 +725,31 @@ def __dtype_from_pep3118(stream, is_subdtype):
if stream.consume('T{'):
value, align = __dtype_from_pep3118(
stream, is_subdtype=True)
elif stream.consume("[numpy$"):
# TODO: Clearly, we would need a registration and a C callback
# probably, i.e. a slot that is called based on the name.
dtype_str = stream.consume_until("]")
module, name, *params = dtype_str.split(":", 2)
numpy_byteorder = {'@': '=', '^': '='}.get(
stream.byteorder, stream.byteorder)
if module == "numpy.dtypes":
if name == "TimeDelta64DType":
unit, num = [int(i, 16) for i in params[0].split(":")]
unit = _time_unit_map[unit]
value = dtype(f"{numpy_byteorder}m8[{num}{unit}]")
elif name == "StringDType":
# Just stores the Python object (ignores any other state).
# (We could always do this, but I think it is nice to avoid
# objects when possible..)
import ctypes
value = ctypes.cast(int(params[0], 16), ctypes.py_object).value
if type(value) != StringDType:
raise SystemError("Critical error, dtype not a StringDType.")
else:
raise NotImplementedError(f"Unknown NumPy dtype: {module}:{name}")
align = value.alignment
else:
raise NotImplementedError(f"Unknown NumPy dtype: {module}:{name}")
elif stream.next in type_map_chars:
if stream.next == 'Z':
typechar = stream.advance(2)
Expand Down
17 changes: 17 additions & 0 deletions numpy/_core/src/multiarray/buffer.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
#include "arrayobject.h"
#include "scalartypes.h"
#include "dtypemeta.h"
#include "descriptor.h"
#include "_datetime.h"

/*************************************************************************
**************** Implement Buffer Protocol ****************************
Expand Down Expand Up @@ -420,6 +422,21 @@ _buffer_format_string(PyArray_Descr *descr, _tmp_string_t *str,
if (_append_str(str, buf) < 0) return -1;
break;
}
case NPY_TIMEDELTA: {
char buf[128];
PyArray_DatetimeMetaData *meta = get_datetime_metadata_from_dtype(descr);
PyOS_snprintf(buf, sizeof(buf), "[numpy$numpy.dtypes:TimeDelta64DType:%x:%x]",
meta->base, meta->num);
if (_append_str(str, buf) < 0) return -1;
break;
}
case NPY_VSTRING: {
char buf[128];
PyOS_snprintf(buf, sizeof(buf), "[numpy$numpy.dtypes:StringDType:%zx]",
(Py_uintptr_t)descr);
if (_append_str(str, buf) < 0) return -1;
break;
}
default:
if (PyDataType_ISLEGACY(descr)) {
PyErr_Format(PyExc_ValueError,
Expand Down
Loading
0