8000 DOC: add doc on ExtensionArray and extending pandas by TomAugspurger · Pull Request #19936 · pandas-dev/pandas · GitHub
[go: up one dir, main page]

Skip to content

DOC: add doc on ExtensionArray and extending pandas #19936

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Mar 6, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Updated docs
  • Loading branch information
TomAugspurger committed Feb 28, 2018
commit 0842fcf12b07065ca20743927b0cb81506f9e585
27 changes: 27 additions & 0 deletions doc/source/ecosystem.rst
Original file line number Diff line number Diff line change
Expand Up @@ -262,3 +262,30 @@ Data validation

Engarde is a lightweight library used to explicitly state your assumptions abour your datasets
and check that they're *actually* true.

.. _ecosystem.extensions:

Extension Data Types
--------------------

`cyberpandas`_
~~~~~~~~~~~~~~

Cyberpandas provides an extension type for storing arrays of IP Addresses. These
arrays can be stored inside pandas' Series and DataFrame.

.. _ecosystem.accessors:
------------------------

A directory of projects providing extension accessors. This is for users to
discover new accessors and for libraries authors to coordinate on the namespace.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

libraries -> library


============== ========== =================
Library Accessor Classes
============== ========== =================
`cyberpandas`_ ``ip`` Series
`pdvega`_ ``vgplot`` Series, DataFrame
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could put double backticks around Series and DataFrame in this table

============== ========== =================

.. _cyberpandas: https://cyberpandas.readthedocs.io/en/latest
.. _pdvega: https://jakevdp.github.io/pdvega/
32 changes: 26 additions & 6 deletions doc/source/extending.rst
Original file line number Diff line number Diff line change
Expand Up @@ -52,18 +52,24 @@ This can be a convenient way to extend pandas objects without subclassing them.
If you write a custom accessor, make a pull request adding it to our
:ref:`ecosystem` page.

Extension Arrays
----------------
.. _extending.extension-types:

Extension Types
---------------

Pandas defines an interface for implementing data types and arrays that *extend*
NumPy's type system. Pandas uses itself for some types that aren't built into
NumPy (categorical, period, interval, datetime with timezone).
NumPy's type system. Pandas iteself uses the extension system for some types
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iteself --> itself

that aren't built into NumPy (categorical, period, interval, datetime with
timezone).

Libraries can define an custom array and data type. When pandas encounters these
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

an -> a

objects, they will be handled properly (i.e. not converted to an ndarray of
objects). Many methods like :func:`pandas.isna` will dispatch to the extension
type's implementation.

If you're building a library that implements the interface, please publicize it
on :ref:`ecosystem.extensions`.

The interface consists of two classes.

``ExtensionDtype``
Expand All @@ -81,10 +87,24 @@ See ``pandas/core/dtypes/base.py`` for interface definition.
``ExtensionArray``
""""""""""""""""""

This is the main object.
This class provides all the ar 8000 ray-like functionality. ExtensionArrays are
limited to 1 dimension. An ExtensionArray is linked to an ExtensionDtype via the
``dtype`` attribute.

Pandas makes no restrictions on how an extension array is created via its
``__new__`` or ``__init__``, and puts no restrictions on how you store your
data. We do require that your array be convertible to a NumPy array, even if
this is relatively expensive (as it is for ``Categorical``).

They may be backed by none, one, or many NumPy ararys. For example,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ararys --> arrays

``pandas.Categorical`` is an extension array backed by two arrays,
one for codes and one for categories. An array of IPv6 address may
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

address --> addresses

be backed by a NumPy structured array with two fields, one for the
lower 64 bits and one for the upper 64 bits. Or they may be backed
by some other storage type, like Python lists.

See ``pandas/core/arrays/base. 8000 py`` for the interface definition.
See ``pandas/core/arrays/base.py`` for the interface definition. The docstrings
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can do a direct reference to code here (e.g. the github link)

and comments contain guidance for properly implementing the interface.

.. _ref-subclassing-pandas:

Expand Down
5 changes: 5 additions & 0 deletions doc/source/internals.rst
5403
Original file line number Diff line number Diff line change
Expand Up @@ -107,3 +107,8 @@ containers (``Index`` classes and ``Series``) we have the following convention:

So, for example, ``Series[category]._values`` is a ``Categorical``, while
``Series[category]._ndarray_values`` is the underlying codes.

Subclassing pandas Data Structures
----------------------------------

This section has been moved to :ref:`ref-subclassing-pandas`.
0