8000 add Welford's algorithm for stable low-memory calculation of mean and variance · Issue #6231 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

add Welford's algorithm for stable low-memory calculation of mean and variance #6231

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
argriffing opened this issue Aug 21, 2015 · 3 comments
Open

add Welford's algorithm for stable low-memory calculation of mean and variance #6231

argriffing opened this issue Aug 21, 2015 · 3 comments

Comments

@argriffing
Copy link
Contributor

Welford's algorithm computes the mean and variance in a way that pays attention to stability while using less memory overhead than the current numpy implementation of variance or standard deviation. See http://stackoverflow.com/questions/32135572, #1696, #3685.
@jaimefrio
Copy link
Member

NumPy's current implementation has two main advantages:

  1. It is noticeably faster (Welford's has to do a division in the inner loop), and not much worse, if at all, when it comes to numerical stability.
  2. It can be coded in pure Python, while Welford's would require Cython or C.

Some time ago I rewrote Pandas' sliding variance to use Welford's, see pandas-dev/pandas#6817, and I looked really hard into putting it also in NumPy, and didn't manage to find a good way of fitting it in. How do you see this being implemented, Alex?

@argriffing
Copy link
Contributor Author

How do you see this being implemented?

I don't know enough about numpy's internals to decide this. If implementing this is impractical due to numpy's architectural constraints then I guess this issue should be closed.

@tvercaut tvercaut mentioned this issue Mar 27, 2021
Closed
@sehoffmann
Copy link

This would be a great addition to numpy; not as a replacement for np.mean and np.var, but as a standalone feature. Online calculations happen all the time (due to resource limitations), and a np.welford function would be convenient so that not everyone has to implement welford on its own over and over again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@argriffing @sehoffmann @jaimefrio
0