8000 df.sample random_state behavior across processes · Issue #29221 · pandas-dev/pandas · GitHub
[go: up one dir, main page]
Skip to content

df.sample random_state behavior across processes #29221

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mstrofbass opened this issue Oct 25, 2019 · 4 comments
Closed

df.sample random_state behavior across processes #29221

mstrofbass opened this issue Oct 25, 2019 · 4 comments

Comments

@mstrofbass
Copy link

Code Sample, a copy-pastable example if possible

import multiprocessing as mp
import pandas as pd

def sample(df):
    df_sampled = df.sample(df.shape[0], replace=True)
    print(df_sampled.index)

if __name__ == '__main__':
    with mp.Pool(4) as p:
        df1 = pd.DataFrame({
            'r1': [0, 1, 2, 3, 4, 5],
            'r2': [0, 1, 3, 4, 1, 9]
        })

        df2 = pd.DataFrame({
            'r1': [0, 39, 5, 3, 4, 5],
            'r2': [0, 1, 1, 9, 4, 4]
        })

        plist = [df1, df2]

        p.map( sample, plist )

Problem description

When you run the above code, the samples generated are identical. After a significant amount of time debugging this, I finally came to the conclusion that it's probably related to the fact that numpy copies the RandomState from the parent process to the child processes, so the random seed used by the two child processes are the same (see: numpy/numpy#9650).

On a similar, but more complicated example, the state wasn't necessarily the same across all processes, but would be shared between subsets of processes (e.g., there might be three states shared by six child processes). So I'm definitely not sure exactly what's going on under the hood, but it's definitely confusing.

I did manage to figure out that it can be solved by explicitly passing a new random state too sample().

I can't find any reference to this anywhere, so I don't know if it's expected behavior or a bug, and so I don't actually know what the problem is. I would suggest that if it's not a bug then there is a documentation problem because this seems unintuitive. If I call df.sample twice in a row in a single thread (without setting the random_state param), I get different samples, so it seems logical that if I call df.sample twice in different threads it would behave similarly and produce two different samples. Thus, I'd suggest that if it's expected behavior it should be explicitly documented somewhere accessible.

I'll also note that this seems to be an issue even if you create the data frame in the child process, so I'm guessing Pandas is using some kind of global RandomState.

This is, arguably, different than the numpy issue inasmuch as Pandas hides the internal implementation around the random seeding, making it much more difficult to debug the actual issue. So while I understand the reasoning behind leaving the behavior as is in the numpy case, I'm not sure their justifications are as strong here.

I'll go ahead and apologize if this is either noted somewhere and I missed it or something that should be understood by me. I spent a decent amount of time trying to solve this issue and couldn't find this behavior documented anywhere and I'm not well-versed in Pandas, so if the problem is on my end, I'm sorry.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.7.3.final.0
python-bits: 64
OS: Darwin
OS-release: 18.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: en_US.UTF-8
pandas: 0.24.2
pytest: None
pip: 19.2.3
setuptools: 41.2.0
Cython: None
numpy: 1.16.4
scipy: 1.3.1
pyarrow: 0.12.0
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.8.0
pytz: 2019.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 3.1.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None
@jreback
Copy link
Contributor
jreback commented Oct 25, 2019

it’s right there in the doc string: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sample.html?highlight=sample#pandas.DataFrame.sample

multi processing behavior of this is out of scope for pandas
as numpy provides these objects

PR ok if you want to enhance the docs with a reference

@jreback
Copy link
Contributor
jreback commented Oct 25, 2019

numpy/numpy#9650

this is defined though subtle b 8000 ehavior in numpy

@jreback jreback closed this as completed Oct 25, 2019
@mstrofbass
Copy link
Author

"it’s right there in the doc string"

What's right in the doc string?

@mstrofbass
Copy link
Author

numpy/numpy#9650

this is defined though subtle behavior in numpy

Thanks for providing a reference I already provided. Very helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jreback @mstrofbass
0