8000 Use fork to fix PicklingError by sudipm-mukherjee · Pull Request #660 · tomopy/tomopy · GitHub
[go: up one dir, main page]
Skip to content

Use fork to fix PicklingError #660

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
sudipm-mukherjee wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Use fork to fix PicklingError #660

sudipm-mukherjee wants to merge 1 commit into from

Conversation

@sudipm-mukherjee
Copy link
@sudipm-mukherjee sudipm-mukherjee commented Nov 22, 2025

The tests are failing for s390x with Python3.14 and the error is:

=================================== FAILURES ===================================
_________ DistortionCorrectionTestCase.test_distortion_correction_proj _________

self = <test_tomopy.test_prep.test_alignment.DistortionCorrectionTestCase testMethod=test_distortion_correction_proj>

    def test_distortion_correction_proj(self):
        test_dir = os.path.dirname(os.path.realpath(__file__))
        file_path = os.path.join(
            os.path.dirname(test_dir),'test_data','discoef.txt')
  
8000
      (xc, yc, list_fact) = load_distortion_coefs(file_path)
        assert_allclose(
>           distortion_correction_proj(
                read_file('distortion_3d.npy'), xc, yc, list_fact)[0],
            read_file('distortion_proj.npy'), rtol=1e-2)

test/test_tomopy/test_prep/test_alignment.py:69: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tomopy/prep/alignment.py:947: in distortion_correction_proj
    arr = mproc.distribute_jobs(
tomopy/util/mproc.py:295: in distribute_jobs
    with closing(mp.Pool(processes=ncore,
/usr/lib/python3.14/multiprocessing/context.py:119: in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
/usr/lib/python3.14/multiprocessing/pool.py:215: in __init__
    self._repopulate_pool()
/usr/lib/python3.14/multiprocessing/pool.py:306: in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
/usr/lib/python3.14/multiprocessing/pool.py:329: in _repopulate_pool_static
    w.start()
/usr/lib/python3.14/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
/usr/lib/python3.14/multiprocessing/context.py:300: in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
/usr/lib/python3.14/multiprocessing/popen_forkserver.py:35: in __init__
    super().__init__(process_obj)
/usr/lib/python3.14/multiprocessing/popen_fork.py:20: in __init__
    self._launch(process_obj)
/usr/lib/python3.14/multiprocessing/popen_forkserver.py:47: in _launch
    reduction.dump(process_obj, buf)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <ForkServerProcess name='ForkServerPoolWorker-2' parent=12317 initial daemon>
file = <_io.BytesIO object at 0x3ffa4eb31a0>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       _pickle.PicklingError: Can't pickle <class 'ctypes.c_float'>: it's not the same object as ctypes.c_float
E       when serializing tuple item 0
E       when serializing multiprocessing.sharedctypes.c_float_le_Array_4096 reconstructor arguments
E       when serializing multiprocessing.sharedctypes.c_float_le_Array_4096 object
E       when serializing list item 0
E       when serializing tuple item 0
E       when serializing tuple item 3
E       when serializing dict item '_args'
E       when serializing multiprocessing.context.ForkServerProcess state
E       when serializing multiprocessing.context.ForkServerProcess object

/usr/lib/python3.14/multiprocessing/reduction.py:60: PicklingError
____________________ TestDistributeJobs.test_retrieve_phase ____________________

self = <test_tomopy.test_prep.test_phase.TestDistributeJobs testMethod=test_retrieve_phase>

    def test_retrieve_phase(self):
        assert_allclose(
>           retrieve_phase(read_file('proj.npy')),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
            read_file('retrieve_phase.npy'), rtol=1e-6)

test/test_tomopy/test_prep/test_phase.py:62: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tomopy/prep/phase.py:123: in retrieve_phase
    arr = mproc.distribute_jobs(
tomopy/util/mproc.py:295: in distribute_jobs
    with closing(mp.Pool(processes=ncore,
/usr/lib/python3.14/multiprocessing/context.py:119: in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
/usr/lib/python3.14/multiprocessing/pool.py:215: in __init__
    self._repopulate_pool()
/usr/lib/python3.14/multiprocessing/pool.py:306: in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
/usr/lib/python3.14/multiprocessing/pool.py:329: in _repopulate_pool_static
    w.start()
/usr/lib/python3.14/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
/usr/lib/python3.14/multiprocessing/context.py:300: in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
/usr/lib/python3.14/multiprocessing/popen_forkserver.py:35: in __init__
    super().__init__(process_obj)
/usr/lib/python3.14/multiprocessing/popen_fork.py:20: in __init__
    self._launch(process_obj)
/usr/lib/python3.14/multiprocessing/popen_forkserver.py:47: in _launch
    reduction.dump(process_obj, buf)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <ForkServerProcess name='ForkServerPoolWorker-13' parent=12317 initial daemon>
file = <_io.BytesIO object at 0x3ff9ce13a10>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       _pickle.PicklingError: Can't pickle <class 'ctypes.c_float'>: it's not the same object as ctypes.c_float
E       when serializing tuple item 0
E       when serializing multiprocessing.sharedctypes.c_float_le_Array_6144 reconstructor arguments
E       when serializing multiprocessing.sharedctypes.c_float_le_Array_6144 object
E       when serializing list item 0
E       when serializing tuple item 0
E       when serializing tuple item 3
E       when serializing dict item '_args'
E       when serializing multiprocessing.context.ForkServerProcess state
E       when serializing multiprocessing.context.ForkServerProcess object

/usr/lib/python3.14/multiprocessing/reduction.py:60: PicklingError
_________________ StripeRemovalTestCase.test_remove_stripe_fw __________________

self = <test_tomopy.test_prep.test_stripe.StripeRemovalTestCase testMethod=test_remove_stripe_fw>

    def test_remove_stripe_fw(self):
        assert_allclose(
>           srm.remove_stripe_fw(read_file('proj.npy')),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
            read_file('remove_stripe_fw.npy'), rtol=1e-2)

test/test_tomopy/test_prep/test_stripe.py:71: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tomopy/prep/stripe.py:125: in remove_stripe_fw
    arr = mproc.distribute_jobs(tomo,
tomopy/util/mproc.py:295: in distribute_jobs
    with closing(mp.Pool(processes=ncore,
/usr/lib/python3.14/multiprocessing/context.py:119: in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
/usr/lib/python3.14/multiprocessing/pool.py:215: in __init__
    self._repopulate_pool()
/usr/lib/python3.14/multiprocessing/pool.py:306: in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
/usr/lib/python3.14/multiprocessing/pool.py:329: in _repopulate_pool_static
    w.start()
/usr/lib/python3.14/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
/usr/lib/python3.14/multiprocessing/context.py:300: in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
/usr/lib/python3.14/multiprocessing/popen_forkserver.py:35: in __init__
    super().__init__(process_obj)
/usr/lib/python3.14/multiprocessing/popen_fork.py:20: in __init__
    self._launch(process_obj)
/usr/lib/python3.14/multiprocessing/popen_forkserver.py:47: in _launch
    reduction.dump(process_obj, buf)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <ForkServerProcess name='ForkServerPoolWorker-22' parent=12317 initial daemon>
file = <_io.BytesIO object at 0x3ff9cf38090>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       _pickle.PicklingError: Can't pickle <class 'ctypes.c_float'>: it's not the same object as ctypes.c_float
E       when serializing tuple item 0
E       when serializing multiprocessing.sharedctypes.c_float_le_Array_6144 reconstructor arguments
E       when serializing multiprocessing.sharedctypes.c_float_le_Array_6144 object
E       when serializing list item 0
E       when serializing tuple item 0
E       when serializing tuple item 3
E       when serializing dict item '_args'
E       when serializing multiprocessing.context.ForkServerProcess state
E       when serializing multiprocessing.context.ForkServerProcess object

/usr/lib/python3.14/multiprocessing/reduction.py:60: PicklingError
_________________ StripeRemovalTestCase.test_remove_stripe_ti __________________

self = <test_tomopy.test_prep.test_stripe.StripeRemovalTestCase testMethod=test_remove_stripe_ti>

    def test_remove_stripe_ti(self):
        assert_allclose(
>           srm.remove_stripe_ti(read_file('proj.npy')),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
            read_file('remove_stripe_ti.npy'), rtol=1e-2)

test/test_tomopy/test_prep/test_stripe.py:76: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tomopy/prep/stripe.py:202: in remove_stripe_ti
    arr = mproc.distribute_jobs(tomo,
tomopy/util/mproc.py:295: in distribute_jobs
    with closing(mp.Pool(processes=ncore,
/usr/lib/python3.14/multiprocessing/context.py:119: in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
/usr/lib/python3.14/multiprocessing/pool.py:215: in __init__
    self._repopulate_pool()
/usr/lib/python3.14/multiprocessing/pool.py:306: in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
/usr/lib/python3.14/multiprocessing/pool.py:329: in _repopulate_pool_static
    w.start()
/usr/lib/python3.14/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
/usr/lib/python3.14/multiprocessing/context.py:300: in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
/usr/lib/python3.14/multiprocessing/popen_forkserver.py:35: in __init__
    super().__init__(process_obj)
/usr/lib/python3.14/multiprocessing/popen_fork.py:20: in __init__
    self._launch(process_obj)
/usr/lib/python3.14/multiprocessing/popen_forkserver.py:47: in _launch
    reduction.dump(process_obj, buf)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <ForkServerProcess name='ForkServerPoolWorker-24' parent=12317 initial daemon>
file = <_io.BytesIO object at 0x3ff9cfe0c70>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       _pickle.PicklingError: Can't pickle <class 'ctypes.c_float'>: it's not the same object as ctypes.c_float
E       when serializing tuple item 0
E       when serializing multiprocessing.sharedctypes.c_float_le_Array_6144 reconstructor arguments
E       when serializing multiprocessing.sharedctypes.c_float_le_Array_6144 object
E       when serializing list item 0
E       when serializing tuple item 0
E       when serializing tuple item 3
E       when serializing dict item '_args'
E       when serializing multiprocessing.context.ForkServerProcess state
E       when serializing multiprocessing.context.ForkServerProcess object

/usr/lib/python3.14/multiprocessing/reduction.py:60: PicklingError
=========================== short test summary info ============================
FAILED test/test_tomopy/test_prep/test_alignment.py::DistortionCorrectionTestCase::test_distortion_correction_proj
FAILED test/test_tomopy/test_prep/test_phase.py::TestDistributeJobs::test_retrieve_phase
FAILED test/test_tomopy/test_prep/test_stripe.py::StripeRemovalTestCase::test_remove_stripe_fw
FAILED test/test_tomopy/test_prep/test_stripe.py::StripeRemovalTestCase::test_remove_stripe_ti

The full log can be seen at https://launchpadlibrarian.net/833008584/buildlog_ubuntu-resolute-s390x.tomopy_1.15.2+ds1-3build1_BUILDING.txt.gz

@sudipm-mukherjee
Use fork to fix PicklingError
05549ef 
Signed-off-by: Sudip Mukherjee <sudipm.mukherjee@gmail.com>
@sudipm-mukherjee sudipm-mukherjee marked this pull request as ready for review November 22, 2025 23:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@sudipm-mukherjee
0