Fix GPU tests for size>=2 #60
Conversation
|
You should make the required sync call in the In short, I believe the following 8000 patch should be very close to all what you need: diff --git a/test/arrayimpl.py b/test/arrayimpl.py
index 5df8cd9..e6b0dbe 100644
--- a/test/arrayimpl.py
+++ b/test/arrayimpl.py
@@ -327,6 +327,10 @@ if cupy is not None:
else:
self.array[:] = cupy.asarray(arg, typecode)
+ def as_raw(self):
+ cupy.cuda.get_current_stream().synchronize()
+ return self.array
+
@property
def address(self):
return self.array.__cuda_array_interface__['data'][0] |
|
Beware of #61, you should rebase your branch and update this PR. |
|
@leofang Did you have a chance to try my suggestion? |
Co-authored-by: Lisandro Dalcin <dalcinl@gmail.com>
Sorry for late reply @dalcinl, I believe you're absolutely right here. I overlooked that |
|
I guess these Open MPI test failures are regressions, right? I mean, they used to work before, right? What I do not like about using a plain |
|
Frankly I don't know if they are regressions or me not doing the job right in the very beginning...😅 Look, we just added the synchronization to ensure the correctness, so perhaps previously we simply did it wrong but got lucky? My take would be to merge as is, and revisit it later if we want to adopt cirun.io to run serious GPU tests on the CI. Currently the new skips are only in effect when we run the GPU tests locally. |
Work in progress.This PR aims to fix the known test errors when there are more than 1 processes. Now multi-process tests can run fine on the same GPU (verified up to
mpirun -n 4). A typical symptom of such errors is random outcomes, which happen because the input arrays are not yet fully ready before MPI accesses their buffers via CUDA Array Interface.