Description
Description
First of all, sorry if this is something that has been already replied to. I searched "tests" in the issues (open and old) and couldn't find any useful information.
I've been unable to understand how to deal with tests involving a spark session with a specific configuration. My code instances a SparkSession with a configuration that I would like to test in my CI pipeline (check that the remote HDFS warehouse is accessible and so on). I'm using local as the resource manager for the sake of simplicity.
If I write a test using xUnit to test this service, I'm getting an error from the JVM Bridge when building the spark session:
System.Exception : JVM method execution failed: Nonstatic method 'getOrCreate' failed for class '1' when called with no arguments
I've seen you have E2E tests with a fixture that I tried to replace with a docker container listening on the port 5567. The output in the container logs show nothing when running the test (but if I don't set up the container the test fails because the port 5567 is not listening).
However, if I run the same test inside the container (sharing a volume with my Test.dll) it works perfectly fine. I'm not sure if what I'm trying to do in my local environment should work or if I am missing something.
Describe the solution you'd like
I think it would be nice to provide a brief explanation in the docs to be able to deal with tests that need to test the integration for a specific implementation of this library.