Support Spark 3.3/JDK8 #3043
Replies: 1 comment 11 replies
-
|
Hi @xumingming! We chose to drop JDK 8 support to keep up with a more recent release of Arrow Java, among other reasons. You can see the discussion at the time here: |
Beta Was this translation helpful? Give feedback.
-
|
@xumingming thank you . We are always looking for new contributors to the comet project and there are a bunch of 'good first issues' I can guide you to work on. |
Beta Was this translation helpful? Give feedback.
-
The resource is not only developers/reviewers, but also CI resources. Our CI matrix is already quite large, and running Spark's test suites for 3.4, 3.5. and 4.0 is typically the bottleneck. The difference in Spark behavior between 3.3 and 4.0 (which we'd like to start treating as a primary target for Comet) is also such that maintaining that many shims and conditional codepaths is concerning to me. Lastly, would you require JDK8 support or are you just asking about 3.3 with JDK 11+? AFAIK Comet cannot support JDK8 anymore as Arrow Java has dropped support for it, and we'd have to backport critical behavior and bug fixes to an older fork of Arrow Java, which seems like a non-starter to me. |
Beta Was this translation helpful? Give feedback.
-
|
@xumingming Unfortunately, I have to agree with @mbutrovich that adding support for Spark 3.3 at this point, given the current number of maintainers, just isn't realistic. If we had more maintainers, then I think it could make sense. Have you tested with a version of Comet that does support 3.3? I would suggest starting there to see if Comet is even viable for you. Adding support for JDK 8 is not possible because the Java implementation of Apache Arrow no longer supports it. Out of interest, what are the main issues preventing you from upgrading to Spark 3.4 or later? |
Beta Was this translation helpful? Give feedback.
-
|
It is also worth noting that Spark 3.3 reached EOL more than 2 years ago, and Spark 3.5 reaches EOL in about 3 months. I would suggest upgrading to pick up important bug fixes and performance improvements. |
Beta Was this translation helpful? Give feedback.
-
|
@andygrove the main issues for upgrading Spark is that we are not justing using community Spark 3.3 directly, we have made many changes to support customized SQL grammar(to be compatible with another engine), to support customized data source etc. The upgrading itself involves a lot of coordination and work. Your advice of trying an old version of Comet that support Spark 3.3 is inspiring and I will try it to see if Comet gives a good perf result. @mbutrovich @andygrove I get that supporting JDK8 is a not easy thing and seems impossible to maintain for Comet. Adding support for Spark 3.3 + JDK11 alone is also desirable and would help a lot. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
We are currently using Spark 3.3/JDK8 in production, don't whether is it possible to support Spark 3.3/JDK8 in comet?
Beta Was this translation helpful? Give feedback.
All reactions