Jython 3 MVP and Roadmap #21
Conversation
|
Ruby (or Jekyll) seem unhappy with my available machine(s), so I haven't been able to test these changes, but the menu changes seem low-risk and the Markdown renders ok on GitHub. Anyway, It's the content I'd like to focus on, here or on the list. |
| * Command-line version. | ||
| * MVP: Launch script or other wrapper. (Generated by `jlink`? C?) | ||
| * The command you run *is* the interpreter, not a launcher. | ||
| * JNI appears to [support this readily]( |
There was a problem hiding this comment.
I once explored this approach, see https://github.com/Stewori/LiJy-launch (and consider to link it here).
However, it incorporates some (non-trivial) code taken from the Java-launcher source, resulting in GPL license with classpath exception.
There was a problem hiding this comment.
Gosh that's complicated, and it worries me because you know a lot more than I about the native interface.
In my ignorance, I would have followed this: https://www.developer.com/java/data/how-to-create-a-jvm-instance-in-jni.html, where the target class would be org/python/util/jython, letting Jython do all the argument processing. The messiness is recompilation for each platform.
However, there is jlink to fall back on and I'm suggesting a native launcher is not MVP.
There was a problem hiding this comment.
The procedure described in https://www.developer.com/java/data/how-to-create-a-jvm-instance-in-jni.html is also at the core of java.exe and LiJyLaunch. However, it appears that java.exe has to do a lot of sanetizing before args are handled to the JNI api. E.g. jdb mode requires args in a special form, wildcard processing is done in the launcher, some system properties are detected, some memory stuff is configured and a bunch of things I do not understand. LiJy inherits this annoying complexity from java.exe's source code. Since the jython.py launcher used to call java.exe, taking this stuff from java.exe source was the only feasile way to provide the same behaviour regarding support for JVM command line args as in jython.py. That java.exe was from Java 8, maybe things changed since then, I never had a look again.
But actually, taking the source from java.exe is a rather painless solution to achieve this goal except for license issues. The easiest way would be to write a dual license, i.e. having the launcher licensed separately. GNU with classpath exception is not too bad. Or we can ask Oracle for permission to relicense this particular code. It is not very much code and not especially genuine. It rather is an annoying and useless work to rewrite/reinvent it just to escape the license.
However, it may be possible to link the required code dynamically from java.exe, see https://stackoverflow.com/questions/39131185/can-a-file-both-be-an-executable-exe-and-a-dynamic-link-library-dll-at-the-s. This would also be preferable for maintenance against upstream changes of java.exe. (I don't know if this approach is actually applicable to java.exe.)
Then jython.exe would link it dynamically and might not be subject to the viral license. I don't know if classpath exception also applies to dynamic linking, but otherwise dynamic linking of JNI would be subject to GNU with cp exception as well. So the approach in https://www.developer.com/java/data/how-to-create-a-jvm-instance-in-jni.html would inherently require relicensing then.
| * Only performance-critical modules are hand-crafted in Java. | ||
| * Modules that take advantage of Java libraries call them from Python. | ||
| * Enthusiasm for writing in Java should be directed to re-implementing | ||
| popular libraries that broaden Jython use (`numpy`, etc.). |
There was a problem hiding this comment.
Maybe mention ctypes, cffi and a cython-backend here.
There was a problem hiding this comment.
I can mention them, but what would we like actually to say?
More broadly, I know calling modules written in C would be useful, but until ideas of the pure Java core settle, I find it difficult to say what's involved. So not MVP, but interested in your thoughts now and when that settling has happened.
There was a problem hiding this comment.
Calling C by pure Java code won't be feasible until Project Panama lands. Currently one inherently has to write platform dependent C glue code or use something like JNA or JNR which is just the same with the only difference being that it was written by others. ctypes, cffi and cython are JNA/JNR-like libs for Python. Having Jython-versions would enable tons of Python modules at a time. This is still not sufficient for NumPy and SciPy but is a requirement. All three are used by NumPy and SciPy. So if you want to support scientific code, getting ctypes, cffi and cython working would be the first step.
That said, each of these is a huge project on its own. So it is likely a better approach to support https://github.com/hpyproject/hpy and hope that these libraries will be made HPy compatible as well. I know that cython announced HPy support. But then, supporting HPy is not feasible with pure Java code either.
There was a problem hiding this comment.
Thanks for this I think it really helpful to publish this. Some minor comments but the navbar bit needs to be fixed before merging. Maybe just have 2 links for now, I think sub menus wouldn't be too hard but its probably not worth worrying about now. If you enable GH pages on your fork they will render a page you can view to check the content.
|
|
||
| ## Performance | ||
|
|
||
| * MVP: speed comparable with CPython (say 2x either way). |
There was a problem hiding this comment.
I'm not sure how critical this is, especially for a MVP. While its nice to be close to CPython in performance (or faster) I don't think many users of Jython this is a requirement. For people requiring the highest performance its unlikely Jython will ever be the interpreter of choice, however Jythons Java integration is compelling even at a lower performance
There was a problem hiding this comment.
I agree in general for MVP. Long-term, I'd like scientific computing to be a proposition without C extensions, if we can manage it.
By some measures, Jython 2 is somewhat faster than CPython, although start-up definitely gives the opposite impression. Full use of the JVM's dynamic language features in JRuby had a massive effect on performance. In my toy, and almost certainly in Jython 3, the byte code interpreter uses MethodHandle but not CallSite. Generated JVM code would use invokedynamic call sites fully, and I'm hoping for something good.
There's a real danger in setting any kind of target that we spend too much time micro-optimising the byte-code interpreter, when a JVM language is the goal. At the same time, if the basic operations of the abstract object API are much slower than in CPython, we're probably laying the wrong foundation. I could be talked down from the particular numbers.
In some simple loops and calls in my toy version I observe about 40% the speed of CPython. It's a bit disappointing, considering how succinct the Java is, but I'm resisting the temptation to delve further.
There was a problem hiding this comment.
Long-term, I'd like scientific computing to be a proposition without C extensions, if we can manage it.
If you talk about Java-ports of NumPy and SciPy, I estimate each one is a project comparable in complexity to Jython itself. In addition one still has to link blas and lapack, which inherently require C glue code. This can be based on existing projects like jblas, but then again, it still uses platform-dependent C code, just written by others. Project Panama may solve this but AFAIK it has not even a planned release date.
|
Long enough for comments, I think. |
Some ideas on MVP collated from our conversations, and an attempt at a roadmap.