Jython 3 MVP and Roadmap #21
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -27,3 +27,7 @@ gh-pages/ | |
| pkg/ | ||
| test/dest | ||
| tmp/* | ||
| scraps/ | ||
|
|
||
| # IDE droppings | ||
| .idea/ | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,238 @@ | ||
| --- | ||
| title: Jython 3 Features and MVP | ||
| --- | ||
| # Jython 3 Features and a Viable Minimum | ||
|
|
||
| This is a discussion document that attempts to describe, | ||
| and to some extent prioritise, | ||
| features for Jython 3, | ||
| based on suggestions collated from various voices on | ||
| [jython-dev](https://sourceforge.net/p/jython/mailman/jython-dev/) | ||
| and in off-list e-mail. | ||
|
|
||
|
|
||
| ## Positioning | ||
|
|
||
| We think people will continue to adopt and use Jython if Jython 3 ... | ||
|
|
||
| * is a modern version of Python, close to standard in its features. | ||
| * runs on a Java platform that is supported in the long-term. | ||
| * integrates cleanly with Java for access to JDK and user libraries. | ||
| * offers correct concurrency (effectively utilising available CPUs). | ||
| * allows code developed on CPython 3.x to run on Jython 3.x. | ||
| * is well-tested for release and supported for bugs. | ||
|
|
||
| The minimum viable product (MVP) is to approach *all* these targets closely, | ||
| that is, it isn't viable if it falls a long way short of any one of them. | ||
| This does not preclude the availability of immature alpha versions. | ||
| (It's a public project, so it is there for anyone to build.) | ||
|
|
||
|
|
||
| ## Version and Forms | ||
|
|
||
| * MVP: A Python 3.8 (?) close to standard in ability to run Python. | ||
| * Only essential differences (e.g. around GC, atomicity). | ||
| * The full syntax (plus some Java twists e.g. to import from Java). | ||
| * Close to the whole standard library (where use is not C-specific). | ||
|
|
||
| * Builds for Gradle/Maven ecosystem. | ||
| * MVP: Slim JAR without bundled dependencies. | ||
| * MVP: Centrally available to cite as a dependency. | ||
|
|
||
| * Provides an executable command | ||
| * MVP: `jython3` command installable on each major OS | ||
| * MVP: subset of commonly-used python3 command options | ||
| * Option-compatible with `python3` (MVP: a subset) | ||
| * Options specific to Jython (JVM options and others) | ||
|
|
||
|
|
||
| ## Performance | ||
|
|
||
| * MVP: speed comparable with CPython (say 2x either way). | ||
| * Higher performance (single-threaded) desirable but not MVP. | ||
| * Future: Work to improve speed. | ||
| * Compile Python to JVM byte code. | ||
| * Faster Python byte code interpreter. | ||
| * Have in mind performance for: | ||
| * scientific computing | ||
| * image, big data and ML libraries | ||
|
|
||
| These choices, if valid, | ||
| make a Python byte code interpreter a legitimate MVP. | ||
|
|
||
|
|
||
| ## Platforms and Environments | ||
|
|
||
| It is difficult to enumerate the possibilities in a MECE way. | ||
| It is multidimensional, although not every combination makes sense. | ||
| Which of the things in this section are MVP? | ||
|
|
||
| ### OS platforms: | ||
|
|
||
| * Windows desktop | ||
| * Linux desktop | ||
| * Mac desktop | ||
| * Raspberry Pi | ||
jeff5 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| * Android (minimum as discussed under "features") | ||
| * Risk: API gaps constrain us, or lead to a special Android version | ||
| * Small JVM devices (e.g. for IoT) | ||
|
|
||
| ### Runtime environments: | ||
|
|
||
| These are imprecise definitions, but the intention is to run everywhere | ||
| Java does, and take full advantage of each environment. | ||
|
|
||
| * Desktop | ||
| * J2EE | ||
| * Risk: Java version support | ||
| * AWS | ||
| * Azure | ||
| * IoT/embedded Java | ||
|
|
||
| At the embedded end of the spectrum, Jython is probably only attractive if | ||
| Python is not available directly (and Java is, obviously). | ||
|
|
||
| ### Significant Integrations | ||
|
|
||
| An unscientific, incomplete list based on projects we have noticed | ||
| (e.g. via a reported bug). | ||
|
|
||
| * Robot Framework | ||
| * ImageJ | ||
| * Pig | ||
| * Ghidra | ||
| * OpenHAB | ||
| * ... ? | ||
|
|
||
| There is a wider Python ecosystem that does not yet use Jython because they | ||
| depend on extensions in C. E.g. there is not much in SciPy without `numpy`. | ||
|
|
||
| How do we find the time and collaborators | ||
| to test integration into these environments? | ||
| Have we enough understanding to avoid unintentionally making it difficult? | ||
|
|
||
|
|
||
| ## Features | ||
|
|
||
| * MVP: Runs on Java 11 SE. Chosen as a minimum because: | ||
| * It is the post-Java 9 workhorse for the time being. | ||
| * It has a rich set of libraries we can exploit in the implementation. | ||
| * LTS version characterises many enterprise Java installations. | ||
| * Enterprises favour security, ease of management. | ||
| * Risk to MVP: J2EE is based on Java 8. Must explore: | ||
| * have I misunderstood this? | ||
| * non-issue if JVM byte code is the same? | ||
| * attention needed to which libraries are on the path | ||
|
|
||
| * Not MVP: Runs on Android 8.0 API level 26: | ||
| * Android 8.0 API level 26 is the first known to support `j.l.invoke`. | ||
| * Constraint on run-time class creation precludes: | ||
| * Compilation from Python to JVM byte code at run-time (`exec()`). | ||
| * Certain approaches to implementation in detail. | ||
| * Needs specialised tool chain. | ||
| * Desirable target, but unknown other obstacles. | ||
|
|
||
| * MVP: Generate and consume CPython byte code: | ||
| * This addresses large modules (JVM class size problem) | ||
| * Makes it possible to adopt modules compiled by CPython (defined version) | ||
| * Would be essential to Android support? | ||
|
|
||
| * MVP(?): Use of `threading` leads to actual concurrency. | ||
| * There is no Global Interpreter Lock (neither a local one). | ||
| * Built-in objects remain internally consistent under concurrent access. | ||
| * The programmer is responsible for synchronisation of his/her code. | ||
| * The value of a shared object seen by another thread may be stale: | ||
| * Behavioural differences from CPython will occur in unsynchronised code. | ||
| * Operations that [happen to be atomic in CPython]( | ||
| https://docs.python.org/3/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe | ||
| ) need not be atomic in Jython. | ||
| * Concurrency is close to a unique advantage: probably MVP. | ||
|
|
||
| * High standard of compatibility with CPython. | ||
| * MVP: `os.name` no longer confuses popular tools (like `virtualenv`). | ||
| * Divergences fixed as discovered. (Adoption of stdlib is a help.) | ||
|
|
||
| * Continue to integrate smoothly with Java | ||
| * MVP: Generally works as in Jython 2. | ||
| * Less magic: an object claiming Java type has the semantics in its Javadoc. | ||
| * Avoid semantic confusion (e.g. `list.pop()` vs `Deque.pop()`) | ||
| * Explicit cast or wrapper to choose Python semantics (possibly?) | ||
| * No special treatment of Swing and AWT names (as now in `PyJavaType`). | ||
|
|
||
| * Support popular libraries (and their dependencies) progressively. | ||
| * MVP: An API that makes extensions possible. | ||
| * Encourage C to Java ports of the most popular (different projects!) | ||
| * Encourage JyNI or HPy experiments. | ||
|
|
||
| * Compile Python source to Java byte code, and: | ||
| * persist compiled Java byte code (in some form). | ||
| * treat Python compiled to JVM as: | ||
| * equivalent to cached .pyc files. | ||
| * resources locatable in a JAR by Jython. | ||
| * classes visible from Java (maybe). | ||
|
|
||
| * Command-line version. | ||
| * MVP: Launch script or other wrapper. (Generated by `jlink`? C?) | ||
| * The command you run *is* the interpreter, not a launcher. | ||
| * JNI appears to [support this readily]( | ||
|
There was a problem hiding this comment. I once explored this approach, see https://github.com/Stewori/LiJy-launch (and consider to link it here). There was a problem hiding this comment. Gosh that's complicated, and it worries me because you know a lot more than I about the native interface. In my ignorance, I would have followed this: https://www.developer.com/java/data/how-to-create-a-jvm-instance-in-jni.html, where the target class would be org/python/util/jython, letting Jython do all the argument processing. The messiness is recompilation for each platform. However, there is jlink to fall back on and I'm suggesting a native launcher is not MVP. There was a problem hiding this comment. The procedure described in https://www.developer.com/java/data/how-to-create-a-jvm-instance-in-jni.html is also at the core of java.exe and LiJyLaunch. However, it appears that java.exe has to do a lot of sanetizing before args are handled to the JNI api. E.g. jdb mode requires args in a special form, wildcard processing is done in the launcher, some system properties are detected, some memory stuff is configured and a bunch of things I do not understand. LiJy inherits this annoying complexity from java.exe's source code. Since the jython.py launcher used to call java.exe, taking this stuff from java.exe source was the only feasile way to provide the same behaviour regarding support for JVM command line args as in jython.py. That java.exe was from Java 8, maybe things changed since then, I never had a look again. But actually, taking the source from java.exe is a rather painless solution to achieve this goal except for license issues. The easiest way would be to write a dual license, i.e. having the launcher licensed separately. GNU with classpath exception is not too bad. Or we can ask Oracle for permission to relicense this particular code. It is not very much code and not especially genuine. It rather is an annoying and useless work to rewrite/reinvent it just to escape the license. However, it may be possible to link the required code dynamically from java.exe, see https://stackoverflow.com/questions/39131185/can-a-file-both-be-an-executable-exe-and-a-dynamic-link-library-dll-at-the-s. This would also be preferable for maintenance against upstream changes of java.exe. (I don't know if this approach is actually applicable to java.exe.) |
||
| https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/invocation.html | ||
| ). | ||
| * `sys.executable` designates the actual executable. | ||
| * Process objects designate the process doing the work (not the wrapper). | ||
|
|
||
| * MVP: Interpreter embeddable in a Java application. | ||
| * Continue JSR-223 support with some clean-up. | ||
| * `PythonInterpreter` API (only) generally similar to Jython 2. | ||
| * Risk: current API is unbounded: too much is public. | ||
| * Reduce the public API aided by Java Jigsaw modules. | ||
| * There will be blood. | ||
| * Risk: much existing guidance invalidated (Jython Book). | ||
|
|
||
| * Clear semantics for `import` from Java | ||
| * Semantics in Jython 2 have repeatedly changed. | ||
| * `import` as written tries the same thing repeatedly | ||
|
|
||
|
|
||
| ## Implementation and API Features | ||
|
|
||
| The Java API available for embedding and to extension writers, | ||
| will be heavily influenced by "internal" implementation choices. | ||
| Looked at the other way, premature API choices may constrain implementation | ||
| freedom in undesirable ways. | ||
|
|
||
| This is especially true of the object model, since for efficiency's sake, | ||
| objects exchanged at the API will be in the implementations we use internally. | ||
|
|
||
| This can be less difficult than in CPython because Java gives us good | ||
| tools for encapsulation: interfaces, packages and modules. | ||
| (Modules are important here. Many implementation details in Jython 2 | ||
| became public API, just to cross our own package boundaries.) | ||
|
|
||
| * MVP: Resolve the object API before 3.x beta. One of: | ||
| * `PyObject` is an interface. | ||
| * Every `j.l.Object` is a Python object directly. | ||
|
|
||
| * MVP: Abstract interface (along the lines of CPython `abstract.h`). | ||
| * Abstract interface for basic operations. | ||
| * The internals of `type` (slots, etc.) are private | ||
| (better encapsulated than CPython). | ||
|
|
||
| * MVP: Clear relationships amongst interpreter, system state and | ||
| thread state. (At least the interpreter and its semantics are API.) | ||
|
|
||
|
|
||
| ## Jython 2 Features not to Reproduce | ||
|
|
||
| * **Not:** Java `List` and `Map` implicitly Python `list` and `map`. | ||
| * This is a tempting feature and it almost works, | ||
| but involves complex guesswork. | ||
| * Propose a brief way to be explicit instead. | ||
|
|
||
| * **Not:** Java package cache. | ||
| * The location of this is problematic for users. | ||
| * It has been the subject of a CVE and follow-up bug. | ||
| * If there is evidence it significantly improves performance, | ||
| without being a security issue, | ||
| a reworked version could be added to 3. | ||
|
|
||
jeff5 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure how critical this is, especially for a MVP. While its nice to be close to CPython in performance (or faster) I don't think many users of Jython this is a requirement. For people requiring the highest performance its unlikely Jython will ever be the interpreter of choice, however Jythons Java integration is compelling even at a lower performance
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree in general for MVP. Long-term, I'd like scientific computing to be a proposition without C extensions, if we can manage it.
By some measures, Jython 2 is somewhat faster than CPython, although start-up definitely gives the opposite impression. Full use of the JVM's dynamic language features in JRuby had a massive effect on performance. In my toy, and almost certainly in Jython 3, the byte code interpreter uses
MethodHandlebut notCallSite. Generated JVM code would useinvokedynamiccall sites fully, and I'm hoping for something good.There's a real danger in setting any kind of target that we spend too much time micro-optimising the byte-code interpreter, when a JVM language is the goal. At the same time, if the basic operations of the abstract object API are much slower than in CPython, we're probably laying the wrong foundation. I could be talked down from the particular numbers.
In some simple loops and calls in my toy version I observe about 40% the speed of CPython. It's a bit disappointing, considering how succinct the Java is, but I'm resisting the temptation to delve further.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you talk about Java-ports of NumPy and SciPy, I estimate each one is a project comparable in complexity to Jython itself. In addition one still has to link blas and lapack, which inherently require C glue code. This can be based on existing projects like jblas, but then again, it still uses platform-dependent C code, just written by others. Project Panama may solve this but AFAIK it has not even a planned release date.