|
| 1 | +--- |
| 2 | +title: Jython 3 Features and MVP |
| 3 | +--- |
| 4 | +# Jython 3 Features and a Viable Minimum |
| 5 | + |
| 6 | +This is a discussion document that attempts to describe, |
| 7 | +and to some extent prioritise, |
| 8 | +features for Jython 3, |
| 9 | +based on suggestions collated from various voices on |
| 10 | +[jython-dev](https://sourceforge.net/p/jython/mailman/jython-dev/) |
| 11 | +and in off-list e-mail. |
| 12 | + |
| 13 | + |
| 14 | +## Positioning |
| 15 | + |
| 16 | +We think people will continue to adopt and use Jython if Jython 3 ... |
| 17 | + |
| 18 | +* is a modern version of Python, close to standard in its features. |
| 19 | +* runs on a Java platform that is supported in the long-term. |
| 20 | +* integrates cleanly with Java for access to JDK and user libraries. |
| 21 | +* offers correct concurrency (effectively utilising available CPUs). |
| 22 | +* allows code developed on CPython 3.x to run on Jython 3.x. |
| 23 | +* is well-tested for release and supported for bugs. |
| 24 | + |
| 25 | +The minimum viable product (MVP) is to approach *all* these targets closely, |
| 26 | +that is, it isn't viable if it falls a long way short of any one of them. |
| 27 | +This does not preclude the availability of immature alpha versions. |
| 28 | +(It's a public project, so it is there for anyone to build.) |
| 29 | + |
| 30 | + |
| 31 | +## Version and Forms |
| 32 | + |
| 33 | +* MVP: A Python 3.8 (?) close to standard in ability to run Python. |
| 34 | + * Only essential differences (e.g. around GC, atomicity). |
| 35 | + * The full syntax (plus some Java twists e.g. to import from Java). |
| 36 | + * Close to the whole standard library (where use is not C-specific). |
| 37 | + |
| 38 | +* Builds for Gradle/Maven ecosystem. |
| 39 | + * MVP: Slim JAR without bundled dependencies. |
| 40 | + * MVP: Centrally available to cite as a dependency. |
| 41 | + |
| 42 | +* Provides an executable command |
| 43 | + * MVP: `jython3` command installable on each major OS |
| 44 | + * MVP: subset of commonly-used python3 command options |
| 45 | + * Option-compatible with `python3` (MVP: a subset) |
| 46 | + * Options specific to Jython (JVM options and others) |
| 47 | + |
| 48 | + |
| 49 | +## Performance |
| 50 | + |
| 51 | +* MVP: speed comparable with CPython (say 2x either way). |
| 52 | + * Higher performance (single-threaded) desirable but not MVP. |
| 53 | +* Future: Work to improve speed. |
| 54 | + * Compile Python to JVM byte code. |
| 55 | + * Faster Python byte code interpreter. |
| 56 | +* Have in mind performance for: |
| 57 | + * scientific computing |
| 58 | + * image, big data and ML libraries |
| 59 | + |
| 60 | +These choices, if valid, |
| 61 | +make a Python byte code interpreter a legitimate MVP. |
| 62 | + |
| 63 | + |
| 64 | +## Platforms and Environments |
| 65 | + |
| 66 | +It is difficult to enumerate the possibilities in a MECE way. |
| 67 | +It is multidimensional, although not every combination makes sense. |
| 68 | +Which of the things in this section are MVP? |
| 69 | + |
| 70 | +### OS platforms: |
| 71 | + |
| 72 | +* Windows desktop |
| 73 | +* Linux desktop |
| 74 | +* Mac desktop |
| 75 | +* Raspberry Pi |
| 76 | +* Android (minimum as discussed under "features") |
| 77 | + * Risk: API gaps constrain us, or lead to a special Android version |
| 78 | +* Small JVM devices (e.g. for IoT) |
| 79 | + |
| 80 | +### Runtime environments: |
| 81 | + |
| 82 | +These are imprecise definitions, but the intention is to run everywhere |
| 83 | +Java does, and take full advantage of each environment. |
| 84 | + |
| 85 | +* Desktop |
| 86 | +* J2EE |
| 87 | + * Risk: Java version support |
| 88 | +* AWS |
| 89 | +* Azure |
| 90 | +* IoT/embedded Java |
| 91 | + |
| 92 | +At the embedded end of the spectrum, Jython is probably only attractive if |
| 93 | +Python is not available directly (and Java is, obviously). |
| 94 | + |
| 95 | +### Significant Integrations |
| 96 | + |
| 97 | +An unscientific, incomplete list based on projects we have noticed |
| 98 | +(e.g. via a reported bug). |
| 99 | + |
| 100 | +* Robot Framework |
| 101 | +* ImageJ |
| 102 | +* Pig |
| 103 | +* Ghidra |
| 104 | +* OpenHAB |
| 105 | +* ... ? |
| 106 | + |
| 107 | +There is a wider Python ecosystem that does not yet use Jython because they |
| 108 | +depend on extensions in C. E.g. there is not much in SciPy without `numpy`. |
| 109 | + |
| 110 | +How do we find the time and collaborators |
| 111 | +to test integration into these environments? |
| 112 | +Have we enough understanding to avoid unintentionally making it difficult? |
| 113 | + |
| 114 | + |
| 115 | +## Features |
| 116 | + |
| 117 | +* MVP: Runs on Java 11 SE. Chosen as a minimum because: |
| 118 | + * It is the post-Java 9 workhorse for the time being. |
| 119 | + * It has a rich set of libraries we can exploit in the implementation. |
| 120 | + * LTS version characterises many enterprise Java installations. |
| 121 | + * Enterprises favour security, ease of management. |
| 122 | + * Risk to MVP: J2EE is based on Java 8. Must explore: |
| 123 | + * have I misunderstood this? |
| 124 | + * non-issue if JVM byte code is the same? |
| 125 | + * attention needed to which libraries are on the path |
| 126 | + |
| 127 | +* Not MVP: Runs on Android 8.0 API level 26: |
| 128 | + * Android 8.0 API level 26 is the first known to support `j.l.invoke`. |
| 129 | + * Constraint on run-time class creation precludes: |
| 130 | + * Compilation from Python to JVM byte code at run-time (`exec()`). |
| 131 | + * Certain approaches to implementation in detail. |
| 132 | + * Needs specialised tool chain. |
| 133 | + * Desirable target, but unknown other obstacles. |
| 134 | + |
| 135 | +* MVP: Generate and consume CPython byte code: |
| 136 | + * This addresses large modules (JVM class size problem) |
| 137 | + * Makes it possible to adopt modules compiled by CPython (defined version) |
| 138 | + * Would be essential to Android support? |
| 139 | + |
| 140 | +* MVP(?): Use of `threading` leads to actual concurrency. |
| 141 | + * There is no Global Interpreter Lock (neither a local one). |
| 142 | + * Built-in objects remain internally consistent under concurrent access. |
| 143 | + * The programmer is responsible for synchronisation of his/her code. |
| 144 | + * The value of a shared object seen by another thread may be stale: |
| 145 | + * Behavioural differences from CPython will occur in unsynchronised code. |
| 146 | + * Operations that [happen to be atomic in CPython]( |
| 147 | + https://docs.python.org/3/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe |
| 148 | + ) need not be atomic in Jython. |
| 149 | + * Concurrency is close to a unique advantage: probably MVP. |
| 150 | + |
| 151 | +* High standard of compatibility with CPython. |
| 152 | + * MVP: `os.name` no longer confuses popular tools (like `virtualenv`). |
| 153 | + * Divergences fixed as discovered. (Adoption of stdlib is a help.) |
| 154 | + |
| 155 | +* Continue to integrate smoothly with Java |
| 156 | + * MVP: Generally works as in Jython 2. |
| 157 | + * Less magic: an object claiming Java type has the semantics in its Javadoc. |
| 158 | + * Avoid semantic confusion (e.g. `list.pop()` vs `Deque.pop()`) |
| 159 | + * Explicit cast or wrapper to choose Python semantics (possibly?) |
| 160 | + * No special treatment of Swing and AWT names (as now in `PyJavaType`). |
| 161 | + |
| 162 | +* Support popular libraries (and their dependencies) progressively. |
| 163 | + * MVP: An API that makes extensions possible. |
| 164 | + * Encourage C to Java ports of the most popular (different projects!) |
| 165 | + * Encourage JyNI or HPy experiments. |
| 166 | + |
| 167 | +* Compile Python source to Java byte code, and: |
| 168 | + * persist compiled Java byte code (in some form). |
| 169 | + * treat Python compiled to JVM as: |
| 170 | + * equivalent to cached .pyc files. |
| 171 | + * resources locatable in a JAR by Jython. |
| 172 | + * classes visible from Java (maybe). |
| 173 | + |
| 174 | +* Command-line version. |
| 175 | + * MVP: Launch script or other wrapper. (Generated by `jlink`? C?) |
| 176 | + * The command you run *is* the interpreter, not a launcher. |
| 177 | + * JNI appears to [support this readily]( |
| 178 | + https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/invocation.html |
| 179 | + ). |
| 180 | + * `sys.executable` designates the actual executable. |
| 181 | + * Process objects designate the process doing the work (not the wrapper). |
| 182 | + |
| 183 | +* MVP: Interpreter embeddable in a Java application. |
| 184 | + * Continue JSR-223 support with some clean-up. |
| 185 | + * `PythonInterpreter` API (only) generally similar to Jython 2. |
| 186 | + * Risk: current API is unbounded: too much is public. |
| 187 | + * Reduce the public API aided by Java Jigsaw modules. |
| 188 | + * There will be blood. |
| 189 | + * Risk: much existing guidance invalidated (Jython Book). |
| 190 | + |
| 191 | +* Clear semantics for `import` from Java |
| 192 | + * Semantics in Jython 2 have repeatedly changed. |
| 193 | + * `import` as written tries the same thing repeatedly |
| 194 | + |
| 195 | + |
| 196 | +## Implementation and API Features |
| 197 | + |
| 198 | +The Java API available for embedding and to extension writers, |
| 199 | +will be heavily influenced by "internal" implementation choices. |
| 200 | +Looked at the other way, premature API choices may constrain implementation |
| 201 | +freedom in undesirable ways. |
| 202 | + |
| 203 | +This is especially true of the object model, since for efficiency's sake, |
| 204 | +objects exchanged at the API will be in the implementations we use internally. |
| 205 | + |
| 206 | +This can be less difficult than in CPython because Java gives us good |
| 207 | +tools for encapsulation: interfaces, packages and modules. |
| 208 | +(Modules are important here. Many implementation details in Jython 2 |
| 209 | +became public API, just to cross our own package boundaries.) |
| 210 | + |
| 211 | +* MVP: Resolve the object API before 3.x beta. One of: |
| 212 | + * `PyObject` is an interface. |
| 213 | + * Every `j.l.Object` is a Python object directly. |
| 214 | + |
| 215 | +* MVP: Abstract interface (along the lines of CPython `abstract.h`). |
| 216 | + * Abstract interface for basic operations. |
| 217 | + * The internals of `type` (slots, etc.) are private |
| 218 | + (better encapsulated than CPython). |
| 219 | + |
| 220 | +* MVP: Clear relationships amongst interpreter, system state and |
| 221 | + thread state. (At least the interpreter and its semantics are API.) |
| 222 | + |
| 223 | + |
| 224 | +## Jython 2 Features not to Reproduce |
| 225 | + |
| 226 | +* **Not:** Java `List` and `Map` implicitly Python `list` and `map`. |
| 227 | + * This is a tempting feature and it almost works, |
| 228 | + but involves complex guesswork. |
| 229 | + * Propose a brief way to be explicit instead. |
| 230 | + |
| 231 | +* **Not:** Java package cache. |
| 232 | + * The location of this is problematic for users. |
| 233 | + * It has been the subject of a CVE and follow-up bug. |
| 234 | + * If there is evidence it significantly improves performance, |
| 235 | + without being a security issue, |
| 236 | + a reworked version could be added to 3. |
| 237 | + |
| 238 | + |
0 commit comments