8000 Jython 3 MVP and Roadmap by jeff5 · Pull Request #21 · jython/jython.github.io · GitHub
[go: up one dir, main page]

Skip to content

Jython 3 MVP and Roadmap #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Oct 24, 2020
Merged

Jython 3 MVP and Roadmap #21

merged 5 commits into from
Oct 24, 2020

Conversation

jeff5
Copy link
Member
@jeff5 jeff5 commented Sep 5, 2020

Some ideas on MVP collated from our conversations, and an attempt at a roadmap.

@jeff5 jeff5 requested review from jimbaker and jamesmudd September 5, 2020 15:53
@jeff5
Copy link
Member Author
jeff5 commented Sep 5, 2020

Ruby (or Jekyll) seem unhappy with my available machine(s), so I haven't been able to test these changes, but the menu changes seem low-risk and the Markdown renders ok on GitHub.

Anyway, It's the content I'd like to focus on, here or on the list.

* Command-line version.
* MVP: Launch script or other wrapper. (Generated by `jlink`? C?)
* The command you run *is* the interpreter, not a launcher.
* JNI appears to [support this readily](
Copy link
Collaborator 8000

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I once explored this approach, see https://github.com/Stewori/LiJy-launch (and consider to link it here).
However, it incorporates some (non-trivial) code taken from the Java-launcher source, resulting in GPL license with classpath exception.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gosh that's complicated, and it worries me because you know a lot more than I about the native interface.

In my ignorance, I would have followed this: https://www.developer.com/java/data/how-to-create-a-jvm-instance-in-jni.html, where the target class would be org/python/util/jython, letting Jython do all the argument processing. The messiness is recompilation for each platform.

However, there is jlink to fall back on and I'm suggesting a native launcher is not MVP.

Copy link
Collaborator
@Stewori Stewori Sep 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The procedure described in https://www.developer.com/java/data/how-to-create-a-jvm-instance-in-jni.html is also at the core of java.exe and LiJyLaunch. However, it appears that java.exe has to do a lot of sanetizing before args are handled to the JNI api. E.g. jdb mode requires args in a special form, wildcard processing is done in the launcher, some system properties are detected, some memory stuff is configured and a bunch of things I do not understand. LiJy inherits this annoying complexity from java.exe's source code. Since the jython.py launcher used to call java.exe, taking this stuff from java.exe source was the only feasile way to provide the same behaviour regarding support for JVM command line args as in jython.py. That java.exe was from Java 8, maybe things changed since then, I never had a look again.

But actually, taking the source from java.exe is a rather painless solution to achieve this goal except for license issues. The easiest way would be to write a dual license, i.e. having the launcher licensed separately. GNU with classpath exception is not too bad. Or we can ask Oracle for permission to relicense this particular code. It is not very much code and not especially genuine. It rather is an annoying and useless work to rewrite/reinvent it just to escape the license.

However, it may be possible to link the required code dynamically from java.exe, see https://stackoverflow.com/questions/39131185/can-a-file-both-be-an-executable-exe-and-a-dynamic-link-library-dll-at-the-s. This would also be preferable for maintenance against upstream changes of java.exe. (I don't know if this approach is actually applicable to java.exe.)
Then jython.exe would link it dynamically and might not be subject to the viral license. I don't know if classpath exception also applies to dynamic linking, but otherwise dynamic linking of JNI would be subject to GNU with cp exception as well. So the approach in https://www.developer.com/java/data/how-to-create-a-jvm-instance-in-jni.html would inherently require relicensing then.

* Only performance-critical modules are hand-crafted in Java.
* Modules that take advantage of Java libraries call them from Python.
* Enthusiasm for writing in Java should be directed to re-implementing
popular libraries that broaden Jython use (`numpy`, etc.).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe mention ctypes, cffi and a cython-backend here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can mention them, but what would we like actually to say?

More broadly, I know calling modules written in C would be useful, but until ideas of the pure Java core settle, I find it difficult to say what's involved. So not MVP, but interested in your thoughts now and when that settling has happened.

Copy link
Collaborator
@Stewori Stewori Sep 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling C by pure Java code won't be feasible until Project Panama lands. Currently one inherently has to write platform dependent C glue code or use something like JNA or JNR which is just the same with the only difference being that it was written by others. ctypes, cffi and cython are JNA/JNR-like libs for Python. Having Jython-versions would enable tons of Python modules at a time. This is still not sufficient for NumPy and SciPy but is a requirement. All three are used by NumPy and SciPy. So if you want to support scientific code, getting ctypes, cffi and cython working would be the first step.
That said, each of these is a huge project on its own. So it is likely a better approach to support https://github.com/hpyproject/hpy and hope that these libraries will be made HPy compatible as well. I know that cython announced HPy support. But then, supporting HPy is not feasible with pure Java code either.

Copy link
Member
@jamesmudd jamesmudd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this I think it really helpful to publish this. Some minor comments but the navbar bit needs to be fixed before merging. Maybe just have 2 links for now, I think sub menus wouldn't be too hard but its probably not worth worrying about now. If you enable GH pages on your fork they will render a page you can view to check the content.


## Performance

* MVP: speed comparable with CPython (say 2x either way).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how critical this is, especially for a MVP. While its nice to be close to CPython in performance (or faster) I don't think many users of Jython this is a requirement. For people requiring the highest performance its unlikely Jython will ever be the interpreter of choice, however Jythons Java integration is compelling even at a lower performance

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree in general for MVP. Long-term, I'd like scientific computing to be a proposition without C extensions, if we can manage it.

By some measures, Jython 2 is somewhat faster than CPython, although start-up definitely gives the opposite impression. Full use of the JVM's dynamic language features in JRuby had a massive effect on performance. In my toy, and almost certainly in Jython 3, the byte code interpreter uses MethodHandle but not CallSite. Generated JVM code would use invokedynamic call sites fully, and I'm hoping for something good.

There's a real danger in setting any kind of target that we spend too much time micro-optimising the byte-code interpreter, when a JVM language is the goal. At the same time, if the basic operations of the abstract object API are much slower than in CPython, we're probably laying the wrong foundation. I could be talked down from the particular numbers.

In some simple loops and calls in my toy version I observe about 40% the speed of CPython. It's a bit disappointing, considering how succinct the Java is, but I'm resisting the temptation to delve further.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Long-term, I'd like scientific computing to be a proposition without C extensions, if we can manage it.

If you talk about Java-ports of NumPy and SciPy, I estimate each one is a project comparable in complexity to Jython itself. In addition one still has to link blas and lapack, which inherently require C glue code. This can be based on existing projects like jblas, but then again, it still uses platform-dependent C code, just written by others. Project Panama may solve this but AFAIK it has not even a planned release date.

Copy link
Member
@jamesmudd jamesmudd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think it looks good to me, but maybe worth waiting on another review. Thanks for writing this up.

@jeff5
Copy link
Member Author
jeff5 commented Oct 24, 2020

Long enough for comments, I think.

@jeff5 jeff5 merged commit e345b37 into jython:master Oct 24, 2020
@jeff5 jeff5 deleted the jython-3-mvp branch August 11, 2021 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants
0