Llvm20 support #235
Merged
Llvm20 support #235
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Begin uodates to LibLLVM to support LLVM20 * A lot has changed in 10 major releases and this is only the beginning
* Native DLL now builds for Win64 but managed code fails for a variety of reasons.
* Support for this was flaky, at best and very tedious.
- It was also unnecessary as the interop isn't intnded for direct use.
* Generated source files then hand edited to support AOT sceanarios
* Moved generated files into AOT as generation "seemed like a good idea at the time"
- Ultimately it makes a mess of things and ends up with the marshalling information etc... in a custom language
- (See also: UbiquityDotNET#214)
- Too much has changed in >ENT AOT and in LLVM to make it worth trying to generalize the code generation and instead rely on the language itself and source generators.
* source generators are REALLY useful for the "handle" types as there are a lot of them and implementing them is just tedious.
* While the cod did finlally work, it ran into the problem of source generator ordering.
- LibraryImportAttribute uses a Source generator that needs to see the whole type to get the `NativeMarshaller` attribute to know how to marshal the type. However, that is ONLY provided by the handle source generator. Thus for things to work. There would need to be some sort of guaranteed ordering of generators, which does not exist at present. (That's a sadly VERY complicated issue in general. [see: dotnet/roslyn#57239])
- Any updates to interop code is done manually from now on. It's too much of a PITA to maintain a generalized source generator that uses a distinct language to represent what is representable in the target language already.
- If new APIs are needed, then they can be added easily enough.
- Down side is if APIs are removed entirely or if the signature changes there's no "net" to catch them...
- Something may be slvageable from the current generator and an processor of the IL metadata files (Cecil?) to detect any mis-alignements...
- It was getting tedious to figure out how to generalize all of the handling of interop
- It is now expressed entirely in C# as it always should have
- splitting the handles and using SourceGen can simplify things even further
- LlvmBindings generator is now reduced to supporting only basic validations of the input headers and generation of the EXPORTS.g.def
- Even that may be removed if the LLVM library is converted to building a DLL for windows or a shared library for non-windows platforms. instead of a bunch of static libs and debug symbols etc...
- Doing that is left for another day...
- Intention is that the handles library is one in the same as the interop library but they are distinct ONLY for the purpose of resolving the source generator ordering in a deterministic manner.
- Lots of back and forth on that but, in the end it just wasn't a right fit for this scenario. (See varios readme.md files for the gory details)
- No testing done yet, and anything that depends on the compiling layers is not yet updated or tested.
- At least it compiles now!
- There are several places where things were simply removed. Especially regarding the legay PassManager as that all needs some major work to support the newer pass builder concetps etc...
…ute as a string from it's ID.
- This is implemented as an extension API to leverage existing functionality.
…fo` sample generating code!
- Everything up to but not incliding optimization and JIT
- Both optimization pass management and JIT have changed significantly since LLVM 10 and still needs some work
* Adjusted all string marshalling (except p/Invokes) to use Encoding based marshalling to ensure correct behavior x-plat.
- P/Invokes will come as a large single commit as there's a LOT to change
- Added tests for string marshalling behavior.
* Converted some properties to provide string.Empty instead of null
- Tests were validating that it was sometimes null even though it is marked as not nullable!
* Added some lower level support for JIT
- THis is hardly close to complete but encountered issues with string marshalling and envoding so went back to deal with that.
… ALL string marshaling.
- The LLVM docs are silent on the encoding so this forces it all through a single marshaller that leverages an encoding (Currently assumes UTF8, but if there is an error in that assumption there is ONLY one place to fix it now)
- Fixed bug in zero termination of marshalled strings using stateful marshaller
* Updated Tests to test for exceptions directly instead of using a custom attribute
* Updated dependencies on ArgValidators to require expclicit refs so things build and run as expected
- Long term goal is to eliminate that completely as the BCL now contains enough support to make most of it obsolete.
…least Chapter 3 sample.
- Also removed it from Interop lib as it was not used there.
- Updated OO wrapper to set it as a private asset as the dependency is an internal implementation detail about to change.
* Updated package references to limit use of the ArgValidators library.
- Slowly removing it as a dependency for any project.
- Ubiquity.NET.Llvm had the remaining dependency
- removed all need in favor of standard arg checks with a few extensions for "Fluent" style use.
* Moved ThrowIfInvalid on generated global handles to an extension method for greater flexibility and fluent support.
* Changed signature of several interop functions to accept a `byte*` as the input to allow use of pre-existing native strings and `LazyEncodedString`
- TODO: Support interop string handles too! ]At least internally]
- The idea is to allow for no marshalling if round tripping strings.
* Added optimizations as a distinct sample to allow testing the optimizations independently from JIT support. * Minor doc updates/corrections * Moved PassBuilder extension APIs header into proper place so that the tooling will see it and generate the correct exports.def for it.
…re NOT specific to LLVM. - Updated string handling to eliminate need of both `LazyEncodedString` and `LazyInitializedString` They were too general and ended up with too much complexity.
* Most of "registraion" is now done in the native code * Library resolver will select the correct name based on the RID and LLVM target.
* Also updated repo-build scripts to new native binary build format
- Allows testing of re-inits etc... of interop Library API as of right now only one init is allowed.
* App MUST track pointer types
- Debug types in library can help but must be used and passed around as there is no caching to resolve to a managed type that includes the information.
* Corrected multiple cases of using reference equality checks for various constructs (mostly a Context).
- Really need an anyalizer to at least warn if performing any reference equality checks when IEquatable<T> is present. (Code is probably wrong)
* Updated/corrected doc comments
…ue equality should be used. * Since this library removed the interning logic of LLVM types reference equality is NOT the same as value equality. So, all types impacted implement IEquatable<T> to allow testing to see if they are the same instance.
…he Kaleidoscope language
* Removed all vestiges of the now really ancient bitmapped attributes
- This was from a pre-LLVM 3.9 era! and compat/transitionling from such an ancient way of doing things is not needed anymore.
- Requres updates to the native extensions to LLVM-C API
* The LLVM wrappers now distinguishes between an Enum attribute (No args) and an Enum with an int parameter.
- The underlying LLVM C++ api differnetiated these but the LLVM-C API combines them.
- LibLLVM and the wrapper now enables distinct testing for either condition.
* Converted attribute related P/Invoke signatures to use byte* instead of string to support use
of LazyEncoded string
* Move CallingConvention Enum to it's own file
* Fixed crashes from analyzer
- See: https://developercommunity.visualstudio.com/t/Keep-getting-errors-while-editing/10895357
* Fixed bugs in LazyEncoded string for empty strings (native)
* Added NativeStrLen to LazyEncodedString so that an accurate string length (not inluding any terminator) is avaliable.
* Fixed bugs in casting size_t values
* Generalized attribute handling for functions and callsites so that any issues for ALL locations are found in one of two places
- LLVM uses different API names for callsites vs functions/declarations.
* Fixed issue with Debug record reporting
* Native library resolver now uses `TryLoad` to handle possibly different locations for the library depending on JIT/AOT runs. NOTE: AOT publishing in VS is broken, dotnet publish command results in a working EXE though. (Although there are a couple of messages that look like errors but are not reported as such, and it does build a functioning app...)
* This brings in latest patches but als removes downstream vulerable dependencies
- A few small source code changes were needed.
* ALL samples are no set to publish as AOT. - Note Publish from within VS using a profile is NOT supported. (It's basically broken AFAICT). `Dotnet publish` kinda sorta mostly works ;) [See:dotnet/runtime#115153]. - it works with the powershell scripting but it is possible to mis configure things and break the tooling... So be careful with local manual builds. * Created a distinct interface for visualizers to allow a "callback" model from within a parser. * Removes file write/behavior from the parser and allows implementers to do whatever (or nothing) * Chapter 2 implements a visualizer that saves the content as a file. * Added AOT friendly custom serializer for a DGML graph. - The implementation coming from the 3rd party library uses XML serializers which requires dynamic reflection. Which breaks AOT scenarios. The library is NOT annotated for AOT support, but is otherwise OK.
- Analyzer is NOT general purpose and only tested to check against this library
- Technically only tested with this library itself. Though it should become a viable component to include as an analyzer with the library so that consumers can catch issues.
* Elimination of old doc articles not in this repo anymore. * Update spelling errors caught by better tool for spell checking more file types.
* Renamed interface to "reporter" to avoid confusion with "logger" as that term has more specific meaning in administrative log file generation which is different from end-user UX.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a massive update to support LLVM20. It was done in a fork to allow for unreleased native lib builds. (One of the problems of keeping them distinct, which is driving a rethink on that point). Now that the native library support has reached a stable point (and an "alpha" release of the libs) this is moving to smaller updates against this larger "feature" branch. Once this is ready for a complete build with docs it will merge to the
developbranch as the general "feature work" is done. Some of the commits are just "SAFETY" commits while others are legit work (most will fit that but sometimes combine things in order to land with a buildable result). Ultimately none of them are relevant when compared to the v10 code base so this is essentially like starting over, thus all commits in this PR should be squashed to a single commit.