Replies: 5 comments 15 replies
-
|
Hi @yashssh! Welcome to the STUMPY community. So, we don't have an active benchmark per se but our most common workhorse example is the following: Depending on your your hardware resources, this should take 5-20 seconds to run on modern multithreaded hardware. Let me know if that helps and if you have any further questions. Please feel free to link to your repo if you'd like to discuss it there instead. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the example @seanlaw! I'm also trying to run the tests using |
Beta Was this translation helpful? Give feedback.
-
|
No, I'm not aware of any way to ignore test failures. Are the tests failing due to your LLVM20 upgrades? |
Beta Was this translation helpful? Give feedback.
-
|
Yes. Across all unit tests I am seeing 2 failures each in Everything else seems to be working fine. How can I create standalone reproducers for these failing tests? Let me know if you want to look at the error message. |
Beta Was this translation helpful? Give feedback.
-
|
Nvm, attaching the failure logs |
Beta Was this translation helpful? Give feedback.
-
|
@yashssh This looks suspicious. What version of LLVM and Numba are you using? I would try using the current stable versions of LLVM and Numba first to see if the assert errors still persist. Also, if you are running things on Linux, please install |
Beta Was this translation helpful? Give feedback.
-
|
I am checking with I don't think I have |
Beta Was this translation helpful? Give feedback.
-
Yes, for running in parallel with |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for looking into it! I couldn't find an easy way to fix the Is this enough to conclude it might not be a multi-threading issue? |
Beta Was this translation helpful? Give feedback.
-
That's a good sign (at least to eliminate problems).
Yes, this likely means that LLVM20 based |
Beta Was this translation helpful? Give feedback.
-
|
I am starting to believe this issue is closely related to fast math optimisations. Is there a way to disable fast math optimisation before executing the tests? I can see the config variable |
Beta Was this translation helpful? Give feedback.
-
|
In case it matters, @NimaSarajpoor had resolved some recent issues with LLVM fastmath but this seems unrelated to those changes since the tests were passing.
The simplest way would be to modify and then reinstall STUMPY: Then, you should be able to run your tests as per usual. |
Beta Was this translation helpful? Give feedback.
-
|
Awesome thank you! Yes indeed, both tests pass after I disable fast math!
I believe since the LLVM version I am testing with is 5 generations ahead it's fair to assume the meaning of some of these fast math flags must have diverged, might necessitate another similar debugging session when the time comes. I'll spend a bit more time on it if I can narrow down the root cause, else I'll try to return when Numba officially moves to LLVM 20+. |
Beta Was this translation helpful? Give feedback.
-
|
It's rather concerning that the LLVM fastmath flags continue to diverge and cause problems instead of maintaining backwards compatibility. Do the tests still fail if you do: Considering the instability, I'm thinking that we may want to be more conservative with the fastmath flags that we use. By the way, out of curiosity @yashssh, may I ask how you came upon the STUMPY package and what it is that you are trying to achieve? |
Beta Was this translation helpful? Give feedback.
-
|
Apologies for the late reply @seanlaw , I got occupied with other things.
Surprisingly not, the failures go away if I mention the flags explicitly like you asked. Looks like this might have something to do with newer fast math flags that were not available till LLVM15.
I'm working on upgrading Numba to LLVM20 so trying to make sure if everything's working well and to collect some numbers around how much performance gains can dependencies expect. I was told STUMPY is a direct user of Numba so started here. If you can point me to other benchmarks/applications that use Numba heavily that would be great! Thanks |
Beta Was this translation helpful? Give feedback.
-
No problem at all @yashssh! Thank you for supporting the
Are you able to share any references to the "newer fastmath flags"? The I really want to know the difference between the
May I ask "who" will be consuming these performance benchmarks and taking action on them? I sincerely do not mean to "shoot the messenger" but the performance gains mean nothing if STUMPY is broken with the new flags. |
Beta Was this translation helpful? Give feedback.
-
I am afraid I don't have an answer right now, I don't have very good grasp over the meaning of all the flags. It was just my guess that issue might be due to some of the newer fast math flags, but let me try spend some more time on it. Also, let me add @gmarkall @stuartarchibald here to answer what exactly is the difference between
For now these are solely for my own experiments to get an early feel on how things might look like with the latest LLVM version in Numba. If there are surprises we can raise them with Numba where either I or someone else can pick them up.
I understand your concerns but I am not sure if I agree with STUMPY might be broken sentiment. Numba is right now on LLVM15 and moving to LLVM20 will not only bring performance gains but also bring in all the bug fixes that went into the compiler. On top of that these upgrades are supposed to be incremental with multiple LLVM version supported at once and not in 1 go giving us time to fix any failures. The other thing is LLVM itself describes fast math flags as enabler for "otherwise unsafe floating-point transformations", they aren't guaranteed to produce correct results and a couple of failures in fast math precision tests due to 5 generation worth of upgrades shouldn't be that alarming(not to discredit their importance or imply they should be ignored). |
Beta Was this translation helpful? Give feedback.
-
|
That's fair. Please let me know if there's anything I can do to help @yashssh |
Beta Was this translation helpful? Give feedback.
-
|
I did a bit more research and below are my findings. When I get rid of Now, I don't understand the motive behind 2 config variables inside Stumpy |
Beta Was this translation helpful? Give feedback.
-
Rather than 3 more
According to the LLVM fastmath documentation, when |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi, I'm working on upgrading
Numbato LLVM20 and right now we are in the middle of upstreaming changes. I want to measure the performance uplifts(if any) after all these LLVM upgrades. Is there a set of benchmarks for stumpy that I can leverage?If not are there some good examples covering major algorithms that I can use to create some micro benchmarks?
Beta Was this translation helpful? Give feedback.
All reactions