-
Notifications
You must be signed in to change notification settings - Fork 76
Upgrading Unleash causes Rust thread panic #288
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
So sorry about that, are there by any chance unicode strings in your contexts? We'll take a look, thank you for reporting this |
I'll have someone do a thorough check to make sure there aren't. But at least in the strings mentioned in the errors, there are not. For example, the full string for one of the errors is |
Hey @lbenedetto, I can't seem to reproduce this at all. Any other context you can provide here? What OS are you running on? Any chance you've tinkered with encoding? Is this a heavy multi-threaded workload? |
Hello, I talked to the person that created the We are running our Java application in a docker base image from Amazon Corretto 21 and the flag is not exposed more multi-threads compared to our other flags |
Hi @hcgoranson could you share with us an export of the feature configuration? If you prefer not to share it here, you can find me or Simon on our community Slack and you can send either or both of us a DM. I'd like to reproduce it on my own, and I'll try out some things but having the same configuration as you may save us a lot of time. |
I did run this example: https://github.com/Unleash/unleash-client-java/tree/new-version-test/examples/cli-example (note I pushed some changes in a branch) against our demo instance using this feature https://app.unleash-hosted.com/demo/projects/default/features/advanced.constraints and I couldn't reproduce the error. From examples/cli-example folder:
I also tried using the name you shared. If you want to try and play around with this code or our demo instance to try to reproduce the issue, please do. |
Hi! I am another colleague of lbenedetto with some more info. So far, we have been unable to replicate the problem on our test environments, neither by using the system as normal, nor generating random inputs for the Unleash client. We tried updating our production system again (this time to 10.2.0) and encountered the same problems. The thread panic comes from Rust code, in the serde_json library. (links to specific line of code) It is difficult to pinpoint exactly what has led to this call, but I believe it is serialising string content into JSON -- perhaps to call the API on the Unleash server. The Rust error message suggests an attempt to split a unicode string in the middle of a multi-byte character. The rust code is iterating and splitting a However, the strings we see in the logs are our flag names, and none of them contain non-ASCII characters. We are not seeing much more useful info in our system logs. This is the most detailed example I have found: |
Hi @throup thanks for the update. We're also unable to reproduce, maybe this happens under high concurrency (some memory management issue that's not easy to catch). Would you let us know how you're setting up Unleash? I'm thinking of the refresh and metrics intervals, whether you're using a local backup or not, so we can test as close as possible to your current setup. |
Hi @gastonfournier .
I presume anything else is "default values".
|
Ok, I've tried the Spring boot sample app: #304 and even tried with customHttpHeader as that's something I was doing differently using apiKey method instead but it also works well. Is there any specific reason why you're using customHttpHeader instead of apiKey? I've used all the same configuration and couldn't reproduce it. Not sure if you provided the detail about the unleash-server you're using as backend. Maybe that can make a difference. Also, is your code running inside a docker container? Maybe that's also something to look at |
Yes, we build a Docker image with our Spring jar in it. Here is an approximation of our Dockerfile with irrelevant stuff removed. FROM amazoncorretto:21-al2023-headless
RUN dnf install gperftools-libs -y && \
dnf update --releasever latest -y && \
echo "networkaddress.cache.ttl=10" >> ${JAVA_HOME}/conf/security/java.security && \
dnf clean all
ENV LD_PRELOAD="/usr/lib64/libtcmalloc.so.4" |
I tried to reproduce this as close as I could and it's still working fine: https://github.com/gastonfournier/tmp-unleash-java-sdk-ygg-sample/actions/runs/14219241407/job/39842991731 I'm using this Dockerfile based on your input: https://github.com/gastonfournier/tmp-unleash-java-sdk-ygg-sample/blob/main/Dockerfile-amazoncorretto There's probably something missing... |
Describe the bug
When we upgraded Unleash java client from version 9.2.6 to 10.0.2, we started seeing errors in production
We reverted the upgrade and the errors went away.
Steps to reproduce the bug
No response
Expected behavior
No response
Logs, error output, etc.
Some of these errors contained recognizable flag names
Screenshots
No response
Additional context
No response
Unleash version
10.0.2
Subscription type
None
Hosting type
None
SDK information (language and version)
No response
The text was updated successfully, but these errors were encountered: