8000 Upgrading Unleash causes Rust thread panic · Issue #288 · Unleash/unleash-client-java · GitHub
[go: up one dir, main page]
Skip to content

Upgrading Unleash causes Rust thread panic #288

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
lbenedetto opened this issue Feb 27, 2025 · 12 comments
Open

Upgrading Unleash causes Rust thread panic #288

lbenedetto opened this issue Feb 27, 2025 · 12 comments
Assignees
@sighphyre
Labels
bug

Comments

@lbenedetto
Copy link
lbenedetto commented Feb 27, 2025 •
edited
Loading

Describe the bug

When we upgraded Unleash java client from version 9.2.6 to 10.0.2, we started seeing errors in production

We reverted the upgrade and the errors went away.

Steps to reproduce the bug

No response

Expected behavior

No response

Logs, error output, etc.

thread '<unnamed>' panicked at /cargo/registry/src/index.crates.io-6f17d22bba15001f/serde_json-1.0.138/src/ser.rs:2083:64:
byte index 2 is not a char boundary; it is inside '\t' (bytes 1..2) of `�	����nt`                                       
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace                                             
fatal runtime error: failed to initiate panic, error 4117039104

Some of these errors contained recognizable flag names

byte index 4 is not a char boundary; it is inside '\u{4}' (bytes 3..4) of `�g���nd_profile_feature_card_notification_LC_enabled`
byte index 4 is not a char boundary; it is inside '\0' (bytes 3..4) of `�����NU_Highlighted_Item_Design`

Screenshots

No response

Additional context

No response

Unleash version

10.0.2

Subscription type

None

Hosting type

None

SDK information (language and version)

No response
@lbenedetto lbenedetto added the bug label Feb 27, 2025
@sighphyre
Copy link
Member

So sorry about that, are there by any chance unicode strings in your contexts?

We'll take a look, thank you for reporting this

@sighphyre sighphyre self-assigned this Feb 28, 2025
@chriswk chriswk moved this from New to Investigating in Issues and PRs Feb 28, 2025
@lbenedetto
Copy link
Author

I'll have someone do a thorough check to make sure there aren't. But at least in the strings mentioned in the errors, there are not. For example, the full string for one of the errors is CE_backend_profile_feature_card_notification_LC_enabled

@sighphyre sighphyre moved this from Investigating to Todo in Issues and PRs Mar 4, 2025
@sighphyre
Copy link
Member

Hey @lbenedetto, I can't seem to reproduce this at all. Any other context you can provide here? What OS are you running on? Any chance you've tinkered with encoding? Is this a heavy multi-threaded workload?

@hcgoranson
Copy link

Hello, I talked to the person that created the CE_backend_profile_feature_card_notification_LC_enabled flag and it was done in the "usual way", so no difference compared to how we usually create flags and no special encoding related setup

We are running our Java application in a docker base image from Amazon Corretto 21 and the flag is not exposed more multi-threads compared to our other flags

@gastonfournier
Copy link
Contributor

Hi @hcgoranson could you share with us an export of the feature configuration? If you prefer not to share it here, you can find me or Simon on our community Slack and you can send either or both of us a DM.

I'd like to reproduce it on my own, and I'll try out some things but having the same configuration as you may save us a lot of time.

@gastonfournier
Copy link
Contributor

I did run this example: https://github.com/Unleash/unleash-client-java/tree/new-version-test/examples/cli-example (note I pushed some changes in a branch) against our demo instance using this feature https://app.unleash-hosted.com/demo/projects/default/features/advanced.constraints and I couldn't reproduce the error.

From examples/cli-example folder:

$ ./gradlew run

> Task :run
16:59:41.070 [main] INFO  i.g.r.FeatureBackupHandlerFile -- Unleash will try to load feature toggle states from temporary backup
Context enabled: 1000
Smaller semver enabled: 0

BUILD SUCCESSFUL in 2s
3 actionable tasks: 2 executed, 1 up-to-date

I also tried using the name you shared. If you want to try and play around with this code or our demo instance to try to reproduce the issue, please do.

@throup
Copy link
throup commented Mar 24, 2025

Hi! I am another colleague of lbenedetto with some more info.

So far, we have been unable to replicate the problem on our test environments, neither by using the system as normal, nor generating random inputs for the Unleash client. We tried updating our production system again (this time to 10.2.0) and encountered the same problems.

The thread panic comes from Rust code, in the serde_json library. (links to specific line of code)

It is difficult to pinpoint exactly what has led to this call, but I believe it is serialising string content into JSON -- perhaps to call the API on the Unleash server.

The Rust error message suggests an attempt to split a unicode string in the middle of a multi-byte character. The rust code is iterating and splitting a &str -- a byte sequence -- which would cause this problem.

However, the strings we see in the logs are our flag names, and none of them contain non-ASCII characters.

We are not seeing much more useful info in our system logs. This is the most detailed example I have found:
Image

@gastonfournier
Copy link
Contributor

Hi @throup thanks for the update. We're also unable to reproduce, maybe this happens under high concurrency (some memory management issue that's not easy to catch). Would you let us know how you're setting up Unleash? I'm thinking of the refresh and metrics intervals, whether you're using a local backup or not, so we can test as close as possible to your current setup.

@throup
Copy link
throup commented Mar 27, 2025

Hi @gastonfournier .
We're running a Spring Boot application. We create a DefaultUnleash bean, with this setup:

unleashConfig = UnleashConfig.builder()
        .appName(appName)
        .fetchTogglesInterval(20L)
        .instanceId(getInstanceId())
        .environment(getEnvironment())
        .unleashAPI(apiUrl)
        .customHttpHeader("Authorization", apiSecrets)
        .build();
unleash = new DefaultUnleash(unleashConfig);

I presume anything else is "default values".

...whether you're using a local backup or not...
I am not sure I understand. We do run Unleash Edge.

@gastonfournier gastonfournier mentioned this issue Apr 1, 2025
Closed
@gastonfournier
Copy link
Contributor

Ok, I've tried the Spring boot sample app: #304 and even tried with customHttpHeader as that's something I was doing differently using apiKey method instead but it also works well. Is there any specific reason why you're using customHttpHeader instead of apiKey?

I've used all the same configuration and couldn't reproduce it.

Not sure if you provided the detail about the unleash-server you're using as backend. Maybe that can make a difference. Also, is your code running inside a docker container? Maybe that's also something to look at

@lbenedetto
Copy link
Author

Yes, we build a Docker image with our Spring jar in it. Here is an approximation of our Dockerfile with irrelevant stuff removed.

FROM amazoncorretto:21-al2023-headless

RUN dnf install gperftools-libs -y && \
    dnf update --releasever latest -y && \
    echo "networkaddress.cache.ttl=10" >> ${JAVA_HOME}/conf/security/java.security && \
    dnf clean all

ENV LD_PRELOAD="/usr/lib64/libtcmalloc.so.4"

@gastonfournier
Copy link
Contributor

I tried to reproduce this as close as I could and it's still working fine: https://github.com/gastonfournier/tmp-unleash-java-sdk-ygg-sample/actions/runs/14219241407/job/39842991731

I'm using this Dockerfile based on your input: https://github.com/gastonfournier/tmp-unleash-java-sdk-ygg-sample/blob/main/Dockerfile-amazoncorretto

There's probably something missing...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sighphyre sighphyre

Labels
bug
Projects
Issues and PRs
Status: Todo
Milestone
No milestone
Development

No branches or pull requests
5 participants
@gastonfournier @throup @lbenedetto @hcgoranson @sighphyre
0