8000 optimize opentelemetry ingest by Vadman97 · Pull Request #9730 · highlight/highlight · GitHub
[go: up one dir, main page]
Skip to content

optimize opentelemetry ingest #9730

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Vadman97 merged 18 commits into from
Jan 7, 2025
Merged

optimize opentelemetry ingest #9730

Vadman97 merged 18 commits into from
Jan 7, 2025

Conversation

@Vadman97
Copy link
Member
@Vadman97 Vadman97 commented Dec 18, 2024 •
edited
Loading

Summary

  • Updates the opentelemetry collector to improve data transfer from the collector to the public graph.
  • Adds profiling to otel.go ingest code in public graph and addresses a number of performance issues:
    • A regex pattern slowing down attribute processing
    • Redis rate limiting in sampling.go causing a bottleneck

How did you test this change?

deploying to prod

new spans

Screenshot from 2024-12-19 00-22-18
Screenshot from 2024-12-19 00-22-09
Screenshot from 2024-12-19 00-22-02

Are there any deployment considerations?

no

Does this work require review from our design team?

no

@changeset-bot
Copy link
changeset-bot bot commented Dec 18, 2024 •
edited
Loading

⚠️ No Changeset found

Latest commit: 36f3976

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

💥 An error occurred when fetching the changed packages and changesets in this PR 
Some errors occurred when validating the changesets config:
The package or glob expression "rrdom" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "rrdom-nodejs" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "rrweb" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note
8000
 that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "rrweb-player" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "rrweb-snapshot" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/all" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/types" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/record" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/replay" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/packer" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/rrweb-plugin-canvas-webrtc-record" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/rrweb-plugin-canvas-webrtc-replay" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/rrweb-plugin-console-record" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/rrweb-plugin-console-replay" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/rrweb-plugin-sequential-id-record" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/rrweb-plugin-sequential-id-replay" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/utils" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@rrweb/web-extension" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "rrvideo" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.

@Vadman97 Vadman97 force-pushed the vadim/otel-ingest-optimization branch from 60cb8aa to 6cace6c Compare December 19, 2024 04:27
@Vadman97 Vadman97 requested review from a team and mayberryzane December 19, 2024 06:21
@Vadman97 Vadman97 marked this pull request as ready for review December 19, 2024 06:21
@Vadman97 Vadman97 force-pushed the vadim/otel-ingest-optimization branch from 1bb5b59 to 5ca6a34 Compare December 19, 2024 20:47
Vadman97
Vadman97 commented Dec 19, 2024
View reviewed changes
@Vadman97 Vadman97 enabled auto-merge (squash) January 6, 2025 20:03
@Vadman97 Vadman97 force-pushed the vadim/otel-ingest-optimization branch from 3ef240c to 45fda27 Compare January 6, 2025 23:21
@github-actions
Copy link
Contributor
github-actions bot commented Jan 6, 2025 •
edited
Loading

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines Covered Coverage Threshold Status
593 530 89% 0% 🟢

New Files

No new covered files...

Modified Files

No covered modified files...

updated for commit: 36f3976 by action🐍

@Vadman97 Vadman97 force-pushed the vadim/otel-ingest-optimization branch from 8ff7331 to ed3f252 Compare January 6, 2025 23:31
@Vadman97 Vadman97 merged commit 8557364 into main Jan 7, 2025
33 checks passed
@Vadman97 Vadman97 deleted the vadim/otel-ingest-optimization branch January 7, 2025 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mayberryzane mayberryzane mayberryzane approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Vadman97 @mayberryzane
0