10000 feat: Configures otel log collector by wprzechrzta · Pull Request #20895 · cloudquery/cloudquery · GitHub
[go: up one dir, main page]
Skip to content

feat: Configures otel log collector #20895

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
Jun 20, 2025
Merged

feat: Configures otel log collector #20895

merged 11 commits into from
Jun 20, 2025

Conversation

wprzechrzta
Copy link
Contributor
@wprzechrzta wprzechrzta commented Jun 17, 2025 •
edited
Loading

Summary

Adds support for open telemetry collector.

Tested on local Tilt cluster

Example otel records

───────
Timestamp:          2025-06-18 07:09:04.000000000
TimestampTime:      2025-06-18 07:09:04
TraceId:
SpanId:
TraceFlags:         0
SeverityText:       info
SeverityNumber:     9
ServiceName:        cloudquery-cli
Body:               table sync finished
ResourceSchemaUrl:  https://opentelemetry.io/schemas/1.26.0
ResourceAttributes: {'service.name':'cloudquery-cli','telemetry.sdk.language':'go','telemetry.sdk.name':'opentelemetry','telemetry.sdk.version':'1.36.0'}
ScopeSchemaUrl:
ScopeName:          cloudquery
ScopeVersion:
ScopeAttributes:    {}
LogAttributes:      {'client':'project:cq-playground','duration_ms':'2274.9195','errors':'0','invocation_id':'a219e90f-c8f2-4f55-84f1-c99fcacb9b69','module':'gcp-source','resources':'35','table':'gcp_storage_buckets'}

Row 55:
───────
Timestamp:          2025-06-18 07:09:08.000000000
TimestampTime:      2025-06-18 07:09:08
TraceId:
SpanId:
TraceFlags:         0
SeverityText:       info
SeverityNumber:     9
ServiceName:        cloudquery-cli
Body:               started call
ResourceSchemaUrl:  https://opentelemetry.io/schemas/1.26.0
ResourceAttributes: {'service.name':'cloudquery-cli','telemetry.sdk.language':'go','telemetry.sdk.name':'opentelemetry','telemetry.sdk.version':'1.36.0'}
ScopeSchemaUrl:
ScopeName:          cloudquery
ScopeVersion:
ScopeAttributes:    {}
LogAttributes:      {'grpc.component':'server','grpc.method':'Transform','grpc.method_type':'bidi_stream','grpc.service':'cloudquery.plugin.v3.Plugin','grpc.start_time':'2025-06-18T09:09:01+02:00','grpc.time_ms':'6126.349','invocation_id':'a219e90f-c8f2-4f55-84f1-c99fcacb9b69','module':'basic-transformer','peer.address':'','protocol':'grpc'}

Row 56:

@wprzechrzta wprzechrzta marked this pull request as ready for review June 17, 2025 15:10
@wprzechrzta wprzechrzta requested a review from a team as a code owner June 17, 2025 15:10
@wprzechrzta wprzechrzta requested a review from blesniewski June 17, 2025 15:10
wprzechrzta
wprzechrzta commented Jun 17, 2025
View reviewed changes
Copy link
Contributor Author
@wprzechrzta wprzechrzta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current implementation relies on optional env variable to configure otel endpoint that will have to be set on sync pod in scheduler.

erezrokah
erezrokah reviewed Jun 17, 2025
View reviewed changes
erezrokah
erezrokah reviewed Jun 17, 2025
View reviewed changes
murarustefaan
murarustefaan reviewed Jun 18, 2025
View reviewed changes
murarustefaan
murarustefaan reviewed Jun 18, 2025
View reviewed changes
murarustefaan
murarustefaan reviewed Jun 20, 2025
View reviewed changes
cli/cmd/root.go Outdated
@@ -224,3 +244,14 @@ func CloseLogFile() {
logFile.Close()
}
}

func otelConfig() (*otel.Config, error) {
insecureEndpoint, err := strconv.ParseBool(env.GetEnvOrDefault("OTEL_ENDPOINT_INSECURE", "true"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's maybe default this to false?

@wprzechrzta wprzechrzta merged commit 146a279 into main Jun 20, 2025
22 of 24 checks passed
@wprzechrzta wprzechrzta deleted the woj/add-logging branch June 20, 2025 10:51
@cq-bot cq-bot mentioned this pull request Jun 17, 2025
Merged
kodiakhq bot pushed a commit that referenced this pull request Jun 23, 2025
@cq-bot
chore(main): Release cli v6.21.0 (#20884)
e1b87a5 
🤖 I have created a release *beep* *boop*
---


## [6.21.0](cli-v6.20.5...cli-v6.21.0) (2025-06-23)


### Features

* Configures otel log collector ([#20895](#20895)) ([146a279](146a279))
* Transform table name for DeleteRecord ([#20870](#20870)) ([8ce11f6](8ce11f6))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@murarustefaan murarustefaan murarustefaan approved these changes

@erezrokah erezrokah erezrokah left review comments

@blesniewski blesniewski Awaiting requested review from blesniewski blesniewski is a code owner automatically assigned from cloudquery/integrations-backend

Assignees
No one assigned
Labels
area/cli
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@wprzechrzta @murarustefaan @erezrokah @cq-bot
0