8000 feat: improve query path observability by Rachelint · Pull Request #1235 · apache/horaedb · GitHub
[go: up one dir, main page]
Skip to content

feat: improve query path observability #1235

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Rachelint merged 29 commits into from
Oct 8, 2023
Merged

feat: improve query path observability #1235

Rachelint merged 29 commits into from
Oct 8, 2023

Conversation

@Rachelint
Copy link
Contributor
@Rachelint Rachelint commented Oct 1, 2023 •
edited
Loading

Rationale

Closes #1211
We have poor ways too observe query status, in this pr, a ton of tools are added for better observability.

Detailed Changes

  • Support slow query / failed query logs.
  • Add metrics about query time range, sst prune rate, push down type and so on...
  • Impl simple distributed query analyze.

Test Plan

Test manually.

@Rachelint Rachelint marked this pull request as ready for review October 1, 2023 10:12
@jiacai2050 jiacai2050 self-requested a review October 2, 2023 07:47
jiacai2050
jiacai2050 reviewed Oct 7, 2023
View reviewed changes
@Rachelint Rachelint force-pushed the improve-query-observability branch from c3aca57 to d998798 Compare October 7, 2023 07:22
jiacai2050
jiacai2050 reviewed Oct 8, 2023
View reviewed changes
Rachelint added 25 commits October 8, 2023 15:19
@Rachelint
add slow log.
625bf3d
@Rachelint
add cache hit/miss for oss cache.
fb44446
@Rachelint
add failed query log.
542d698
@Rachelint
impl simple analyze.
d106493
@Rachelint
add sub table query hotspot.
6f3d861
@Rachelint
add push down stats.
94c4b85
@Rachelint
stats query time infos.
fe6ca96
@Rachelint
add metric for sst filter rate.
e949ad3
@Rachelint
Revert "stats query time infos."
7b872f4 
This reverts commit 49c537d.
@Rachelint
new way to stats time info for tables.
105e910
@Rachelint
modify default slow query threshold to 5s, and support setting throug…
6d2614f 
…h http api.
@Rachelint
stats the sst prune for whole partitioned table.
4af1a32
@Rachelint
fix failed log.
7793ca6
@Rachelint
fix clippy.
be8ee27
@Rachelint
support enable/disable table level sst metrics.
8a7e348
@Rachelint
fix clippy.
d1e6a95
@Rachelint
fix potential panic about query time range related metrics.
ab77096
@Rachelint
rename PushDownStatus to PushDownEvent.
dcf3995
@Rachelint
refactor the maybe table level metrics.
368f990
@Rachelint
eliminate unnecessary clone of table name.
7e03cab
@Rachelint
add some todos.
2097320
@Rachelint
optimize LogDispatcher's judgement logic.
c3b1133
@Rachelint
update snapshots for resolver testing.
b4a8ac6
@Rachelint
modify comments.
69dfecc
@Rachelint
fix enable_table_level_metrics judgement.
e192d03
@Rachelint Rachelint force-pushed the improve-query-observability branch from 14e66d0 to b33df8a Compare October 8, 2023 07:28
jiacai2050
jiacai2050 approved these changes Oct 8, 2023
View reviewed changes
Copy link
Contributor
@jiacai2050 jiacai2050 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Rachelint Rachelint merged commit 6d0c2b9 into apache:main Oct 8, 2023
@Rachelint Rachelint mentioned this pull request Oct 9, 2023
Merged
Rachelint added a commit that referenced this pull request Oct 9, 2023
@Rachelint
fix: bug about logging nothing (#1250)
ee70ab4 
## Rationale
Bug leading to logging nothing is introduced by pr #1235 , I fix it in
this pr by overwritting `tag` default value to empty.

## Detailed Changes
+ Hide all detailed impl about logging in `logger` crate.
+ Overwrite  default `tag` value(set by `log` crate) to empty.

## Test Plan
Test manually.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jiacai2050 jiacai2050 jiacai2050 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Query debugging improvement tickets

2 participants

@Rachelint @jiacai2050
0