10BC0 feat: memtable bitmap for null by ShiKaiWi · Pull Request #1046 · apache/horaedb · GitHub
[go: up one dir, main page]
Skip to content

feat: memtable bitmap for null #1046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jiacai2050 merged 7 commits into from
Jul 6, 2023
Merged

feat: memtable bitmap for null #1046

jiacai2050 merged 7 commits into from
Jul 6, 2023

Conversation

@ShiKaiWi
Copy link
Member
@ShiKaiWi ShiKaiWi commented Jul 4, 2023 •
edited
Loading

Rationale

For a table which contains many nullable columns, many nulls may cause huge memory consumption in the memtable, leading to small ssts where null values will be compressed.

Detailed Changes

  • Introduce a bitset for continuous row encoding
  • Use u32 to encode the offest in the encoded bytes rather than u64

Test Plan

Existing unit test. And test it with tsbs's data, here is result with write buffer 32MB:

description version encoded size of one row number of rows in memtable sst's size
20 col (not null) main 325B 70000+ 1.8M
20 col (not null) current 295B 78000 2M
20 col + 200 null col main 2135B 12100 460K
20 col + 200 null col current 332B 70000 1.5M

tanruixiang
tanruixiang reviewed Jul 4, 2023
View reviewed changes
@ShiKaiWi ShiKaiWi force-pushed the feat-memtable-bitmap-for-null branch from 1446144 to 92ce7a0 Compare July 5, 2023 08:51
jiacai2050
jiacai2050 reviewed Jul 5, 2023
View reviewed changes
jiacai2050
jiacai2050 reviewed Jul 6, 2023
View reviewed changes
@ShiKaiWi ShiKaiWi force-pushed the feat-memtable-bitmap-for-null branch from f19988d to ba9fd02 Compare July 6, 2023 10:40
jiacai2050
jiacai2050 approved these changes Jul 6, 2023
View reviewed changes
Copy link
Contributor
@jiacai2050 jiacai2050 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jiacai2050 jiacai2050 merged commit c6c306d into apache:main Jul 6, 2023
@ShiKaiWi ShiKaiWi deleted the feat-memtable-bitmap-for-null branch July 6, 2023 12:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tanruixiang tanruixiang tanruixiang left review comments

@jiacai2050 jiacai2050 jiacai2050 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ShiKaiWi @jiacai2050 @tanruixiang
0