FFFF Consider adding batch support for `DELETE /responseStore` broker API · Issue #17643 · apache/pinot · GitHub
[go: up one dir, main page]

Skip to content

Consider adding batch support for DELETE /responseStore broker API #17643

@yashmayya

Description

@yashmayya
  • Context is in Fix ResponseStoreCleaner bugs causing unbounded response store growth and cleanup failures #17622.
  • In cases where there's a very large number of cursor responses stored on the brokers, the periodic controller job that is responsible for cleanup can get bogged down because it currently does a GET call on each broker to retrieve a full list of cursor responses and then issues an individual DELETE call for each cursor response that is past expiry.
  • We can consider adding a batch delete API to make this controller-based broker cleanup job more efficient. Basically, we can avoid even fetching the entire list of stored cursor responses - maybe through an API like DELETE /responseStore?expiredBefore={currentTimeMs} which pushes the expiration logic to the broker where it belongs. The controller can act purely like a cron trigger in that case.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    0