clickhouse-driver integration eats inserted data when generator is used #4657
Description
How do you use Sentry?
Self-hosted/on-premise
Version
2.34.1
Steps to Reproduce
In a Django project using clickhouse-driver (0.2.9) to interact with an external database I recently upgraded sentry sdk. Inserting data into clickhouse stopped working afterward. The reason was that we are using a generator as a source of data for the INSERT query and the sentry clickhouse-driver integration consumes the generator before it is passed to the clickhouse-driver. Thus an INSERT is sent to clickhouse with empty data.
This is a barebones example where using a generator to pass the data results in no data being written to the database.
records = [{"aa": 1, "bb": 2}]
data = (rec for rec in records) # generator
client.execute("INSERT INTO
73ED
XXX (aa, bb) VALUES ", data)When I replaced the generator with a generator which would throw an Exception, I got the following traceback:
File "/opt/virtualenvs/celus/lib/python3.12/site-packages/clickhouse_driver/client.py", line 376, in execute
rv = self.process_insert_query(
File "/opt/virtualenvs/celus/lib/python3.12/site-packages/clickhouse_driver/client.py", line 607, in process_insert_query
rv = self.send_data(sample_block, data,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/virtualenvs/celus/lib/python3.12/site-packages/sentry_sdk/integrations/clickhouse_driver.py", line 142, in _inner_send_data
db_params.extend(data)
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/virtualenvs/celus/lib/python3.12/site-packages/hcube/backends/clickhouse/backend.py", line 64, in ff
raise ValueError('sh*t, someone ate my data')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Which shows the point where the generator gets consumed.
Expected Result
Data being written to clickhouse.
Actual Result
No data being written into clickhouse.
Metadata
Metadata
Assignees
Labels
Projects
Status