Streams without a status message #72367
Description
Connector Name
Big Query
Connector Version
BigQuery v3.0.16
What step the error happened?
During the sync
Relevant information
[source-declarative-manifest] Streams do not emit terminal stream status messages
Connector Name
source-declarative-manifest
Connector Version
7.6.5 (used in Airbyte Cloud)
Destination Connector
destination-bigquery:3.0.16 (CDK v3)
What step the error happened?
During the sync operation (replication)
Description
When using source-declarative-manifest with a declarative YAML source that defines multiple streams (or even a single stream), the connector does not emit terminal stream status messages that are required by destinations using CDK v3 (like destination-bigquery:3.0.16).
The sync fails with the error:
io.airbyte.cdk.TransientErrorException: Input was fully read, but some streams did not receive a terminal stream status message.
If the destination did not encounter other errors, this likely indicates an error in the source or platform.
Streams without a status message: [applications]
Expected Behavior
The source-declarative-manifest should automatically emit stream status messages (specifically STREAM_STATUS messages) when a stream finishes reading all records, similar to how native Python connectors do.
Actual Behavior
The connector reads all data successfully (authentication works, pagination works, records are read), but does not emit the terminal stream status message that destinations with CDK v3 require. This causes the destination to fail even though all data was successfully read.
Relevant Information
YAML Configuration
The declarative source YAML defines streams using DeclarativeStream with:
- OAuth2 authentication (Basic Auth + client_credentials)
- Pagination using
DefaultPaginatorwithPageIncrement - Simple record extraction from API responses
Error Details
Error during CHECK operation:
- ✅ CHECK passes successfully
- ✅ Connection test works
- ✅ Discover works
Error during SYNC operation:
- ✅ Authentication works
- ✅ Data is read from API
- ✅ Records are sent to destination
- ❌ Stream status messages are NOT emitted
- ❌ Destination fails waiting for stream status
Logs
2026-01-27 00:26:42 error io.airbyte.cdk.TransientErrorException: Input was fully read, but some streams did not receive a terminal stream status message.
If the destination did not encounter other errors, this likely indicates an error in the source or platform.
Streams without a status message: [applications]
at io.airbyte.cdk.load.state.SyncManager.markInputConsumed(SyncManager.kt:116)
at io.airbyte.cdk.load.state.PipelineEventBookkeepingRouter.close(PipelineEventBookkeepingRouter.kt:300)
at io.airbyte.cdk.load.task.internal.InputConsumerTask.execute(InputConsumerTask.kt:117)
Workaround Attempts
- ✅ Tried with single stream (removed second stream) - Still fails
- ✅ Verified authentication is working correctly
- ✅ Verified pagination is working correctly
- ✅ Verified data is being read successfully
- ❌ No YAML configuration option exists to force stream status emission
Related Issues
This appears to be the same issue affecting other connectors:
- #61419 - Similar error with
source-clickhouse
Impact
- Severity: High - Blocks all syncs using
source-declarative-manifestwith destinations on CDK v3 - Affected Users: Anyone using declarative YAML sources with modern destinations (BigQuery 3.x, Snowflake 3.x, etc.)
- Workaround: None available in Airbyte Cloud (cannot downgrade destination versions)
Environment
- Platform: Airbyte Cloud
- Source:
source-declarative-manifest:7.6.5 - Destination:
destination-bigquery:3.0.16 - Data Volume: ~170 records (small dataset, not a scale issue)
Additional Context
The issue is that source-declarative-manifest does not implement the emission of stream status messages that are required by the new CDK v3 destinations. This is a code-level limitation, not a YAML configuration issue.
Suggested Fix:
The source-declarative-manifest connector should be updated to emit STREAM_STATUS messages when each stream completes, similar to how native Python connectors handle this.
Relevant log output
Contribute
- Yes, I want to contribute
Internal Tracking: https://github.com/airbytehq/oncall/issues/11046