When I am exporting data into airbyte then into BigQuery, it outputs into a single table with a _table column, including other tables data. The schema is correct, so when _table = another table besides the one I'm exporting it has a bunch of null values. Not sure what's going on here as this is the only data source I'm having issues with.
#Airbyte stream exporting into strange format
4 messages · Page 1 of 1 (latest)
Hi @rare bloom. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
This sounds like it’s coming from how Convex’s streaming export API is being consumed by Airbyte, not from Convex “mixing” tables incorrectly.
From the Convex side:
- The streaming export API returns documents from all tables by default, and each document includes a
_tablefield indicating which Convex table it came from. [Streaming export] - You can optionally filter by a single table using the
tableNamequery parameter onlist_snapshotanddocument_deltas. [Streaming export; document_deltas]
If your Airbyte → BigQuery setup is:
- Calling
list_snapshot/document_deltaswithouttableName, and - Writing all returned rows into a single BigQuery table,
then you’ll see exactly what you describe: one BigQuery table, a _table column, and rows from multiple Convex tables, with many nulls where the schema doesn’t match.
What you can check / adjust (based on the docs):
-
Confirm whether the export is filtered by table
Ensure the connector (or any custom code using the Convex streaming export API) is passingtableName=<your_table>when it calls:GET /api/list_snapshotGET /api/document_deltas[Streaming export]
-
Confirm how Airbyte is mapping streams to BigQuery tables
Convex’s export API exposes tables; Airbyte then maps those to “streams” and then to destination tables. If Airbyte is configured to:- Treat “all Convex tables” as a single stream, or
- Use a “single destination table” per connection,
you’ll get the merged table with
_tableas a discriminator.
The Convex docs don’t go into the Airbyte-side configuration details, so I can’t say exactly which Airbyte setting to flip, only that from Convex’s perspective you should either:
- Filter by
tableNamein the export API calls if you truly only want one Convex table, or - Expect multiple Convex tables in the output and have Airbyte split them into separate BigQuery tables based on stream/table name.