#Airbyte stream exporting into strange format

4 messages · Page 1 of 1 (latest)

rare bloom
#

When I am exporting data into airbyte then into BigQuery, it outputs into a single table with a _table column, including other tables data. The schema is correct, so when _table = another table besides the one I'm exporting it has a bunch of null values. Not sure what's going on here as this is the only data source I'm having issues with.

supple duneBOT
#

Hi @rare bloom. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:

#

This sounds like it’s coming from how Convex’s streaming export API is being consumed by Airbyte, not from Convex “mixing” tables incorrectly.

From the Convex side:

  • The streaming export API returns documents from all tables by default, and each document includes a _table field indicating which Convex table it came from. [Streaming export]
  • You can optionally filter by a single table using the tableName query parameter on list_snapshot and document_deltas. [Streaming export; document_deltas]

If your Airbyte → BigQuery setup is:

  • Calling list_snapshot / document_deltas without tableName, and
  • Writing all returned rows into a single BigQuery table,

then you’ll see exactly what you describe: one BigQuery table, a _table column, and rows from multiple Convex tables, with many nulls where the schema doesn’t match.

What you can check / adjust (based on the docs):

  1. Confirm whether the export is filtered by table
    Ensure the connector (or any custom code using the Convex streaming export API) is passing tableName=<your_table> when it calls:

  2. Confirm how Airbyte is mapping streams to BigQuery tables
    Convex’s export API exposes tables; Airbyte then maps those to “streams” and then to destination tables. If Airbyte is configured to:

    • Treat “all Convex tables” as a single stream, or
    • Use a “single destination table” per connection,

    you’ll get the merged table with _table as a discriminator.

#

The Convex docs don’t go into the Airbyte-side configuration details, so I can’t say exactly which Airbyte setting to flip, only that from Convex’s perspective you should either:

  • Filter by tableName in the export API calls if you truly only want one Convex table, or
  • Expect multiple Convex tables in the output and have Airbyte split them into separate BigQuery tables based on stream/table name.

If you share how your Airbyte connection is configured (especially how many streams it sees and how they’re mapped to BigQuery tables), the Convex team could help further in Discord, but that’s outside what’s documented here.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: