#Aggregate (Convex Component) fails if multiple documents are added at once

25 messages · Page 1 of 1 (latest)

crimson jacinth
#

If I add multiple documents at once, I get the following error:
Documents read from or written to the "btreeNode" table changed while this mutation was being run and on every subsequent retry. Another call to this mutation changed the document with ID "j97257gsj6gkr786bghe6nd2zn772qjw". See https://docs.convex.dev/error#1

Adding them one by one does not trigger this error.

Errors and Warnings | Convex Developer Hub

This page explains specific errors thrown by Convex.

vernal capeBOT
#

Thanks for posting in #1088161997662724167.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.

    - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
    - Use [search.convex.dev](https://search.convex.dev) to search Docs, Stack, and Discord all at once.
    - Additionally, you can post your questions in the Convex Community's #1228095053885476985 channel to receive a response from AI.
    - Avoid tagging staff unless specifically instructed.

    Thank you!
worldly karma
#

Are you correctly awaiting the ‘insert’?

Also what loop are you using to run through all inserts

crimson jacinth
#

yes i am. i tried it using insert and also using trigger

worldly karma
crimson jacinth
#

`import { Triggers } from "convex-helpers/server/triggers";
import { DataModel } from "../_generated/dataModel";
import {
balanceAggregate,
liquidNetWorthAggregate,
netWorthAggregate,
tokenBalanceAggregate,
tokenNetWorthAggregate,
usdcBalanceAggregate,
usdtBalanceAggregate,
} from "../aggregates/wallet";

const triggers = new Triggers<DataModel>();

// Wallet Aggregate Triggers
triggers.register("wallets", balanceAggregate.trigger());
triggers.register("wallets", usdcBalanceAggregate.trigger());
triggers.register("wallets", usdtBalanceAggregate.trigger());
triggers.register("wallets", tokenBalanceAggregate.trigger());
triggers.register("wallets", liquidNetWorthAggregate.trigger());
triggers.register("wallets", tokenNetWorthAggregate.trigger());
triggers.register("wallets", netWorthAggregate.trigger());

export const wrapDB = triggers.wrapDB;`

#

whenever i insert multiple documents into wallets it triggers and therefore triggers the error (does not matter what the update is)

nova ruin
#

Can you show the code for adding multiple documents at once, and the code for doing them one at a time? (Is there a Promise.all client-side or in an action or in a mutation?) OCC errors are possible when multiple mutations write the same data at the same time. See https://www.convex.dev/components/aggregate#read-dependencies-and-writes

Incidentally, this looks problematic, although probably not causing your problem:
export const wrapDB = triggers.wrapDB; should be export const wrapDB = (ctx) => triggers.wrapDB(ctx);

Convex
Aggregate

Keep track of sums and counts in a denormalized and scalable way.

crimson jacinth
#

` await sequentialPromiseAll(
addresses.map(async (address) => {
const entry = await ctx.runQuery(
internal.functions.wallet._getByUniqueFields,
{
input: {
address,
chain: "solana",
},
},
);

    if (!entry) {
      throw new Error("Wallet not found");
    }

    await ctx.runAction(api.functions.wallet.updateWallet, {
      input: {
        address,
      },
    });
  }),
);`

it was promise all, but promise all basically could barely run because everything was failing. i switched to sequential (basically awaiting each one, and it still fails, but less.

i have tested and confirmed that this only happens if i send muliple transactions at once, and if i space it out it will also work. but for my project i cannot afford to purposely slow it down so much as i have a high volume of data.

crimson jacinth
nova ruin
#

See the description of read and write dependencies. It can handle high volume in certain circumstances, like it you have a lot of data and high maxNodeSize, or if you use Namespaces, or if you do multiple writes within a single mutation

#

The code you pasted looks like it's parallelizing queries and actions. But aggregate writes can only happen in mutations. I recommend you do large batches of aggregate writes in a single mutation, and avoid parallelizing

crimson jacinth
#

`export const _updateMany = internalMutation({
args: {
input: zc(z.array(partialSchemaWithId)),
},
handler: async (ctx, args) => {
const { input } = args;

const updatedEntries = await Promise.all(
  input.map(async (inputEntry) => {
    const { _id, ...entryUpdate } = inputEntry;

    const existingEntry = await ctx.skipRules.table(TABLE_NAME).getX(_id);
    const updatedEntry = { ...existingEntry, ...entryUpdate };

    if (isEqual({ ...existingEntry }, updatedEntry)) {
      return existingEntry;
    }

    await existingEntry.replace(updatedEntry);
    const entry = await ctx.skipRules.table(TABLE_NAME).getX(_id);

    return entry;
  }),
);

return updatedEntries;

},
});`

nova ruin
#

and you're only calling _updateMany once?

crimson jacinth
#

yes

#

ok wait a moment

#

i think the new convex function was not deployed properly

#

let me double check

#

might have fixed it. i will report back the next time i do a high volume write (all in one transaction)

crimson jacinth
# nova ruin and you're only calling _updateMany once?

unfortunately...

Uncaught Error: Uncaught Error: Too many bytes read in a single function execution (limit: 8388608 bytes). Consider using smaller limits in your queries, paginating your queries, or using indexed queries with a selective index range expressions.

#

i will try to find some reasonable middle ground for batching

nova ruin
#

Sounds good. You should be able to split it across several mutations, as long as only one runs at a time

#

(in general, aggregate mutations can run in parallel, but not very many and there might need to be certain tweaks to avoid conflicts)

crimson jacinth