Rate Limiting

[lament saddle]

I have an internal action that I am calling using a migration. The action calls an external API that has a rate limit of 3000 requests per minute, I want to make sure that the action will never exceed 3000 calls per minute and the action gets queued and waits if there is more than 3000 requests in that minute.

[icy igloo]

Hi @lament saddle. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:

---

You can implement rate limiting in your Convex application to ensure your internal action doesn't exceed the external API's limit of 3000 requests per minute. Convex provides a rate limiter component that's perfect for this use case.

Here's how you can implement it:

1. Install the Rate Limiter Component

First, install the rate limiter package:

npm install @convex-dev/rate-limiter

2. Configure the Component

Create or update your convex/convex.config.ts file:

// convex/convex.config.ts
import { defineApp } from "convex/server";
import rateLimiter from "@convex-dev/rate-limiter/convex.config";

const app = defineApp();
app.use(rateLimiter);

export default app;

3. Set Up the Rate Limiter in Your Action

Here's how you can implement the rate limiting in your internal action:

import { RateLimiter, MINUTE } from "@convex-dev/rate-limiter";
import { internalAction } from "./_generated/server";
import { internal } from "./_generated/api";
import { components } from "./_generated/api";

// Initialize the rate limiter with your configuration
const rateLimiter = new RateLimiter(components.rateLimiter, {
  externalApiCalls: { 
    kind: "fixed window", 
    rate: 3000, 
    period: MINUTE,
    shards: 10 // Helps with throughput
  },
});

export const myAction = internalAction({
args: {
// Your action arguments
skipCheck: v.optional(v.boolean()),
},
handler: async (ctx, args) => {
if (!args.skipCheck) {
// Reserve capacity for this request
const status = await rateLimiter.limit(ctx, "externalApiCalls", {
reserve: true,
});

  if (status.retryAfter) {
    // If we need to wait, schedule the action to run later
    // Add some jitter to avoid thundering herd
    const withJitter = status.retryAfter + (Math.random() * 1000);
    return ctx.scheduler.runAfter(
      withJitter,
      internal.yourModule.myAction,
      {
        ...args,
        skipCheck: true, // Skip check when we run again
      }
    );
  }
}

// Now make your API call, knowing we're within rate limits
// Your API call code here

},
});

This implementation:

1. Uses a "fixed window" rate limit of 3000 requests per minute
2. Uses the `reserve: true` option to reserve capacity for future execution
3. Automatically schedules the action to run later if the rate limit is reached
4. Adds jitter to prevent all delayed requests from hitting at the same time
5. Uses sharding to improve throughput while maintaining the rate limit

The `reserve` option is particularly important here as it allows your action to "get in line" for future execution rather than just failing when the rate limit is hit [Stack post on rate limiting](<https://stack.convex.dev/rate-limiting#making-llm-requests-with-reserved-capacity>).

For a migration that's calling this action many times, this approach ensures your requests will be properly spaced out and won't exceed the 3000 per minute limit of your external API.
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------