@drowsy ivy Suppose I wanna run some effects on some events like generating thumbnail after file upload is done , is their any specific way of doing it?? Like some event listeners or pub sub kinda thing
#Event listener on file uploads
topaz crest
drowsy ivyBOT
Let me check the knowledge base for you. One moment! :books:
Yes, you can achieve this by using Convex's server-side mutations and scheduling functions. Here's a general approach:
-
File Upload: You can upload files programmatically in your Convex functions or upload individual files using the File storage tab on the Data page. You can generate an upload URL using the
storage.generateUploadUrlfunction of theMutationCtxobject. This URL expires in 1 hour and should be fetched shortly before the upload is made. source (https://docs.convex.dev/file-storage/upload-files#generating-the-upload-url) -
Trigger Function After Upload: After the file is uploaded, you can trigger a function to generate a thumbnail. This can be done by creating a mutation that runs after the file upload is successful. For example, in a React Native or Expo application, after successfully uploading a file and getting the
storageId, you can call a mutation to save custom metadata associated with the file in Convex. source (https://stack.convex.dev/uploading-files-from-react-native-or-expo) -
Scheduled Functions: If you need to delay the execution of the function, you can use Convex's scheduler. For example, you can schedule a function to run after a certain delay to generate the thumbnail. source (https://docs.convex.dev/scheduling/scheduled-functions)
Here's an example of how you can schedule a function to run after a file upload:
// Mutation to save custom metadata for the file
export const saveRecording = mutation({
args: { storageId: v.id("_storage"), author: v.string() },
handler: async (ctx, args) => {
const recId = await ctx.db.insert("recordings", {
storageId: args.storageId,
author: args.author,
format: "audio",
});
ctx.scheduler.runAfter(0, internal.transcript.transcribe, { recId });
return recId;
},
});
In this example, the internal.transcript.transcribe function is scheduled to run immediately after the saveRecording mutation is called. You can replace internal.transcript.transcribe with your own function to generate a thumbnail.
Please note that Convex does not currently support event listeners or a pub-sub system for triggering functions. However, you can achieve similar functionality using the methods described above.
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
topaz crest
@drowsy ivy Scheduling functions from mutations is atomic with the rest of the mutation. This means that if the mutation succeeds, the scheduled function is guaranteed to be scheduled. On the other hand, if the mutations fails, no function will be scheduled, even if the function fails after the scheduling call.
how to do it??
Give me example code for storing files in the file storage and storing the file authorised user in the db
drowsy ivyBOT
Let me check the knowledge base for you. One moment! :books:
Sure, here's how you can do it:
First, you need to generate an upload URL for storing files. This can be done using the storage field in the mutation context. Here's an example:
import { mutation } from "./_generated/server";
import { v } from "convex/values";
export const generateUploadUrl = mutation({
args: { filename: v.string() },
handler: async (ctx, args) => {
const url = await ctx.storage.generateUploadUrl(args.filename);
return url;
},
});
In this example, generateUploadUrl is a mutation that takes a filename as an argument and returns an upload URL.
Next, you can store the file's metadata and the authenticated user's information in the Convex database. This can be done using the db and auth fields in the mutation context. Here's an example:
import { mutation } from "./_generated/server";
import { v } from "convex/values";
export const storeFileMetadata = mutation({
args: { filename: v.string(), url: v.string() },
handler: async (ctx, args) => {
const user = ctx.auth.user;
const fileId = await ctx.db.insert("files", {
filename: args.filename,
url: args.url,
uploadedBy: user.id
});
return fileId;
},
});
In this example, storeFileMetadata is a mutation that takes a filename and a URL as arguments, and stores them in the files table in the Convex database along with the ID of the authenticated user.
Please note that the auth field is automatically propagated when calling queries and mutations from the action. You can learn more about it here (https://docs.convex.dev/auth/auth0#user-information-in-functions).
Finally, you can schedule a function to run in the future using the scheduler field in the mutation context. Here's an example:
import { mutation, internalMutation } from "./_generated/server";
import { internal } from "./_generated/api";
import { v } from "convex/values";
export const scheduleFunction = mutation({
args: { fileId: v.id("files") },
handler: async (ctx, args) => {
await ctx.scheduler.runAfter(5000, internal.files.process, { fileId: args.fileId });
},
});
In this example, scheduleFunction is a mutation that schedules the internal.files.process function to run 5 seconds after the mutation is called. The function is passed the ID of the file to process.
Please note that the internal.files.process function is just an example and you would need to replace it with your own function.
You can learn more about scheduling functions here (https://docs.convex.dev/scheduling/scheduled-functions).
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
topaz crest
@drowsy ivy here what will be the type of the file here
const uploadFile = internalAction({
args:{fileName:v.string(),file:},
async handler(ctx, args) {
const uploadURL = await ctx.storage.generateUploadUrl()
const result = await fetch(uploadURL, {
method: "POST",
body: args.fileName,
});
const { storageId } = await result.json();
},
})
drowsy ivyBOT
Let me check the knowledge base for you. One moment! :books:
In the uploadFile internal action, the file argument would typically be of type v.blob(). This is because files are usually represented as binary data (blobs) when being uploaded. However, the exact type can vary depending on how you're handling file uploads in your application.
Here's how you might define the uploadFile internal action with a file argument of type v.blob():
const uploadFile = internalAction({
args: { fileName: v.string(), file: v.blob() },
async handler(ctx, args) {
const uploadURL = await ctx.storage.generateUploadUrl(args.fileName);
const result = await fetch(uploadURL, {
method: "PUT",
body: args.file,
});
const { storageId } = await result.json();
return storageId;
},
});
In this example, the file argument is expected to be a blob representing the file to be uploaded. The fetch function is used to send a PUT request to the generated upload URL with the file as the request body. The storageId of the uploaded file is then returned from the function.
Please note that this is a simplified example and actual file upload handling can be more complex, especially when dealing with large files or handling potential upload failures.
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
drowsy ivyBOT
Let me check the knowledge base for you. One moment! :books:
I apologize for the confusion. You're correct, Convex does not currently support v.blob() as a type for argument validation.
In Convex, files are typically handled as ArrayBuffer objects, which are represented as v.bytes() in Convex's type system. This is because ArrayBuffer objects can efficiently represent binary data, which is useful for handling files.
Here's how you might define the uploadFile internal action with a file argument of type v.bytes():
const uploadFile = internalAction({
args: { fileName: v.string(), file: v.bytes() },
async handler(ctx, args) {
const uploadURL = await ctx.storage.generateUploadUrl(args.fileName);
const result = await fetch(uploadURL, {
method: "PUT",
body: args.file,
});
const { storageId } = await result.json();
return storageId;
},
});
In this example, the file argument is expected to be an ArrayBuffer representing the file to be uploaded. The fetch function is used to send a PUT request to the generated upload URL with the file as the request body. The storageId of the uploaded file is then returned from the function.
Please note that this is a simplified example and actual file upload handling can be more complex, especially when dealing with large files or handling potential upload failures. source (https://docs.convex.dev/database/types#convex-values)
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
topaz crest
@drowsy ivy can we access fetch api from internalmutation
drowsy ivyBOT
Let me check the knowledge base for you. One moment! :books:
No, you cannot use the fetch API directly within an internalMutation. According to the information from the Discord support thread, Convex does not allow fetching from outside resources during the query or mutation. This is because queries and mutations are pure functions that cannot integrate outside data.
However, you can use an action to fetch all those resources and save them into Convex in a mutation. Actions can have side effects, so that would work if you want to do the merge server-side. You can have the action use runQuery, and then have the action merge in the external data from APIs. You can invoke this action from your app and use the return value.
Here's an example from the Discord support thread:
// Keep your query
// Have the action use `runQuery`, and then have the action merge in the external data from APIs
// You can invoke this action (not `internalAction`) from your app and use the return value
Please note that this is a simplified example and you would need to replace runQuery and internalAction with your own query and action respectively. source (https://discord.com/channels/1019350475847499849/1184357172142952448)
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
topaz crest
@drowsy ivy how to use runMutation inside the internalAction
drowsy ivyBOT
Let me check the knowledge base for you. One moment! :books:
You can use the runMutation method inside an internalAction to call a mutation. The runMutation method is part of the ActionCtx object that is passed as the first argument to the action's handler function.
Here's an example of how you can use runMutation inside an internalAction:
export const runMigration = internalAction(
async ({ runMutation }, { name, cursor, batchSize }) => {
let isDone = false;
while (!isDone) {
const args = { cursor, numItems: batchSize };
({ isDone, cursor } = await runMutation(name, args));
}
}
);
In this example, runMigration is an internalAction that calls a mutation in a loop until a certain condition is met. The runMutation method is used to call the mutation with the given name and arguments. The mutation's name and arguments are passed as parameters to the runMutation method. source (https://stack.convex.dev/migrating-data-with-mutations#looping-from-an-action)
Please note that the runMutation method returns a promise of the mutation's result. You can use the await keyword to wait for the promise to resolve before continuing with the next iteration of the loop.
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
topaz crest
@drowsy ivy how to call internal actions inside a mutation
drowsy ivyBOT
Let me check the knowledge base for you. One moment! :books:
In Convex, you can't directly call an action from a mutation. This is because mutations in Convex are transactional and are prohibited from having non-transactional side effects like calling other cloud services or actions.
However, you can schedule an action to execute after a mutation commits. This ensures that the communication with the outside world only happens if the mutation has successfully run. Here's an example:
await ctx.scheduler.runAfter(0, internal.sources.addEmbedding, {
source,
texts: chunks.map(({ text }) => text),
});
In this example, the internal.sources.addEmbedding action is scheduled to run immediately after the mutation commits. The action is only invoked if the mutation successfully commits. source (https://stack.convex.dev/pinecone-and-embeddings#kick-off-a-background-action)
Please note that the internal.sources.addEmbedding action is just an example and you would need to replace it with your own action.
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
topaz crest
@drowsy ivy how to use playwright with actions
drowsy ivyBOT
Let me check the knowledge base for you. One moment! :books:
Based on the information from the Discord support thread, it appears that Playwright is not currently supported in the Convex Node.js or normal actions environment. This is due to certain limitations and requirements of the Playwright library that are not compatible with the Convex environment.
Here's the relevant conversation from the Discord support thread:
**Trusted Support Staff 8**:Community Member 36 Following up on this, although it may have worked for some users (?) Playwright is not supported in either the Convex Node.js or normal actions environment. It seems like Vercel doesn't support this either for similar reasons.
This means that you won't be able to use Playwright directly within a Convex action. If you need to perform web scraping or browser automation tasks, you might need to consider other methods or tools that are compatible with Convex. source (https://discord.com/channels/1019350475847499849/1196023188421881908)
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
topaz crest
how to install @playwright/browser-chromium in convex actions
@drowsy ivy how to install @playwright/browser-chromium in convex actions
drowsy ivyBOT
Let me check the knowledge base for you. One moment! :books:
Based on the information from the Discord support thread, it appears that Playwright, including the @playwright/browser-chromium package, is not currently supported in the Convex Node.js or normal actions environment. This is due to certain limitations and requirements of the Playwright library that are not compatible with the Convex environment.
Here's the relevant conversation from the Discord support thread:
**Trusted Support Staff 8**:Community Member 36 Following up on this, although it may have worked for some users (?) Playwright is not supported in either the Convex Node.js or normal actions environment. It seems like Vercel doesn't support this either for similar reasons.
This means that you won't be able to install or use the @playwright/browser-chromium package directly within a Convex action. If you need to perform web scraping or browser automation tasks, you might need to consider other methods or tools that are compatible with Convex. source (https://discord.com/channels/1019350475847499849/1196023188421881908)
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
topaz crest
@drowsy ivy I want to run webscraping inside the actions what should I go with then
drowsy ivyBOT
Let me check the knowledge base for you. One moment! :books:
While Convex does not support Playwright or Puppeteer for web scraping, you can use other methods to scrape data in Convex actions.
One approach is to use the fetch API to make HTTP requests and then parse the response data. This can be done directly within an action. Here's an example:
export const fetchSingle = internalAction({
args: {
url: v.string(),
},
handler: async (ctx, { url }) => {
const response = await fetch(url);
const text = parsePage(await response.text());
if (text.length > 0) {
await ctx.runMutation(internal.ingest.load.updateDocument, { url, text });
}
},
});
In this example, the fetchSingle action fetches a webpage and parses its content into text. The parsed text is then saved in the database using a mutation. source (https://stack.convex.dev/ai-chat-with-convex-vector-search#ingest-data-loading--splitting-data)
Another approach is to use a library like Cheerio for parsing HTML. Here's an example:
export const fetchAndEmbedSingle = internalAction({
args: {
url: v.string(),
},
handler: async (ctx, { url }) => {
const loader = new CheerioWebBaseLoader(url);
const data = await loader.load();
// ...
},
});
In this example, the fetchAndEmbedSingle action uses the CheerioWebBaseLoader to load and parse a webpage. source (https://stack.convex.dev/ai-chat-using-langchain-and-convex#ingest-data-loading--splitting--embedding-data)
Please note that these are simplified examples and actual web scraping can be more complex, especially when dealing with large amounts of data or handling potential errors.
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
topaz crest
@drowsy ivy So my problem statement is that I am building a file upload system, user will upload file then
Record get saved as a transaction
Then an action run for file uploading and here error can happen and I have not applied any strategy to recover here
Now I also want to add a service for thumbnail generation. Here the service is present in different api. Should I go for actions or crons
Give me a strategy
drowsy ivyBOT
Let me check the knowledge base for you. One moment! :books:
Based on your problem statement, here's a strategy you can follow:
- File Upload and Record Creation: When a user uploads a file, you can create a record in the database as part of a mutation. This operation is transactional, meaning that if the mutation succeeds, the record is guaranteed to be created. If the mutation fails, no record will be created.
export const saveFileMetadata = mutation({
args: { filename: v.string(), url: v.string() },
handler: async (ctx, args) => {
const user = ctx.auth.user;
const fileId = await ctx.db.insert("files", {
filename: args.filename,
url: args.url,
uploadedBy: user.id
});
return fileId;
}
});
source (https://docs.convex.dev/functions/mutations)
- File Upload Action: After the record is created, you can schedule an action to upload the file. This action can handle the actual file upload process, including any error handling or retry logic.
export const uploadFile = internalAction({
args: { fileName: v.string(), file: v.bytes() },
async handler(ctx, args) {
const uploadURL = await ctx.storage.generateUploadUrl(args.fileName);
const result = await fetch(uploadURL, {
method: "PUT",
body: args.file,
});
const { storageId } = await result.json();
return storageId;
},
});
source (https://docs.convex.dev/functions/actions)
- Thumbnail Generation: After the file is uploaded, you can schedule another action to generate the thumbnail. This action can call the external API to generate the thumbnail.
export const generateThumbnail = internalAction({
args: { fileId: v.id("files") },
async handler(ctx, args) {
// Call the external API to generate the thumbnail
// ...
},
});
- Error Handling: If an error occurs during the file upload or thumbnail generation process, you can handle it within the respective action. For example, you can retry the action or log the error for later review.
export const uploadFile = internalAction({
args: { fileName: v.string(), file: v.bytes() },
async handler(ctx, args) {
try {
const uploadURL = await ctx.storage.generateUploadUrl(args.fileName);
const result = await fetch(uploadURL, {
method: "PUT",
body: args.file,
});
const { storageId } = await result.json();
return storageId;
} catch (error) {
// Handle the error
// ...
}
},
});
source (https://docs.convex.dev/functions/actions#error-handling)
Please note that these are simplified examples and actual file upload and thumbnail generation can be more complex, especially when dealing with large files or handling potential errors.
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
topaz crest
@drowsy ivy stream file uploads with resume and pause
drowsy ivyBOT
Let me check the knowledge base for you. One moment! :books:
I'm sorry, but based on the provided knowledge sources, Convex does not currently support streaming file uploads with resume and pause functionality. The file upload process in Convex is handled via HTTP actions or upload URLs, and there's no mention of streaming, pausing, or resuming uploads in the documentation.
Here's how you can upload files via an HTTP action:
const fileUploadAction = internalAction({
args: { fileName: v.string(), file: v.bytes() },
async handler(ctx, args) {
const uploadURL = await ctx.storage.generateUploadUrl(args.fileName);
const result = await fetch(uploadURL, {
method: "PUT",
body: args.file,
});
const { storageId } = await result.json();
return storageId;
},
});
source (https://docs.convex.dev/functions/actions)
And here's how you can upload files via upload URLs:
const uploadURL = await ctx.storage.generateUploadUrl(fileName);
const result = await fetch(uploadURL, {
method: "PUT",
body: file,
});
const { storageId } = await result.json();
source (https://docs.convex.dev/file-storage/upload-files#uploading-files-via-upload-urls)