I wrote a seed data script to migrate products into the database, however createProductsWorkflow takes over 10 minutes per product that has thousands of variations.
I have 30 products that can have over 11 thousand variations each.
#Is there a way to improve performance of `createProductsWorkflow`?
green crystal
green crystal
looking at the database, there's nothing to indicate it's a database population problem, nor do I see anything in workflow_execution..it just hangs, for ever before going to the next product.
green crystal
Forked the product module to add console.logs and attempt to debug.
The product workflow completes but just sits there and doesn't complete the workflow for over 10-15 minutes depending on the amount of variations.
[createProducts_] - No variants to upsert for product ID: prod_01JGQA6PTEGK2XENKTKYGF4461
[createProducts_] - Images field is not an array for product ID: prod_01JGQA6PTEGK2XENKTKYGF4461
[createProducts_] - Completed processing related entities for product ID: prod_01JGQA6PTEGK2XENKTKYGF4461
[createProducts_] - Completed upserting all related entities.
[createProducts_] - Successfully created all products.
[createProducts] - Created products: serializing
[createProducts] - Created products: emitting event
[createProductsStep], Products created
muted tapir
@green crystal If i have to be honest with you i don't think that the default medusajs product module architecture is best for your use case
Just think about fetching the products, there is no default pagination for product variants
Have you considered splitting up your products, and linking similar ones together in a different way?
11 thousand variants is insanely large, i can't really imagine what your selling, but maybe my imagination is lacking
green crystal
@muted tapir The uber eats/door dash example "medusaeats" is a simply way to demonstrate this example. I know right now the medusaeats example only has single items, but in reality any real restaurant will have a default dish + many forms of customizations. Simple example is a pizza, or salad, there are base "pepperoni" and falafel salads but there's myraid of customization options. Chipotle might be another example, and so on.
muted tapir
But it won’t have 11000 variants for any single product
You wrote 11 thousand each
That’s like ordering a bowl of rice and specifying how many grains of rice you want, in which case you would use quanity
green crystal
Here's a clearer example
Total Variations is a combination of PortionSize options * Protein options * Veggies options * Sauce type options * Carbs options
it's quite common to have these level of granular customization for most food places, chipotle, cava, etc.
I can totally agree that the default product workflow and variation architecture medusa provides is inadequate solution to model such a menu, was just hoping maybe I was missing something, since the database is seeded properly but the workflow does not exit for a while, causing the seeding script to get stuck between each item.
green crystal
I think my path forward here is to build out a custom module.
Current module crashes with out of memory when trying to edit an existing product with that many variations.
A little rough, but I will implement a new variation module.
Thanks for the feedback.
muted tapir
Have you considered using the bulk file upload btw?
I see what youre saying, i actually have a similar problem
All of my products have add ons, and without the addons some products have 900 variants. So if all products needed a variant for every one of lets say 10 possible add ons, then I would hit the few thousand variants
So what I did was created service items and displayed them to the user as addons and i kind of linked them to their parent product with metadata, but in reality, the addons are just products/line items
However i don't think that is your problem though
@green crystal it's called the import products workflow
Also check this out:
https://github.com/medusajs/medusa/releases/tag/v2.1.3
Line items with custom prices.
If i were do redo my "service items" solution then this is something I would consider
However, it does seem like your issue is with the workflow timeouts on the main import, which is an easy problem to solve, and there are many ways to go about it. The whole point of these durable workflows is to make sure that long operations execute or fail predictably
green crystal
@muted tapir stayed up until 6am, I forked the products module, ripped out all the variation module logic.
Implemented my own data model for variations, which in fact extended cart line items to support my use case.
muted tapir
wow