Hi all 👋 I've been exploring porting a large blog to Astro. Unfortunately, I've been running into speed and memory issues when the number of docs gets into the thousands. I've been using plain markdown docs (not MDX) with content collections. Issues with npm run dev and npm run build. Are there any tips for working with large collections? Or alternatively, guidelines for doc/collection limits?
#Scaling to thousands of markdown docs with content collections
dense sequoia
orchid wave
I started an initiative to analyze scalability to thousands of pages in this project https://github.com/MicroWebStacks/astro-examples#25_markdown-scale
basically a python script generates a different number of pages, builds different configurations then saves the results in json I published some examples results here https://microwebstacks.github.io/astro-examples/
well, don't expect miracles, you never know exactly when it stops until you try and it fails, but when it come to such large scale project a serious analysis would be required before, with multiple fallback strategies
- split website on multiple sub-sites under different sections
- fallback on full ssr with managed cache which can give best of both worlds for large scale websitex ssg+ssr, see my proof of concept, I never deployed it, just a study https://github.com/MicroWebStacks/astro-examples#20_ssr-cache-proxy
I started with mdx and md, .mdoc was on the pipeline but the progress now stalled, prs are welcome 😉
dense sequoia
Thanks @orchid wave, this is a great resource! I'll dig into it more after work 🙏
dawn jackal
Idea from #1107772135658750115 message
If you don't need all the content during development, you could implement a toggle in getStaticPaths that uses import.meta.env.MODE to detect the environment and if it's development return a subset of pages (basically only what is needed in dev to do what you want)
https://docs.astro.build/en/guides/environment-variables/#default-environment-variables
Obviously this assumes getStaticPaths is the limiting factor here