Running into heap limit for a small site

[sharp sigil]

I have a fairly small site (around 69 pages) which was building just fine with astro@2.0.0 with the default memory (2096 MB on my machine).

Recently, I tried to migrate to astro@2.0.6. astro build threw the following error.

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 00007FF73592012F node_api_throw_syntax_error+175407
 2: 00007FF7358A5AF6 SSL_get_quiet_shutdown+63110
 3: 00007FF7358A6EB2 SSL_get_quiet_shutdown+68162
 4: 00007FF73633FF24 v8::Isolate::ReportExternalAllocationLimitReached+116
 5: 00007FF73632B282 v8::Isolate::Exit+674
 6: 00007FF7361AD0DC v8::internal::EmbedderStackStateScope::ExplicitScopeForTesting+124
 7: 00007FF7361AA2FB v8::internal::Heap::CollectGarbage+3963
 8: 00007FF7361C0533 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath+2099
 9: 00007FF7361C0DDD v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath+93
10: 00007FF7361C972A v8::internal::Factory::AllocateRaw+810
11: 00007FF7361DCB6D v8::internal::FactoryBase<v8::internal::Factory>::NewHeapNumber<0>+685
12: 00007FF7361E1018 v8::internal::FactoryBase<v8::internal::Factory>::NewRawTwoByteString+72
13: 00007FF7361D7676 v8::internal::Factory::NewStringFromTwoByte+262
14: 00007FF7361D76DA v8::internal::Factory::NewStringFromTwoByte+26
15: 00007FF73604FE8E v8::internal::Intl::CompareStrings+37406
16: 00007FF735FF959D v8::internal::LookupIterator::wasm_value_type+106925
17: 00007FF735FFAB21 v8::internal::JSV8BreakIterator::GetAvailableLocales+4433
18: 00007FF7362DC9A4 v8::internal::Builtins::code+133284
19: 00007FF7363DD6C1 v8::internal::SetupIsolateDelegate::SetupHeap+558449
20: 00007FF7363AE212 v8::internal::SetupIsolateDelegate::SetupHeap+364738
21: 00007FF6B71A5926

A retry with 4GB memory gave the same error. I was finally able to build the site at 6GB memory allocation but this seems a huge regression to me.

• I am only using Astro's sitemap integration
• The Remark and Rehype plugins that I'm using worked fine with astro@2.0.0 (the build was getting generated with 2GB of memory)
• I am not using any frontend framework; it's plain HTML, web components, Sass, thing

I have been trying to create a reproduction of this issue but no luck so far. It is happening for my site consistently though.

I do have some pages where I have embedded content with very small images are inlined as base64. Could they be the root of this issue?

[sharp sigil]

Ok. I filtered out some of the pages with inlined bas64 images and it seems this is indeed an issue with embedded content in the HTML. Still not able to create a reproduction either on StackBlitz or CodeSandbox 😢

The issue happens when the images are embedded as data uri directly in the markdown file.

[slow dagger]

That could be the root cause, I'm currently working on benchmarking Astro SSG/SSR for markdown and in SSR as example all markdown html is duplicated raw inside js files.
https://microwebstacks.github.io/astro-examples/
maybe this could give you an insight before you scale pages. I only so far worked on page numbers with small content but did not yet explore pages with big size content and images
https://github.com/MicroWebStacks/astro-examples#25_markdown-scale
here example built SSR page chunk

just to give an idea, images kept aside 10K pages result in "7.95 MB" build size but I did not check the max memory consumption though...

[formal dagger]

I suggest opening an issue so we can track this

[edgy crane]

Open a New Issue
https://github.com/withastro/astro/issues/new/choose

[sharp sigil]

@slow dagger I ran into this issue with a site of ~120k pages but never expected this to happen for a small site (all of which is around 5MBs)

[slow dagger]

I see, as happydev suggested let us track this in an issue. If you isolate example big pages, I'd take similar load in my benchmark to identify if it's a bug or just memory limit and how to overcome the bottleneck.
Can you try to reproduce it with a minimal exampl and dummy content ?

[slow dagger]

A github repo even non reproducing the error would be good enough to start with, also in the issue provide details about your os, cpu and memory sizes

[sharp sigil]

Yes, I'm trying to create an isolated example. The problem with my issue is that the page is not very huge. The embedded image is around 200kb. I even tried to recursively create pages with the exact same image to isolate the problem but that is not triggering the issue. I might have to debug this even further.

[slow dagger]

that's a good sign, not being able to reproduce an issue is a sign it still might be related to something else, unknown 😦

[sharp sigil]

I've created an issue with a reproduction here: https://github.com/withastro/astro/issues/6135

[slow dagger]

did you try the nodejs argument "--max-old-space-size" before, might be a different kind of leak here but just to know if it can work with astro and how to use it ?

[slow dagger]

https://github.com/withastro/astro/issues/6135#issuecomment-1417644949

[sharp sigil]

I did actually. That's how I was able to build the site with 6GB. Originally, it was capped to 2GB on my machine

[slow dagger]

Could you please help me how did your cmd look like ? How did yoy pass the argument through astro build ?

By the way, as in the link I posted above to the github issue, your problem is not memory in this case, it's just the base64 image in md.

[slow dagger]

got it thanks to Nate NODE_OPTIONS="--max-old-space-size=3000" astro build

[sharp sigil]

Yes, this is what I eventually used to pass the build. Although, I am now approaching it differently by externalizing the tweets and loading them lazily on demand.

[slow dagger]

note that limitation only relate to markdown not to .astro using .json given that you'd be tempted to move to ssr. But well client side loading is also a way out.