#accessing with mobile app seems to crash microservices

1 messages · Page 1 of 1 (latest)

terse steppe
#

I recently upgraded the mobile app on my android pixel 4. I had been using fdroid, but it was too slow and I wanted to get on 1.84, so I uninstalled it with froid and installed from apk. I also upgraded the server to 1.84. Since then when I load the app on the pixel it will try and start the backup process and load the timeline, eventually it suddenly looks like it loads most of the timeline but then loses its connection to the server. If I try to load an asset it will take me back to the login screen and from there if I try to login it gives 404, then bad gateway. It looks like microservices is running out of memory or something and my server redeploys it. The app is configured with 64gb ram.
If i don't load the app on my pixel, I can use the web version on my computer without issue and the server seems stable. I am wondering if I some how reinstalled the app incorrectly?

jagged fractal
#

Can you share the logs?

terse steppe
#

Hi, thanks so much for the response. I have attached some logs. My system runs on truenas scale, which is using kubernetes. As I mentioned, the system seems fine when using only web. Something else that I thought was a bit strange was in the mobile app, after I installed 1.84 I was looking for the link sharing function and it doesnt seem to appear, when I click on the share icon I can share as always to individual apps, but no options to create a link.

📎 Immich_log_2023-11-03T081222.285735.csv 📎 immich_immich-machinelearning-7848cc56f8-rw5vr_immich.log 📎 immich_immich-typesense-6cd75b7f79-jc5k7_immich.log 📎 immich_immich-web-8d744fb66-bh9l2_immich1.log 📎 immich_immich-microservices-7854d8dc49-klc45_immich2.log 📎 immich_immich-9c6b58cc8-tsgbz_immich1.log
jagged fractal
terse steppe
#

Thanks Alex 🙂

jagged fractal
#

okay, so the mobile app is on the correct version

#

Link sharing functionality only available on the assets that are uploaded to Immich, local asset doesn't have that option

#

Can you add this to the .env file and restart the stack? NODE_OPTIONS="--max-old-space-size=8192"

#

I am pretty sure that nodejs always run on 4GB of ram default unless specified otherwhise with the option

terse steppe
#

I will try this shortly, I am working now 😦

#

Thank you

jagged fractal
#

no worry, let me know how it goes

terse steppe
#

ended up working too much today and didnt get to try this. adding environment variables on truenas scale is done via ui. (and its not obvious, to me anyway, if it has been deployed)
I also noticed some kubernetes 'stuff' that needed to be sorted out.(it uses cloudnative pg and the pods were not scaling for some unknown reason.) I fixed that and it looks like things are slightly better (lol) but the app is still behaving strangely. I will attempt to add the node_options soon. Thanks again.

terse steppe
#

Thanks agian Alex, which container needs these options added? microservices? And do you happen to know if it matters with the quotes or not?
There seems to be an issue with Truenas Scale not saving the variable when I add it. I can add it in the container like this export NODE_OPTIONS="--max-old-space-size=8192" but those options don't take effect (as far as I know) until the container is recreated. So I have raised this bug with truecharts developers. And I will report back. Are you able to shed some light on why this might have happened when reinstalling the app on my device? my only guess is that suddenly it tries to process all assets again and this requires more memory?
Thank you

terse steppe
#

in the microservices container it looks like node is running with very little ram? i might verifying this incorrectly though, this is in bytes?!
> process.memoryUsage() { rss: 36651008, heapTotal: 5128192, heapUsed: 3541560, external: 2123913, arrayBuffers: 10500 }

terse steppe
#

nevermind, this command seems to give more accurate heapsize. Since I cannot set this on pod creation in kubernetes, is it possible to restart the app after i export the node_options environment variable in the running pod?
> v8.getHeapStatistics() { total_heap_size: 5128192, total_heap_size_executable: 262144, total_physical_size: 4341760, total_available_size: 4342498800, used_heap_size: 3742376, heap_size_limit: 4345298944, malloced_memory: 147552, peak_malloced_memory: 172416, does_zap_garbage: 0, number_of_native_contexts: 1, number_of_detached_contexts: 0, total_global_handles_size: 8192, used_global_handles_size: 2752, external_memory: 2138087 }

terse steppe
#

So I noticed that the heapsize is increased if I run export of NODE_OPTIONS it looks increased?
> v8.getHeapStatistics() { total_heap_size: 5128192, total_heap_size_executable: 262144, total_physical_size: 4857856, total_available_size: 8636609192, used_heap_size: 4596816, heap_size_limit: 8640266240, malloced_memory: 147552, peak_malloced_memory: 680848, does_zap_garbage: 0, number_of_native_contexts: 1, number_of_detached_contexts: 0, total_global_handles_size: 8192, used_global_handles_size: 4224, external_memory: 2167802 }

but the out of memory error seems to be happening around 4gb? so I am guessing this needs to be sent via environment variable when the pod is created for it to get used....

`<--- Last few GCs --->

[8:0x237da310000] 484923 ms: Mark-Compact 4046.2 (4133.6) -> 4031.4 (4134.8) MB, 7018.85 / 0.00 ms (average mu = 0.088, current mu = 0.011) allocation failure; scavenge might not succeed
[8:0x237da310000] 492036 ms: Mark-Compact 4047.5 (4134.8) -> 4032.6 (4136.1) MB, 7046.20 / 0.00 ms (average mu = 0.050, current mu = 0.009) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xc99960 node::Abort() [immich_microservices]
2: 0xb6ffcb [immich_microservices]
3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
...
12: 0x14f7b76 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
13: 0x7f6daea19ef6
Aborted`

jagged fractal
terse steppe
elder beacon
#

the env var goes to the node vm, not immich

terse steppe
#

Thanks Marc, that is where I will try to set it then.