Search problem after upgrading to v1.91.4 | Immich | Page 1

upper dagger Dec 22, 2023, 11:23 PM

#

After upgrading to v1.91.4 from 1.90 I have an issue with search (using clip):

Search fails (times out after 3 minutes)

I tried rebuilding smart search but it's going very slow and when I check the logs for immich-microservices I see this in the logs:

[31m[Nest] 1  - [39m12/22/2023, 11:12:31 PM [31m  ERROR[39m [38;5;3m[JobService] [39m[31mUnable to run job handler (smartSearch/clip-encode): TypeError: fetch failed[39m

[31m[Nest] 1  - [39m12/22/2023, 11:12:31 PM [31m  ERROR[39m [38;5;3m[JobService] [39m[31mTypeError: fetch failed

    at Object.fetch (node:internal/deps/undici/undici:11730:11)

    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)

    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:102:31)

    at async /usr/src/app/dist/domain/job/job.service.js:113:37

    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)

    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)[39m

[31m[Nest] 1  - [39m12/22/2023, 11:12:31 PM [31m  ERROR[39m [38;5;3m[JobService] [39m[31mObject:[39m

{

  "id": "fa877739-6e97-46af-a5f8-1a37df71e335"

}

stark mortarBOT Dec 22, 2023, 11:23 PM

#

:wave: Hey @upper dagger,

Thanks for reaching out to us. Please follow the recommended actions below; this will help us be more effective in our support effort and leave more time for building Immich immich .

References

Container Logs: docker compose logs docs
Container Status: docker compose ps docs
Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
Typesense: You can't fix this. Just wait until we get rid of it. Sorry.

Checklist

:blue_square: I have verified I'm on the latest release(note that mobile app releases may take some time).
:blue_square: I have read applicable release notes.
:blue_square: I have reviewed the FAQs for known issues.
:blue_square: I have reviewed Github for known issues.
:blue_square: I have tried accessing Immich via local ip (without a custom reverse proxy).
:blue_square: I have uploaded the relevant logs, docker compose, and .env files using the buttons below or the /upload command.
:blue_square: I have tried an incognito window, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

If this ticket can be closed you can use the /close command, and re-open it later if needed.

upper dagger Dec 22, 2023, 11:24 PM

#

machine-learning container has this log:

[2;36m[12/22/23 23:16:34][0m[2;36m [0m[34mINFO    [0m Booting worker with pid: [1;36m155199[0m                    

[2;36m[12/22/23 23:16:40][0m[2;36m [0m[1;31mERROR   [0m Worker [1m([0mpi[1;92md:1551[0m99[1m)[0m was sent code [1;36m132[0m!             

[2;36m[12/22/23 23:16:40][0m[2;36m [0m[34mINFO    [0m Booting worker with pid: [1;36m155204[0m                    

[2;36m[12/22/23 23:16:45][0m[2;36m [0m[1;31mERROR   [0m Worker [1m([0mpi[1;92md:1552[0m04[1m)[0m was sent code [1;36m132[0m!             

[2;36m[12/22/23 23:16:45][0m[2;36m [0m[34mINFO    [0m Booting worker with pid: [1;36m155209[0m

and immich-server container have this log:

[31m[Nest] 1  - [39m12/22/2023, 11:19:02 PM [31m  ERROR[39m [38;5;3m[TypeError: fetch failed

    at Object.fetch (node:internal/deps/undici/undici:11730:11)

    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)

    at async SearchService.search (/usr/src/app/dist/domain/search/search.service.js:63:35)] [39m[31mTypeError: fetch failed[39m

covert lynx Dec 22, 2023, 11:43 PM

#

The ML worker is being sent SIGKILL. How much RAM do you have? Edit: Actually it’s SIGILL, probably because of no AVX

upper dagger Dec 22, 2023, 11:44 PM

#

16gb - 2gb used on that node

#

buut.. dmesg does show something interesting: traps: gunicorn[3895862] trap invalid opcode ip:7fc6eb8045f2 sp:7fff88a6fb20 error:0 in libtorch_cpu.so[7fc6db12f000+16075000]

covert lynx Dec 22, 2023, 11:45 PM

#

Aha

upper dagger Dec 22, 2023, 11:45 PM

#

the cpu on that node is a bit old. Lemme see if I can move it to a different node

covert lynx Dec 22, 2023, 11:45 PM

#

Do you have Tag Images enabled by chance?

upper dagger Dec 22, 2023, 11:46 PM

#

no, that's disabled

covert lynx Dec 22, 2023, 11:47 PM

#

Hm that’s interesting, so torch itself can cause issues even when it isn’t being used

upper dagger Dec 22, 2023, 11:48 PM

#

cpu on that is AMD Phenom(tm) II X4 965 Processor

covert lynx Dec 22, 2023, 11:48 PM

#

The good news is that pytorch has been removed from ML entirely, so the next release won’t have this issue

upper dagger Dec 22, 2023, 11:49 PM

#

I wonder why it didn't happen on earlier versions. Maybe they just randomly got assigned to nodes with more recent cpu's

#

(running it in kubernetes, which has a bunch of used and onsale nodes :D)

covert lynx Dec 22, 2023, 11:50 PM

#

Maybe yeah, could also be from a dependency version bump

upper dagger Dec 22, 2023, 11:52 PM

#

I tagged the newer nodes and set a required tag affinity on it. Let's see now

#

yeah, seems to work now! Thanks

#Search problem after upgrading to v1.91.4

References

Checklist