[SOLVED] Migration from `1.5.8` to `1.5.9` gets stuck with no feedback

[sage wren]

in Appwrite console shows the corrct version in the footer

[weak orchid]

What do you mean? Would you please provide more details and context?

[valid spear]

I think I had that too. It took really long on my dev server. No progress even after 2h, eventho my dev server had barely any data

Nothing in the logs

[sage wren]

After running the update script everything seems okay, but when running docker compose exec appwrite migrate it gets stuck saying Starting Data Migration to version 1.5.9 and nothing happens. I checked the docker logs and nothing there as shown in the screenshots

left it for almost 6h and nothing happened. When I access the console it works and in the footer shows the correct version

this is the logs from appwrite container

I am running Appwrite on Debian Bookworm

[sage wren]

just one more thing to add, redis was consuming a lot of memory until i restarted the computer. now seems to be back a normal consumption

[weak orchid]

And then it eventually finished?

[valid spear]

No, I stopped after 3h.

I think that server did not have that many data to begin with. Like 1k documents so it should have never taken that long

[weak orchid]

So weird...I tested migrations...

[valid spear]

I could see that my redis server was working when the migration was running. Maybe it would have finished but I saw no progress after 3h and that server only had 1k documents. It should never take that long no?

Maybe it could not migrate because of the changes in functions?

[sage wren]

No it did not finish

I cancelled the operation and restarted everything

@valid spear in previous migrations that I ran it took no more than 1 min

[valid spear]

Did you go from 1.5.7 to 1.5.8 to 1.5.9?

[sage wren]

yes

[weak orchid]

My test definitely showed a lot of output like usual

[valid spear]

For me it it did take like half an hour but on my production server I have tons of data

[valid spear]

Nah, no output for me

[sage wren]

from 1.5.7 the migration did not ran because of a bug I saw opened here so I did not report it

when I ran the migration from 1.5.7 to 1.5.8 I had the same issue mentioned here https://discord.com/channels/564160730845151244/1270988067091779644

Mariadb logs seems fine

[valid spear]

I went through all logs, none has shown any errors

[sage wren]

what os are you using?

[valid spear]

Debian 12

It is running in docker so the OS should not play a role here

[sage wren]

yea, but sometimes could be docker itself :/

[valid spear]

Very, very unlikely

[sage wren]

are this warnings relevant?

[valid spear]

No

[sage wren]

I just run tests for 2 apps using this instance of appwrite and everything seems fine.

this instance we use it for dev/staging

[weak orchid]

I'm worried the migration didn't finish 😬

[sage wren]

yes me 2 😅

if there's anything I can test or do on my end to help debugging please let me know

[waxen zenith]

Same problem here. I updated my local appwrite (Docker Desktop on Windows) from 1.5.7 -> 1.5.8 (Migration Error) -> 1.5.9 without any Problems. On the remote Testing server running Debian, I tried updating from 1.5.7 to 1.5.9 and now it is stuck on Migration showing "Starting Data Migration to version 1.5.9"

I'm able to login to the console. Everything seems to work exept for Functions. Error "404 Function with the requested ID could not be found." when I try to access the functions page

Update: Same 404 on visiting the Functions page on the local instance where the migration was successful

[sage wren]

even with new created functions?

[waxen zenith]

I also get this error when i do not have a function at all. Just created a new project to test it..... 404 when i click on Functions

[valid spear]

Thanks for clarifying, I assumed it might have been because of the functions because Appwrite did introduce new changes to functions under the hood I think.

[sage wren]

same here

[errant thorn]

Exactly the same behaviour for me (migration and functions). The production apps I have running seem to not experience any problems at the moment. Let me know if I can test anything

[sage wren]

I have deleted everything and started new instance of Appwrite and everything works nice but the functions still giving 404 error

[bleak phoenix]

hey, i also have the issue (functions are also 404) tho on what i can see the migrate is using some CPU and also i'm not sure at 100% but it also talk a lot to the redis container

when reading CPU from portainer you can divide by two to have the same value on the CLI commands

[sage wren]

logs

[manic nacelle]

is there any error with specification or similar?

[bleak phoenix]

on what i have i only see the migrate message who is "stuck"

redis logs are fine

appwrite logs shows nothing (except the 404 for the functions like sam)

i also checked the function worker, migration worker, executor, function scheduler and no error there

[bleak phoenix]

i think i know why after looking at the code

looking at this file : https://github.com/appwrite/appwrite/blob/1.5.9/src/Appwrite/Platform/Tasks/Migrate.php

line 92 you define the $projects variable with the console project in it

but you don't add new projects, there is only a foreach line 107 and a skip for the console project

oh mb i said nothing

i didn't see the define line 135

ummmm i don't think this is supposed to happen

i added 2 logs inside the foreach to see the projects ids that was currently proccess and it's the console project ...

i think this is the issue

[sage wren]

does this solves the functions 404?

[bleak phoenix]

i don't think the 404 can be resolved if you already have it. I think what happen is that the migration was ongoing for the console project and broke the Functions tab.

So i think to fix it we will need some rollback the database before the migration and edit a bit of code to make the console project skip the migration

For now the better will be to stay in 1.5.8 with a database rollback, wait for the appwrite team to fix it (or a contributor), and after that upgrade to the newer version with the patch

[sage wren]

fair enough. thank you 🙏

[bleak phoenix]

i'm currently taking a backup and i will check by setting the condition of internalProjectId to be equal to console since for me it's like that

it's just long for me cuz my backup is a 100GB compressed tar so it's long to load and extract

[bleak phoenix]

so i'm currently running the migration again with the console project skipped and i have now a message to say Migrated 1/6 projects... (on the UI i have 5 btw but i think it's + console) and then i think it's stuck there (for now 7 minutes)

i will let it like that for a bit more but i don't think the internal project id was the only cause

[weak orchid]

so is it stuck on the clear cache line?

[weak orchid]

no no. functions 404 is totally unrelated. It's fixed as part of https://github.com/appwrite/appwrite/issues/8510

[bleak phoenix]

oh ok ok

[bleak phoenix]

i will check, i will add some logs to see where it's stuck rn

[weak orchid]

I just tested again and it does work:

[bleak phoenix]

looks like an infinite loop

let me log the keys to see

[weak orchid]

ouch jesus

thanks

anyone able to create a github issue for this? 🙏🏼

[bleak phoenix]

so ye it's not false but empty

[bleak phoenix]

sry i'm editing in the container directly

[weak orchid]

that's kind of odd iterator isn't null though...

[bleak phoenix]

looking directly in the redis i can see some datas

[weak orchid]

what was the code for htis?

[bleak phoenix]

this (or this file : https://github.com/appwrite/appwrite/blob/1.5.9/src/Appwrite/Platform/Tasks/Migrate.php)

[weak orchid]

still makes no sense to me why iterator is not null, but keys is empty 😵‍💫

[bleak phoenix]

true

[weak orchid]

🤣

SCAN family functions do not guarantee that the number of elements returned per call are in a given range. The commands are also allowed to return zero elements, and the client should not consider the iteration complete as long as the returned cursor is not zero.

Source: https://redis.io/docs/latest/commands/scan/#number-of-elements-returned-at-every-scan-call

[bleak phoenix]

(for the scan with the match i forgot to do with 0 but it's also empty)

[weak orchid]

FYI, I've created an issue: https://github.com/appwrite/appwrite/issues/8518

[weak orchid]

can you try this patch and see what happens? https://github.com/appwrite/appwrite/pull/8519/files

[bleak phoenix]

yep it works now

you can see the keys: not at 0 anymore, on some others projects i can see 2 or 3 keys but they are deleted in redis

[bleak phoenix]

as i saw in the project page it will be available in the next version 1.5.10, do you have an idea of when it might be released ??

[weak orchid]

It's been unofficially released: https://hub.docker.com/layers/appwrite/appwrite/1.5.10/images/sha256-aba97f0a183d1dff8fa08899bebf5ffbb5079c2cf585f0c4e984b2732a8d8380?context=explore

[bleak phoenix]

oh pog

🎉

(also the 404 for the functions page is patched too)

and no data was lost

[waxen zenith]

Same here. Everythings seems to work again on my local installation

[sage wren]

works like charm. Thanks a lot 🙏

functions also works as expected

[valid spear]

For me too

[weak orchid]

Thanks for your patience! 🙏