Error after upgrading (Unable to login)

[robust snow]

Hi, I upgraded to version 1.5.0 and after that I was getting errors while logging in, I created a similar docker instance and restored my backup to version 1.1.0, it say that "Workspace not found" while trying to log in, is my data lost, how do I fix this?

the primary need to upgrade was that workflows stopped working on the older version I was using

I am not exactly sure of what exact version of twenty I was on previously, although it was surely above version 1.0.0

[mossy turret]

Hey @robust snow you need to incrementally upgrade your twenty instance across minor versions so it can proceed to execute each upgrade and migrations required by each version

You can check your previous version by looking at your workspace version in database located in core.workspace.version

[robust snow]

what query shall I run

and I believe my portainer messed up so it got upgraded to the latest at once which must have caused the problem

is it fixable though?

if yes, then how

[mossy turret]

You should restore the backup you have to its version

And proceed to upgrade incrementally

To the latest version

[robust snow]

so I can create a new instance with the version in my db then?

[mossy turret]

Double checking that your workspace version got updated to the just installed version is always a plus

[mossy turret]

I'm not sure to understand the question

To fix your issue you need to determine what version is coming from your backup

[robust snow]

the original docker stack is messed up as of now, should I reinstall that stack with the same version as my db or create a new stack

[robust snow]

how do I check that?

[mossy turret]

restore this backup from this twenty version

[mossy turret]

looking to your workspace version located at schema core table workspace column version

[robust snow]

can u give me a query, I am not too sure if I understand

[mossy turret]

SELECT version FROM core.workspace

Or using docker exec for example

docker exec -e PGPASSWORD="${PG_DATABASE_PASSWORD}" "${CONTAINER_NAME}" \
    psql -h localhost -U "${PG_DATABASE_USER}" -d "${PG_DATABASE_NAME}" -t -c \
    "SELECT version FROM core.workspace LIMIT 1;" | xargs)

Assuming you only have one workspace

Or if you have any access to a db GUI explorer following path i've shared

[robust snow]

can I get this from the backup file?

also seems like a syntax error in the query, I am not really used to postgresql

seems like I am still on 0.55.8

[mossy turret]

Then you will need to restore this backup in a 0.55.8 twenty version and proceed to incremental upgrade to next minor version

[robust snow]

sure

let me try

[robust snow]

Cannot read properties of undefined (reading 'reduce') error while logging in on v0.55.8

[robust snow]

this was the output when I tried to query

WARNING:  database "default" has a collation version mismatch
DETAIL:  The database was created using collation version 2.36, but the operating system provides version 2.41.
HINT:  Rebuild all objects in this database that use the default collation and run ALTER DATABASE "default" REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
0.55.8

[robust snow]

I tried multiple versions from 1.0.0 to 1.5.0, none of them seems to work, 0.55.8 also seems not to work, error says Cannot read properties of undefined (reading 'reduce')

[mossy turret]

are you still using a twenty postgres pg spilo database image ?

[robust snow]

this is my docker compose file

  db:
    image: postgres:16
    volumes:
      - db-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: ${PG_DATABASE_USER:-postgres}
      POSTGRES_PASSWORD: ${PG_DATABASE_PASSWORD:-postgres}
    healthcheck:
      test: pg_isready -U ${PG_DATABASE_USER:-postgres} -h localhost -d postgres
      interval: 5s
      timeout: 5s
      retries: 10
    restart: always

[robust snow]

Please help

[mossy turret]

Your db instance seems to be the correct one

[mossy turret]

I would like more context please

[robust snow]

please let me know what else is needed

currently v0.55.8 gives me the error - Cannot read properties of undefined (reading 'reduce')
v1.1.0 was working but certain elements like workflows were broken
when I upgraded, it went straight to v1.5.0 and broke everything
luckily I backed up the db with the command
I am unable to get this working on the version 0.55.8 and 1.0.0 and above

[mossy turret]

You twenty instance might have got corrupted by upgrading from 0.55.8 to 1.5.0

I would recommend restoring a backup of a clean 0.55.8 instance

[robust snow]

let me try that once

[mossy turret]

And proceed to upgrade incrementally

[robust snow]

Workspace not found on new instance of v0.55.8

and multi workspace is disabled

[mossy turret]

While restoring a backup of 0.55.8 ?

One that has not been attempted to get migrated to 1.5.0

could you please clear your cache
yarn command:prod cache:flush

[robust snow]

yes

[robust snow]

this needs to be run in the server container right?

[mossy turret]

Yes

[robust snow]

let me try

[robust snow]

Workspace not found same error

do I need to restart the container after the flush?

I made this backup on v1.x.x

which was somehow upgraded from v0.55.8

I do have a backup from before that, but it might not have some changes init

let me try with that one, if it works, it works

¯_(ツ)_/¯

[mossy turret]

nope

[mossy turret]

I'm not sure to understand

It's not a 0.55.8 dump then

[robust snow]

yes

[mossy turret]

Please restore a backup of the version of twenty you try to run

[robust snow]

I had installed v0.55.8 initially, but when I was upgrading it wrongly got upgraded to v1.x.x directly, it somehow worked, but now I am unaware of the version that was v1.x.x

and cant find it in the db also

v1.x.x was working somewhat fine, but then I again made the mistake to upgrade to v1.5.0 directly without sequential upgrades

[mossy turret]

the workspace ?

or the workspace version

[robust snow]

the version that is v1.x.x

[robust snow]

version

[mossy turret]

I'm not sure to undestand, you can't find it because it's 0.55.8 or because there's nothing ?

[robust snow]

it says 0.55.8 but it was not 0.55.8

and the one which was actually on v0.55.8 says that You're not member of this workspace.

can we vc if you are available?

[robust snow]

i feel i fked up the upgrade

and now think my db is fked

[robust snow]

Please help bro

[robust snow]

with the query u sent me it shows 0.55.8, but I was not using 0.55.8 while I backed this version up, 0.55.8 was the version I initially installed

[robust snow]

Hi

[robust snow]

@mossy turret let me know when you are available, my crm is dead, please help ☹️

[mossy turret]

Hey there, please try restoring a backup that has not been corrupted in the correspoding twenty version

Then proceed to incremental upgrade

[robust snow]

sure

[robust snow]

hey, even with the best backup, it says that You're not member of this workspace.

I had that backup on v0.55.8

[mossy turret]

Are you sure you haven't changed your credentials since then ?

Could you please try creating a new workpsace member

[robust snow]

nope

[robust snow]

how do I do that?

User does not have access to this workspace

[robust snow]

this happens when I try to signup

even if I try to change the pass on my previous account, it says You're not member of this workspace.

[robust snow]

maybe I can show it to u better on a vc stream on discord

[robust snow]

Hi

@mossy turret bro really sorry to bother you, nothing seems to work, please help me with the suitable solution 🙁

[mossy turret]

Hey all good have you tried clearing your cache using yarn command:prod cache:flush ?

[robust snow]

yes

[mossy turret]

Honestly I'm not sure understand where your workspace is at exactly

Could you please share the exact process you're following in order to restore your backup on the previous version

Do you correctly restart all your containers etc

echo $APP_VERSION in twenty-server will show what curent twenty version is running

[robust snow]

gives blank

all are running

TAG: v0.55.8

is set in docker compose

[robust snow]

should I try to restart again?

yea, it is not working

not letting me log in

I am sure I havent changed the username or the password

[mossy turret]

Please run echo $APP_VERSION is the twenty-server container

To double check it's in the expected version

After changing the tag you indeed need to restart your instance

[robust snow]

I dont even get a version

[robust snow]

this is a new instance, which I started on v0.55.8 and then restored the backup on

but this is what portainer shows

and this is what I see when I print all env vars

[mossy turret]

So it seems like your workspace has correctly been restarted on the good version

Adding a new workpsace member either using the invite link or classic password email is not working either ?

[robust snow]

I dont know how to do that

[robust snow]

and I cant log in from my admin account

[mossy turret]

Please find your invite hash in your database at core.workspace.inviteHash

https://your_twenty.com/invite/YOUR_HASH

and try creating a new workspace member this way

[robust snow]

ERROR: column "invitehash" does not exist LINE 1: SELECT inviteHash FROM core.workspace LIMIT 1;

[mossy turret]

select "inviteHash" from core.workspace

it's camel case

[robust snow]

public invite link is disabled

[mossy turret]

Please toggle core.workspace.isPublicInviteLinkEnabled to true

[robust snow]

Entity "apiKey" does not have a primary column. Primary column is required to have in all your entities. Use @PrimaryColumn decorator to add a primary column to your entity. on the crm page

[mossy turret]

Would need more context

Were the workspace member creation + logging successful in the first place ?

[robust snow]

I had only 1 user initially, that was the admin, I didn't try adding any other users, but my main admin account was working originally...

[mossy turret]

Which means you just got this new error out of the blue ?

[robust snow]

yes...

[mossy turret]

What about the process we were on creating a new user through the invite hash

[robust snow]

this error

[mossy turret]

You're missing a migration for sure

Around api keys

yarn database:migrate:prod

[robust snow]

uhm, I have 3 backups, all 3 of them seem broken then

[robust snow]

on server right?

[mossy turret]

yes

[robust snow]

do I need a restart?

[mossy turret]

Normally no, but just do by default at this point

[robust snow]

should I try to login or try to create a new user

[mossy turret]

I don't know how you got the error

Please try reproducing to the same point

[robust snow]

anyway none of it works 😟

[mossy turret]

Do you still have a live production available ?

Exporting everything as csv and re-importing in fresh new install

Seems to be the solution here

Depending on the complexity of your twenty instance

[robust snow]

see

1. I made a new instance of v0.55.8
2. Restored my backup
3. Tried logging in, but unable to login
4. Ran yarn command:prod cache:flush
5. Tried loggin in, but unable to login
6. Checked app version, which was as expected
7. Tried using hash to create new user.
8. Got Entity "apiKey" does not have a primary column. Primary column is required to have in all your entities. Use @PrimaryColumn decorator to add a primary column to your entity. on the crm page
9. We are here now

[robust snow]

no 💀

[robust snow]

had many complex fields too and workflows also

[mossy turret]

Please try restoring your bakcup on the exact version you were

I assume it wasn't 0.55.8

I think you will be able to access your instance on uper version

[robust snow]

the db says 0.55.8 but it was something like 1.x.x

I am not exactly sure of that 1.x.x version

[mossy turret]

If you have any versioning of your .env

[robust snow]

i got it from the twenty github as an example, it used latest

and thats where everything got messed up

[mossy turret]

Ok I see we should definitly remove this dockerhub privisioning

[robust snow]

I tried all major releases v1.0.0, v1.1.0, v1.2.0, v1.3.0, v1.4.0, v1.5.0

[robust snow]

😢

and on restarts also, it auto upgraded

that is what fished up

[robust snow]

do you believe it is possible to restore my crm...?

[mossy turret]

Yes for sure, in the worst case scenario we still have the data in the dump

Would you consider sharing a dump of your instance ?

[robust snow]

uhm, sure, it doesnt contain any secrets right?

[mossy turret]

Credentials secrets no, private data such as emails or inserted data yes

[robust snow]

uhm data may be considered private

may we hop on a vc

[mossy turret]

We can plan a vc

[robust snow]

sure lmk

[mossy turret]

We can have a call tomorrow 10:15am CEST timezone ?

[robust snow]

works

[mossy turret]

Cool ! see you tomorrow

[robust snow]

sure 😄

You are GMT +2 right?

[mossy turret]

CEST

[robust snow]

alright

[mossy turret]

Hey @robust snow I'm sorry but I'll have to postone our call to 25 sep 3pm CEST if it's also ok for you

[robust snow]

alright

[robust snow]

Hi, just a reminder...

[mossy turret]

Hey will be there on 3pm today CEST

[robust snow]

alright

[robust snow]

in #1147463229086380145

[robust snow]

@mossy turret sup bro 👋 found something?

[mossy turret]

Hey @robust snow, unfortunately nothing new on your issue I would recommend as discussed together to manually export your few workflows and data model in a fresh new twenty instance

[robust snow]

alright, also whats the ETA for the next update, so I will plan to start with the new update...

[mossy turret]

Next version should be published on dockerHub by next Friday

[silver siren]

Just for posterity reindec database use

psql -d "postgres" -U "postgres"
ALTER DATABASE postgres REFRESH COLLATION VERSION;
REINDEX DATABASE;
\c template1
ALTER DATABASE template1 REFRESH COLLATION VERSION;
REINDEX DATABASE;
\c default
ALTER DATABASE "default" REFRESH COLLATION VERSION;
REINDEX DATABASE;