proceeding with chain halt (even tho Prop385 failed quorum)

[peak folio]

good to hear that we can continue

[gaunt crown]

proceeding with chain halt (even tho Prop385 failed quorum)

[queen vigil]

yeah lets go please

[vapid grove]

Yes, makes sense to me.

[hollow holly]

As long as everyone sets their halt heights we should be good.

[lilac sphinx]

Looking forward to see forked chain in action and where we end up

[tall mason]

Of course, I couldn’t assemble it because it didn’t actually work; during voting for this item it gave a VPs error..

[hollow holly]

Yeah, it'll be fun. The last fork I was part of took 5 hours

[queen vigil]

wow, your role is so cool

[tall mason]

the role is so cool and so is the scoring on the leaderboard

[rotund niche]

Yes, lets go please

[kindred flare]

Good solution, especially since we have already voted for the hard fork in proposal 316)

[near tapir]

after last chain helt i cant unjail my validator it keep giving me rejected by vps error how can i fix this issue guys

[peak folio]

hard fork is the solution

[somber lava]

so, do we set a halt-height?

[latent quarry]

because of failed prop, i think yes we should.

[somber lava]

restarted with, worth adding as an announcement I suppose
namada node ledger run-until --block-height 229614 --halt
not actual atm

[fallen turtle]

restarted my Validator with this command will suspend node at 229614

namadan ledger run-until --block-height 229614 --suspend

[somber lava]

btw, wondering what is the difference, in the end it is the same halt/stop/etc.

[cosmic mason]

Hello brother, do you have any information about the next batch of S-class submissions ?

[somber lava]

no

[jaunty pawn]

hey everyone, how would you all feel about postponing the hard fork until Wednesday if we could provide binaries for all steps that would remove the need for anyone to build the Namada repo from source? I think this could be really nice and possibly make the hard fork run more smoothly. Let me know your thoughts please, and thanks for staying engaged!

[strong fable]

I think this is a good option. Let's wait one more day)

[fallen turtle]

I will vote for this . totally agree wit you

I think i will restart my node again without halt part till its fully decided

[fallen turtle]

restarted my node without --block-height 229614 . till we decide when is the exact time .

[jaunty pawn]

Btw, for moving the hard fork until Wednesday, feel free to stop and restart your node just with namadan ledger run until we pick a new block height if you are currently running with run-until

[gaunt crown]

@here

[peak folio]

Yup good

[rotund niche]

I think this is a good option

[shut nest]

I am in favour

[vocal jetty]

agreed

[round bear]

ok

[vernal grail]

Cool! Anything to make this go smoothly! No objection here!

[compact hill]

It's ok to me

[zenith shadow]

Sorry to ask. But did we still get the roids for voting? I mean we tried to vote tho

[untold raven]

agreed

[tall badger]

well, lets do it

[signal pivot]

it's okay

[distant wolf]

np, team

[vapid grove]

thank you 👍

[unkempt scroll]

Good to go

[river chasm]

the further along the week we go, the more problematic if any issues arise that will delay eventually. we also need to consider when it ends and if all v2 activity should be during a major holiday for a good part of the world. but that being said, I'm ok with wednesday, but tuesday better imo (and I don't see why a full day is needed to build binaries)

[rapid roost]

I agree with this. A line needs to be drawn in the sand to delineate when the SE ends, as well as which steps are left to perform. Otherwise, I feel we risk a growing sense of apathy and risk credibility the longer this draws out.

[keen aspen]

I agree, I think providing binary code makes it easier for everyone to upgrade painlessly.

[iron basin]

No problem for me, anytime

[placid garden]

go go

[placid garden]

what the command for?

[somber lava]

start the node with halt at the target height

[placid garden]

🫡

[eternal sparrow]

so should i run this command? my validator is still jailed because of the bug

[somber lava]

please pay attention to the announcements: #📣-se-announcements message
no need to run with halt height atm
crossed out that message to don't confuse ppl

[jaunty pawn]

Please take a peek at the updated instructions and newly provided files for hard forking: https://github.com/anoma/namada-shielded-expedition/pull/357

[river chasm]

where do we see the complete file and not just the diff? sorry I'm a bit of a git noob

or has it been merged already?

[iron basin]

Here's the actual link to the hard fork instruction: https://github.com/anoma/namada-shielded-expedition/blob/brent/hard-fork-instructions/HardFork.md

[jaunty pawn]

@river chasm some of the needed files (the new wasm files and the make-db-migration binary) are in a dir called hard_fork. This is also mentioned in the HardFork.md

[eternal sparrow]

should i make some preparations for the hardfork?

[somber lava]

#1221630593499009164 message you can read to understand what to do, ask if not clear, wait for the upgrade block announce
p.s. apmk 🙂

[river chasm]

is this the updated one? I wasn't sure if the pr had been merged already

[river chasm]

good stuff. only point I'm missing is a bit of clarification on the relationship between namada ledger, namadan ledger, and namadan node

[iron basin]

Yes, it's updated, the pull request was merged just an hour ago.

[river chasm]

alright

[indigo magnet]

Did I understand correctly that if a person decides to completely reinstall the node, then after your fork he will simply install the updated version 0.32.1? This means no migration, just re-installation.

[rigid tendon]

It would be great if you could announce the new exact block height before reaching the old announced height of 229614

[past olive]

What happens after the hardfork, to the validators that dont upgrade ?

[iron basin]

They will be abandoned, useless.

[past olive]

I was thinking in terms of the competition 🙂

[river chasm]

if not running a validator but just a node, I don't see any issues with this? but I'm not the authority..

[river chasm]

good question

cause for disqualification? 👀

[gaunt crown]

thanks! okay i'll try to do this, it shouldn't be too much more than 24 hours before the target time

[tall badger]

Do we need to copy over priv_validator_state.json before running v0.32.1 ?

[river chasm]

you need to read the entire instructions doc

[silent junco]

I looked at the manual. I didn't understand at what point we should update the binaries to 0.32.1? After the migration or immediately after the node halt? And then proceed with the migration

[river chasm]

I would do it at the appropriate step in the process (once you start using that version).

[river chasm]

re the question right above, maybe it could be clarified at which point it would be good to update the namada binaries in bin-directory to new versions? (for me it looks like this is right before performing the actual state migration?)

[iron basin]

After, the target block height reached and the node stopped.

[river chasm]

the guide doesn't specify..

[iron basin]

As I can see and understand, update Namada Binary to v 0.31.10 before chain halt, v0.31.10 has additional migration features patched. Once the chain halts, Get migrations.json then you should upgrade to v0.32.1 and continue the state migration.

Took me a little bit but I got it 🙂

[somber lava]

you don't need to update and restart the node
you can just use that new binary in parallel for the introduced commands

[river chasm]

again, guide doesn't specify when they should be upgraded systemwise. this isn't difficult

[somber lava]

the same as it was with consensus-key change until the command was not merged into the main branch
there was a specific branch that allowed to create a binary with the required command

[river chasm]

not disagreeing with your take, just could be included specifically

[iron basin]

Anyway, I backed up the Shielded Expedition folder, anything can happen 🤣

[past olive]

@gaunt crown @jaunty pawn What happens to my node, If I forget to suspend, and my node goes passed the expected height of the hardfork ? Would I need to resync from scratch or something like that ?
Do we need to prepare for snapshots just before / at the suspend height for people that might get into that situation ?
How does one start/restart a node from a snapshot after the upgrade, if the structure of the database changes ?

[shut nest]

Are we executing all these command at the home directory level if I understand correctly? Create a folder and generate these required files there and update database all from the home directory level?

[shut nest]

We actually built from source over there. Here we can directly git clone the binaries. I would say we can copy and keep the binaries in separate folders for the three different branches, execute commands, and then move binaries for 0.32.1 in /usr/bin folder after hardfork. Another way is to change bashrc so correct binary is invoked when execute Namada commands. Does this sound correct?

[shut nest]

Is that right, if are modifying database in the base repo ‘.local/share/namada/shielded-expedition.88f17d1d14’ wouldn’t that mean once database is modified only the new binary is restarted and new chain progresses? We won’t be able to run two binaries simultaneously

Or do you mean, have separate binaries already copies so they can be executed easily without update binary at /usr/bin level

[jaunty pawn]

Hi everyone, going to try to answer some questions here

Regarding upgrading binaries for your system, I see some of you have it set up that binaries are in your /usr/bin aka in your path such that you can call namadan without a path for example? This sounds like a feature of one's unique setup. As some of you assumed, once the first instance of namadan-0.32.1 is mentioned in the HardFork.md, you can use the new binaries universally for everything, since we are on a new chain after this

Perhaps I'll try to clarify this

But I wanted to be as general as possible since perhaps some operators don't put binaries in a folder like /usr/bin. After all, these binaries aren't running in the background - you use them to execute certain functionality like running a node, and they can exist wherever you'd like them.

@past olive if you forget to suspend your node and don't upgrade, your node is not going to be able to progress past the hard fork height anyway. If you are too late to the upgrade, it is possible you might be left behind, but ultimately there will be some post-hard fork snapshot from which you can sync. No syncing from scratch will be required

@past olive a snapshot should basically just be an entire base directory you get from someone else from which you can simply do namadan ledger run

[past olive]

ok, thx for the explanation (Im asking because Im not sure to be available at 2pm UTC tomorrow, and I want to know my options 😅 )
Can I do the upgrade later in the evening / night for instance ?

[jaunty pawn]

@shut nest you can execute all these commands from where you usually execute your commands. Your home dir is probably fine, I just don't want to give specific advice since many oeprators may have different file structures. But specifically, the make-db-migration will look for a folder called wasm relative to wherever you run the command from, so make sure that and the two files within it are in the appropriate place

[jaunty pawn]

I'm not sure if you will be able to do the upgrade much later than everyone else because part of the upgrade involves advancing the chain 2 blocks, which obviously requires enough voting power online. You can certainly try it out (this is part of the experimentation), but if it does not work and the chain has already forked, then you'll be able to sync from a snapshot

[past olive]

This is going to be quite an adventure...

Do you think we have enough real active VP to advance the chain those 2 blocks, given that we didnt even reach quorum on the proposal 385 ? Isnt there a risk being stuck there for hours if we dont have enough shielded-100 online ?
Would it be worth distributing naan to all active Pilots so we garantee having enough VP ?

[river chasm]

you set your node to halt at specified height before this time

[past olive]

yeah but if Im unable to run the rest of the commands in due time, does it have any use ?

[river chasm]

do we risk then as we are in fact doing this that the latecomers who are right after the 2/3 vp is online missed the slot?

[river chasm]

hm idk lol

[jaunty pawn]

Yeah basically at the update-db step, you should run the command and then do nothing, waiting until enough VP is online. Once that happens, your node will progress two blocks and then either halt or stall (may or may not need to CTRL+C). After that, you all need to start as normal again with namadan-0.32.1 ledger run, which itself needs enough VP to progress ofc

So there could be some latent periods in these steps, but eventually we will get through

A fun adventure!

I do think we have enough real active VP to advance the chain given that the chain is making blocks still

[river chasm]

but if this process starts right when enough vp is online, what happens to those who come on five minutes later?

it then becomes a game of "who is faster at the console"

or has the faster hardware

if I understand the process correctly

[jaunty pawn]

I think those who might get "left behind" would only be those who did not do update-db yet before enough voting power has done namadan-0.32.1 ledger run, effectively completing the hard fork

If you did update-db and then go to sleep, while everyone else started the new chain already, you should still be able to simply ledger run and sync back up

[river chasm]

but all this will take place in the matter of minutes right?

or does the update-db step take a longg time?

[jaunty pawn]

It could take place in a matter of one minute, or it could be hours, we are going to find out

the update-db step takes perhaps less than one minute to actually run

every step takes a very short time to run

[river chasm]

I see a scenario happening here where a lot of people get left behind even if they are ready at 14utc

[jaunty pawn]

It is all possible, which is why we will work to have post-hard fork snapshots as soon as possible

[river chasm]

it'll be a rat race to run the right commands quick enough

[jaunty pawn]

I hope it'll be a rat race to run the right commands so that we can fork efficiently

[river chasm]

would it be possible to put in a delay in the software so there's like a coordinated hour-long wait before upgrade is done simultaneously? that would be pretty cool 🙂

[river chasm]

I feel iffy about the potential of people getting left out

[jaunty pawn]

@gaunt crown and I discussed coordinating some period after chain halt where we tell people NOT to update yet, in order to allow us to produce and upload a migrations.json with a shasum and let operators try it themselves and see if they produce the correct one

And I was envisioning something like a 1-hr long period as well

[river chasm]

that does sound like a good idea imo. it still does not solve the fundamental issue of what happens when you say "go" if there's potential for milliseconds making a difference if you're in or out. again unless I totally misunderstand this

[gaunt crown]

yeah like imagine this happening on mainnet
we'll have rough social consensus about whether or not we have a good state dump to proceed with, independently produced by many sources

and it will be clear if an individual operator has somehow produced a different state dump, because the hash won't match

it will take a few minutes and we'll see that like 98% of operators will report the same hash, and we'll be good to proceed

and even then, it will take probably at least 1hr and maybe like 2hrs before we have enough voting power to resume signing blocks

[river chasm]

my issue is with the succeeding step, when we try and produce those two blocks. if there is potential that anyone not in on that in realtime gets "left out".

though I'm not 100% if that's actually the case

[shut nest]

Brent in that case, do you recommend cloning binary for each branch in separate folder and running it for respective step. Once finalized, ensure you are running 0.32.1 nice migration has been completed and database has been modified?

[river chasm]

I think most would want to copy the binaries to /usr/local/bin.

especially if running as a service with default service params

as per namada docs

[gaunt crown]

you're correct

if someone isn't online during this period, we will have a post-fork snapshot that they can use

[shut nest]

Yeah, I might be available for an hour or so only, so if I update-db and run using 0.32.1 it should automatically progress forward when enough VP is online most likely in a few hours. Then it will halt and we restart to start validating blocks in the new chain, correct?

[shut nest]

Yeah, that can be done for the last binary is what I meant 0.32.1

[river chasm]

what I'm saying is, won't this issue also apply to people who are online? but happen by chance to be among the latecomers? (maybe semantics)

[gaunt crown]

good question, one sec

[shut nest]

Another option is replace as you go

[river chasm]

I mean you are correct if by online you mean the server is on and waiting, but as I said above, it'll be a race to get their server online first and then inevitably some will be left out

[river chasm]

that's what I would do I think

it's just two binaries after all

[shut nest]

Yeah, 1) so now since we know the block we can execute chain halt command and it should ensure chain halt happens that the respective block. This should automatically used 0.31.9 as current binary. Once chain halts we clone 0.31.10 and replace binaries in /usr/bin and execute command to generate migration.json 3) Once migration.json has been verified we clone 0.32.1, update binaries in /usr/bin, run update db command, which might have an hour based on how long VP is achieved. 4) then we can stop and restart our systemd service once this halts

@jaunty pawn one question, I see ‘conversion_state.txt’ generation in step 1 for ‘generating migration.json’ but I don’t see it being used directly anywhere. Just checking, this file will be generated at the same folder level where we execute all commands (can be home directory or another directory and this is what you refer as working directory right ), and we just check it’s correctly generated right? Guessing it’s used internally for other commands, but wanted to check

[gaunt crown]

it will take a while before these two blocks are produced, like 1 - 2 hours probably before there's enough voting power to advance those two blocks

[river chasm]

ok cool

so everyone has time to "join"

[gaunt crown]

i'm guessing 1 - 2 hours for enough voting power, but we don't actually know

i think that everyone will have time to join tho

[river chasm]

ok perfect

[shut nest]

Yeah should take a few hours is my guess as last time

[past olive]

@willow lily

[jaunty pawn]

@shut nest the make-db-migration assumes that conversion_state.txt is in your current directory from where the binary is run

[gaunt crown]

okay thanks for feedback! no more tagging Brent for now, pls 🙂
he's going to focus on finishing things up to merge the instructions

[shut nest]

Done, and I think it’s pretty clear now, atleast for me. Thanks Brent and Gavin for making things clear and answering all questions. I think we are all looking forward to getting this done smoothly. Should be fun!

[willow lily]

hey there
@jaunty pawn if you could take a look at my doc I think it could smooth-up the migration for many pilots it will be nice
https://gist.github.com/darklankou/21c38d2f894d138ebc79c92ecebb63f8
it already installs the binaries needed.

and for the rest I will need it to migrattion to check everything is ok but it should be

[jaunty pawn]

Hard fork instructions are merged: https://github.com/anoma/namada-shielded-expedition/blob/main/HardFork.md

[zenith shadow]

So I can easily follow this right?

[jaunty pawn]

@willow lily just from a quick skim, you can replace all instances of my personal branch brent/hard-fork-instructions with main or whatever it is now. The PR is merged in the repo

@zenith shadow I still think it is a good idea to look at the hard fork instructions on the namada-shielded-expedition repo as well

[past olive]

A few typo / missing spaces, but nice break down to avoid tripping !
I'd add "Read the official hard-fork-instructions then you can follow this guide for a smoother experience" with a link at the top of the doc

[willow lily]

yep will do

[willow lily]

updated with main links
looking for typos

[willow lily]

you should always read the doc indeed

[river chasm]

small detail, why aren't we using --halt?

[somber lava]

curious the difference between --halt and --suspend

--halt                         Halt at the given block height
--suspend                      Suspend consensus at the given block height

[river chasm]

interesting

[vernal grail]

Hmm sounds like a stop and pause hehe. Wonder the same, maybe it's related to how much the node stops, perhaps resuming from a suspended state is quicker than a halted state?

[river chasm]

are we foreseeing any issues with indexer btw?

[eternal sparrow]

stuck?

[placid garden]

no , my indexer is synced with latest block

[river chasm]

that's not my question sir

[indigo magnet]

In theory, you will have to update the indexer database, right? After all, the blocks will start with 1.

[river chasm]

I don't think the block numbers will reset?

anyone?

[indigo magnet]

So wasn't there a migration from 0 to 1? Then the blocks seemed to reset.

[finite raptor]

what will happen to post-genesis validators? will they need to recreate validator?

[river chasm]

no that was a regenesis

which was not v0 to v1 in any case

[river chasm]

read instructions

[shut nest]

Its go time

[iron basin]

Time to work :}

[past olive]

lets get to it

[somber lava]

🤓

[round bear]

34bd7815ebc4d2344bd875ce4ae847bc42feb03099c69313a646a40a3aa2ea99 migrations.json

[shut nest]

When running make-db-migration getting this error

./make-db-migration: line 1: version: command not found ./make-db-migration: line 2: oid: command not found size: '789601472': No such file

I have conversion_state.txt and wasm folder in the same directory and I moved make-db-migration from hard-fork folder also to my current directory befoe running the command

@round bear Any suggestions? What am I doing wrong

@granite hare Sorry fpr the ping, any suggestion for what I am missing?

[past olive]

What hash did you run ? shasum ?

[willow lily]

d3f95fae0cb50ad928947e9eb177834b migrations.json

[shut nest]

Any chance you can answer my question, Having rtouble when using make-db-migration executable?

[past olive]

34bd7815ebc4d2344bd875ce4ae847bc42feb03099c69313a646a40a3aa2ea99 migrations.json

Got the same with a sha256sum.
My node is waiting for the 2 blocks

Thanks a lot to @willow lily for the easy guide to follow in parallel of the hardfork guide ❤️
✅

[willow lily]

I got it on a full node
yopu can download the file from my validtaor here
wget https://namada.lankou.org/migrations.json

[shut nest]

Yeah, but wanna try generating it as well. Can you send a link to your guide, maybe I am missing something?

[past olive]

https://gist.github.com/darklankou/21c38d2f894d138ebc79c92ecebb63f8

here it is !

sha256sum migrations.json

[river chasm]

or wait y'all using sha256?

[round bear]

i used sha256sum, sha1 is b68d08158a0914f091c81f55a3658b620a028fd4

sorry for confusion

[river chasm]

let me check lol

confirm 34bd7815ebc4d2344bd875ce4ae847bc42feb03099c69313a646a40a3aa2ea99

[shut nest]

Fixed it

[finite raptor]

where is source code make-db-migration ? I have a problem with libssl.so.3.

[past olive]

Can you elaborate for the others that might encouter the issue ?

[willow lily]

you can dload the binary directly

[shut nest]

I redownloaded the make-db-migration directly, using wget and regenerated conversion_state.txt

as also files in wasm folder

[willow lily]

interesting I will write it into my guide

[shut nest]

Nothing different, just did these steps in a separate folder and it worked, rather than working at home directory level

[past olive]

🤔 I did it in home, no issue. Are you in /root or /home ?
Or maybe an issue while downloading ?

[shut nest]

Its worth mentioning doing these steps in a separate folder is preferable, with only binary, and wasm folder even when generating conversion_state.json

[willow lily]

I've updateed my proc with any new informations we have
https://gist.github.com/darklankou/21c38d2f894d138ebc79c92ecebb63f8

[shut nest]

how to check if migration.json is correct?

[past olive]

Yeah, that's the actual hardfork procedure 😄 Im ok, every worked first shot for me. Im just trying to gather the issues for others to be able to jump the pitfalls without sweating too much 🤣

[past olive]

sha256sum

[silent junco]

shasum migrations.json

[past olive]

Expect this : #1221630593499009164 message

[willow lily]

m5dsum / sha256 sum it

34bd7815ebc4d2344bd875ce4ae847bc42feb03099c69313a646a40a3aa2ea99  migrations.json

md5sum migrations.json 
d3f95fae0cb50ad928947e9eb177834b  migrations.json```

[past olive]

well done !

[shut nest]

Feels good, let me now do this for RPC node too

[finite raptor]

@granite hare ?

[shut nest]

Are you guys moving to next step or waiting for team to confirm this?

[somber lava]

#📣-se-announcements message

[shut nest]

Waiting as per announcement, lol, Raven is so fast 😅

Thanks

Looks like we can proceed to next step, trying it on my RPC node first

dry run giving no errors

[somber lava]

++

[shut nest]

Moving to live run

[past olive]

is that your live run output ?

[river chasm]

dry

[shut nest]

dry run output

Wanted to check with folks before taking the leap of faith 🙂

[past olive]

😄

I already jumped into the unknown 😄

[shut nest]

This is my live run output

[shut nest]

Broadcaster successfully started ✅

[jaunty pawn]

These are the right things to see!

#📣-se-announcements message

[shut nest]

I need to head out now. Can I leave the node for now. And restart when VP is reached as it might take a while? Will start it as systemd service when I restart it

[river chasm]

what do we do now?

safe to suspend and start ledger?

[shut nest]

I thinkj we need to wait till VP is reached and it moves 2 blocks on new fork

[river chasm]

it already did though?

[past olive]

1 block commited

Committed block hash: 1aac6fd71340fa82a030741d13b6ec9f0590e569f5e12505b2fbf0914db3c877, height: 237908