restart

[upper ridge]

Doing the restart tmw afternoon is a big problem for me. Probably won't be able to participate in that

[rapid token]

Thanks for the update Gavin!

[stone estuary]

that's fine, minus one opponent 🧐 🤣
ps. forever yours with love ❤️‍🔥 you know I am joking

[upper ridge]

nice

[upper ridge]

No but seriously, why do we need to suddenly bring weekends into this

[rapid token]

Network stops for no one 😉

its 24/7

Anyway, everyone has a timezone issue regardless of when it happens

[upper ridge]

look at the context here - we're already overtime in this expedition and have pretty much put our lifeblood into this

[rapid token]

I stayed up super late for all previous restarts 😅

[feral light]

let's tell @burnt mesa to stop the chain on weekends

[rapid token]

At this point sleep is optional

[upper ridge]

well some of us have other commitments

[stone estuary]

better ask @lapis latch

[old salmon]

I agree. Mainnets are one thing, testnets are a different story.

[lapis latch]

I'm retired bruh

._.

[old salmon]

And I've never been a big fan of "sleep is for the weak" narrative...

[feral light]

well, Liver. there are things we need to talk about

🤣

[crisp helm]

lollllll

[sullen flame]

sleeplessness makes me weak

[old salmon]

same here

Unless it makes a big difference to the timeline, I'd personally advocate to pushing the restart to Monday.

[naive violet]

I don't see a big deal about the weekends, but I get it

[upper ridge]

we could do tmw morning / am if they can make it, that would give time to set up the stuff before weekend hits but personally I agree with you, monday is better

[sullen flame]

if you are planning to sync from scratch, devs are about to post the pre-release (it's probably building rn)
and you can start syncing it now so that it's ready by tomorrow

[brittle plinth]

The issue is not the weekend .. Main issue is there is no fixed timeline for that plan.. we can't wait all day for that.

[upper ridge]

this too

[upper ridge]

that's the version that will become release?

[upper ridge]

and it tends to shift. today was going to be restart. all of this is understandable ofc, but everything I said above too

[old salmon]

But doesn't that mean that as soon as enough stake syncs the chain goes live again? Meaning, there is no genesis time any longer.

[full veldt]

I feel you @upper ridge . I wasn't available all this afternoon so I was kinda secretly hoping for a postpone restart 🙂 we all have a life

[upper ridge]

I'm guessing the way to control it is to tell you guys when to start your validators? after all you have by far the most vp

[upper ridge]

was wondering about the same though

[old salmon]

Who's "you guys" in this context?

[upper ridge]

sorry, shielded-100

the cool guys 🙂

[trim spruce]

🧟

[old salmon]

Last weekend I suggested that validators agree to not start until Monday, since it was already Friday evening UTC at the time of my suggestion.

Could try that again but am aware of not going to the well too many times 😅

[upper ridge]

guessing it would have to be on official announce basis for it to work

and even then people could probably just restart if enough

can't we just all restart the chain tonight when pre-release hits? 😏

[rapid token]

We all have other commitments

[old salmon]

Yes, telling computers what to do is easy (for now), people are harder...

[old salmon]

That's precisely the point.

[upper ridge]

clearly not on those specific times. it wasn't a jab at anyone if it came across like that

[trim spruce]

The solution is as simple as effective: we just break the chain again tomorrow.

[burnt mesa]

We have to

[full veldt]

#1215054434925813907 message
@sullen flame the command you mention is namada ledger run-until --block-height I think

[tranquil ether]

well not an easy choice

• on one side I want to continue building and I need a working BC to do it
• I'm tired as everybody here
  if there is snaoshit ready we could restart tommorrow morning UTC, I think maybe

[stone estuary]

I used that one
namada node ledger run-until --block-height <#> --halt

[tranquil ether]

question is waht is slowing things down preventing from restart ?

this will happen tomorrow afternoon

anyway no hard feelings, I'm glad you have taken the mic to help on communication @sullen flame it's really smoother for us

[sullen flame]

okay! after much talking and some conflicting info, @here this is my understanding

first, the caveats: members of the eng team and Spork are each syncing with v0.31.9. if it goes well, eng team will cut the release. you can use the software as prerelease, but yeah, it hasn't fully been tested until someone has successfully synced (and a couple of other things)

if you do want to get a jump start and you don't mind using prerelease software, here's a proposed decision tree:

1. sync from scratch with the v0.31.9 release https://github.com/anoma/namada/releases
   Steps:
   a) set many peers/seeds, start syncing, wait like 12 hours or something

2. sync with v0.31.6 from some block before the latest block height 90044 (roll-back is broken, but fixed in v0.32.0 i'm told)
   Steps:
   a) build and update with the v0.31.9 release https://github.com/anoma/namada/releases
   b) start syncing and wait for voting power to come online

3. sync from snapshot - this is probably the easiest way, thanks to @glacial nexus, but also yeah there's a trust assumption
   Steps:
   a) download snapshot and use the v0.31.9 release https://github.com/anoma/namada/releases
   b) see msg from KrEwEdk0 (below) to place the snapshot
   c) start syncing and wait for voting power to come online

msg from @glacial nexus:

Snapshot at height 90000 https://namada-se-rpc.citadel.one/snap/namada-se-90000.tar.lz4 Contains db directory to be placed in chain id root dir and data directory to be placed under cometbft dir priv_validator_state.json is removed from the snapshot to prevent validators from double signing on accident as node won't start without this file Also made snapshot with tx index for RPC nodes: https://namada-se-rpc.citadel.one/snap/namada-se-90000-tx_index.tar.lz4

[sullen flame]

okay good news, @muted rapids was able to successfully sync to 90044 with v0.31.9
i've locked the thread temporarily so that my msg above is seen and not buried

[sullen flame]

could you use #🏕-testnet-campfire? it's running v0.31.6

okay! @here is the v0.31.9 prerelease: https://github.com/anoma/namada/releases

thread is unlocked, what's up

[feral light]

so confused

what we need to do 🤣

[brittle plinth]

re-sync use 0.31.9 and stop at the block before 90044 and wait for 0.32.0 :))
use namada node ledger run-until --block-height <#> --halt

[stone estuary]

just restarted with 0.31.9 after the snapshot, synced to 90044 and started to sync all rounds for 90045 (validator node)
checking rpc as well
looks fine, just need to get last round and then get VP to continue, afaik

[sullen flame]

nah, not v0.32.0 just yet. wait for tomorrow and we'll help coordinate more voting power to come online

v0.32.0 will come only after governance proposal. it will be more involved bc it will be a hard fork

[lapis latch]

Can just go up till 90044-90045 right?

[upper ridge]

um no? I think?

[lapis latch]

will upgrade to 0.31.9 now and use Citadel one's snapshot.

[sullen flame]

i think so, yeah

just we don't want too many genesis validators doing this or we'll start making blocks again haha
want to give people notice before that happens

[lapis latch]

Aah right, got it

[upper ridge]

lfg lol

[lapis latch]

I have barely any VP lol.

9k-10k, but won't start it yet then

[upper ridge]

do we need to sync from scratch, or do we not need to do that? inquiring minds want to know 🙂

[lapis latch]

Gonna be using citadel one's snapshot atm

Though probably have enough time to sync from scratch

[stone estuary]

#shielded-expedition message too, just the snapshot is without index, will be ready soon as well, just for the sake of more sources

[rapid token]

How long did the whole sync from start take? Thinking of doing it for my validator

[lapis latch]

I actually wanna go and sync from scratch as well for my validator.

[upper ridge]

10 hrs more or less. maybe a little more

[lapis latch]

Is citadel one's with tx enabled tho?

[upper ridge]

one of them are yes

[rapid token]

Do we have tx enabled snapshot, I prefer otherwise to sync from scratch then

Enough time right

[upper ridge]

yup citadel one has

[upper ridge]

depends on when restart

[lapis latch]

Will this cause trouble if one let's say has config set to no tx, but uses a snapshot with tx enabled?

[stone estuary]

pure depends on server, 4-6hours

[lapis latch]

Just checking for potential edge case

[upper ridge]

when you say tx you mean indexer right?

[glacial nexus]

node just won't index any new txs

[lapis latch]

Ah okay, good

Thankss

[stone estuary]

it is just an additional folder in data: tx.index

you can even remove it

[lapis latch]

Talking bout the tx_index.indexer="kv" setting

[upper ridge]

yup exactly.

do we know how to deal with indexer in this?

the actual indexer I mean

[rapid token]

Restarting from scratch then, easier imo and less chances to mess up

[upper ridge]

also, if we sync all the way up to 90044 or whichever it was, do we risk getting the "bad block" now or no such risk after update?

[stone estuary]

not easier, time wasting as for me, but for the sake of safety to don't catch any corner case seem yes
I am ok to get corner case, will post it as S-tier then 😄

[upper ridge]

thinking of resyncing too

but idk maybe more comfy to just use snap

if anyone does resync with new version, can't y'all do a snapshot around 90k

[stone estuary]

my rpc is synced to 65000 on 0.31.6 and syncing after to the head with 0.31.9

[rapid token]

I have your snapshot too, just checking if I can use this for both rpc and validator node

[lapis latch]

I'm actually super occupied programming, so using that snapshot for one of my nodes is perfect

Will use it for rpc node

[oak kettle]

should i resync from scratch? i'm already jailed tho

[upper ridge]

do we risk anything being changed in the tx-indexer file so as to mess up the indexer read of the data?

[crimson pier]

I will sync my Validator from scratch . 🫡

[stone estuary]

I guess no, coz it will not index those blocks that are already in db

[upper ridge]

yes but going forward?

also, would the last block be included somehow or is all good with the indexer just resuming?

[stone estuary]

it is not finalized, should not be processed by indexer, but you can resync it as well))

[glacial nexus]

what was the latest consensus round?

[upper ridge]

90044?

[stone estuary]

1237

btw, once you get to 90045 as we had 1237 failed rounds it is worth changing below values to make it faster, otherwise each round is 30s (~10 hours) to get to the last
as after the previous upgrades to sync faster, and then move back

[glacial nexus]

don't change timeout_propose_delta and timeout_prevote_delta, it will result in nil-prevotes

Only timeout_precommit_delta needed for faster ctachup and won't screw your prevotes

and any other parameters

timeout_propose = "3000ms"
timeout_propose_delta = "500ms"
timeout_prevote = "1000ms"
timeout_prevote_delta = "500ms"
timeout_precommit = "1000ms"
timeout_precommit_delta = "0ms"
timeout_commit = "10000ms"

this is what you need to have

[stone estuary]

removed mine)

[rapid token]

Having issues untar the lz4 snapshot

Can someone post a command for doing this. Raven, the common you posted is giving issues

[upper ridge]

gonna resync my rpc I tbink

[plush spire]

People have issues, software has bugs. What is your error message and linux distro

[upper ridge]

huh?

no errors here

[rapid token]

Resync is giving issues. I cleaned up and reinitialized and it’s giving issue chain id not initialized

[lapis latch]

I'm at 600, just started hehe.

[plush spire]

What does "issues" mean.

[upper ridge]

looks to be starting fine here from scratch

[lapis latch]

Gonna do my rpc now.

[stone estuary]

local file: tar -Ilz4 -xf "$filename" -C "$path"
remote: curl -L "$link" | tar -Ilz4 -xf - -C "$path"

[plush spire]

"there was an error message and I clicked ok" "what was it" "idk"

[rapid token]

Path should be the shielded testnet folder right

Not the config/data one correct

[stone estuary]

yep, you can use some temp and move after

[upper ridge]

lz4 -d folderABC.tar.lz4 -c | tar xvf -

[lapis latch]

I prefer doing a temp folder first

[rapid token]

Good idea. Download snapshot, let me try this

[lapis latch]

Will use the snapshot for rpc

[stone estuary]

$HOME/.local/share/namada/shielded-expedition.88f17d1d14 - this is the path on my side

[lapis latch]

validator is going at it from scratch.

[stone estuary]

but for the sake of safety, use temp

[plush spire]

for the sake of safety, sync from scratch

[lapis latch]

Hehe true, I might just sync from scratch.

Still contemplating.

[upper ridge]

with an rpc/fullnode, is there any point to preserving old priv_validator key?

[lapis latch]

Nah, I just let it burn

🔥

[glacial nexus]

no, but if you'll use my snap it doesn't have it
But for node to start you need this file. On rpc node you can just place an empty json there
{}

[stone estuary]

interesting, we will have growing rounds while not enough vp seems

[rapid token]

What all do I need to move from temp, both cometbft and db folder right?

[stone estuary]

cometbft/data + db

[upper ridge]

I may have figured out what causes the indexer to halt btw

[upper ridge]

chain growing already?

[lapis latch]

Oh?

I'm not even sure if I had this still btw.

[upper ridge]

you don't while the chain is halted ofc

[lapis latch]

Hahahaa

Touché, touché

[stone estuary]

I would not call that growing 😄
@glacial nexus do you have growing rounds ?

[upper ridge]

you implied we would have growing rounds - what was that abt

[glacial nexus]

i'm just catching up on my rpc
round 1606 and looking at prevotes timestamp it's still not the latest round

[stone estuary]

ah, I guess I just had it stopped before, that's why I had only 1237

[lapis latch]

At 5k blocks now, going quite fast.

[stone estuary]

all rounds from the last week 😄

[rapid token]

Did you init again with new version? For some reason this was giving me isssus. Trying the snapshot and taking a bit to start

[glacial nexus]

oh yeah, i stopped my val on round 1935 and it was on march 2nd
so there a a looot of rounds to catch up

[lapis latch]

Ah, I stopped node, installed binaries, renamed my old chain folder, joined network again using utils command, then copied over everything I wanted to preserve and ran it again.

The usual.

[upper ridge]

always goes fast in the start...

[leaden scaffold]

Thanks @glacial nexus for the snapshot, I think I'll try that first!

[rapid token]

Yeah, did all this and gave issues for rpc node. Maybe let me try for validator node

[glacial nexus]

sync time is 5-8 hours depending on the hardware and if you have tx indexing enabled or not

[upper ridge]

think I'll just go with snapshot on my validator

[rapid token]

Trying it again

[lapis latch]

Will use snapshot for rpc tho

[upper ridge]

@sullen flame when exactly is the restart being planned?

[stone estuary]

but we did it already, only matter of VP is now

[sullen flame]

we're just about to hash the state dump from two different nodes that we synced to see if they match

if they match, we should announce a time tomorrow

hashes match 🎉

[upper ridge]

who controls the announcements controls the vp

[upper ridge]

as early as possible (within reason) would be appreciated

[upper ridge]

are any checks being run on whether using a snapshot made on 31.6 is good or not?

[sullen flame]

yeah Spork is doing that now

[lapis latch]

Used @glacial nexus's snapshot for rpc. Blob ZENODE is back on track: https://zenode.app/explorer/namada.

Thank you dude!

[glacial nexus]

Does state dump take into account not finalized block?
Because if it doesn't, hash might be the same as we have right now on 0.31.6

3132 is the latest consensus round

[broken bolt]

idekubagus has been warned

Reason: Posted a link

[rapid token]

Using my old address book fixed the issue

Syncing from scratch in both for now

[sullen flame]

can pls you state dump and shasum it? not sure how long that will take

Spork is done for the eve, we're so tired haha

[lapis latch]

Hahah I understand

on it

[leaden sun]

Syncing from scratch

[lapis latch]

801131a49b18ff09bcc76210b1ff0c3f52bf2c0a db_dump_90044.toml

[sullen flame]

801131a49b18ff09bcc76210b1ff0c3f52bf2c0a db_dump_90044.toml

[lapis latch]

:)!

[sullen flame]

💥💥💥💥💥

[glacial nexus]

now can someone do the same with nodes that were not upgraded or rolled back?

well, i can, one sec

shasum db_dump_90044.toml 
5a2c22a43c19df6df919e733893e450455b094b3  db_dump_90044.toml

yeah. all good then

[lapis latch]

goodd

[sullen flame]

wat? this doesn't match

[lapis latch]

Think that's good, he did it on an older version

[glacial nexus]

i made it from a node on 0.31.6 that wasn't rolled back
to confirm that state dump differs from upgraded nodes

[sullen flame]

thank you for checking these!!

[upper ridge]

what this mean? citadel one's snapshots are good?

[sullen flame]

okay so Spork is going to write up instructions tomorrow morning and then we'll organize a restart

people can use krewedk0's snapshot if they want, also we can likely host a snapshot

[lapis latch]

Yes, and Citadel's check to also see whether old version differs from 0.31.9 makes sure we don't get state inconsistencies again (as many are still on v0.31.6 and a handful on v0.31.9).

[upper ridge]

Gavin, do we have to restart on a friday afternoon?

[sullen flame]

maybe 2p CET? that's 8a ET

[rare dirge]

we are up with snapshot

I[2024-03-07|20:14:05.504] Timed out                                    module=consensus dur=1s height=90045 round=7 step=RoundStepPrecommitWait
I[2024-03-07|20:14:08.517] Timed out                                    module=consensus dur=1s height=90045 round=8 step=RoundStepPrecommitWait
I[2024-03-07|20:14:10.413] Timed out                                    module=consensus dur=1s height=90045 round=9 step=RoundStepPrecommitWait
I[2024-03-07|20:14:12.079] Timed out                                    module=consensus dur=1s height=90045 round=10 step=RoundStepPrecommitWait
I[2024-03-07|20:14:13.851] Timed out                                    module=consensus dur=1s height=90045 round=11 step=RoundStepPrecommitWait
I[2024-03-07|20:14:15.450] Timed out                                    module=consensus dur=1s height=90045 round=12 step=RoundStepPrecommitWait

[sullen flame]

i think so, but will announce first thing tomorrow

[upper ridge]

urg

[rare dirge]

must i leave mine up or stop and start again tomorrow?

[upper ridge]

better that than later though. I'm just very weary of how timelines tend to shift and suddenly we're long into the weekend

some of us will be left out if we can't start it beforehand

[full veldt]

Do WE have a table off how much VP we have ? Coz I can see that people are on standby at block 90045, which means they are passed the faulty one.
So as soon as there is enough VP , it's gonna be rolling

[lapis latch]

Validator node of mine is at 20k blocks atm. Goes quite fast.

[stone estuary]

3k+ rounds yet)

[rare dirge]

what, rounds goes to 3k?

[lapis latch]

pff.

[stone estuary]

yes, coz we had running chain all this time which tried to finalize 90045 block

[stone estuary]

here

3132 is the last

~1-2 per sec when syncing

[upper ridge]

we risk it suddenly starting

[stone estuary]

it more looks like not halt but a consensus problem, coz during halt nothing happens

[stone estuary]

this is what I wrote above, once enough VP, the chain will start moving

[upper ridge]

what's the r/r of a full resync of validator vs snapshot restore?

[rare dirge]

ok. will just leave running until tomorrow

[stone estuary]

full resync 5-7 hours, snapshot - 1h for validator to get rounds, I guess, maybe 1.5h
or a u about risks?

[rapid token]

13k here and syncing as well

Rpc also decided to sync from scratch, around same block

[proven meteor]

Do we need to set the block height halt flags before block 90044 ( namada node ledger run-until --block-height <#> --halt ) or not?
Because I am also reading that when VP is reached there is a chance that the chain will move on. Maybe we should all agree on what to do?

[thin wave]

ok waiting for tmrw 🚀

[edgy sierra]

Waiting ☝️

Thank you!

[proven meteor]

Thanks. Will also wait for tomorrow.

[old salmon]

Full disclosure, we need to leave ours running as there are a number of commitments tomorrow that can't be rescheduled, especially without a defined start time. If that changes we'll provide an update.

[rapid token]

Same here. Leaving mine running as well

[upper ridge]

still think booting on a friday afternoon is ungood. but yeah will do the same if it hits with a time I'm not available

[upper ridge]

was asking about risk-reward

think I'll chance it on snapshot, getting tired of syncing and looks like it should work on all accounts from accounts above

[sullen flame]

yes, this is a good idea

[upper ridge]

For the record, regarding some things said earlier, I've put every single thing I've had into this for the past five weeks, spent the same number of half-sleep nights to do the stuff i've done here (like a lot of us which tbh is stupid but hey that's how it is). I don't feel it's fair to question people's commitment over having stuff that can't be canceled. (whether or not one thinks it's fair to postpone over weekend is something else)

[timber zealot]

@sullen flame
not all that good, assembles, swears at RUSTUP_TOOLCHAIN, need to be warned that this is possible
Compiling rocksdb v0.22.0 (https://github.com/heliaxdev/rust-rocksdb?rev=4dc7f4fdfa17e923d3078d51261e3db66707754d#4dc7f4fd) error: environment variable RUSTUP_TOOLCHAINnot defined at compile time --> crates/namada/src/vm/wasm/compilation_cache/common.rs:90:53 | 90 | concat!(env!("CARGO_PKG_VERSION"), "_", env!("RUSTUP_TOOLCHAIN")), | ^^^^^^^^^^^^^^^^^^^^^^^^ | = help: usestd::env::var("RUSTUP_TOOLCHAIN")to read the variable at run time = note: this error originates in the macroenv` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: could not find DefaultHasher in hash
--> crates/namada/src/vm/wasm/compilation_cache/common.rs:84:41
|
84 | let mut hasher = std:#️⃣:DefaultHasher::new();
| ^^^^^^^^^^^^^ could not find DefaultHasher in hash
|
help: consider importing one of these items
|
7 + use borsh::__private::maybestd::collections::hash_map::DefaultHasher;
|
7 + use crate::sdk::borsh::__private::maybestd::collections::hash_map::DefaultHasher;
|
7 + use namada_core::borsh::__private::maybestd::collections::hash_map::DefaultHasher;
|
7 + use namada_sdk::borsh::__private::maybestd::collections::hash_map::DefaultHasher;
|
and 1 other candidate`

RUSTUP_TOOLCHAIN

[tame parrot]

What file I need to backup before sync from scratch

[versed light]

just run sudo curl https://sh.rustup.rs -sSf | sh -s -- -y . $HOME/.cargo/env again and build

[sullen flame]

if you're building namada from source, then you'll need to upgrade the rust toolchain

[prisma umbra]

hey, there is no official image for v0.1.39
i created a PR for it, do you think it can get merged before the restart? or we should build or own image
https://github.com/anoma/namada/pull/2847

[tame parrot]

Hello, What the file I need to backup when sync from scratch

[scarlet rose]

so confused now

[turbid oxide]

Do we start syncing or use snapshot for validator node? I see Gavin last announcement not to do it, can someone explain please, bit confusion.

[rapid token]

I am syncing from scratch. However, using snapshot should work as well.

#1215348664554360833 message

[turbid oxide]

no halt height to set?

[rapid token]

I think it’s okay to not set halt height. When enough voting power comes online, we should start moving

You can set a height too if you prefer and restart later.

[oak kettle]

can't see node status after updated to 0.31.9, tried using curl -s localhost:26657/status also won't detect

[broken bolt]

deadsea0x has been warned

Reason: Posted a link

[oak kettle]

Status query failed with Error {
msg: "HTTP error",
source: "error sending request for url (http:// 127.0.0.1:26657/): error trying to connect: tcp connect error: Connection refused (os error 111)",
}

however logs showing that i'm syncing blocks

[upper ridge]

which command are you running?

[oak kettle]

curl -s localhost:26657/status | jq

[oak kettle]

returned this message

[oak kettle]

sorry, when i ran namadac status command returned this message

not showing anything when using curl

[upper ridge]

ok that makes more sense

you sure you're on 31.9 right?

[oak kettle]

have any idea what happened?

[upper ridge]

no 🙂

[oak kettle]

root@Ubuntu-2204-jammy-amd64-base ~/.local/share/namada/shielded-expedition.88f17d1d14 # namada -V
Namada v0.31.9

[upper ridge]

except the error is super similar to the cli thing that was introduced with 31.6 which I thought was fixed now

[oak kettle]

is there any alternative how to monitor my node?

[upper ridge]

what happens if you put a --node https:// (insert localhost and such)

[timber zealot]

Yes, I mean to warn those who still have the old version).

[oak kettle]

same, nothing changes

might be port issue?

i'll look into it

[timber zealot]

updated and started the synchronisation from scratch.

[oak kettle]

can you run this command:
curl -s localhost:26657/status | jq

[timber zealot]

my block is increasing, because the synchronisation is from scratch)

[oak kettle]

yeah, but can you run that command? my node can't after updated to 31.9

[timber zealot]

Yes, I can, but what error are you getting?

[oak kettle]

it's not showing any errors

not showing anything actually, just returned nothing

😢

[upper ridge]

you sure it's actually running?

[oak kettle]

yes, i'm picking up blocks right now

height 8000

[upper ridge]

let me try on mine and see what happens, one sec

works fine here, outputs json with status

so idk

[oak kettle]

😢

or should i have to wait until the node fully synced?

[upper ridge]

no shouldn't matter

mine isn't synced either atm

[oak kettle]

i'll resync from scratch again then😆

[upper ridge]

could be a firewall issue idk, but seeing as it fails on localhost, seems unlikely

doubt that will fix it tbh. you could try renaming the namada-folder, rejoin from scratch and see if it helps. then restore back if it doesn't

i've seen some weird errors if i've had other nodes like osmosis running, idk if that could be an issue?

[oak kettle]

it is resolved

i deleted the entire network folder and rejoin

thanks for the helps @upper ridge

[upper ridge]

good stuff! didn't help much I'm afraid haha 🙂

[oak kettle]

how long does it take to sync from scratch?

[rapid token]

12 hrs or so

Would recommend snapshot if you are starting now

[oak kettle]

do you have snapshot?

[upper ridge]

there's one above in the chat from citadel.one

both with index and without

[rapid token]

#1215348664554360833 message

[oak kettle]

alright, i'll try

Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.

what's this 😆

[upper ridge]

what are you trying to do? ie which command?

[oak kettle]

trying to download the snapshot
instead using wget command, i used curl haha 😂

[upper ridge]

mhm was just about to say wget

[oak kettle]

sorry 😆

[proven meteor]

Reading that some are running the nodes without setting up an halt height. This way we won't get an orchestrated start. Maybe an idea to make an announcement about running with halt height?

[upper ridge]

tbh I think most just want to start as soon as possible

whether that's a good thing or not

[burnt mesa]

Can I just wait and start when we don't need the halt anymore. That's I can start when blocks production has resumed?

[upper ridge]

you're crew right?

[burnt mesa]

Yes

[upper ridge]

should be fine I think

[proven meteor]

See #1215348664554360833 message

[upper ridge]

though tbh I would just leave it running

[burnt mesa]

But no blocks will be produced right? Till genesis validators join or something like that

[upper ridge]

that's my understanding yes

until enough voting power joins to start producing blocks

[oak kettle]

downloaded snapshot, and now it seems stuck at block 90044. what's the latest block btw?

logs showing error reconnecting to peer

[upper ridge]

that's where you will be stuck 🙂

[proven meteor]

90044 is latest

[oak kettle]

so i just have to wait until it picks new blocks?

oh yeah i read from the announcement to wait for genesis validator to sign new blocks 😃

[upper ridge]

it's not just the genesis, the mention of them is simply because they have massive vp compared to the rest

[oak kettle]

and meanwhile I'm with 0 VP
i'm jailed

🤖

[sullen flame]

also we'll share our snapshot soon

Spork is working on our coordination instructions, and then we'll announce our proposed restart time. we're thinking 14:00 utc

[lapis latch]

https://tenor.com/view/taking-a-photo-ted-templeton-sr-the-boss-baby-family-business-take-a-snapshot-capture-a-photo-gif-22838184

[upper ridge]

would be great if timeframes can be kept today. (still not excited about this booting today, but if times are kept I can probably be part of the restart - if not it's a slim chance)

[lapis latch]

That flashlight hurts my feeble eyes actually. Damn.

[full veldt]

yeah, thanks for the flashlight Zen

[upper ridge]

time to get out into the sun 🙂

[lapis latch]

I'm trying to take a different approach in causing troubles in this expedition. Induce some widespread seizure.

[rapid token]

Crashing some other things I see instead of nodes then

[lapis latch]

For real, I actually burned my eye once lol. I stared into the sun for too long. My eyes would blur out everything I focused on for 3-4 weeks.

[upper ridge]

will this be the move from v0 to v1? or is that later with the hardfork?

[sullen flame]

if you're not a genesis validator, i think it's safe for you to leave your validator running (instead of paused)

really the coordination timing (estimated 14:00 utc) will be for the genesis validators

i will include this in the announcement

[upper ridge]

and then network will pick up immediately if things go as planned, or will take some time to propagate?

[rapid token]

Pickup once enough are signing blocks

Need 66% signing for new blocks to be produced

[lapis latch]

Yes, our VP should really be negligible

[rapid token]

Baby VP more like it

[upper ridge]

let's say all genesis are in place at the set time (which may be too hopeful but). How long till the chain starts moving?

[rapid token]

I think immediately if 66% VP is reached

[upper ridge]

if you do the math, post gen have a lot too. but imo a lot aren't participating

so don't think we should fade our contribution here either

[lapis latch]

It's about last active set though

[upper ridge]

tbh I just want to develop more tools, but not many roid in that 🙂

[lapis latch]

So even if we have a lot of roid in unison, we still won't sign the blocks to make the chain progress

So mostly pre genesis ones will matter since most of those were in the active set

[lapis latch]

Heheh, perhaps switch up the purpose of the things you make while still trying to leverage the skillsets you have

[upper ridge]

mebbe. but done what I can there for now

[upper ridge]

I was in active set

so did y'all just replace data and db folder and it worked off the cuff, or is a join-network reset better?

[sullen flame]

it should be fine for you to be synced and ready to make blocks

we'll need tens of validators with enough voting power to come online, which i think we can arrange with the genesis validators

[upper ridge]

you think starting at the proposed time is realistic?

[sullen flame]

yeah, eastern US time will be 9a, CET will be 3p, HK will be 10p

[stone estuary]

what's up here? 🤓

[upper ridge]

someone want to weigh in on this?

[upper ridge]

morning, liver

[stone estuary]

namada_metrics_consensus_round_voting_power_percent{chain_id="shielded-expedition.88f17d1d14",vote_type="prevote"} 2261.24531865957```
2k% 😄

[ripe spoke]

I'm seeing this:
err="error signing vote: step regression at height 90045 round 3132. Got 2, last step 3"

after using snapshot

[urban vapor]

@sullen flame We are running a pre-genesis node, but jailed, all our allocated tokens are slashed, now only few tokens are staked on our validator, should we do the coordination as other normal genesis validators?

[sullen flame]

hi! yes

[urban vapor]

roger

[stone estuary]

I have this on RPC
2024-03-08|01:03:17.600] Timed out module=consensus dur=26m9s height=90045 round=3132 step=RoundStepPropose
wondering why metrics return so huge % for prevotes and precommits

[urban vapor]

we re-sync our node from scratch, and the indexer also need re-sync from scratch?

[upper ridge]

anyone familiar with this error? �ERROR�namada_apps::node:📒�Tendermint error: Failed to initialize CometBFT: No such file or directory (os error 2)

[glacial nexus]

do you have priv_validator_state.json file inside of cometbft/data dir?

[stone estuary]

snap or resync? if snap check that dirs are in place, e.g. Cita's snap has some guidance where to put what, and mine directly has the same tree inside as the local

and yeah, state.json

what is the current active vp?
metrics show some dumb values and the last completed round is with 67+ seems, but the new is not started yet afaik

[sullen flame]

i'm told
"that seems fine while you're syncing to the latest round"

[tranquil ether]

you're a sceintist for sure !

Another prominent scientist who temporarily injured his eyes by looking at the Sun (though not to observe sunspots) was Isaac Newton. When he was about 22 year.....

[lapis latch]

Oh I did not know this!

My reasoning was more spiritually inclined. The doctors had a good laughter and were intrigued when they looked into my eye though. Was completely orange looking. It healed luckily man. I was super scared back then that I had ruined my sight.

[tranquil ether]

it should give you the filname no ?

restart: up and ready but not genesis and jailed ... so .. I will do psycoholigcal care for node runners

[lapis latch]

Or not perse spiritually, more like trying to find ways to restore my mental state through rather alternative measures.

[tranquil ether]

galileo too

(newton)... When he was about 22 years old, he looked at the reflection of the Sun in a mirror, while standing in a darkened room — a situation almost guaranteed to maximize damage by dilating the pupil. (The similarity to the case of Fechner is instructive: both were deliberately trying to provoke after-images; both observed the Sun well up in the sky from inside a dark room; both suffered photophobia and retreated to a darkened room for some time; and both eventually recovered normal vision.)

Fetchner - > galileo

you're definietely a scientist !

[lapis latch]

I underestimated the sun lol

[lapis latch]

Interesting!

[broken bolt]

l_ankou has been warned

Reason: Posted a link

[tranquil ether]

htt p s://aty.sdsu.edu/vision/others.html

[lapis latch]

Ah damn, I wasn't quick enough to click the link.

[lapis latch]

Thank you, will read it now

[full veldt]

Do you know the threshold too?

[lapis latch]

Lollll

Greaves says, that in measuring the diameter of the sun he hurt his sight, “insomuch that for some days after, to that eye, with which I observed, there appeared, as it were, a company of crows flying together in the air at a good distance. At the first I did verily believe I saw a company of crows flying in the air

I also had this 😂. I saw like black spots in my sight everywhere I looked. Funny description haha.

[chilly jackal]

I updated it now. But it still doesn't find the peers. ast_block_height":"90044" is stuck here. What can I do?

[upper ridge]

the cometbft directory somehow isn't created. only on one vps, was setting up a spare node to play with the new indexer

[upper ridge]

nope

[tranquil ether]

waht i you create the dir yourself ?
dir permission creation problem maybe ?

[upper ridge]

shouldn't be, checked it, but should maybe check again

nope no permission problems

it's an auxiliar server so nbd, just odd

going to get my main up and running now

[stone estuary]

those values are weird, the threshold is 66.(6)% (2/3)

"BA{219:___________________________________________________________________________________________________________________________________________________________________________________________________________________________} 0/156523009298949 = 0.00"```

[upper ridge]

ok so my priv-validator-state is set at block 90045, isn't that a bit odd?

shouldn't it be 44?

what was the status of 44, was it accepted or no?

if all validators go with a file like this and the idea is to create a new 44, won't that be a problem?

or am I imagining things?

height": "90045",
"round": 2492,
"step": 2,

what happened with 90044? was that "good" or "bad"?

[trim spruce]

#1215348664554360833 message

[stone estuary]

3132 rounds should be)

[old salmon]

I synced from scratch starting yesterday and am seeing -

SIGNED_MSG_TYPE_PRECOMMIT(Precommit) 000000000000 000000000000 @ 2024-03-08T11:50:13.375363731Z}" err="error signing vote: round regression at height 90045. Got 275, last round 3130"

[stone estuary]

wait for the last round, 3132
after that you will have only this
2024-03-08|01:03:17.600] Timed out module=consensus dur=26m9s height=90045 round=3132 step=RoundStepPropose

[old salmon]

I see, it's still progressing but very slowly. I started syncing over 12 hours ago.

[upper ridge]

hm yes, but wasn't 44 the problematic block?

[trim spruce]

Yes, but apparently the engineers deem this intended behavior.

[upper ridge]

🤔

that doesn't make sense to me

or is it that the block wasn't the problem but the way the software parsed it?

[trim spruce]

I think it means the validators are in the process of proposing a new block after 44

Not sure what happens with block 44 itself

[upper ridge]

if block 44 is somehow corrupted, I don't see how we can progress to building block 45

[trim spruce]

The offending transaction in block 44 has a data field with an unexpected length. I guess v0.31.9 handles that edge case, so it can process 44.

[upper ridge]

hm makes sense if so

but if that's the case why do we even need to resync the whole thing and not pick up where we left off?

because the former client didn't store it correctly?

not fully convinced

[rare dirge]

my node got to
module=consensus dur=26m9s height=90045 round=3132 step=RoundStepPropose

But not progressing, looks to be just looking for peers

[trim spruce]

I wondered about that too. Could be that the v0.31.9 also contains other changes such as a database upgrade, so the old format has to be converted to the new format.

Updated RocksDB dependency. For a shared libary users make sure to link against version v8.10.0.

[upper ridge]

idk I can't get this to square with gavin's description of the bug

[trim spruce]

Kind of nasty that a RocksDB update makes it necessary to resync from the start.

[upper ridge]

guess we'll see if it all works out well

[upper ridge]

shouldn't. if a snapshot from 31.6 can be used (as they are), I don't see why resync is needed

[lapis latch]

I think this doesn't get created when the tendermint mode is set to Full instead of Validator.

But I might be wrong

[upper ridge]

don't think that's correct

hasn't been an issue on my other rpc node and it's sibling I used to create a snapshot yesterday. just this one vps that's doing this for reasons I don't understand

[lapis latch]

Is it running tho?

[upper ridge]

no it exits and never runs

[stone estuary]

~10h in such a state already 🙂 and nothing happens
I assumed we should have some VP growing but it's not...

[upper ridge]

you got any opins on my musings about the blocks above?

[rare dirge]

so we just wait?

[stone estuary]

90044 was finalized in the chain, afaik
90045 can't get consensus, and not moving now even with rounds
I would wait the team with some instructions of clarifications

[rare dirge]

cause it is not doing anything hard to see if in limbo and waiting or a problem? 😂

Anyone got that pic showing who in concensus?

[stone estuary]

everyone, on both rpc and validator

let's go to the new thread 😄 #📣-se-announcements message