the elusive bug

[minor nexus]

the block in question had three IBC transactions:

1. accepted
2. failed (out of gas)
3. failed (invalid event)

why was the event invalid in Transaction 3? it shouldn't have been. it's because the event from Transaction 2 wasn't dropped, and it looked at this invalid event as belonging to Transaction 3

this was a problem for the chain because Transaction 3 was only invalid in the newer client version, and in v0.31.6 the Transaction 2 event did not emit. wat?

the newer version's gas metering is a bit cheaper than v0.31.6, and because it's a bit cheaper, it was able to execute one more line in the newer version than in v0.31.6--and this line was the invalid IBC event. so in v0.31.6 the Transaction 3 is valid, and new version Transaction 3 is invalid

when a tx fails, we drop the write log (ie. don't write to storage) but we do not drop the IBC event (and we should). this is the bug: we must drop events when transactions fail

[ocean sorrel]

better Thursday than Friday, right).

Unjailed's fixed, right?

[charred bane]

@minor nexus Thank you for the breakdown!

[midnight glen]

Thanks Gavin for the explanation. Looking forward to SE resuming soon!

[charred bane]

The context and detail is very helpful 🙏

[noble whale]

nice

[remote rivet]

Ouch, that was a mouthful.
Well done team for catching that, and to the person who triggered it. I've seen some people spamming IBC txs, so that might explain 😄

[midnight glen]

This is why we have testnets to catch devious bugs like this

[jagged agate]

Thanks for updating us 🙏

[remote rivet]

agreed !

yeah, thx for the update, at such a late hour !

[finite rose]

don't think unjail gets fixed until the hardfork in 32.0?

[finite rose]

does this mean we restart on 31.6 tmw or that we get a new version? good catch

[minor nexus]

the elusive bug

[minor nexus]

the unjail fix will come with the governance proposal and hard fork

[minor nexus]

v0.31.9 should get cut tomorrow for the restart

v0.32.0 soon after for the hard fork

[ocean sorrel]

unjail will be fixed already in 0.32, it turns out.

[halcyon crescent]

Is the rollback command of any use? Seems to take a lot of time and CPU to rollback 1 block though.

rollback has been tested last year and confirms the high CPU usage, but in the end it seemed to work.
#🔧︱validator-testnet message

[surreal flint]

please fix the faucet

[snow crow]

I actually did that transaction 😅

[snow crow]

Will I get roids if I show proof? I broke the chain and I'm likely to do it again 😅😅😅

[bright zealot]

I think the team should not let everyone wait for that long to debug next time. It's almost a week already. Revert should be considered fast the first time. SE will keep go long way if we debug issue like this

If the Team wanna have a long testnet for debugging issues better to create another testnet event for that than SE. That's my thought.

[pliant coral]

Well done team!

[minor nexus]

whaaat realy? hhaha i was curious if we could track down who did it

[minor nexus]

i don;t think that there's a ROID category for this, but yeah if you can break the chain again i will reward you with a bounty

[mellow falcon]

When is shielded-expedition expected to end? thank sir

[sweet remnant]

nice one ........ catchy bug indeed

[sweet remnant]

yes there is, security "stop the chain"

let's be fair, I broke the faucet and report it I got points, @snow crow deserves it's points too if he proove the tx

[finite rose]

doubt it was done purposely

[finite rose]

there's a security to submit proof on how to. not to actually halt chain (and tbh given what was done here was specifically upgrading to a client we were told not to touch, I don't think it should be rewarded, nothing personal against favour)

[minor nexus]

we'll give at least one week notice, my guess is 2 weeks

talked with eng team, we think chain restart coordination could begin in ~6 hours

we've got ~4 hours more of syncing to do, then maybe ~1h to cut releases

[halcyon crescent]

Rollback was not possible?

Not that I am impatient, just wondering about the technical reason.

[finite rose]

we got snapshots available if that can help speed the process

[kind saffron]

set your balance to 10000000 and create a snapshot

[finite rose]

ssshhhh 😂

[minor nexus]

what's the block height of your snapshot?

[finite rose]

still syncing but I'm at 74k approx atm, so can create one from there and any place up (once I get further up). @mortal sinew posted one in #shielded-expedition at around 90k I believe.

lmk if you need a cut at a certain block height..

[sour marten]

there is also 90k snap (both w/ and w/o tx index)
#🛡︱se-100 message

[finite rose]

could be so kind to share it here for us plebs? 🙂

w index would be super useful

[sour marten]

Snapshot at height 90000
https://namada-se-rpc.citadel.one/snap/namada-se-90000.tar.lz4
Contains db directory to be placed in chain id root dir and data directory to be placed under cometbft dir
priv_validator_state.json is removed from the snapshot to prevent validators from double signing on accident as node won't start without this file

Also made snapshot with tx index for RPC nodes:
https://namada-se-rpc.citadel.one/snap/namada-se-90000-tx_index.tar.lz4

[finite rose]

excellent, the latter would be very helpful

if they don't decide to start chain below that height ofc

[finite rose]

you have any idea how the indexer itself would behave to a slight rollback?

[sour marten]

if we're going to restart same way as the last halt, we just need to re-execute a block we're currently trying to reach consensus on. And this block wasn't indexed yet

[minor nexus]

what version client version are you syncing with?

[minor nexus]

what version client version was this snapshot made with?

[sour marten]

0.31.6

[finite rose]

31.6

[finite rose]

sure, I was thinking the plan may be to roll back some extra blocks. which would leave us with redundant blocks in the indexer. I'm not sure if it's made to handle that defensively or if we would need to fully resync (and or do sketch stuff in the db itself)

[sour marten]

we can't rollback blocks without changing chain-id
which essentially will be a hard-fork and a new chain launch

[finite rose]

hm ok

[sour marten]

i mean rollback more than the current block
cometbft will connect to peers with the highest block and nodes will return to the same state we're in currently. Assuming they won't apphash trying to reach current block

[finite rose]

fair

I suppose my question above would apply once we get to the hardfork then, but no need to cross that bridge until we get to it I suppose

[mortal sinew]

missed conversation 😄 yep, mine is with 0.31.6, but it is without tx.index for usual users, and will have with it as well soon
so, at least we have two sources that ppl can use

[midnight glen]

We planning to go with a new version or 0.31.6, since the bug was identified

[finite rose]

new

[midnight glen]

Yeah, hence wondering if the snapshots will still be good for that

[finite rose]

remains to be seen

maybe team will make snapshots made with new client available to us

if so, I hope they make one with indexer as well 👀👀

[halcyon crescent]

For the testnet that is acceptable, but how would that play out with mainnet?

"Social consensus"?

How would the community verify the snapshots have not been tampered with?

[finite rose]

maybe there's a theoretical s-class waiting for you there 🙂

[halcyon crescent]

Aha!

[mortal sinew]

was it a question? 🧐

[snow crow]

I'd try the same thing I did again hope it breaks it again

[midnight glen]

yes, I am not sure about this since folks are talking about 0.31.6 currently 😅

[snow crow]

Wasn't on purpose but I'll do it again and see if it breaks

[finite rose]

updated my comment above. I think I feel that given the only reason this happened was prematurely upgrading to a client version we were told not to touch because it could break the network, I lean towards thinking it shouldn't be rewarded.

otherwise it's pure mayhem whenever an unstable version is put out

but idk I'm not the one making the rules 😅 gg on your part if you get rewarded

[finite rose]

btw, will we be given a couple of hours warning before it all goes down?

[remote rivet]

I'd tend to lean on your side too pretoro. I dont know if "unintentionnaly breaking the chain" is a category that should be rewarded 🤔
Finding bugs, logging them through github, hell even putting up a PR if you have the skill is something I'd gadly reward if it were me. but unintentionnal outage seems a bit far fetched

[finite rose]

I think unintentional and pure luck are ok. I'm more weary if the cause is doing something we have been told would wreck the network and we should keep our hands off. but again just my opin

[remote rivet]

correct

[mortal sinew]

the only existing working now, yeah

[finite rose]

all of this being said though, I'd say submit it and let team decide

not wanting to gatekeep people's points

[minor nexus]

snapshots will only work for our purposes if made with the unreleased v0.31.9

eng team is working on that now. Spork is going to use eng team's snapshot to test and then write instructions (to import from snapshot) so that we'll be ready to coordinate the restart

and yeah then maybe we can get blocks made without having to resync from genesis 🤞

fyi @finite rose

[sour marten]

if tx in question was included in the latest pre-halt block, can't you just use a 0.31.6 snap and upgrade at some height after snap was done?

if 0.31.9 is not consensus-breaking, there is no need to resync whole state from genesis using it

idea of this 90k snap was to restore state using it, sync up to pre-halt block (or right away if it won't apphash between 90k and consensus block) and upgrade to binary version that fixes non-determinism

i'm just worried that we might not restart today if you just starting to resync for a new snap. It took me 8h to sync with tx indexing enabled

[minor nexus]

we're at 70k / 90k on one machine, another (more powerful) machine is at 25k / 90k blocks

but yeah we gotta sync to test before cutting the v0.31.9 release

[finite rose]

good stuff

[finite rose]

are they syncing with indexer support?

otherwise it'll be a horrible mess for those of us running indexer

[snow crow]

I didn't use a new version I was on version 0.31.6 I'm still there I can show you

[mortal sinew]

exactly what I thought preparing the snapshot

[midnight glen]

How long does it take to download the snapshot you provided? Thanks

[halcyon crescent]

Gavin explicitly said snapshots will only work for our purposes if made with the unreleased v0.31.9 that makes me wonder why a 0.31.6 snapshot up to 90043 is unusable. Is it because the engineering team want to check if the changes in 0.31.9 yield the same state, or is the database changed?

In the latter case, can we still verify integrity?

[sour marten]

Only thing that potentially might imply it is rocksdb version bump

[halcyon crescent]

So hashes would still be intact, just a compatibility change

[sour marten]

Otherwise i think it's perfectly fine to use snap from 0.31.6 and i'll test it on rpc node to see if prevote hashes will match with team's snap

[minor nexus]

argh! they don't think so

[finite rose]

that's really bad for all apps running indexer

[finite rose]

6gb roughly

[finite rose]

the bug gavin described was caused by a tx made with the newer version I believe?

so I doubt you could have made that tx if you were still on 31,6

unless I totally got it all wrong

[kind saffron]

R.I.P. namascan

[finite rose]

and simple votes checker and transactions checker

[sweet remnant]

ok so it means the snapshots from v 0.31.6 are useless / unusable ? and we have to wait from yours

[mortal sinew]

I'll try anyway 😄

[midnight glen]

keep us posted.

[mortal sinew]

need a binary to try, so waiting for the official announcement and then will see if it works with that snap

[minor nexus]

okay i hear you, i will chat with eng team

[minor nexus]

okay maybe we can use this! i'm hearing from eng team that you can use this snapshot (assuming that you trust @sour marten haha)

the issue is that we need to use v0.32.0 before cutting this as a release:

but for everyone else's purposes, the snapshot above should work

[remote rivet]

So should we wait for 0.32.0 to start our nodes ? Or what ? Do we relaunch today 😄 ? coz tomorrow it's friday...

[midnight glen]

Maybe we need to breajk the spell and start on a friday

[finite rose]

no pls

[finite rose]

do we even need to use snapshot? won't turning it back on again after new consensus forms have node pick up where it should?

[mortal sinew]

#📣-se-announcements message

time to talk in new thread 😄

[finite rose]

just made one 🙂

[mortal sinew]

sorry for that piece of sarcasm)

[left shuttle]

hi, im using 31.6 latest height 90044, so we dont need this snap ? and just upgrade to 32?

[mortal sinew]

everyone have the same as you, and exactly that block that was stuck is bad, that's why or resync or use snapshot which is for only only partially confirmed as working solution

[finite rose]

but will that be necessary once a new consensus forms?

[minor nexus]

dang it, new thread haha

okay gonna close this one after i've answered Qs

[left shuttle]

ow ic. thx

[minor nexus]

there's a prerelease about to be published, you can use that

[minor nexus]

i imagine yes.. but let me check

[covert brook]

What about the spreadsheet

[minor nexus]

which?

[covert brook]

About to tasks submissions

[remote rivet]

Maybe

[minor nexus]

is 90044 the last block before the chain halt? if so, it's no good

if you could roll back, then you wouldn't have the problem block. and when we have v0.32.0, we'll be able to roll back

[finite rose]

it is, but that block never finished I think? or is the issue it did and it's wrong?

[finite rose]

there's a rollback command in cli that apparently works to roll back a single block....

[halcyon crescent]

Yes, takes about half an hour depending on your machine

[halcyon crescent]

Wait, the rollback in 0.31.6 is no good?

[minor nexus]

thanks! yes i will announce that

[minor nexus]

let me ask

[remote rivet]

Ok, so assuming this sounds like a good path to prepare?

• stay on 0.31.6
• start sync from scratch or from snapshot
• stop sync before reaching 90044
• update to 0.32
• finalize sync to the tip
• wait for consensus and VP to increase

Ama right?

[minor nexus]

i'm gonna lock this thread in a minute, wanna make sure everyone in the other thread is in the loop

[finite rose]

aren't we upgrading to 31.9? not 32?

[mortal sinew]

why did I read it as "punished"? 😄

[minor nexus]

this should work, except for the update version, which is the 0.31.9 prerelease (not v0.32.0)
make sure to use whatever command stops the chain before the latest block (people have said that latest height is 90044)

[ocean sorrel]

it's just time for you to get some sleep.)

[minor nexus]

k gonna lock, see y'all in https://discord.com/channels/833618405537218590/1215348664554360833