Sequencer error: Nonce not matching | Mina Protocol | Page 1

drowsy gulch Sep 23, 2024, 9:39 PM

#

I am getting lots of error logs like this:

Is it because of the migrations from previous versions? Do I need to wipe all db for it?

📎 message.txt

#

I am on sovereign mode

#

📎 message.txt

#

how can I just prune on startup to wipe db?

#

How do I solve the nonce error? Sequencer keeps giving that error now

fast violet Sep 24, 2024, 6:45 AM

#

You can prune the DB by using --pruneOnStartup like you do already

#

It looks like you send wrong nonces with your transactions, the logs look to be consistent on the sequencer side

#

Do you send multiple transactions per block?

drowsy gulch Sep 24, 2024, 8:32 AM

#

fast violet Do you send multiple transactions per block?

Yes I am

#

What should be the interval?

#

Also --pruneOnStartup feels like not working it says found 5 migration file on prisma and looks like it migrates

#

starterkit-postgres | 2024-09-23 21:41:31.828 UTC [77] LOG: could not receive data from client: Connection reset by peer

#

Is this also normal?

#

tx sent 18, onchain value is 32 btw these values are changing all the time and it keep spamming the logs

fast violet Sep 24, 2024, 11:28 AM

#

migration have nothing to do with pruning, pruning just deletes all data from the tables

#

migrations create the tables

fast violet Sep 24, 2024, 11:28 AM

#

drowsy gulch tx sent 18, onchain value is 32 btw these values are changing all the time and i...

we dont sort transactions when pulling them from the mempool, so maybe the order is fucked up there?

drowsy gulch Sep 24, 2024, 12:02 PM

#

I am not sure, but I am sending those requests from one api

#

To be able to only send 1 transaction per block, is there a configuration to enable? Because tests are passing. It means it actually works if I do it 1 tx per block

#

also inmemory mode generates less error and that way I was able to debug an error. The thing is there is an assertion that keeps happening in loop

#

    public async submitUserAnswerInternal(answerID: AnswerID, answer: UserAnswer): Promise<void> {

        const exam = (await this.exams.get(answerID.examID));
        assert(exam.isSome, "Exam not found");
        assert(exam.value.isActive.equals(UInt64.from(1)), "Exam is not active");
        await this.answers.set(answerID, answer);
    }

    @runtimeMethod()
    public async submitUserAnswers(answersInput: UserAnswersInput): Promise<void> {
        for(const answer of answersInput.answers) {
            await this.submitUserAnswerInternal(answer.answerID, answer.answer);
        }
    }

#

I do this

drowsy gulch Sep 24, 2024, 12:19 PM

#

fast violet we dont sort transactions when pulling them from the mempool, so maybe the order...

If you dont sort it then yes, it is the reason

#

And also then I assume the transaction 1 in the block does not update the state for transaction 2 in the same block right?

fast violet Sep 24, 2024, 12:20 PM

#

no it does

#

thats what protokit does in comparison to the L1 🙂

drowsy gulch Sep 24, 2024, 12:21 PM

#

uh, then sorting would be nice there

fast violet Sep 24, 2024, 12:21 PM

#

drowsy gulch If you dont sort it then yes, it is the reason

but yeah, thats a big issue. We want to build the proper mempool soon

#

how soon, I'll have to circle back to you on that one

drowsy gulch Sep 24, 2024, 12:21 PM

#

What should I do to 1 transaction per block

#

I need to sort this out in 2 days

#

If I manage to make it 1 tx per 1 block from sequencer configuration that would be a quick solution for now I would be happy with it

#

@fast violet

fast violet Sep 24, 2024, 12:24 PM

#

i mean i think you should be able to replace the mempool and add sorting there

#

its really easy actually

#

you can look at PrivateMempool.ts in the framework repo, copy it, add sorting there

#

then inject it in sequencer.ts in the starter-kit

#

under the token "Mempool"

drowsy gulch Sep 24, 2024, 12:25 PM

#

Ok will try, is there a quick way to do 1 tx per block though?

#

If I fail in time

fast violet Sep 24, 2024, 12:27 PM

#

for your tests or the UI?

drowsy gulch Sep 24, 2024, 12:28 PM

#

For production

#

I want sequencer to generate 1 tx per 1 block

#

like testingAppChain

fast violet Sep 24, 2024, 12:28 PM

#

for production, i'd say do the mempool thing and only return one tx at a time

#

but for that you'll need to sort it as well haha

drowsy gulch Sep 24, 2024, 12:28 PM

#

oh... XD

#

Ok it is not a problem I understood the problem so it would work yeah

fast violet Sep 24, 2024, 12:29 PM

#

yeah - i am surprised nobody ran into that stuff earlier

drowsy gulch Sep 24, 2024, 12:29 PM

#

thanks a loit

#

Yeah, I am surprised too

#

For example zknoid should have that problem

#

The most interesting thing is try catch does not catch that problem

#

Because it skips the tx

#

But because order is wrong, they all fail with assertions

#

I saw a comment somewhere saying, if multiple assertions happening because of the same reason , it gets skipped without error?

#

When I do these transactions protokit client gives me ok

#

I caught the real assertion fail logs from inmemory debug mode

fast violet Sep 24, 2024, 12:46 PM

#

true, ill ask them

#

they haven't had the error, but also don't do multiple txs in a block

drowsy gulch Sep 24, 2024, 3:09 PM

#

@fast violet Btw should I use nonce for sorting? Also Do I have to build local framework myself or I just import mempool as a file from starterkit works?

fast violet Sep 24, 2024, 3:14 PM

#

yes sort by sender and nonce

#

just copy paste it, create a new mempool

#

then i can show you how to inject it

#

no need for building locally

drowsy gulch Sep 24, 2024, 3:18 PM

#

Is sorting on nonce should be greaterThan? Or a.nonce.sub(b.nonce)

#

Not sure if this works :

    const pendingTransactions = await this.transactionStorage.getPendingUserTransactions();
    return pendingTransactions.sort((a, b) => Number(a.nonce.sub(b.nonce)));

drowsy gulch Sep 24, 2024, 3:44 PM

#

Is this enough?

📎 message.txt

#

@fast violet

#

export const baseSequencerModules = {
  ...apiSequencerModules,
  Mempool: PrivateMempoolWithSort,
  BlockProducerModule: BlockProducerModule,
  BlockTrigger: TimedBlockTrigger,
} satisfies SequencerModulesRecord;

#

Btw in our case if we sort by nonce it is enough. We only have one user that is express server

drowsy gulch Sep 24, 2024, 4:23 PM

#

I am using it now with inmemory environment , it does not generate any errors

#

But I am afraid from sovereign

drowsy gulch Sep 24, 2024, 7:07 PM

#

@fast violet yeah the issue actually did not fixed. But the I see that the log does not affect anything and state is persistent . Didnt understand why though

drowsy gulch Sep 24, 2024, 8:36 PM

#

Hey It does not work with multiple transactions

#

Can you send me a quick solution for that?

#

When I use development mode it calculates the first user though

drowsy gulch Sep 24, 2024, 9:26 PM

#

📎 message.txt

#

📎 message.txt

drowsy gulch Sep 25, 2024, 12:45 PM

#

@fast violet I guess I will solve this error by using Zk Programs more

#

Zknoid does it

#

I am getting inspiration now

fast violet Sep 27, 2024, 8:28 AM

#

Sorry for the delay, the error above is kinda weird...

#

especially the second one I've never seen before

#

it shouldnt need a zkprogram solution, protokit was exactly engineered for stuff like that

#

is there a reproduction test you can give me that's minimal? Then I'd be able to debug the issue effectively

drowsy gulch Sep 28, 2024, 7:36 AM

#

there is no error in tests

#

because tests are happening with one block per transaction

#

so thats why I could not predict this error

#

@fast violet I can give you our repo it has an express server to communicate

#

We send two transaction with 1 - 2 seconds

#

I am also having worse error when I run protokit with sovereign docker:up

#

If I do that it just gives Cannot Convert False to BigInt error

#

But If I run only database and sequencer and run express api without docker it works.

#

But that is not a problem right now. The biggest problem is that when I send publishAnswers and checkScore transactions in the same block it just gives error most of the time

#

If I run sequencer in memory without docker it works better

#

But still many of the times it gives error. I guess it is because multiple transactions per block and inmemory is faster for block production I suppose

#

I am sure that if I just can enforce 1 tx per block that should solve it

#

Can you show me an example about it on PrivateMempool?

fast violet Sep 28, 2024, 8:05 AM

#

Yeah that would be good to try, to confirm the suspicion on what the problem is

drowsy gulch Sep 28, 2024, 8:05 AM

#

    public async getTxs(): Promise<PendingTransaction[]> {
        const pendingTransactions = await this.transactionStorage.getPendingUserTransactions();
        return pendingTransactions.sort((a, b) => Number(a.nonce.sub(b.nonce))).reverse();
    }

#

I tried sorting too but I am not sure that was solving anything

fast violet Sep 28, 2024, 8:06 AM

#

What you do is after the sort you basically you only return the first tx

drowsy gulch Sep 28, 2024, 8:07 AM

#

so in getTxs I only return one tx and that will be enough right? But is getTxs the right function?

#

Please send me a simple line for that I don't know If am doing the best practice or the right thing there 😄 We have a presentation on Monday with Mina

#

And I should not have this problem because logically there is no problem at the app, all tests passing

drowsy gulch Sep 28, 2024, 8:33 AM

#

@fast violet

fast violet Sep 28, 2024, 8:40 AM

#

yeah its really weird

#

I'll try to recreate it, but kinda busy this weekend unfortunately...

#

const first = pendingTransactions.sort((a, b) => Number(a.nonce.sub(b.nonce))).reverse().at(0);
return first !== undefined ? [first] : [];
```

drowsy gulch Sep 28, 2024, 8:41 AM

#

Thanks a lot

#

❤️

fast violet Sep 28, 2024, 8:42 AM

#

i dont know if the sort() and reverse outputs it ascending or descending tho

drowsy gulch Sep 28, 2024, 8:46 AM

#

me too XD

#

It does in normal javascript

drowsy gulch Sep 28, 2024, 2:32 PM

#

I am not sure if that is working now because I need to test it on production as well, but for extra measure I also implement a queue logic into my api to not spam the protokit

#

Also I was thinking if looping inside 120 element sized array is a bottleneck for the protokit transaction?

#

Should I reduce it for now? Would that increase the block production by any significance?

#

Because what I am doing is looping inside 120 element sized array to calculate the score

#

I also found out that actually there might be an important assertion that I am not seeing clearly

#

And everything is happening because of it. Most of the time the first student gets the score correctly and others become undefined because I am doing it in a 9 second cron job

#

student by student

#

why student by student because when I tried multiple requests at once with timeout. It didnt work

fast violet Sep 28, 2024, 2:40 PM

#

hmm, okay

#

depends what you do in the loop, if thats a lot of work or not

#

and esp also how many state writes and reads you do

drowsy gulch Sep 28, 2024, 5:46 PM

#

    public async getUserAnswers(examID: Field, userID: Field): Promise<[Field[], Field[]]> {
        const exam = (await this.exams.get(examID)).value;
        let userAnswers: Field[] = [];
        let correctAnswers: Field[] = [];
            for (const question of exam.questions) {
            const answerID = new AnswerID(examID, question.questionID, userID);
            const answer = Provable.if(
                (await this.answers.get(answerID)).isSome, 
                UserAnswer, (await this.answers.get(answerID)).value, 
                new UserAnswer(Field(0), Field(0))
            );
            userAnswers.push(answer.answer);
            correctAnswers.push(question.correct_answer);
        };

        return [correctAnswers, userAnswers];
    }

    public calculateScore(correctAnswers: Field[], userAnswers: Field[]): Field {
        let scoreController = new ScoreController(Field(0), Field(0));
        for (let i = 0; i < correctAnswers.length; i++) {
            const newScore = Provable.if(
            correctAnswers[i].equals(userAnswers[i]).and(correctAnswers[i].equals(Field(0)).not()), 
            ScoreController, 
            new ScoreController(scoreController.corrects.add(Field(1)), scoreController.incorrects) , 
            new ScoreController (scoreController.corrects, scoreController.incorrects.add(Field(1))));
            scoreController = new ScoreController(newScore.corrects, newScore.incorrects);
        }
        return scoreController.corrects;
    }

    @runtimeMethod()
    public async checkUserScore(userID: Field, examID: Field): Promise<void> {
        const [correctAnswers, userAnswers] = await this.getUserAnswers(examID, userID);
        const score = this.calculateScore(correctAnswers, userAnswers);
        await this.userScores.set(new UserExam(examID, userID, UInt64.from(1)), score);
    }

#

User answers array and questions array size is 120

#

Btw I now realized that I can't wait for protokit to finish transaction and block pushed? How can I wait for that from client?

#

server.post("/check-score", async (req, res) => {
  const examina = client.runtime.resolve("Examina");
  const examID = Poseidon.hash([Field(Buffer.from(req.body.examID).toString("hex"))]);
  const userID = Poseidon.hash([Field(Buffer.from(req.body.userID).toString("hex"))]);
  const tx = await client.transaction(serverPubKey, async () => {
    await examina.checkUserScore(userID, examID);
  });
  tx.transaction = tx.transaction?.sign(serverKey);
  await tx.send();
  setTimeout(async () => {
    const userScore = await client.query.runtime.Examina.userScores.get(new UserExam(examID, userID, UInt64.from(1)));
    const userScore0 = await client.query.runtime.Examina.userScores.get(new UserExam(examID, userID, UInt64.from(0)));
    const userScore2 = await client.query.runtime.Examina.userScores.get(new UserExam(examID, userID, UInt64.from(2)));

    console.log("User score calculated: ", userScore?.toJSON());
    console.log("User score calculated is active 0: ", userScore0?.toJSON());
    console.log("User score calculated is active 2: ", userScore2?.toJSON());
    res.json({ score: userScore ? userScore.toJSON() : userScore0 ? userScore0.toJSON() : userScore2 ? userScore2.toJSON() : "User score not found"});
  }, 1000);
});

#

Most of the time these logs are undefined

#

And then some time later userScore2 is the right score

#

But I need to poll for that score to not be undefined

#

A lot of timeouts there. It is not cool

#

After this check-score I also execute a getScore function to get the score again from different endpoint with timeouts and some time later I get the score

drowsy gulch Sep 28, 2024, 8:38 PM

#

@fast violet so my main question is: What is the best practice to wait for the transaction and block to be succesfull

fast violet Sep 29, 2024, 6:24 AM

#

hmm, normally you can't do that - since its more the ethereum transaction model where you push your tx and then it gets included eventually

#

we probably should add functionality to poll and wait for a txn

#

what you can do in the meantime is to create your own function where you have a interval that polls the latest block height every n seconds
You do that via (await appChain.query.network.unproven)?.block.height

#

after that I found out that we actually don't have a query where you can query blocks or anything like that. We have that in our gql API, but unfortunately not packaged nicely for you to use

#

the chain store in the starter kit actually uses that raw graphql API for that, maybe you can reuse that code to do it?

#

https://github.com/proto-kit/framework/issues/204 Created this one though for the future

drowsy gulch Sep 29, 2024, 10:32 AM

#

Hey, now I solved it but it is such a workaround that if I show what I am doing your eyes would be in tears now I dont have any problem

#

I am using redis jobs and using timeout a lot, I saw that in protokit.dev you also use 8 seconds timeout so I thought if I wait long enough and try enough times with queries to get the updated data succesfully. I saw that graphql implementation too. But I think it just polls for new blocks every second? I am also now just polling the get score query per 1 second till I get the score. But definitely I think transaction being succesful needs a better handling. As client side when sending the transaction it feels like we are awaiting for transaction status and the status comes when block is produced like in other blockchains. But it is not the case and I think should be pointed out in your documentation. It is not obvious and I only now actually understanding after months of development with it because I experienced this because the transaction completion takes too long for the particular function. If I would only use 10 element sized array I would not discover this too.

#Sequencer error: Nonce not matching