Trade offs between executing code inside and outside the circuit | Mina Protocol | Page 1

stoic coyote Oct 3, 2023, 9:13 AM

#

I need advice on how to structure an app in terms of where to execute part of the core code logic: zkOracle or smart contract.

Ideally, zkOracle should function as proxy relaying data from off-chain sources to smart contracts. So all of the core code logic should be executed on smart contracts inside the circuit. But it's tempting to place more logic inside the zkOracle because it's easier to run outside the circuit using plain JavaScript. Moreover, executing some logic inside the circuit can be really difficult.

For example, I'm dealing with string manipulation to extract a DoB from a personal identification number. Outside the circuit (inside zkOracle in plain JavaScript), I can easily manipulate the string to extract the DoB. Inside the circuit, facing obstacles.

#

Plain JS outside of circuit

// personal identification number
const pno = 'LT-39509100123'

const startIdx = pno.indexOf('-') + 1
const pnoDigits = pno.substring(startIdx)

const century = Math.floor(Number(pnoDigits.charAt(0)) / 2) + 18
const yy = pnoDigits.substr(1, 2)
const mm = pnoDigits.substr(3, 2)
const dd = pnoDigits.substr(5, 2)

const DoB = `${century}${yy}-${mm}-${dd}`
const DoBTimestamp = Date.parse(DoB)
console.log(DoBTimestamp) // 810691200000

o1js inside the circuit

import { Encoding } from 'o1js';

// personal identification number
const pno = Encoding.stringToFields('PNOLT-39509100123');
console.log(pno)
// [
//   0,
//   Uint8Array(32) [
//     80, 78, 79, 76, 84, 45, 51, 57, 53,
//     48, 57, 49, 48, 48, 49, 50, 51,  1,
//      0,  0,  0,  0,  0,  0,  0,  0,  0,
//      0,  0,  0,  0,  0
//   ]
// ]

// is it even possible to slice Array of Fields, concatenate selected slices and convert it to ints?
console.log(pno[0].value)
console.log(pno[0].value[1][0])

// error on build:
// Element implicitly has an 'any' type because expression of type '0' can't be used to index type 'number | FieldVar | Uint8Array'. Property '0' does not exist on type 'number | FieldVar | Uint8Array'.
// console.log(pno[0].value[1][0])

My options are as follows:

Put this logic inside the zkOracle and output a DoB timestamp as part of it's response.
Use zkOracle to relay pno to smart contract and try coming up with ways to execute this logic inside the circuit.

Considering the trade offs of simplicity and proper app logic which option would you recommend?

pale maple Oct 3, 2023, 9:30 AM

#

I'm not sure that you can really have a trade-off between simplicity and proper app logic 😄
If logic needs to be in the zk proof, you simply have to find a way to make it provable.
If logic doesn't need to be in the zkp, then there's no reason to put it there

#

In principle, you can do everything you do in your plain JS example also inside the circuit, on variable size arrays with indices that are variables

#

but there's no point doing that on constant field elements

#

instead of doing stuff on constant field elements you can just do them on plain JS values - that's the same

#

so I can start give you pointers on how to achieve that kind of string manipulation on variable Field elements in the circuit, but I'd really want to understand if you need it first

#

A generally useful reference for dynamic array and string parsing in provable code is this: https://github.com/gretzke/zkApp-data-types

#

My take is -- string parsing shouldn't be difficult. If it is, and there are real use cases, then we should add stlib methods to make it easy. It's not a huge lift

#

Have you looked at CircuitString from o1js btw?

stoic coyote Oct 3, 2023, 9:42 AM

#

pale maple Have you looked at `CircuitString` from o1js btw?

Thanks for sharing your thoughts!

Yeah, I've looked into it. But did not found a way to make it work. So went back to Array of Fields (Encoding.stringToFields).

finite knoll Oct 3, 2023, 9:42 AM

#

Poseidon.hash(string.split(‘’).charCodeAt(0).map(charCode => Field(charCode))

xD very naive

pale maple Oct 3, 2023, 9:43 AM

#

but that's exactly the stuff you don't need

finite knoll Oct 3, 2023, 9:43 AM

#

but using built in encoding is better ^*

pale maple Oct 3, 2023, 9:43 AM

#

I think the question is more about how to do parsing on an input array of field elements which represent characters

#

so, CircuitString

#

you can't do parsing (like, indexOf("-")) on the Encoding representation or on a Poseidon hash

finite knoll Oct 3, 2023, 9:45 AM

#

i think the question here is more what is it that needs to be proven? my first hunch would be to say the identification number can be just a single field input to the circuit

pale maple Oct 3, 2023, 9:45 AM

#

I think the objective here is representing a string as an array of characters, where each character is a Field element, and do stuff like string.indexOf("-") and string.substring(1,2) on it

#

the gretzke library is by far the most advanced example of this that I've seen in o1js so far

#

but I've also seen very advanced examples in circom which is more painful with abstractions and data types than o1js 😄 so a lot is doable

stoic coyote Oct 3, 2023, 9:47 AM

#

finite knoll i think the question here is more what is it that needs to be proven? my first h...

Proof of age. Where data is off-chain, relied by zkOracle. It contains personal identification number expressed in a way that needs string parsing to extract date of birth.

finite knoll Oct 3, 2023, 9:50 AM

#

i see i see, so you extract the age from the ID

stoic coyote Oct 3, 2023, 9:51 AM

#

finite knoll i see i see, so you extract the age from the ID

Exactly. And i'm struggling with doing this inside the circuit. Feeling very tempted to do this as part of zkOracle, not zkProgram.

finite knoll Oct 3, 2023, 9:52 AM

#

if you do what gregor is suggesting you end up with this:

ID -> Field[] -> myCircuit(id: Field[])

And now you need to know what character from the ID you want to be parsing, by accessing the Provable.array argument of the circuit

#

e.g. id[0] == age, or id[0] = day id[1] = month id[2] = year

#

from there you can compute the age, using only the encoded ID as input

stoic coyote Oct 3, 2023, 9:56 AM

#

Got it. I think this is exactly what I was trying to do. I did this using Encoding.stringToFields but instead I should be using CircuitString, right?

Trying to do this I ran into some troubles. Let me get back with a code example.

pale maple Oct 3, 2023, 10:02 AM

#

I know that CircuitString is limited right now but there's nothing stopping us from adding methods to it

#

The reason Encoding.stringToFields makes no sense is that it packs many bytes (= characters) into one number. But for your usecase you want character-level information readily available. For example, to test if the ith character equals "-", you want to to something like char.equals(Character.from("-"))

#

Oh btw, everything that's statically known makes stuff much easier

#

Like, if you know that the 5th and 6th characters are the two digits of the month, then you should use that info and use constants (5, 6) instead of doing it dynamically

stoic coyote Oct 3, 2023, 10:12 AM

#

pale maple The reason `Encoding.stringToFields` makes no sense is that it packs many bytes ...

Okay, din't realize that Encoding.stringToField did this.

#

import { Field, method, Experimental, Encoding, Provable } from 'o1js';

const program = Experimental.ZkProgram({
  methods: {
    parseData: {
      privateInputs: [Provable.Array(Field, 1)],
      method(pno: Field[]) {
        console.log(pno) // [ Field { value: [ 1, 86 ] } ]
      },
    },
  },
});

// run
const { verificationKey } = await program.compile();
const pno_ = Encoding.stringToFields('LT-39509100123')
console.log(pno_) // [ Field { value: [ 0, [Uint8Array] ] } ]
const proof = await program.parseData(pno_)

#

Interestingly, if I console.log the result of Encoding.stringToFields outside of zkProgram, it contains an array of Uints8. Inside the zkProgram it logs just a single value.

pale maple Oct 3, 2023, 10:14 AM

#

yes - the two cases you're observing are constants vs variables

#

check out the implementation https://github.com/o1-labs/o1js/blob/main/src/lib/field.ts#L132

stoic coyote Oct 3, 2023, 10:14 AM

#

Oh man I don't know so much 😄

pale maple Oct 3, 2023, 10:15 AM

#

https://github.com/o1-labs/o1js/blob/main/src/lib/field.ts#L54-L71

stoic coyote Oct 3, 2023, 10:15 AM

#

Thanks, let me try to understand this, and then do the same with CircuitString. Will see if I can make that work.

pale maple Oct 3, 2023, 10:17 AM

#

you don't need to know the internals of the field representation to solve your task btw. just that they are variables on which you can do addition and multiplication and basically nothing else. so every programming task has to be solved by formulating it as a combination of addition and multiplication (and of course all the helpers like field.equals() on top that you already have in o1js :D)

stoic coyote Oct 3, 2023, 10:57 AM

#

Trying to do this using CircuitString. How Do I convert substrings to ints? Is it even possible? Should I take a single char and then convert it to int?

import { Field, method, Experimental, Encoding, Provable, CircuitString } from 'o1js';

const program = Experimental.ZkProgram({
  methods: {
    parseData: {
      privateInputs: [CircuitString],
      method(pno: CircuitString) {
        const year = pno.substring(4, 6)
        const month = pno.substring(6, 8)
        const day = pno.substring(8, 10)
        console.log(year.values.length) // 128, why not 2?
        // how do I convert these CircuitStrings to Ints?
      },
    },
  },
});

// run
const { verificationKey } = await program.compile();
const pno_ = CircuitString.fromString('LT-39509100123')
const proof = await program.parseData(pno_)

stoic coyote Oct 3, 2023, 11:58 AM

#

I guess my last question is badly phrased. Let me try with a different example. Lets say I want to take single character (Field) from CircuitString and check if its equal to something. The following fails, because the the last constraint is unsatisfied. How do I correctly substring CircuitString?

import { Field, method, Experimental, Encoding, Provable, CircuitString } from 'o1js';

const program = Experimental.ZkProgram({
  methods: {
    parseData: {
      privateInputs: [CircuitString],
      method(pno: CircuitString) {
        const digit = pno.substring(3, 4) // that is "3"
        const digit_ = digit.values[0].value // is this how you do it?
        digit_.assertEquals(Field('3'))
      },
    },
  },
});

const { verificationKey } = await program.compile();
const pno_ = CircuitString.fromString('LT-39509100123')
const proof = await program.parseData(pno_)

stoic coyote Oct 3, 2023, 12:46 PM

#

Also, CircuitString.fromString('abc012') maps each char to a value of UTF-16 encoding. For example:

character a is mapped to Field(97)
character 0 is mapped to Field(48)

I assume this means that hashmap structure must be implemented to map the values back.

For example CircuitString.fromString('0') will actually hold a value of Field("48") and I would have to map it to Field(0) if I want to do any math with it, right..?

gray fox Oct 3, 2023, 12:51 PM

#

You don't need map, just substract 48.

stoic coyote Oct 3, 2023, 12:57 PM

#

gray fox You don't need map, just substract 48.

Yeah, but the number in the original string can be any single digit number :/ Its not going to be zero-only.

#

No wait, 0 is 48, 1 is 49 😄 Okay got it.

gray fox Oct 3, 2023, 12:59 PM

#

You also probably want to check that it's in the correct range

wise ridge Oct 3, 2023, 1:21 PM

#

@stoic coyote can you tell me about the zkOracle part? (Or point me to code). Are you providing your own zkOracle by controlling the API yourself and having it sign responses? Or are you ingesting someone else's data that is signed? Or something else?

stoic coyote Oct 3, 2023, 2:13 PM

#

wise ridge <@749314817449394186> can you tell me about the zkOracle part? (Or point me to c...

Sure, keep in mind this is quick and dirty first version: https://github.com/id-Mask/zk-Oracle/blob/ac37a9dc41028020fc5242d67ae7f066cc3eaf17/server.js#L137-L172

I'm deploying my own zkOracle that is basically a wrapper around some other API that expose the data I need.

GitHub

zk-Oracle/server.js at ac37a9dc41028020fc5242d67ae7f066cc3eaf17 · i...

Contribute to id-Mask/zk-Oracle development by creating an account on GitHub.

#

@pale maple @finite knoll @gray fox thanks for helping me out! I've got what I wanted.

The key part here was to understanding the difference between Encoding.stringToFields and CircuitString, and then use the simple trick to subtract 48 from UTF-16 encoded values.

For future reference, below is an incomplete example of how to parse date from personal id.

import { Field, method, Experimental, Encoding, Provable, CircuitString, Circuit } from 'o1js';

const program = Experimental.ZkProgram({
  methods: {
    parseDoB: {

      privateInputs: [CircuitString],
      method(pno: CircuitString) {

        // century
        const firstChar = pno.substring(6, 7)
        const firstDigit = firstChar.values[0].value.sub(48)
        let century = Field(18)
        century = Circuit.if(firstDigit.greaterThan(2), century.add(1), century);
        century = Circuit.if(firstDigit.greaterThan(4), century.add(1), century);
        Provable.log('Century: ', century)

        // decade
        const secondChar = pno.substring(7, 8)
        const decade = secondChar.values[0].value.sub(48)
        Provable.log('Decade: ', decade)

        // year
        const thirdChar = pno.substring(8, 9)
        const year = thirdChar.values[0].value.sub(48)
        Provable.log('year: ', year)

        // date / timestamp
        const date = century.mul(100).add(decade.mul(10).add(year))
        Provable.log('date: ', date)

        return date
      },
    },
  },
});

const { verificationKey } = await program.compile();
const pno = CircuitString.fromString('PNOLT-39509100123')
const proof = await program.parseDoB(pno)

#Trade offs between executing code inside and outside the circuit

Plain JS outside of circuit

o1js inside the circuit