Cannot compile SmartContract returned by the function in case the SmartContract uses ZkProgram | Mina Protocol | Page 1

empty ibex Nov 29, 2023, 6:02 PM

#

I'm receiving the error message:

 TreeVerifier.compile() depends on TreeCalculation, but we cannot find compilation output for TreeCalculation.
    Try to run TreeCalculation.compile() first.

      41 |   it(`should compile contracts`, async () => {
      42 |     await TreeCalculation.compile();
    > 43 |     await TreeVerifier.compile();
         |                        ^
      44 |   });
      45 | });
      46 |

when trying to compile SmartProgram and ZkProgram returned by the function. Both SmartProgram and ZkProgram use MerkleWitness, and the height of the Merkle Tree is the argument to the functions:

import { describe, expect, it } from "@jest/globals";
import { Field, MerkleWitness, ZkProgram, method, SmartContract } from "o1js";

function TreeCalculationFunction(height: number) {
  class MerkleTreeWitness extends MerkleWitness(height) {}

  const TreeCalculation = ZkProgram({
    name: "TreeCalculation",
    publicInput: Field,

    methods: {
      check: {
        privateInputs: [MerkleTreeWitness, Field],

        method(root: Field, witness: MerkleTreeWitness, value: Field) {
          const calculatedRoot = witness.calculateRoot(value);
          calculatedRoot.assertEquals(root);
        },
      },
    },
  });
  return TreeCalculation;
}

function TreeVerifierFunction(height: number) {
  const TreeCalculation = TreeCalculationFunction(height);
  class TreeProof extends ZkProgram.Proof(TreeCalculation) {}

  class TreeVerifier extends SmartContract {
    @method verifyRedactedTree(proof: TreeProof) {
      proof.verify();
    }
  }
  return TreeVerifier;
}

const TreeCalculation = TreeCalculationFunction(4);
const TreeVerifier = TreeVerifierFunction(4);

describe(`Merkle Tree contracts`, () => {
  it(`should compile contracts`, async () => {
    await TreeCalculation.compile();
    await TreeVerifier.compile();
  });
});

#

When I use plain SmartContract and ZkProgram, without them being generated by the function, everything works:

import { describe, expect, it } from "@jest/globals";
import { Field, MerkleWitness, ZkProgram, method, SmartContract } from "o1js";

class MerkleTreeWitness extends MerkleWitness(4) {}

const TreeCalculation = ZkProgram({
  name: "TreeCalculation",
  publicInput: Field,

  methods: {
    check: {
      privateInputs: [MerkleTreeWitness, Field],

      method(root: Field, witness: MerkleTreeWitness, value: Field) {
        const calculatedRoot = witness.calculateRoot(value);
        calculatedRoot.assertEquals(root);
      },
    },
  },
});

class TreeProof extends ZkProgram.Proof(TreeCalculation) {}

class TreeVerifier extends SmartContract {
  @method verifyRedactedTree(proof: TreeProof) {
    proof.verify();
  }
}

describe(`Merkle Tree contract`, () => {
  it(`should compile contracts`, async () => {
    await TreeCalculation.compile();
    await TreeVerifier.compile();
  });
});

#

The question: How can I generate the SmartContract and ZkProgram without repeating the code for each Merkle Tree height?

empty ibex Nov 29, 2023, 7:05 PM

#

It looks like the both contracts should be returned by the same function for it to work:

import { describe, expect, it } from "@jest/globals";
import { Field, MerkleWitness, ZkProgram, method, SmartContract } from "o1js";

function TreeFunction(height: number) {
  class MerkleTreeWitness extends MerkleWitness(height) {}
  const TreeCalculation = ZkProgram({
    name: "TreeCalculation",
    publicInput: Field,

    methods: {
      check: {
        privateInputs: [MerkleTreeWitness, Field],

        method(root: Field, witness: MerkleTreeWitness, value: Field) {
          const calculatedRoot = witness.calculateRoot(value);
          calculatedRoot.assertEquals(root);
        },
      },
    },
  });

  class TreeProof extends ZkProgram.Proof(TreeCalculation) {}

  class TreeVerifier extends SmartContract {
    @method verifyRedactedTree(proof: TreeProof) {
      proof.verify();
    }
  }
  return { calculation: TreeCalculation, verifier: TreeVerifier };
}

const height = 5;
const { calculation: TreeCalculation, verifier: TreeVerifier } =
  TreeFunction(height);

describe(`Merkle Tree contract`, () => {
  it(`should compile contracts`, async () => {
    await TreeCalculation.compile();
    await TreeVerifier.compile();
  });
});

The code above works.

@echo vessel is it a bug or a feature?

echo vessel Nov 29, 2023, 11:15 PM

#

Seems like a weirdness of jest to me, can you reproduce the problem in a plain script ran with nodejs?

empty ibex Nov 30, 2023, 8:57 AM

#

echo vessel Seems like a weirdness of jest to me, can you reproduce the problem in a plain s...

The same error with plain typescript:

/Users/mike/Documents/DeFi/MinaNFT/minanft-lib/.yarn/cache/o1js-npm-0.14.2-66e53db98b-a1f7a03bab.zip/node_modules/o1js/dist/node/bindings/compiled/_node_bindings/snarky_js_node.bc.cjs:6815
         throw err;
         ^

Error: TreeVerifier.compile() depends on TreeCalculation, but we cannot find compilation output for TreeCalculation.
Try to run TreeCalculation.compile() first.
    at /Users/mike/Documents/DeFi/MinaNFT/minanft-lib/.yarn/cache/o1js-npm-0.14.2-66e53db98b-a1f7a03bab.zip/node_modules/o1js/dist/node/index.cjs:13636:15
    at Array.map (<anonymous>)
    at picklesRuleFromFunction (/Users/mike/Documents/DeFi/MinaNFT/minanft-lib/.yarn/cache/o1js-npm-0.14.2-66e53db98b-a1f7a03bab.zip/node_modules/o1js/dist/node/index.cjs:13629:34)
    at /Users/mike/Documents/DeFi/MinaNFT/minanft-lib/.yarn/cache/o1js-npm-0.14.2-66e53db98b-a1f7a03bab.zip/node_modules/o1js/dist/node/index.cjs:13509:52
    at Array.map (<anonymous>)
    at compileProgram (/Users/mike/Documents/DeFi/MinaNFT/minanft-lib/.yarn/cache/o1js-npm-0.14.2-66e53db98b-a1f7a03bab.zip/node_modules/o1js/dist/node/index.cjs:13509:27)
    at Function.compile (/Users/mike/Documents/DeFi/MinaNFT/minanft-lib/.yarn/cache/o1js-npm-0.14.2-66e53db98b-a1f7a03bab.zip/node_modules/o1js/dist/node/index.cjs:15166:81)
    at main (/Users/mike/Documents/DeFi/MinaNFT/minanft-lib/experimental/treecompile.ts:40:22)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

Node.js v20.8.1

using the code

import { Field, MerkleWitness, ZkProgram, method, SmartContract } from "o1js";

function TreeCalculationFunction(height: number) {
  class MerkleTreeWitness extends MerkleWitness(height) {}

  const TreeCalculation = ZkProgram({
    name: "TreeCalculation",
    publicInput: Field,

    methods: {
      check: {
        privateInputs: [MerkleTreeWitness, Field],

        method(root: Field, witness: MerkleTreeWitness, value: Field) {
          const calculatedRoot = witness.calculateRoot(value);
          calculatedRoot.assertEquals(root);
        },
      },
    },
  });
  return TreeCalculation;
}

function TreeVerifierFunction(height: number) {
  const TreeCalculation = TreeCalculationFunction(height);
  class TreeProof extends ZkProgram.Proof(TreeCalculation) {}

  class TreeVerifier extends SmartContract {
    @method verifyRedactedTree(proof: TreeProof) {
      proof.verify();
    }
  }
  return TreeVerifier;
}

async function main() {
  const TreeCalculation = TreeCalculationFunction(4);
  const TreeVerifier = TreeVerifierFunction(4);
  await TreeCalculation.compile();
  await TreeVerifier.compile();
}

main();

#

I am still not sure it is a bug, as this behavior has some logic. When I create several interconnected contracts inside the function, the SmartContract should ensure it relates to the proper ZkProgram.

echo vessel Nov 30, 2023, 1:40 PM

#

aah I see now -- right, it doesn't work because the zkprogram you compiled is not the same JS value as the one your contract takes

#

didn't realize this earlier

#

not a bug. not sure it's a feature either 😛 but an expected inconvenience 🙂

empty ibex Dec 1, 2023, 8:16 AM

#

echo vessel aah I see now -- right, it doesn't work because the zkprogram you compiled is no...

Btw, is it possible, in principle, to serialize ZkProgram to pass it to the backend as a JSON object for recursive proofs calculation, or backend should be recompiled every time ZkProgram is updated or a new ZkProgram is added?

echo vessel Dec 3, 2023, 9:38 AM

#

empty ibex Btw, is it possible, in principle, to serialize ZkProgram to pass it to the back...

You mean, instead of creating the zkprogram methods from code, create them from a serialized object?

#

It's not possible in general because a zkprogram method can contain arbitrary JS which is used to compute witnesses in the prover. So the prover is fundamentally not serializable

empty ibex Dec 3, 2023, 9:40 AM

#

echo vessel You mean, instead of creating the zkprogram methods from code, create them from ...

Yes, create ZkProgram on the local computer, then call JSON.stringify(ZkProgram.toJSON()), send this string to AWS lambda, and call ZkProgram.fromJSON(JSON.parse(str)).compile()

echo vessel Dec 3, 2023, 9:42 AM

#

Got it. It's a similar problem to sending an arbitrary JS function to a backend. It does work for some functions - by stringifying their code (fun.toString()) -- but breaks if they are not self contained, e.g. use something from their closure, which most functions do

empty ibex Dec 3, 2023, 9:44 AM

#

echo vessel Got it. It's a similar problem to sending an arbitrary JS function to a backend....

So in this case we should look somewhere to dynamic imports, like calling require("MyZkProgram") in JS code

echo vessel Dec 3, 2023, 9:45 AM

#

Yeah. You could probably make it work to send a full JS bundle to the backend and require it dynamically

#

Probably that bundle should not contain o1js 😄

#

Because it's so big

empty ibex Dec 4, 2023, 8:30 AM

#

echo vessel Probably that bundle should not contain o1js 😄

There is no need to include o1js in the bundle; it can also be loaded dynamically. In my tests, o1js dynamic loading takes 1.532 seconds. All that is needed is to put a small 2KB .js file into the AWS S3 bucket.

For it to work, I import o1js two times - first using import type to make TypeScript code compile and then, on AWS lambda await import to actually load it.

The following code works and successfully compiles ZkProgram and SmartContract on AWS lambda. It is being called as

await example("contracts", "TreeFunction", 5);

First, it copies contracts.js from S3 to lambda instance /tmp folder, then dynamically imports both contracts.js and o1js, calls the TreeFunction that returns ZkProgram and SmartContract, and compiles both of them:

const { PROVER_KEYS_BUCKET } = process.env;

export async function example(
  contractName: string,
  functionName: string,
  height: number
) {
  const contractsDir = "/tmp/contracts";
  const files = [contractName + ".js"];
  // Copy compiled from TypeScript to JavaScript source code of the contracts
  // from S3 bucket to AWS lambda /tmp/contracts folder
  await loadCache(PROVER_KEYS_BUCKET!, contractsDir, files);
  await listFiles(contractsDir);

  const contracts = await import(contractsDir + "/" + contractName);
  const o1js = await import("o1js");
  const { TreeCalculation, TreeVerifier } = await contracts[functionName](height, o1js);
  await TreeCalculation.compile();
  await TreeVerifier.compile();
}

The contracts.js is compiled from contracts.ts:

import type {
  Field,
  MerkleWitness,
  ZkProgram,
  method,
  SmartContract,
} from "o1js";
import type * as O1js from "o1js";

export async function TreeFunction(height: number, o1js: O1js) {
  const { Field, MerkleWitness, ZkProgram, method, SmartContract } = o1js;

  class MerkleTreeWitness extends MerkleWitness(height) {}
  const TreeCalculation = ZkProgram({
    name: "TreeCalculation",
    publicInput: Field,

    methods: {
      check: {
        privateInputs: [MerkleTreeWitness, Field],

        method(root: Field, witness: MerkleTreeWitness, value: Field) {
          const calculatedRoot = witness.calculateRoot(value);
          root.assertEquals(calculatedRoot);
        },
      },
    },
  });

  class TreeProof extends ZkProgram.Proof(TreeCalculation) {}

  class TreeVerifier extends SmartContract {
    @method verifyRedactedTree(proof: TreeProof) {
      proof.verify();
    }
  }
  return { TreeCalculation, TreeVerifier };
}

To summarize, instead of serializing ZkProgram/SmartContract to be sent to the backend, it is possible to send plain JS code of the ZkProgram/SmartContract to the backend and compute an unlimited number of recursive proofs)))

echo vessel Dec 4, 2023, 8:33 AM

#

btw if you want type-safety, you can probably add

import type * as O1js from "o1js";

// ...

function TreeFunction(o1js: O1js) {
  // ...
}

empty ibex Dec 4, 2023, 8:46 AM

#

echo vessel btw if you want type-safety, you can probably add ```ts import type * as O1js fr...

I've amended the contracts.ts code. It works perfectly, and there is no need to redeploy the backend, I've just copied new contracts.js to S3 to make this change))

#Cannot compile SmartContract returned by the function in case the SmartContract uses ZkProgram