#Is it possible to configure custom limits for the standalone quickstart?

165 messages · Page 1 of 1 (latest)

gentle bluff
#

It would be really nice to be able to set limits a bit higher than futurenet/testnet so you could see what limits you were actually hitting and by how much in order to tweak your contracst and calls. Is this currently possible?

#

Generally it would be really nice to have more profiling insight into budget spends on "actual" networks vs Rust tests

royal mason
#

yes and no - you can perform a network upgrade and we even have a script for that, but I'm not sure how easy it would be to execute with quickstart

#

it would be great if you could hack it for quickstart

gentle bluff
royal mason
#

(actually, seems like there is some commented-out code for standalone as well)

#

yes, that's what has been used for the upgrade

gentle bluff
#

I'm also guessing currently there's no real way to see how any given invocation stacked up against these values in case of simulation failure correct?

#

Would be nice if standalone had a just force it through to see where you limits will/did exceed

pliant light
#

Standalone alrady has the high limit override enabled, so it should have much higher limits than testnet.

#

@royal mason quickstart standalone != stellar-core standalone.

royal mason
#

simulation will need some limits, as its a sandbox and it shouldn't run arbitrarily large programs. but it could be somewhat larger than mainnet

#

oh, what is it then?

pliant light
#

This is why we renamed quickstart standalone recently to "local".

#

Quickstart local is just a local network running with mostly same config as testnet, except accelerated, and high limit override is enabled.

gentle bluff
pliant light
royal mason
#

I think high core limit overrides help much though (if anything, these can be somewhat confusing). simulation overrides seem much more useful

gentle bluff
#

Ultimately looking for the "official" way to test contracts when you're getting errors like this

host invocation failed\n\nCaused by:\n    HostError: Error(Budget, ExceededLimit)\n    DebugInfo not available\n
royal mason
#

IDK if we enable/collect debug events in standalone quickstart... but we should

pliant light
#

Then review the budget and see what is big.

royal mason
#

BTW this is an execution error, which means that you've exceeded the limit you've set in tx

pliant light
royal mason
#

(we need some debug guide for the errors... I know it's confusing)

gentle bluff
#

Even when testing against compiled WASM

royal mason
#

that's concerning

gentle bluff
#

This also won't throw if you go beyond limits like max_read_entries_tx or max_write_entries_per_tx

royal mason
#

preflight adds about 15% on top of the simulated instructions, so we should hopefully be reasonably safe. but could you quantify 'significant'?

pliant light
#

When you say different, do you mean different to what's actually required, or different to what preflight is saying you need?

gentle bluff
#

Latter. What preflight says is needed vs what Rust test says was used. Trying to get some concrete values. My contracts is…slow

#

I may have been lying

Rust test

Cpu limit: 18446744073709551615; used: 22852853
Mem limit: 18446744073709551615; used: 3425979

Simulation

"cost": {
    "cpuInsns": "24379542",
    "memBytes": "3405269"
},
royal mason
#

as I've mentioned above, simulation is expected to return higher values than the tests because it just adds some percentage on top of simulated values in order to account for metering divergence and possible ledger state changes

#

(metering divergence due to recording storage and recording auth, that is)

pliant light
#

Rust test is prob correct on cpuinsns?

royal mason
#

it has the same divergence issues. the only way to be correct (for a given ledger state) is to build a full transaction and run it in the enforcing mode (which isn't feasible in tests and still won't necesserily reflect the reality where e.g. more signatures are used for multisig)

#

but it should be close enough

gentle bluff
#

Same invocation but with higher resource consumption

Rust test

Cpu limit: 18446744073709551615; used: 47684029
Mem limit: 18446744073709551615; used: 18138765

Simulation

"cost": {
    "cpuInsns": "31068040",
    "memBytes": "8962315"
},

Seems like significant divergence to me. Especially given the test is actually trending higher

pliant light
#

You need to reset the budget after each run in a test, otherwise it accumulates.

gentle bluff
#

I'm doing this

env.budget().reset_unlimited();

let id = client
    .glyph_mint(&u1_address, &None, &map![&env], &Some(width as u32))
    .unwrap();

env.budget().print();
pliant light
#

The map needs creating before reset_unlimited.

#

I guess. I'm not sure actually.

gentle bluff
#

Doing

let map = map![&env];
env.budget().reset_unlimited();

let id = client
    .glyph_mint(&u1_address, &None, &map, &Some(width as u32))
    .unwrap();

env.budget().print();

Makes a very marginal difference from 67665746 to 67664478 cpu and 39437229 to 39437165. I'm still seeing roughly 10M more CPU and 1M MEM used on the actual network vs the Rust test side.

pliant light
#

This is very surprising.

#

@grand vapor @thin lynx Would you expect this? ☝🏻

gentle bluff
#

fwiw I'm sending quite a number of events in this invocation and interacting with storage and a SAC token as well

#

That was a lie the smaller numbers are on the Rust test side NOT the network side. (I corrected my statement above)

#

I'm also noticing my CPU result fluctuates running the same test in Rust. Which seems odd?

pliant light
#

That seems odd.

gentle bluff
#

The only randomness are my Address. Unless env.crypto().sha256 has some randomness as well, which I would doubt

pliant light
#

Sha256 shouldn't have any randomness. The system should be entirely reproducible, and not being reproducible would be a bug.

pliant light
#

Even the prng is seeded in tests with a fixed reproducible seed. The only thing that's random is Addresses are randomly generated, but that shouldn't cause any cost difference.

gentle bluff
#

@pliant light is it possible to extend the expiration defaults for local? It's very short atm

pliant light
#

@somber marsh could we make all the limits configurable? Maybe instead of the high limit being a Boolean, it could be a toml section letting us set every parameter including expiration?

#

Or is this something we'd be able to do if we used the upgrade?

somber marsh
#

We could, but there would be ~20 values to override so it would get messy just for testing. The upgrade mechanism is the right way to do this, but we need to add some tooling to make it easier to use.

royal mason
#

I might be missing something, but it seems to me like the issues that are being discussed here happen due to exceeding tx limits, not network limits. i.e. it has nothing to do with the network limit overrides.

pliant light
#

I agree upgrades would be better, mostly because we could even support changing the limits in the quickstart local network while the network is running without having to restart. But improving upgrades seem to be a ways off.

royal mason
#

again, could you please clarify the reason for configuring these limits?

#

in my mind there are only a few valuable setups: high limits on everything, matching mainnet, matching testnet. why should one ever need to e.g. raise the memory limit, but nothing else? the tx will still fail on the relevant network and network limit enforcement is really trivial to reproduce off-chain when necessary (e.g. in preflight)

gentle bluff
#

Running transaction against an actual network is very valuable when you can't check valid limits in Rust tests and returned errors on actual networks don't tell you what or by how much you exceeded a limit

pliant light
royal mason
#

right, that matches my intuition. but matching network is inconvenient to achieve via toml config and for high limits we already have a flag, don't we?

#

FWIW 'matching the network' is more involved than matching the limits. you'll also probably want to have the same fees and the same budget calibration

pliant light
#

High limits just sets a little bit high. Also when we add features that are bespoke to testing, we end up with bugs like what happened for the last two weeks where it didn't work the same as a network and was buggy. Since upgrade config is something that already exists, seems reasons to rely on that existing behavior and it's maximally flexible.

royal mason
#

I tottally agree here, which is why I don't want to add more test-only flags to core (which seems to be the suggestion here?). I want to off-load this to the upgrade script

gentle bluff
#

I'm honestly slightly indifferent as long as it's simple for devs. I will say the local quickstart is a bit handicapped with things expiring in what I feel like is 30 minutes

royal mason
#

hmm, I think that's useful - you probably should bump your entries. we could of course add an option to increase the default lifetime significantly for the local instances, but really we're not quite building a sandbox here - the end goal of the contract development is the mainnet deployment and you do need to worry about lifetimes

gentle bluff
#

30 minutes is painful for local. It should be easily configurable

#

Both lower and higher

royal mason
#

lower - maybe, higher - why? it probably matches mainnet, but it's 5x faster due to faster ledger closing. again, you should probably be bumping your entries, so I'm not sure if that's a 'bug' or a 'feature'

gentle bluff
#

Mainnet is going to be 30 minutes? 😳

#

For local it's absolutely a bug or a missing feature at least

#

local should let you pretty much do whatever with fees and limits

#

I should be able to isolate the interest I want to currently test against, for me right now that's fees and metering, but if I keep hitting expired entries every 30 minutes that's very frustrating. I'll deal with bumping later.

royal mason
#

it's 4096 ledgers, so whatever time it takes to apply that many ledgers (about 6 hours)

gentle bluff
#

I'm pretty sure local is closing ledgers like every second

#

It's definitely not taking 6 hours before my entries are expiring

royal mason
#

I should be able to isolate the interest I want to currently test against,
again, I'm not strictly opposed to making these modifiable. however, I'm not sure if that's the right development philosophy in general and I'm not sure it should be promoted. I'd hope most of the things are achievable within the unit testing framework and you'd go to the local network to figure out the tx side of the interactions

royal mason
gentle bluff
#

most of the things
Well they aren't right now. Many of the fees are not easily accounted for currently, if we can fix that then sure

royal mason
#

basically if you're running the WIP contracts against the localnet, then it's likely because we have gaps in the testing infra. I wonder what these gaps are

gentle bluff
#

It's a dev environment, if I wanted to test the prod I'd run on prod

#

Again though it should just be configurable, even if my a script vs toml configurations

royal mason
#

but the bump is a part of your contract logic, isn't it?

gentle bluff
#

For instance entries sure but not permanent ones no

royal mason
#

why?

#

imagine you have a token contract and your balance expire within 5 hours. would it be a good UX for your users?

gentle bluff
#

I don't want to? idk, I'm not testing that piece right now. I want to do that in a separate Stellar transaction later? It shouldn't matter. I should be able to test in a local env the specific part of my contract without having to worry about specific network configurations. Again local is already very close with this by allow much larger limits

royal mason
#

my point here is rather not that you should/shouldn't be able to do something, but more so about flagging the state archival. unlike common stuff, like instructions/gas, this might be not and immediately obvious concept. I'm a bit concerned about hiding it completely during the development process only to make it blow up during the release

gentle bluff
#

What we have now is a weird build the whole thing for prod before you test any piece of it but then when things start to break it's quite tricky to find where it broke

#

We need to be able to build and test incrementally separate parts of the contract without having to care about every other piece

royal mason
#

things start to break it's quite tricky to find where it broke
yes, that definitely sucks - we need to do better on the preflight side

gentle bluff
#

We may ultimately be saying the same thing but where you care more about archives I currently care more about fees

#

The difference is archieves actually have some tooling to be able to recoup in case you mucked up. Restore, bump, etc. But if you mucked up on fees, your just stuck, so that is far more concerning to me right now

#

Especially because you can't appropriate test to find these issues

royal mason
#

FWIW bumps are not free

gentle bluff
#

Sure, but you can do them without the need to call the contract, so you contract isn't bricked

#

You can very easily brick a contract right now if you don't get your fees right

#

And there's not clear way to test what fees you're using or hitting outside of just raw CPU and MEM

royal mason
#

ok, noted - besides the limit tweaking let's really get down to the fee issues you have

gentle bluff
#

In a nutshell it's easily measuring these 👆

  • First in a Rust test
  • And in a simulation
Screenshot_2023-10-11_at_2.07.15_PM.png
#

CPU and RAM we have via env.budget().print(); though as I've noted I'm not finding these to be terribly accurate to the actual network

#

But for the rest we really have nothing as if you go over during sim it just errors with unhelpful messages

royal mason
#

yes, that's fair

#

tx size is tricky to do in tests. everything else can actually be computed

gentle bluff
#

The workaround I'm using is testing in local which doesn't adhere to these limits and lets you go over (avoiding the error) and then you can check the response against this table to see where you've gone over

royal mason
#

(with a caveat of not having the real ledger state- but the same is true for local network as well)

gentle bluff
gentle bluff
gentle bluff
#

Which again the sim side of testing would catch even if the Rust test couldn't

royal mason
#

yes, that's why I'm not that concerned about it

gentle bluff
#

For example prior to 20-rc I was emitting a bunch of events. This 2 kb limit crushed my contracts. And I had no idea how to surface that. Until I ran in local then boom. I saw my events sum was well over 2 KB

royal mason
#

FWIW 2KB limit can be increased

#

that's unfortunately the first time someone complained about it

gentle bluff
#

The point is less that it's a problem, I may be an outlier (though 2kb is really limited)
The point for now is I had no clear avenue for how to find the source of the exceededLimit simulation error

#

(I'll open a separate issue for the 2kb limit to discuss)

royal mason
#

The point for now is I had no clear avenue for how to find the source of the exceededLimit simulation error
we need a separate issue for that as well. I think that just returning the diagnostic events could solve this

gentle bluff
royal mason
#

yes, it might need some env-side support, but I think plumbing the events (or at least the last diagnostic event that contains the error) should help with this

pliant light
#

Events are already accessible in tests. Is that what you're needing?

gentle bluff
royal mason
#

we need to a) reset events for every invocation (I think there is a separate issue for that) and b) add a getter for encoded events size

gentle bluff
#

Along with all the other limits as well. Ideally all collected back into the budget tool we already have

royal mason
#

b) should also filter out diagnostic events

pliant light
#

I didn't keep up with the thread because I had to go AFK. If we can make config updates easier to do I can incorporate them into quickstart so we can configure whatever we need on the quickstart side. If we can move away from bespoke testing functionality would be awesome, so we don't need core builds to tweak things.

gentle bluff
royal mason
#

yeah, please open an issue for soroban-tools, thanks

#

event observation is useful for testing the logic, but we also need a simple way to check the size. the same goes about storage - there is a way to see what's stored, but that's not convenient for verifying the total IO consumption

gentle bluff
pliant light
#

@gentle bluff Would it makes sense in your mind to disable expiration limits for local by default (or set them so high no typical developer would encount an expired entry), and provide an option to enable expiration so that developers can test with expiration when they're ready.

Reading the comments I'm getting a sense that folks are encountering expiration before they're ready, or maybe when it is irrelevant.

In my own experience when I'm focused on contract development I don't care about expiration because for the most part it just results in me having to stop what I'm doing and run extra commands. When I'm focused on client/dapp development I do get to a point where I care about expiration because I want to make sure the wallet or application I'm developing knows how to restore.

royal mason
#

I'm getting a sense that folks are encountering expiration before they're ready,
ugh, I guess it's nice to help with on-boarding... but I feel like it would be more productive to highlight this to the devs as soon as possible - it's really easy to miss expiration in the SDK and then if you make it easy in local it could result in devs wasting time building something that is not compatible with state archival. disabling expiration should at least be a conscious choice IMHO...

pliant light
#

building something that is not compatible with state archival
What are some examples of something that is not compatible?

gentle bluff
#

Well not bumping in your contract anywhere would be an example.

#

But yeah I would like an opt out of expiration flag for sure

pliant light
#

But that isn't a compatibility issue. You can restore, the expiration isn't destructive.

royal mason
#

it's pretty easy to shoot yourself in a leg with temp entries

pliant light
#

Incompatibility would mean your contract is entirely broken, or just won't work.

#

Temp entries are a good example of why we should keep expiration on.

royal mason
#

also just thinking about user flows - while everything is possible to do in theory, in practice owner-less protocols need to think about how they maintain their state. maybe it's not strictly 'incompatible' but I can imagine certain design approaches being less feasible given the existence of state archival

pliant light
#

What's an example of that?

royal mason
#

basically anything with cold storage + no ownership might be tricky. I don't have a specific example though

pliant light
#

I'm just wanting to get my mind seeing the same scenarios.

gentle bluff
#

Either way I think it’s the more rare case especially early in the dev cycle/journey so having hard mode on by default feels wrong

pliant light
#

Can the default expiration of temp entries and persistent entries be set separately?

royal mason
#

again, I'm not opposed to maybe not disabling the state archival completely (that would require too much core hacking), but at least setting arbitrarily large min expiration such that it's not noticeable unless you develop your contract for years. I just don't want to make it default and hide state archival completely

royal mason
#

it is in fact different

pliant light
#

We could benefit from temp entries expire, persistent entries don't.

royal mason
#

yeah, we could set persistent to be 10 years in the future

pliant light
#

Then include an option on quickstart --enable-state-archival.

#

To reduce persistent down to something shorter.

royal mason
#

yeah

pliant light
#

Once we can configure things we can let people set the vaue instead of being a boolean.

royal mason
#

btw with new terminology ('state archival') we can really only apply it to persistent entries, because temp entries are not archived

pliant light
#

That's perfect.

royal mason
#

so I think state archival implies persistent anyway. and temp entries are just temporary - I think it's a well understood concept and there shouldn't be an issue with temp entries being gone forever

pliant light
#

To make this happen, is the right thing to add a new config to stellar-core, or hang tight for the new config updater tool and we'll use that in quickstart?

royal mason
#

if this requires a new core build, I'd say let's do the proper (update) thing. there is a bunch of stuff to merge in core and I don't think we have much time for test-only settings