State Expiration: Proposal to Simplify Exclusive Storage

[tranquil silo]

@subtle sigil and I would like to propose a modification to State Expiration to reduce the complexity and operational cost of using exclusive storage.

Proposal: https://docs.google.com/document/d/1v4zCiT1xitKb8pMqYROeAiz1IEElv4ySRxqrQKsVnz8

cc @dapper path @late oriole @tall onyx

[tall onyx]

I think this is pretty much what we had in mind for a while in terms of avoiding proofs for the most use cases. the main difference is that in the previous discussions we assumed that proofs would still be needed past the initial eviction epoch.

I like the overall idea, the main issue is to make sure that it's we actually never need per-user unique entries

[tranquil silo]

the main issue is to make sure that it's we actually never need per-user unique entries
Anything we need to do to validate that, nows the time. 👍🏻

[tall onyx]

and of course we'll need to more work to support per-contract global storage (it is a nice thing to have in general)

so the one use case I can think of is signatures with unlimited lifetime. are these actually ever useful? I don't know - I would be happy to not support that

[harsh nymph]

are these actually ever useful?
That's a very good question. The only situations I can think of where these are useful are for escrow style account and presigned timelocked transactions to unlock funds. This is a well explored pattern in Stellar. However, in Soroban-land you could create a simple timelocked vault contract that is restorable and doesn't require the long lived signature

[tall onyx]

yeah, you could actually implement any sort of 'forever' storage via a separate contract. I guess the requirement should be that an arbitrary signature has unlimited expiration

[tranquil silo]

This is also not prohibitive to impl, it is just inconvenient. You could bump your nonce forever, or a long time, with a far future ledger bound expiry.

[harsh nymph]

You could bump your nonce forever, or a long time
I think we should be careful about suggesting that the solution for protocols that involve long periods of no interactions is to add interactions

[tall onyx]

that won't be true at least for built-in nonces; they'll have to have a well-defined expiration ledger because the signature itself will have to be defined with expiration ledger.

[subtle sigil]

We had some more discussion about this internally and realized there are some problems if we make "different creation epochs" mean "different instances". In particular it makes the SAC creation path and, more generally, any sort of factory pattern a bit awkward, because the factory or SAC-creator has to pick parameters to create an instance at the moment of creation and that'll differ from one creation to another, eg. if you create in 2023 you get a different instance than if you create in 2024. and if the factory is like a long-lived multiplexor it has to keep track of which epoch it created a given sub-instance with in order to route requests to it.

I proposed a followup idea which is to basically make instance-eviction definitionally lag data eviction: we expire instances when they expire, but we evict them only after their contract data LEs are all evicted -- basically extend the refcounting we're going to do between the instance => wasm edge in the graph to also count data => instance edges in the graph (these relationships are all acyclic so merely counting references should be enough)

if we do so, we create a circumstance where you can recreate instances, but only when they're being recreated into essentially a vacuum of no-live-data, and then subsequent revivals of live data under the recreated instance are at least going to be consistent with it. I'm not 100% sure this would solve all possible problems in this space though -- in particular I'm not sure it'd solve the "instance takeover by recreation with default admin" scenario -- but I thought I'd mention it as a possible conceptual building block we could consider.

[vernal portal]

Ref counting is an interesting idea. I’ll have to think some more about how this could be used to sidestep rent. Like keep a few bytes alive in order to keep 100KB alive type of situation.

[tall onyx]

basically this would mean that if no-one is using/bumping the contract, it effectively might be considered deleted, as there is no guarantee that it will have the same behavior after recreation

[vernal portal]

Mmm yeah maybe you’re right: this may not actually change things much in practice. Users have to be somewhat active in both cases. I think what we’re really saying here is that if we want to build on top of “mergeable” semantics and people need “unique” kind of behavior they have to pay rent for something on a regular basis.

[subtle sigil]

I think if you want the answer to be "you have to pay rent regularly" you need a good answer for the followup "ok but what if they don't"

[tall onyx]

yeah, if you want to keep a contract indefinitely, you need to bump at least some entry. I think it's ok-ish for e.g. tokens or AMMs (if nobody ever uses them, then probably the impact of potential recreation is minimal or non-existent). but for some other contracts that might be an issue

[subtle sigil]

like practically speaking: if someone doesn't pay rent on something "they have to pay rent on" and it's evictable and then it gets evicted and we don't have any mechanism that prevents recreation then we haven't got anywhere: someone can recreate it and take over the contract after eviction.

the point of the negative evidence was to prevent that

[sonic idol]

These are valuable in any system that tracks a credit / liability to a user (like Aave, Maker), because the user will not actively pay rent to keep their credit / liability entries alive.

It might be OK to put precedence on the protocol to keep these entries up to date on rent, but currently there is no simple way to find these entries, so it's not reasonable for the lenders / owners of the protocol to bump these.

[tall onyx]

hmm, is there an issue about recreating the liability entry? similarly to balance, it can just be merged (I suppose the party that wants to do the merge is different, but that's fine)

[sonic idol]

if a credit / liability entry gets evicted, whatever the user did with the protocol to take that credit / liability in the first place can be removed, as the protocol won't know about the liability / credit.

For Aave and Maker, the user could remove their collateral before the merge could occur. There might be potential to add a user lock if there appears to be a mismatch (user did not repay but their liability is gone) that prevents them from removing collateral, but I haven't thought through if that system would be exploitable.

[tall onyx]

sorry, I don't quite understand what's going on here. but what I read seems more like an issue of eviction, not uniqueness

[sonic idol]

The unique entry solved this issue because you could run a proof of existence if a liability balance did not exist and you expected one.

Although, I do understand the current concerns requiring ESS (assuming this is the node that tracks the eviction store) to be highly available and low latency

I can think on the user lock workaround more, the more I think about it now the more confident I am it will be safe.

[tall onyx]

cool

[sonic idol]

cc: @vagrant flume and @rough spindle - this change also could concern both of your projects

[sonic idol]

@harsh nymph Found a path forward that does not rely on any unique storage, although it deviates from a more traditional Aave / Compound interface for lending pools. (a benefit of this is that its more efficient from a VM Instatation front, too). I wasn't able to find a solution that kept the Aave / Compound interface that didn't rely on at least some unique storage per user.

Left my thought process in a doc if any1 has similar issues: https://docs.google.com/document/d/1wEnXhwGciFMFpNymIlB96giSCQlqSIyVDm_UZNArLYs/edit#heading=h.kb2r0dpnyvo5

The TL;DR is any credit / liability state (or something a user wants to expire) needs to be coupled with whatever state could be exploited in the event it does expire (like collateral) in a single ContractDataEntry.

[tall onyx]

this seems like a generally good idea although I understand that this might break the token abstraction. I wonder, would duplicating the balances solve this issue as well? (i.e. the entry in the main contract contains 'logical' balances in the same entry and tokens have their own balances and are usable outside of the main contract)

[sonic idol]

Yeah, there is potentially room to split collateralized deposits into the storage silo and leave non-collateralized deposits in a separate token.

Building a contract abstraction to make any "bToken" tradable still might be more effective, as there is nothing that should prevent the transfer of collateralized tokens either, provided the user has a large enough health factor.

Trading bTokens isn't a large use case, so I would need to do some research into how aTokens (aave's collateral token) are being used across chains.

[harsh nymph]

This does seem more efficient both from a vminstatiation perspective and also from a storage utilization perspective

@sonic idol what if you would implement a multi-token standard (EIP1155ish)?

[sonic idol]

That is roughly the idea to at least allow for transferability, but I don't expect there to be liquidity pool support or anything for these.

[harsh nymph]

I don't want to stir things up too much but this does raise an interesting question: should the standard soroban token interface be a multi-token? It does seem that between state expiration, vm instantiation cost and storage optimizations that grouping stuff together is a good overall strategy and our standards should encourage that

[tall onyx]

I'm not sure I understand that well enough, but I suppose that means that there will be just a single SAC instance that manages all the tokens? this is a bit wasteful in terms of storage space (+1 address per every entry) and also probably a bit more awkward to use

[harsh nymph]

That's a good point @tall onyx . You don't have to use it - even if the standard interface is multi-token the SAC implementation can be one contract per asset that only supports a single token id (side note: the popular ERC721 nft token standard is indeed a multi-token standard - it supports an arbitrary number of tokens per contract). But yeah it does mean that you have to add a token id to all token interactions, which is not ideal

More on ERC1155: https://docs.openzeppelin.com/contracts/3.x/erc1155

[tall onyx]

I don't quite get it TBH

arbitrary number of tokens would mean that storage has to store balances in separate keys, at which point that doesn't have any benefits vs just having multiple token instances. it does make sense for limited sets of tokens though, so that a all the balances are in the same entry

[tranquil silo]

@subtle sigil I have an alternative suggestion to fixing the issue for SAC:

Maybe a little controversial, but do we really win much by expiring SACs? Given a SACs interaction with classic Stellar, and given classic Stellar ledger entries don't expire, it seems reasonable to me that the SAC admin and meta data never expire.

SAC balances that are for non-stellar accounts can still be Mergeable.

The storage cost of a SAC should be tiny. They don't cause a WASM code entry to be kept on ledger.

[tall onyx]

we may eventually want classic entries to expire as well. also the concern still remains for regular contracts basically, any factory contract (e.g. AMMs) would want similar semantics where contract ID is predictable and deterministic

[tranquil silo]

Factory contracts are pretty different problem to the SAC which has global forever identity. I would think the one year lifetime (cc @subtle sigil) would be sufficient for most planning around factory contract IDs.

[tall onyx]

why do you think SAC should have a global forever identity, but, e.g. a uniswap-style AMM that swaps two SAC tokens shouldn't?

I think the use case is very similar

in one case it's inconvenient to distinguish between USDC-2023 and USDC-2024, in another case it's inconvenient to distinguish between USDC-BTC-2023 and USDC-BTC-2024

[tranquil silo]

1. Because of the tie to classic. Classic entries already have globally forever IDs. E.g. account IDs are not epoch contrained, and asset IDs are part account ID.

2. Also the win of expiring SAC meta/admin entries is tiny compared to the win of entries that have a different scale (entries that are per user like balances) or entries that are connected to entries of larger size (like contract wasm).

At least for the SAC case we're making perfect the enemy of good.

@vernal portal thoughts?

[tall onyx]

I'm really not a fan of such inconsistency/special-casing. on one hand, classic state not expiring is something that might change in the future. on the other hand, if we feel that non-stable ids are problematic, then chances are a lot of contract developers would think the same about their contracts.

so instead of special-casing SAC, I'd say we should rather look for generic solution available to all the contract devs. like we could consider providing an option for making the contract instance entry to never expire for much higher fee. so the cheap option would be expirable ID with epoch and there would be an expensive option for a non-expirable id

[sonic idol]

What is the gain for coupling the eviction window to the contract ID? It seems like the WASM hash, contract Id, and rent can be separated. Is there something I am missing?

It wouldn't be a big deal for a uniswap style factory to just recreated the same USDC-XLM pool, for example, at the previous contract ID in the event it expires.

[tall onyx]

What is the gain for coupling the eviction window to the contract ID?
it makes it impossible to recreate the contract (during eviction epoch it will be in the ledger, after that it won't be create-able anymore)

It wouldn't be a big deal for a uniswap style factory to just recreated the same USDC-XLM pool, for example, at the previous contract ID in the event it expires.
it would be an issue for any contract with meaningful global state (e.g. you wouldn't want someone to re-create your token with the same ID as before, but a different admin).

in case of the factory itself, you wouldn't want someone to re-create the factory itself with a different wasm.

[tranquil silo]

It wouldn't be a big deal for a uniswap style factory to just recreated the same USDC-XLM pool, for example, at the previous contract ID in the event it expires.
👍🏻 And this is supported by the proposal. If the pool with its epoch constrained ID expires, it can be restored. Restoring would restore the same wasm, and all instance/unique data that is stored in the contract instance.

[tall onyx]

right, the only downside is that it will be possible to create a different USDC-XLM pool. given that we'd hope new assets will be brought to the network it means that e.g. USDC-XLM pool will be instantiated in 2023, but, say, CHF-XLM, will only be instantiated in 2024 etc. hence if you live in 2025 you'd need to know when was every pool introduced and basically require a directory (with stable ids the pool id is known without a directory)

[tranquil silo]

The method by which IDs are currently generated is so complex, that I think folks will rely on directories, or websites, to tell them what the ID of a specific pool is in any case.

Or rather, dapps and apps will take care of that.

[tall onyx]

then why should SAC be special-cased?

[tranquil silo]

There should only be one Soroban address for each Stellar asset.

Otherwise an issuer might create the SAC for their token, configure an admin and discard their issuer key. Then someone might come along and recreate the SAC in another epoch which would by default have the issuer as the admin. It's reasonable for the issuer to be in control of who the admin is at all times.

[tall onyx]

hmm, I'm still not sure why this wouldn't be an issue for all the other contracts

[subtle sigil]

I think (if I understand correctly) @vernal portal 's idea is that "if we're going to need a directory anyways, a factory can use any unique salt it likes including just like nonces or random noise and stick the resulting identity in its directory and save us the trouble of committing to any design"

[tranquil silo]

Can you unpack that a bit, I don't see the big picture. Is that instead of an epoch derived ID?

[tall onyx]

that would require the factory itself to be non-recreatable though

[harsh nymph]

It does seem like SAC is the only case in which there's a hard requirement to have a unique canonical address. Given how lightweight the SAC is it's very palatable to just not expire it

[tall onyx]

I don't disagree about that, but shall we also expose this functionality to all the contracts in general?

this is also not a straightforward change btw - there needs to either be some special contract data storage tier for SAC instance only or there should be some special separate entries with the flag 'SAC instantiated' (as we unfortunately don't have the 'asset' ledger entry)

so if we need to do some non-trivial amount of work, might as well make it general

[tranquil silo]

I don't think this needs to be a general solution. We had a general unique data solution, and it was too costly. This proposal is about compromising for the majority of use cases.

My understanding from hearing @vernal portal and @river pilot discuss this in the past that making it possible to make other records to never expire is undesirable because folks will exploit it to keep spam on chain. Someone will always be willing to pay it. Folks can just keep rent bumping if they really need it in any case?

Also, an 'expire never' feature is something we could add in v1.1 if we find it is needed?

[tall onyx]

that's fine - the rent will increase as time goes. the issue is allowing the entry to stay forever for a fixed, relatively small price

btw I believe currently SAC can be instantiated for arbitrary assets, even if they don't exist on-chain (@late oriole please correct me if I'm wrong)

[tranquil silo]

Asset existence isn't really a thing on Stellar. Anyone can create a trust line for any asset, whether it has been issued or not. Anyone should be able to create a SAC before it is issued, or before a trust line exists for it, or before the issuer exists.

[tall onyx]

exactly, which is why you essentially can create an unlimited number of persistent ledger entries

[tranquil silo]

Avoiding that is I think perfect vs good. There are other ledger entries folks can create that will never expire.

[tall onyx]

if we think that's not an issue, then why not allow using the same hypothetical mechanism (still not sure how exactly it works) for all the contract ID entries?

[river pilot]

I think persisting the whole contract instance is problematic, but what if we just persist the key? If I understand correctly, the ideal solution would be that all Contract instances are unique but don't require proofs of nonexistence on creation. Is it that big of a deal if we just persist expired contract instance keys? ETH currently has 44 million contract instances deployed since genesis, which would require about 1.5 GB to persist (assuming 32 byte keys with some db overhead). We still expire the contract instance entry and send it to ESS since that could contain unique entries and be somewhat larger, but just persist the key. I don't think this is DOS-able. Each deployment persists around 40 bytes, and I imagine the cost of deployment is too high to make this feasible (but I could be wrong). This also doesn't incentivize storing arbitrary data via contract instances because the entry body is still expired and incurs a restoration cost. The only way this could be gamed is to provide a persistent boolean value that a contract doesn't have to pay rent for via the existence/non-existence of a given contract ID. Does the deployment cost outweigh this benefit, and is a persistent bool value useful enough to be used in this way? We could not expose the existence/non-existence of a contract ID to the host so that contracts can't use instances as a persistent bool which is probably good enough. Oracles could still do it but I doubt that's a worthwhile application.

I don't love the idea of persisting anything, but the set of contract IDs should be the smallest data set we have on the network. I think if we can guarantee that contract instances are unique without needing the ESS, that provides a sort of "root of truth" that makes other expiration problems a lot easier and simplifies contract deployment greatly

[tall onyx]

since that could contain unique entries and be somewhat larger,
entries don't necessarily have to contain anything beyond the executable (i.e. +32 bytes/entry) - everything else can be just logically tied to the contract instance via some mechanism. but that's more of an implementation detail indeed

btw if these are just keys, they just shouldn't even be usable if the payload has expired, so there is even no way to store 1 bit of info with that

we could also make this reasonably expensive and provide an alternative of using an epoch-based id for much cheaper. this would discourage throw-away contracts (like claimable balances) from using the persistent storage

[river pilot]

I'm still somewhat skeptical of a epoch-based id, just because it seems super complicated and footgunny for end developers (based on what I've caught up on so far). Can you talk a little more about the throw-away contract case you're trying to protect against? I'm more inclined to just have unique contract instance and temporary and leave out any epoch based "mergeable" case

[river pilot]

Yeah I was talking more about the hacky case where the key itself encodes the bool, i.e. If contractIDExists(id), do xyz

[tall onyx]

I think we can define two sets of contracts: ones that care about their ID being unique and deterministic (e.g. SAC, arguably AMMs and factory-deployed stuff like that, probably NFTs) and another set of contracts that don't really care about ids having any special properties beyond being non-recreatable (anything unique like smart wallets, temporary stuff like claimable balances or offers etc.) I don't have a good idea on what are the sizes of these sets, but seems like both have some real use cases

[tall onyx]

this check could just panic if contract entry hasn't been recovered (and it's not even like we have such host fn)

[river pilot]

We could expose some deployment option like "deterministic contract ID" that costs more and persists the contract ID and the cheaper "nondeterministic contract ID" that's cheaper and includes expiration epoch. I just don't know if this setting is too niche and complicated for actual developers, it seems pretty in the weeds @tranquil silo . We might be able to add this later, but at least for now I think deterministic id that persists should probably be the default (and maybe only option)

[tall onyx]

yeah, 2 options is what I'm suggesting. 'nondeterministic' is not exactly the right term here, probably should be something like 'stable' and 'unstable' or something like that. that could also be just the default mode and another mode reserved to users that actually care about their contract ids. but I'm not opposed to just having all ids unique in v1 either, as long as everyone else is on board with that

[tranquil silo]

I'm more inclined to just have unique contract instance and temporary and leave out any epoch based "mergeable" case
The intent with the epoch ID is to make contract instances essentially unique, not mergeable. They're still restorable, but could never exist in both places at once. If keeping "expired" contract IDs on chain is feasible, that sounds good to me too. Let's do that, it's simpler?

I don't think we need more options for folks. Contract instances that have different lifetimes could harm interoperability. Developers also need to make a pretty involved choice. I can imagine tutorials just always suggesting unique contract instances because they're safer.

So ideally contract instances are always unique, somehow, without requiring any proofs. And instance data attached to them is also unique.

Is storing a big set of expired contract IDs on ledger feasible?

[river pilot]

Yeah the easiest way to do that is just persist a contract ID key tombstone on expiration in the Bucket List. This means that the BucketList is guaranteed to grow over time, but the growth rate is the deployment rate of contract instances, which should be the smallest rate of any data type on the network

Since we're storing unique entries in the contract instance now, this means that creation of any unique data and deploying any contract never requires a proof. Proofs are only necessary when a given entry has actually expired and is in the ESS

[tranquil silo]

@subtle sigil Is this reasonable to you? It seems very reasonable to me.

[river pilot]

I feel like 40 mb of persistent storage per million contract deployments is very doable, especially since we still expire all data in the instance to prevent gaming

[tranquil silo]

I added the idea as proposal 6 to https://docs.google.com/document/d/12CCDtKkvghAZozXpEEPVCfLNmfzrykdNcqf0Eh_N-CI.

[tall onyx]

a bit annoying thing about this is the fees. I guess we can charge 0.5 XLM+ (to be not cheaper than classic) and do that based on the footprint (so it would be basically a value on top of the fee per RW entry)

[harsh nymph]

the growth rate is the deployment rate of contract instances, which should be the smallest rate of any data type on the network
I'm unsure about this statement. I think that by making contract instances the only unique ledger entries we might be encouraging contract implementations to bias towards creating more contract instances. Let's imagine two potential claimable balances implementations. In one there's a single ClaimableBalances contract that persists each individual claimable balance in a mergable ledger entry. The second has a ClaimableBalanceFactory that deploys a unique ClaimableBalance contract instance for each claimable balance. Which ones would devs prefer? I'm not sure - maybe the latter because they don't have to think about merging. But I do think that If we're keeping around every id for every contract ever created and devs bias towards creating contract instances then this might become a problem

[tranquil silo]

It's not necessarily better to do it that way though, because mergeable claimable balances would actually be fine. But I could see someone doing it that way because they think it is simpler.

[subtle sigil]

wouldn't a reasonable claimable balance system want to use the (much cheaper) temp entry types anyway? since there's no point storing an entry nobody claims?

[tranquil silo]

I don't think that's the example we should worry about, but there is probably a contract that could exploit this and would be reasonable, maybe preferred, to be one instance per use. 😬

[harsh nymph]

@subtle sigil yes, I agree that's what a reasonable implementation should look like but I'm not sure we should build on it

[harsh nymph]

yeah claimabale balances are one example. smart wallets also comes to mind as something that could be implemented in both ways

[tall onyx]

account contracts (aka smart wallets) have to be separate instances

[harsh nymph]

I'm not advocating against this, just saying that the expectation that contract instances count will grow slow is problematic

[tall onyx]

one other potential issue is that if the contract entries are going to have price dependent on the ledger size, someone might want to book a bunch of contract ids to sell them later. although I guess we could make contract updates pay the instance cost difference...

[river pilot]

I mean we have the max ledger entry lifetime already to combat agaisnt that. Also we just store the instance key permanently, so you still have to pay the restoration fee to actually use the instance, which is a function of instance size and current rent_fee price. All that to say I don't think buying IDs to "rent out" is financially effecient or possible

Do we currently have a specific contract deployment fee, or is it just charged the regular instruction and read/write fees? Adding a deployment specific fee could help account for the persistent fingerprint left behind on expiration and deincentivize weird "contract instance as storage" exploits

[tall onyx]

that's what I've referred to above when mentioning fees. actually, I think if we charge the additional fee any time we see contract instance in RW footprint, we would ensure that buying out ids is likely infeasible

[river pilot]

You could charge the same fee on restoration, like a deploy or re-deploy fee to defeat the exploit.

I guess your footprint approach already does this implicitly

[tall onyx]

yeah. it's probably the simplest option we could do. distinguishing instantiation and update is a bit more complicated (it's not that hard, but introduces more awkwardness in how fees are computed). maybe we still have to do that, not sure yet

[river pilot]

This is also one of those things we can change in V2 when we fully turn on state expiration. For example, if we see a lot of the ClaimableBalanceFactory type of behavior and need to not persist contract IDs, we could retroactively apply one of the other proposals we have been talking about. For example, in V2 would could implicitly have all the preexisting contract IDs have an archive epoch of 0. We would still need some additional backwards compatible logic, but it shouldn't be too complicated to implement a change like this (I think). Until we see actual usage patterns to see how contract instances are used, I'm inclined to go with the persistent fingerprint route