frail imp
State Expiration is coming in Preview #10. The semantics are going to stay with us for a long time (well forever actually) so we want to ensure things make sense. Please take a look at the attached doc and comment with feedback and suggestions. This is your last chance :). https://docs.google.com/document/d/1a8tKr58w7fApzt20F0okWUkiHJk0BbS2Qmer5LLlKvQ/edit#heading=h.g6cjreyrpbp7
lime oyster
“popped” or restored in a last in first out order
Does this mean restoring an entry, requires restoring entries that have been expired since my entry was expired? E.g. if 2000 entries have been expired since mine was, do I need to restore 2001 entries to get at mine?
green coral
> “popped” or restored in a last in first out order
Does this mean restoring an ...
FIFO is per key
lime oyster
Recreateable storage exposes a pop host function that restores a given entry from the ESS and returns it to be used within the smart contract. While the pop function removes the given entry from the ESS, it does not automatically add the entry back to the BucketList. Instead, it returns the entry to the smart contract function, allowing the smart contract to define custom merge logic before writing the entry back to the BucketList.
This sounds really interesting. I really like the idea of contracts being responsible for merging, but it also seems to present some challenges. Some questions:
• Who is responsible for validating an entry from the ESS is valid and not a forgery?
• How will stellar-core coordinate with ESS when a contract calls the host function?
• The ESS probably shouldn't delete an entry until 1-2 ledgers have passed, who will coordinate that?
• Who is responsible for ensuring a restored entry is done so atomically when Soroban supports concurrency? The footprint probably can't guarantee that entries are restored serially and the same ESS entry could be given to a contract twice in the same ledger where those invocations are processed concurrently. Or will the ESS entry key be required to be included in the footprint?
• Is there anything that prevents a contract from being given multiple entries for the same key to merge? e.g. Given 5 entries to merge for the same key.
• The host interface for merging feels very low-level. I think we could explore ways to make it so that contracts when retrieving entries are given multiple values, and contracts should define logic for merging them. e.g. A recreateable contract data get operation could always return a list of values, forcing contract developers to merge data whenever accessing the data.
lime oyster
• The host interface for merging feels very low-level. I think we could explore ...
This could also help with disconnecting getting data back in the bucketlist from contracts executing the merge.
green coral
I think one thing that we should probably look into is the "manual bump" experience from the wallet: I suspect that entries are going to follow a power low of sorts (because: humans), where most entries on ledger are not going to be very active, and therefore "autobump" will be useless as a mechanism to keep them alive, so people will probably want to manually bump their entries (probably every time they use them), by some amount that is context specific -- in some cases people will probably manually bump by some large number (like 6 months), where as in others by much more modest numbers (because the recovery cost is not bad).
smoky steppe
Or will the ESS entry key be required to be included in the footprint?
it has to be in order to be merged, right?
lime oyster
> Or will the ESS entry key be required to be included in the footprint?
it has...
Contracts can do whatever they want in code. There's no guarantee a contract will call set with the same key, or even read the same key.
I guess contracts that don't open themselves up to processing an ESS entry twice. But that isn't obvious to the contract developer.
smoky steppe
hmm, maybe, but in any case nothing prevents us from treating the proof as RW footprint for the sake of concurrency
lime oyster
My question really is what prevents ESS consumption more than once. Something will need to. I think that something has to be the environment/core.
Assuming the answer to my first question on responsibility is environment/core.
smoky steppe
sure, but that's a different issue, right? one issue is to invalidate ESS entry when it is externalized, another issue is concurrency (which is simpler)
but it's also not clear to me as to what is the flow for removing ESS entries from the archive, I've also left a comment in the doc
and therefore "autobump" will be useless as a mechanism to keep them alive,
unless autobump bumps entry for a significant amount of time, that is
forcing contract developers to merge data whenever accessing the data.
I've been thinking about this as well - as I've noted in the doc I believe that merge function should be mandatory with restorable storage, otherwise we'd need to implement some default behavior, which will likely break someone at some point. maybe the recreatable storage interface must have a merge function as a callback, so it's not possible to bypass defining it and every storage access is guaranteed to do the merge when needed
lime oyster
> forcing contract developers to merge data whenever accessing the data.
I've be...
The thing I'm not sure about is that the system can't assume the developer will call set with the new value. There might not be a new value to call set on. Or a contract might bail and not do anything. Can the two recreateable entries live in the bucketlist and then merge attempted later?
The way the doc is written sounds like the bucketlist assumes there's only one entry and so merge has to happen immediately at the point of reintegration.
If that's the case, the API can't be implicit where read results in merging as happenstance. It needs to be more explicit, where the contract actively knows in this moment I am responsible for merging.
smoky steppe
The thing I'm not sure about is that the system can't assume the developer will call set with the new value.
but that's ok, isn't it? then the restoration is basically a no-op, which is of course a bug, but shouldn't cause any significant issues beyond the scope of the contract
lime oyster
It's an irrecoverable bug. ESS will delete the entry. The system appears to have allowed restore, but not.
Pretty massive bug if token balances can vanish.
Maybe core/environment can only coordinate deletion from ESS if a matching key is written?
smoky steppe
but that's a logic bug, isn't it? we can't prevent those
like they can just do max instead of sum etc.
but as for ESS deletion mechanism, that's still not something I understand. ESS needs some way to know that the proof has been used before deleting it
or core, if proofs are stored on-chain (which was a bit of a surprise to me)
lime oyster
It doesn't seem like a logic bug to me. Contract was given multiple entries, but didn't end up needing to do anything with them and went on its way.
This is the distinction bewteen implicit and explicit. If the contract has an explicit function merge, then yes its a logic bug because it was told to merge and didn't.
smoky steppe
right, which is why I'm suggesting to require a merge as a callback for any restorable storage access
lime oyster
If the contract has implicit merging whenever it calls get, then no its not a logic bug, because it was told to perform some other action, but that action ended up being a noop, so it's not really at fault for also not merging when it was implicitly supposed to do that.
I was thinking the callback would just merge but not set.
It's quite surprising if the framework calls set / saves data magically.
smoky steppe
hmm, but that's what the definition of merge is?
take n entries, replace with a single entry...
lime oyster
You can merge data without saving it under the same key.
smoky steppe
I'm not sure that's expected semantics
this is the same entry, it shouldn't be written elsewhere
lime oyster
This is implicit vs explicit again.
What if the noop causes the get to never get called?
And therefore the callback never to get triggered.
smoky steppe
that's ok, the proof shouldn't be invalidated
lime oyster
If that's the case, then implicit will work fine in the noop case, no matter how it is implemented.
It's not clear to me from the doc that's how it works though.
smoky steppe
I'm not sure yet how this should work either
I left my suggestion in the doc
basically no matter the actual implementation, I think the proof should only be invalidated when the contract has acknowledged it and that fact has been externalized
copper jetty
I'm a fan of Dima's proposal for required merge callbacks on every call to env.storage().recreatable().get(). A little context on how restoration proofs actually work that was ommited. Whenever someone wants to restore an entry, they submit a RequestForRestoration that gets recorded on ledger. An ESS node then sees the request and submits the proof, which includes the entry. This proof gets recorded on chain. You can think of this as a wrapped ledger entry, the entire entry is there but not useable as a ledger entry. With Dima's proposed call back, every call to get would check to see if a wrapped ledger entry (I.e. the proof) exists. If it does, the ledger entry is unwrapped and given as an input to the callback. The callback can then merge the entry according to contract implementation. Note that this doesn't necessarily require a write, a correct implementation may just take the newer version and ignore the older version. Perhaps instead of calling this merging, we should call it the conflict resolution callback. After the proof is used (or unwrapped), the entry is deleted permanently from the ESS
I think I prefer the required call back over my pop function. I was envisioning the pop function being used inside a dedicated "merge" function that contracts define. This is a footgun though since contracts may just not implement this or call the pop function anywhere, making the entry unrecoverable. By requiring the callback, merges happen automatically as part of a normal function call and don't require dedicated merge functions
An example: say a user has a balance with 5 USDC in the Bucket list and a balance with 10 USDC in the ESS. The USDC contract correctly sums balances in there conflict resolution callback. All the user has to do to trigger the merge is submit a request for restoration for the balance in the ESS. Then any contract call that accesses the entry will trigger the call back, resulting in a single entry with 15 USDC. I like this a lot since merging appears to be pretty automatic from the invoker perspective and we require good design behavior on the side of contacts
lime oyster
Note that this doesn't necessarily require a write
smoky steppe
they submit a RequestForRestoration that gets recorded on ledger
why does this have to be stored in ledger though? it feels like you're trying to use ledger as communication channel between core and ESS, which is kind of counter-productive compared to something like an RPC
I've also left a big comment about this in the doc
lime oyster
Conveniently it decouples the steps, so that ESS and contracts aren't more tightly coupled.
smoky steppe
the steps have to be decoupled in any case
lime oyster
we should call it the conflict resolution callback
After the proof is used (or unwrapped), the entry is deleted permanently from the ESS
@copper jetty How does the network know it is safe to do that? A contract might, using the callback, read an entry (merged in the process), but not write it because in that moment it doesn't need to mutate the data. As a result, it might not be safe to delete the entry from the bucketlist at least.
lime oyster
> Note that this doesn't necessarily require a write
<:plus1:1007758932212711494...
We might need to always write I think to make deletion safe?
smoky steppe
the current flow (in my understanding) is
I propose to replace the parts where the requests and proofs are being sent on-chain with just RPCs between dapps and ESS instance(s). there still will be some lag of course, but at least the data flow is much more straightforward because instead of waiting for ledger to externalize certain value you just wait for RPCs between services
A contract might, using the callback, read an entry (merged in the process), but not write it because in that moment it doesn't need to mutate the data
in my undrestanding the callback would be vec[LedgerEntry]->LedgerEntry fn and host will take care of the write
copper jetty
> After the proof is used (or unwrapped), the entry is deleted permanently from ...
Yeah, once an entry is unwrapped it's considered restored and is deleted. If the contract doesn't do the correct thing with the unwrapped entry, that's just a contract bug that we can't really avoid. I don't think I understand what you mean by "in that moment it doesn't need to mutate data." Under what circumstances would the call back not mutate data? If the contract is using recreatable entries and a restoration comes through, it's the contracts responsibility to manage the restoration via the callback. Can you provide an example as to when a merge would occur but the contract wouldn't mutate data for a valid reason? Again, a valid conflict resolution callback does not necessarily require a write, so that's not a valid protocol level check to see if it's "safe" to delete the entry from ESS
smoky steppe
Again, a valid conflict resolution callback does not necessarily require a write,
I don't agree. the write is always implied, even if nothing changes in the value. I don't think it's a good idea to overload the callback with functionality. it has very well-defined purpose and semantics.
lime oyster
If the contract doesn't do the correct thing with the unwrapped entry, that's just a contract bug that we can't really avoid.
In principle maybe, not in practice. If the callback itself doesn't necessitate a write, there's nothing inherently in the nature of implementing the callback that tells a developer they need to also write the merged data. If it's just in documentation, it'll be a footgun.
If the result of the callback is written, and that's something the framework / sdk / environment does, great, I think that plugs that hole.
If we're expecting developers to know to always call set after calling get (which has a callback that may or may not be called), then footgun.
It might be helpful if we talk about a concrete interface for the callback.
smoky steppe
I think writing the callback result unconditionally is a really simple approach to this issue. I'm really not sure about the restoration flow though. good thing is that probably this doesn't need to get into v1 release, so we have some time to think more about it
copper jetty
Yeah we just need to figure out the callback UI right now. How about the call back takes two entries as input, the BucketList version and the ESS version. It then must return a single value which is automatically written
Idk what the types should be though or how this will actually look like at the SDK level, but intuitively I think this makes sense
smoky steppe
yeah, this sounds reasonable. the type would be RawVal and then it can be specialized on the SDK side to any value type the user needs.
copper jetty
I think the proof interface is still up for debate, but I think we're in agreement that we won't expose any more of the request interface to contracts so we should be good to continue with this and be feature complete for mainet launch wrt state expiration
smoky steppe
yeah, requiring merge callback would be good enough. it should even be testable even without the real expiration flow
copper jetty
If someone wants to formalize this and replace the pop host function of the semantics doc with this that would be great. SDF should be able to change write permissions
smoky steppe
yeah, sure
smoky steppe
one other consideration would be to completely decouple entry restoration from the contract execution. ESS could do the whole restoration process (so it won't even be necessary to store the proofs on-chain in the first place). this would just work fine for the unique entries. for the recreatable entries we could introduce a reserved function that would perform the merge. the only difference would be that besides the 2 values it would accept the key. the downside is that it is a bit easier to forget adding it (we could do some basic off-chain verification though), but the upside is that there is no bundling of entry restoration and regular transactions
lime oyster
yeah, this sounds reasonable. the type would be `RawVal` and then it can be spec...
I'd make it type V to match the V in the get function. It's good practice I think to require / assume that these types maintain type consistency. I think for the most part contracts will be type consistent anyway.
smoky steppe
from the host side it would be RawVal though. SDK can impose additional type checks
updated the doc. I'll think more about this though
lime oyster
The callback would return a RawVal that replaces the ledger entry value?
smoky steppe
yeah
lime oyster
Moving a conversation here that was happening offline. We should discuss the names we use for different concepts in this thread. Many of the terms we've used in that doc at the top of this message have required explanation, or created confusion. The below vocabulary is an alternative way we could describe the same concepts, that came out of discussion between @craggy maple @sweet locust @green coral @smoky steppe @lime oyster .
The following terms have been suggested:
Temporary Storage
Exclusive Storage
Mergeable Storage
Suspension Ledger
Suspended Entry
Evicted Store
Evicted Entry
To refer to the following concepts and behaviors:
All entries suspend at some point, their Suspension Ledger.
Entries are either Temporary, Exclusive, or Mergeable.
After the Suspension Ledger of a Temporary entry is reached, it is deleted from the ledger. It is never recoverable.
To create a Temporary entry, it can simply be written to the ledger. The only place Temporary entries live is in the ledger.
After the Suspension Ledger of an Exclusive or Mergeable entry is reached, it is marked as Suspended, is no longer available to contracts, and after sometime removed from the ledger and added to the Evicted Store.
Exclusive entries can exist in the ledger, or in the Evicted Store, but not at the same time.
To create an Exclusive entry the client must provide proof the entry is not in the Evicted Store.
To recover an Exclusive entry from the Evicted Store to the ledger, a client can do so with a proof. The contract does not need to be involved.
Mergeable entries can exist in the ledger and in Evicted Storage at the same time, and multiple versions can exist in Evicted Storage.
To create a Mergeable entry, it can simply be written to the ledger.
To recover a Mergeable entry from Evicted Store to the ledger, a client can do so with a proof, and the contract must be prepared to merge the entry with the entry already in the ledger as part of its read operation of the entry.
None of this is decided. This is a discussion in progress, and just a recent point in the discussion.
What do folks think?
distant dew
Mergeable entry feels slightly more confusing than Recoverable Entry. It seems like a common case where a Mergeable Entry will get moved to the evicted store, and a user will want to bring it back to the ledger.
This isn't really a merge, and is more a recovery. It's clear enough to state that contracts will handle recovery logic if the entry was recreated on the ledger after eviction.
No hard feelings either way, tho. I think this is clear enough!
Also think just calling a Mergeable Entry an Entry would be appropriate, since this one feels somewhat hard to name as its the "base case".
smoky steppe
hmm, I don't actually think your understanding is correct here, which kind of points that we still have some work to do on clarity. there are two kinds of entries that can be recovered (exclusive and mergeable). mergeable entries might exist on-chain and in archive at the same time (and also be merged when needed, e.g. to recover the full token balance), while exclusive entries are guaranteed to only exist in one place (either ledger or archive) and hence they don't need to be merged. exclusive entries are actually semantically almost the same as the 'regular' non-expirable entries we have now
distant dew
That is how I understood it. For me, it's easier to think of mergeable as the base case, since exclusive is a mergable entry they can only exist in one place at a time, and a temporary entry is a mergeable entry that can't be sent to the evicted store and is deleted on eviction.
This is likely where the confusion arose?
smoky steppe
I think 'mergeable' case is the most exotic one and is probably mostly useful for entities like balances. in order to use it you'll need to specify the merge callback as well. so it would be weird for me to make it the base case
sweet locust
Yeah. We initially discussed "recoverable" and "unique recoverable" as names, but anytime someone asks a question about a "recoverable entry" the next question has to be "do you mean recoverable or unique recoverable?". Which is needlessly confusing. So the goal was to have a distinct name for each scenario where they couldn't accidentally be confused by someone new, who may not be familiar with every case.
smoky steppe
if we go that way I'd say unique/exclusive should become the default 'recoverable' storage. unless you are doing tokens, you should probably be choosing between unique recoverable (e.g. to store your admin entry) and temp storage
lime oyster
In the design discussion today there appears to be consensus for:
Temporary Storage
Exclusive Storage
Mergeable Storage
Expiration Ledger
Expired Entry
Evicted Store
Evicted Entry
Ref of doc we used for brainstorming: https://docs.google.com/document/d/1pWNT4IFx3-C8pQMcDMxQGX7-PyQbac-1AFLisCZM1vM
sweet locust
At risk of this thread becoming even longer, how should a user discover whether they need to restore or extend an entry?
Expired Option 1 (Pessimistic): simulateTransaction could deny access to expired entries and maybe the contract call would fail (or maybe not). The response could include those entries the contract tried to access but was denied.
Expired Option 2 (Optimistic): simulateTransaction could allow access to expired entries, and the contract call would succeed. The response could include those entries the user needs to extend before sending the transaction.
Evicted: It's probably impossible for Soroban-RPC to know about any evicted entries. So... The contract would need to signal that to the client in some out-of-band way, I guess? 🤷♂️ Maybe contracts could return a special error-code, like return Err(Error::EvictedEntry(key)) or something? Although, how would the contract know if the entry was evicted or never existed before?
