#State Expiration Storage Keys

293 messages · Page 1 of 1 (latest)

molten pendant
#

To support state expiration, there are now three different types of ContractData unique, recreateable, and temporary. Temporary entries are permanently deleted when they expire, while unique and recreateable entries are sent to the deep state store on expiration. Recreateable entries may have many different version of the same key in the state store, while unique entries are guaranteed to only have a single version that exists between the ledger and the deep state store.

We need to decide how to treat keys for these different entry types. Initially, the most secure option was for each entry type to have a seperate key space. This means that for the key 0x1, UNIQUE(0x1), TEMPORARY(0x1), and RECREATEABLE(0x1) are all valid, completely independent entries that can exist at the same time. The reason for this is to prevent a TEMPORARY or RECREATABLE entry from "shadowing" a UNIQUE entry. If there was a shared key for all entry types, a buggy contract may create the entry UNIQUE(0x1). After some time, this entry expires and is sent to the deep state store. According to the security guarantees of unique entries, it should be impossible to create a new entry with the same key and "shadow" the entry in the deep state store. However, if we do not enforce separate key spaces, a buggy contract may accidentally shadow the entry by creating TEMPORARY(0x1). Because TEMPORARY entries have no strong security guarantees, they do not check the deep state store before creating the entry, allowing the UNIQUE entry to be shadowed and violate its uniqueness property. The initial intention of having separate key spaces is to prevent such bugs and ensure that UNIQUE entries are always unique even if the contract has this class of bug.

#

However, this exposes a weird interface where both the key and entry type must be provided for every get/set operation. Instead of using a flag parameter, the SDK could abstract this by exposing a separate class for each storage type, i.e. TempStorage::set, UniqueStorage::set, etc. Additionally, this allows for another class of bug, where the contract developer thinks that the key 0x1 only refers to a single entry when it actuality it can refer to two totally separate entries UNIQUE(0x1) and TEMPORARY(0x1) .

So the question is, do we have a clunkier interface with separate key spaces for each type and guarentee that UNIQUE entries are always truly unique? Or do we provide a unified key space to simplify the UX, but allow buggy contracts to violate UNIQUE entry properties if used incorrectly? @glossy garnet @velvet nova @verbal cloud

glossy garnet
#

just to provide an example for the context:
doing set(nonce_key, UNIQUE, 1) in one contract call, then set(nonce_key, TEMP, 0) and then get(nonce_key, TEMP) would lead to pretty much same (bad) consequences for independently of whether key has storage class or not. the fundamental issue is the ability to store the same key with different storage types

velvet nova
#

I think we'd need to avoid having the type of storage as a parameter like ☝🏻, as it looks like an attribute, rather than part of the identity.

#

Today we have:

env.storage().set(key, val)
#

Moving the type outside the parameters would help I think:

env.storage().unique().set(key, val)
env.storage().temp().set(key, val)
env.storage().recreatable().set(key, val)
glossy garnet
#

it's basically equivalent to what I had

#

I think if the key contains the storage type, it should be explicitly a part of the key (and vice versa)

velvet nova
#

LedgerKey is an implementation detail. It might be helpful for folks to think of different types of storage as different locations.

glossy garnet
#

given

   Foo,
   Bar
}```
there is nothing preventing you from doing 
`env.storage().unique().set(DataKey::Foo, 0); env.storage().temp().get(DataKey::Foo)` etc.
#

while LedgerKey is an implementation detail, there is a very clear 1:1 mapping from storage keys to ledger keys (just add current contract id and you're done). now things become tricky

#

I would prefer either having the storage type annotation on the key itself (and have storage type in ledger key) or do runtime check for the storage type mismatches (and not have storage type in ledger key).

verbal cloud
molten pendant
#

I don't know if there's a good way to annote the key itself give that we allow users to use arbitrary SCval as keys. We would need to specify a key type such as
struct Key
{
SCval ID;
Type type;
}

And then modify the SDK to take a key of this type. Unfortunately any type of runtime check will be nondeterministic based on what entries have expired, since you would only be able to detect a type mismatch if an entry has not yet expired. We may be able to do something a little more sophisticated in testing. For example, tests could log individual key type pairs on access and throw an error if there's an inconsistentcy in a given test. Still, this is dynamic analysis so it requires good tests to catch errors and could never catch all bugs of this class.

quaint briar
#

@molten pendant could you please clarify that for me ? To support state expiration, there are now three different types of ContractData unique, recreateable, and temporary. Temporary entries are permanently deleted when they expire, while unique and recreateable entries are sent to the deep state store on expiration. Recreateable entries may have many different version of the same key in the state store, while unique entries are guaranteed to only have a single version that exists between the ledger and the deep state store. . So, I understand that temporary entries are just being deleted from the node. That's very straight forward. What's about Unique and Recreatable ? do they get deleted as well ?

molten pendant
#

Yeah, so everything gets deleted from the BucketList on expiration. The different types refer to the entry's deep state store (formerly known as archive) behavior. Temporary entries don't get sent to the deep state store, but UNIQUE and RECREATEABLE entries do. The difference between UNIQUE and RECREATEABLE entries is their security guarantees. UNQUE entries are guaranteed to always be unique, which means that a single key can either exist in the BucketList or in a single place in the deep state store, but never both. RECREATEABLE entries do not have this guarantee, so there could be an arbitrary number of versions of the key in the deep state store and up to one version in the BucketList at any given time. To enforce the uniqueness property, we require a "proof of nonexistence" in order to create UNIQUE entries. This is expensive, so it is only required for UNIQUE entries. The concern is that if we don't have separate key spaces, the same key that was initially used to create a UNIQUE entry could be later on be used again for a RESTORABLE or TEMPORARY entry and bypass this uniqueness safety check. If we have separate key spaces, this is not possible.

proper robin
#

@molten pendant I see from the initial doc that state expiration covers not only contract data, but WASM blobs and contract instances. Let's say a defi project team wants to deploy a defi primitive protocol (say a lending contract) for the benefit of the ecosystem. After the initial contract deployment: does rent still need to be paid over time to keep this service available for the community? Can anyone pay the rent or does it have to be someone with keys to the contract account?
I'm thinking of a situation where there's a valuable protocol that people start building on it (from other invoking contracts to UI), the team eventually moves on and over time and people forget about it (but still use it) and suddenly one day, rent is exhausted and things break. Is this an expected scenario (that as the ecosystem matures, we expect there to be instrumentation to manage)?

molten pendant
#

Anyone can pay rent. Additionally, entries receive a small rent bump on access. This design is specifically for the example you gave, where the initial team that deployed the protocol would not be responsible for on going rent payments. Rather, whoever uses the primitive would pay a small fee towards rent on every invocation of that smart contract, and would also pay for a small bump on any entries that the given smart contract call accesses. In this way rent is decentralized with no central authority being responsible for keeping entries live on the ledger.

glossy garnet
#

e would need to specify a key type such as
struct Key
{
SCval ID;
Type type;
}

but that's what happens in core, right? of course we could say that we have three different separate 'storages'... but that seems really footgun-y to me. to be clear, the annotation could live in SDK and just ensure that the proper keys get used with the proper storage types. I just don't know if it's actually possible to implement

molten pendant
#

The issue is it's impossible to enforce that "the proper keys get used with the proper storage types" because the same key with a different storage type. may exist in the archive. We would need to segment the keys statically somehow. Including the flag in the key is the most general segmentation strategy. I don't think there's a way to do this safely without exposing this to the user at some level, so I guess the question is whether we go the "safer" collision free route or allow storage type key collisions with a simpler ux

#

In either case this is only an issue in buggy contracts, and the contract probably won't function properly if either class of bug is present, so maybe I'm over designing too much here.

glossy garnet
#

yeah... it just seems to me really weird to have different versions of the same key, semantically lifetime seems to me like a metadata attribute. I'm not too concerned about the existence of the key in the archive (I might be wrong though).

molten pendant
#

We can't have a runtime check work 100% of the time because of the archive, but we can still have a runtime check for type mismatches if the key exists in the bucket list. This "best guess" runtime check would probably be good enough to root out this kind of bug at the until testing stage anyway

glossy garnet
#

to be clear, what I mean by the 'annotations' is sdk-only feature. it would just ensure that the key is used with a proper lifetime and make it hard to reuse the same key with different storage types

#

of course if contract is updated and starts using a different storage tier then things might break. or if you're not using the contracttype keys at all. but that's ok-ish

molten pendant
#

So are you suggesting keeping the separate keyspaces on the core side or getting rid of the key space separation in core and exposing these annotations via SDK?

glossy garnet
#

I'm inclined a bit more on the side of not having storage type in the ledger key (and hence allow contracts to shadow archived unique keys). then we maybe could limit the entry type annotation to only times when the keys are created (thus limiting the ways to mess up in the contract code). but I'm not feeling super strongly on this one.
on a side note, it seems like this is yet another benefit of decoupling entry modification from creation

molten pendant
#

Yeah, with the current set interface every modification would need to include the type annotation.

glossy garnet
#

any thoughts on this, @velvet nova ?

molten pendant
#

From a state expiration perspective, the best solution seems to be separating create and modification, requiring type anotation only during creation, and having a single key space for all types core side

#

If we can't separate create and set for other reasons, I think option 2 should be anotation on every storage call

glossy garnet
#

yeah, that's what I would prefer as well. the only footgun that remains here is for the case when there are multiple creation code paths that use different types and even for that the chance of actually shadowing the archived value is pretty low (vs a chance of noticing the key collision on creation)

velvet nova
#

I think we should go with the route that has the same key provided at create and set, where set can create or mutate. Code won't always know if it needs to create or mutate. So we'll end up with cases where folks need to check first.

#

Today we have:

env.storage().set(key, val)
#

Something like this feels reasonable:

env.storage().unique().set(key, val)
env.storage().temp().set(key, val)
env.storage().recreatable().set(key, val)
#

In the XDR side of ☝🏻 having the LedgerKey include the type+scvalkey seems fine?

molten pendant
#

Yeah I think from a UX perspective to is seems reasonable that env.storage().unique() is completely separate from env.storage().temp()

velvet nova
#

I agree. I understand there may be some confusion that you can store the same scval key in two places, but this is not too different to storing two files with the same name in different folders.

#

I think as long as we never surface storage without the qualifiers in APIs, should be okay?

#

The other place will be the soroban-rpc API, but again, as long as we include context should be okay.

#

The nice thing about that too is that when someone is reading, they know what type of storage they are reading from. Which might be important for the reader to know.

#

A reader reading from unique/recreatable storage that gets an error may have a different course of action to reading from temp storage.

molten pendant
#

Yeah, this makes archive preflight behavior easier as well. I.e., you know that a temp key will never be archived, while a unique key might

velvet nova
#

I'm not sure if this fits into this thread specifically, but I'm more concerned with how a developer will determine if they need to use unique vs recreatable. The decision tree for that feels not straight forward. In fact, @molten pendant if you have a decision tree for that, I'd love to see it 😄

molten pendant
#

Yeah I'm working on a semantics doc and barebones "user guide" today that gives everything an official name and talks about when each storage type should be used

glossy garnet
#

especially for temp storage

#

(for unique/recreatable IIUC we should have some reasonable default)

molten pendant
#

All lifetime extensions, including initial lifetimes, are handled via the footprint

glossy garnet
glossy garnet
molten pendant
#

The contract can define hints though, so the full interface would look like env.storage().unique().set(key, val, numLedgersHint)

glossy garnet
#

right, the hint... but then in case of modification it does nothing, right?

#

also, I'm feeling a bit lost in all the discussions, but for the temp storage it seems necessary to be able to provide the exact minimum lifetime (not just a hint). otherwise it's not possible to implement temp storage nonces

molten pendant
#

Right now the hint is just extending the lifetime by the bump amount and works the same for creation and modification, but I see where this would be problematic. Like when creating for the first time, it would be reasonable to allocate 6 months up front then do small incremental bumps, but that's tricky if you only have set. What we could do is have the hint define a suggested minimum and the actual bumpAmount is the difference between the current lifetime and the recommended lifetime. I think this makes a lot more sense actually than a raw bump ammount

#

Can you flesh out your example? If the user doesn't follow the recommended hint and the entry gets deleted early, I assume the contract just won't work. Like in your nonce example, if the invoker doesn't follow the hint and the nonce gets deleted, then whatever the invoker was attempting to do with the nonce just wouldn't work right?

velvet nova
glossy garnet
glossy garnet
molten pendant
#

I think if a nonce being deleted can lead to signature replay, it shouldn't be a temporary entry to begin with though? I don't see how a minimum bound on the entry lifetime would address the issue unless I'm misunderstanding your example

glossy garnet
#

it would disallow the replay 🙂

#

here is how temp storage nonces would work: when you sign anything you specify the nonce (some unique random value; probably just a random u64 is good enough) and the signature expiration (say, one week). when this gets executed, we will store the nonce in a key (address, nonce) in a temp entry that lives at least until the signature expires

molten pendant
#

Got it, so the nonce needs a guarantee that it survives at least as long as the signature can

glossy garnet
#

I think we've discussed that it's important to provide minimum lifetime guarantees for the temp entries, otherwise they're pretty useless

#

that's not true for other 2 entry types as they don't disappear forever

molten pendant
#

So what do we do from a UX perspective? Do we enforce the "hint" for temp entries but allow it to be overridden for restorable entry types? And should we have any bounds to prevent greifing attacks where you make a temp entry with a very long lifetime that the invoker must pay?

#

I think this will also be difficult with only a set function. During preflight, we can check the current rent balance and conditionally bump rent if it falls bellow some threshold, so env.storage().storageType().set(key, val, minLifetime) works for restorable types. However, we can't do conditional bumps during tx application for parallelism issues, so how can a temporary entry define the minimum lifetime without an explicit create function?

glossy garnet
#

right, that's another argument for having a separate function for creating new entries

#

another argument is ability to have more efficient footprints

#

but then there is a different point of confusion, which is delete-create vs modify (they should ideally provide the same possiblities in terms of lifetime extension)

molten pendant
#

Actually we might be able to do this on the backend with set. Essentially, the "hint" is the minimum lifetime. For restorable entry types, this is always ignored on TX apply. For temp entries, this is ignored except when the entry is created for the first time.

glossy garnet
#

yeah, that's an option

molten pendant
#

It does seem foot-gunny though, especially since the exact same function call would sometimes enforce the minimum lifetime but not always. Thoughts on revisiting create @velvet nova?

#

The way we could communicate it is that the "hint" provides a minimum lifetime whenever the entry is accessed. This value can be overridden by the invoker via the footprint except for temporary entry creation. It may be worthwhile to expand this to any creation event. On modification, the lifetime can always be overridden by the invoker

#

One possibility is that Leigh mentioned code paths where it is unclear if the entry is being created for the first time. I think in cases where the minimum lifetime guarantee is important, such as nonces, you generally know that the entry is being created for the first time. Maybe a compromise could be that set can still create or modify an entry with set but the lifetimeExtension hint is never enforced and can be overwritten. We then have a special function only for temp entries called createWithLifetime that does enforce the minimum lifetime and panics if the entry already exists. That way we still have the more generic set interface more most cases, but when lower bound lifetimes are important you can explicitly use the createWithLifetime function

#

It may be too complicated though, having two ways to create storage entries

velvet nova
#

I don't think I understand the problem completely. Can you assume I know nothing: Why does the lifetime parameter need to be interpreted differently in create vs mutate, and for the different storage types?

molten pendant
#

The issue here is specifying a minimum lifetime. For recreateable entries that can be restored after they expire, minimum lifetimes don't matter since the entry can be arbitrarily restored. For temporary entries, this is not the case. In Dima's example, a temporary entry may be used as a nonce to prevent a replay attack. Say there is a singature that is valid for one week, the temp entry nonce must live at least once week to effectively prevent a replay

glossy garnet
#

in terms of create vs mutate, maybe it doesn't need to be interpreted differently. in terms of different storage types, there needs to be some way of ensuring minimal temp entry runtime (and conversely this is probably not needed or even harmful for restorable netries)

velvet nova
#

Temp or recreatable, if the lifetime isn't at least minimum, if there isn't something a developer can have a concrete guarantee about, feels like that'll create confusion.

molten pendant
#

Yeah the create vs mutate is an implementation concern. On creation, it's possible to enforce a minimum lifetime because there is no read/write contention on the entry since it doesn't exist yet. However on modification, we don't want to allow enforcing a minimum lifetime because this leads to read/write dependencies. The worst case scenario we're trying to avoid is a situation where txs that should be parallel aren't because of conditional lifetime extensions. If you imagine something like the ERC-20 token contract, the contract is used by many different TXs in basically every block. If every TX checked the lifetime of the contract instance to conditionally bump it, every token transfer would have a read/write dependency, even if they were transferring completely different tokens.

velvet nova
#

E.g. if I use recreatable, but then make some assumption in my overall system that it won't need to recreated until N time, and then it expires in N-1, that would be surprising.

glossy garnet
#

you shouldn't make any assumptions on recreatable storage

#

the entry restoration is external to the contract

molten pendant
#

You can actually make assumptions on restorable types, it'll just get taken care of in preflight. If your contract relies on the existence of a recreateable or unique entry which is in the archive, it's fine if the contract call fails. The idea is that it will fail in preflight, which will then give you a list of keys you need to go and restore

#

The reason why temp entries need the minimum is that there's no restore path and the data is permanently gone

glossy garnet
#

what I mean is that semantically non-temp entries are always 'in ledger' (even though you sometimes need to jump through the hoops to get them back), while temp entries semantically have limited lifetime and it makes sense to specify this lifetime.

molten pendant
#

We could just say that footprints can't override temp entry lifetime extensions, only recreatable and unique entries. We can do conditional lifetime extensions on temp entries, they just need to be in the read/write set. I think this should be fine, since I doubt temp entries would be bumped very often so we wouldn't have the data dependency issues like I was talking about

#

Same interface except the hint isn't a hint for temp entries, but still acts as a hint for recreatable and unique

#

This would allow us to keep set the way it is as well

#

The hint should also be an optional value in case you don't want to provide a hint/lifetime extension check

glossy garnet
#

but for temp it probably should be non-optional...

#

otherwise this sounds reasonable, albeit confusing

molten pendant
#

I think it should still be optional. Say you want an entry to last at least 128 ledgers from it's creation time. On creation, you'd want to call set(val, key, 128). On follow up calls, you wouldn't want to bump it again so you just leave it null

glossy garnet
#

true... but then in order to set the lifetime once I'd need my code to distinguish create vs modify cases

molten pendant
#

Yeah, but I still think it should be optional in case users are able to distinguish cases, or I guess they could also just set minimumLifetime to 0 but same thing essentially

#

In conclusion, we have

env.storage().type().get(key, Optional<minimumLifetimeHint>)
env.storage().type().set(key, val, Optional<minimumLifetimeHint>)

where the footprint can override the minimumLifetimeHint for UNIQUE and RESTORABLE entry types, but not for TEMPORARY

glossy garnet
#

I'm still a bit concerned about using the default value for the temp entries. maybe we could just fail when creating a temp entry without explicit lifetime..

velvet nova
#

If the lifetime was only specified in the footprint, the contract would be simpler? What's the motivation for having both the footprint and contract share lifetime ownership?

glossy garnet
#

because these are different 'lifetimes'

#

restorable entries have 'rent lifetime', i.e. how much time they will stay in the ledger before archival (not removal). semantically these are persistent entries and 'rent lifetime' is just a function of the rent payments. contracts can't really know what their users want to do with the rent and hence it makes sense to have ownership outside of the contract. from the contract standpoint these are really just persistent entries most of the time.
temp entries have just 'lifetime', after which they're gone. their use cases are very different - stuff likes nonces, allowances, oracles etc. where the contract writer has a very specific idea on what the minimum lifetime should be and users probably rarely need to bump or customize it

#

so the ownership doesn't need to be necessarily shared (besides maybe the hint), but it makes sense to have different owners for different storage types

velvet nova
#

Then maybe the API should be:

env.storage().unique().set(key, val)
env.storage().unique().get(key)
env.storage().recreatable().set(key, val)
env.storage().recreatable().get(key)
env.storage().temp().set(key, val, lifetime)
env.storage().temp().get(key, lifetime)
#

Actually, here's another idea:

# 
env.storage().temp().set(key, val)
env.storage().temp().setLifetime(key, lifetime)
env.storage().temp().get(key)
#

The lifetime can be required for temporary value, and it can have a reasonable default, and then if contracts need to use something different they can call an explicit function.

molten pendant
#

I think we should still have

velvet nova
#

We may benefit from having different terms for these two types of lifetimes.

#

Temp lifetimes vs recreatable/unique archivetimes?

glossy garnet
#

yeah, that would probably be beneficial

molten pendant
#

Yeah I agree

#

I think we still need env.storage().unique().set(key, val, archiveTime) though, since we still want the contract to be able to suggest lifetime bumps during preflight

glossy garnet
#

but there also was some demand for the contract-specific hints for recreatabl entries... IIRC it would allow contract to hint that e.g. during a token transfer it makes sense to bump sender balance, but not receiver balance

velvet nova
glossy garnet
#

probably?

#

that might actually make things cleaner

velvet nova
#

Setting an entry, creating an entry, bumping its lifetime, these are all independent actions (except for the specialty create which is a set and lifetime set).

#

I think it'd make the API easier for me to understand, speaking subjectively.

glossy garnet
#

agree

molten pendant
#

So we would have the following?

env.storage().unique().set(key, val)
env.storage().recreatable().set(key, val)
env.storage().temp().set(key, val)
env.storage().temp().setLifetime(key, lifetime)
env.storage().unique().suggestArchiveetime(key, archiveTime)
env.storage().recreateable().suggestArchivetime(key, archiveTime)
velvet nova
# 
env.storage().unique().set(key, val)
env.storage().unique().setArchiveTime(key, expiry)
env.storage().unique().get(key)
env.storage().recreatable().set(key, val)
env.storage().recreatable().setArchiveTime(key, expiry)
env.storage().recreatable().get(key)
env.storage().temp().set(key, val)
env.storage().temp().setLifetime(key, lifetime)
env.storage().temp().get(key)
glossy garnet
#

setArchiveTime -> hintArchiveTime?

velvet nova
#

Is a temp's lifetime guaranteed exactly?

#

My understanding is no.

molten pendant
#

Yes

velvet nova
#

Oh ok.

#

It can be an exact ledger and then it is inaccessible after that ledger? Even if that ledger is outside a checkpoint boundary, etc?

molten pendant
#

Well not exactly let me rephrase. It is guarenteed exactly, but anyone can extend it

#

Correct

glossy garnet
#

minimum lifetime is guarnateed though, right?

velvet nova
#

If it's min lifetime, then I think set vs hint is less important for the archive case.

#

Since they're all fuzzy in different ways.

glossy garnet
#

well, I'm actually thinking that set is a bit confusing, because set implies a time point, while what you in fact do is 'increase by N ledgers'

#

but that's not too important - I think the general approach looks reasonable

molten pendant
#

I think temp entry should be increaseLifetime, other types should be hintArchiveTime since the footprint can override the hint

velvet nova
#

increaseLifetime for temp 👍🏻

molten pendant
#

A follow up question: should we allow arbitrary temp entry lifetime bumps? Currently the plan is for all entries to be bumpable manually via an op where you list the keys to bump, but if contracts are directly managing temp entry lifetimes we might not want this

#

Especially if temp entries are used in a security context, maybe all temp bumps should have to be initiated by contract code?

glossy garnet
#

hmm, increase is confusing in a different way: e.g. consider you create an entry with 'default' lifetime of 16 ledgers and then you call increaseLifetime(64). is the lifetime 64 or 80 now?

velvet nova
#

hintArchiveTime feels... ambiguous to behavior. The contract developer will hint to what the archive rent should be, but then the client can.. modify it? 🤔

molten pendant
#

Yeah that's the idea

velvet nova
#

Is that client flexibility important?

glossy garnet
velvet nova
#

I imagine most clients will just execute on the hint value without modifying it.

molten pendant
glossy garnet
#

well, imagine token balance. wallet might opt to bump rent when it reaches certain threshold

velvet nova
glossy garnet
#

the contract hint seems more useful as a boolean flag TBH...

glossy garnet
# molten pendant It would be 80

this is kind of a weird interface, isn't it? maybe SDK could introduce a helper like 'create_with_lifetime` that does the correct math...

glossy garnet
molten pendant
glossy garnet
#

so I wouldn't consider that a requirement

velvet nova
# velvet nova Is that client flexibility important?

I think it's going to be difficult to do the right thing with archive times if they are just suggestions from the contract side. Contract dev specifies one value, but client can set it lower, higher, as a contract dev I feel like what's the point in me setting it.

glossy garnet
molten pendant
#

so we have

env.storage().unique().set(key, val)
env.storage().unique().get(key)
env.storage().recreatable().set(key, val)
env.storage().recreatable().get(key)
env.storage().temp().set(key, val)
env.storage().temp().setLifetimeToAtLeast(key, lifetime)
env.storage().temp().get(key)
#

Yeah creation is fine because that's a data dependency already. I just don't want to introduce a bunch of dependencies where you modify a preexisting entry, but the only thing you change is the expiration

#

So by "not common", I meant extending lifetime of a temp entry after the initial creation lifetime is set

glossy garnet
glossy garnet
velvet nova
#

If there are core capabilities of tokens that wallets are expected to interact with, they should probably be functions that get called?

#

In which case, contracts can set the archive times in functions.

glossy garnet
#

yes, that's what I suggested then - contracts can provide functions like get_entries_for_rent_bump(address: Address) that can be just called off-chain

velvet nova
#

Or maybe that's an antipattern.

#

Extending rent of things that can be restored, for the sake of extending them... feels like someone is treating Stellar as a DB.

glossy garnet
#

(this way you don't need to expose the archive lifetime logic to contracts at all)

#

the issue is that there is no reasonable default

velvet nova
#

Is it correct that the restore process has to be accessible by any user?

glossy garnet
#

like if you use your wallet, say, once per two weeks and the default bump is just 1 week. is it right that you'd need to restore your balance every time you use the wallet?

velvet nova
#

Would there be a downside to restoring?

glossy garnet
#

yeah, IIUC you should be able to get the restorable entries from the preflight

#

it would probably come with some additional fees + some additional latency

velvet nova
molten pendant
#

Yeah, so restoration is a somewhat expensive process, which gets more expensive the longer something has been in the archive

#

The biggest cost is latency, where it can take several blocks to get your entry back from the archive. You submit an unarchival request in block 0, someone needs to produce the proof (which is quick for entries recently archive) and submit it, so the absolute fastest you could access an archived entry is 3 blocks

velvet nova
#

How will that expense map as a function / compared to the cost of rent on chain? Will restoring be less expensive than keeping something on ledger for the same period of time?

molten pendant
#

The two aren't really correlated. Currently the plan is to use Bitcoins reward model where the request for unarchival also has some reward in XLM. The first node to provide the proof gets the reward, so the fee is expected to follow market demand

#

The rent fee is a network config based on the current BucketList size so there's no direct correlation between rent costs and restoration costs

#

Generally speaking though, something used every 2 weeks should not have to be unarchived

#

Generally speaking, think that restoring something takes 30 minutes. We don't know what actual numbers will look like but that's what I've been using for my mental model

#

Maybe the bool flag approach is best, then the footprint just has a single "bumpAmmount" applied to all flagged values

glossy garnet
#

to take this point ad absurdum if restoration was super seamless, there wouldn't be a need for ledger at all (everything could be immediately archived)

molten pendant
#

Also the proof process is expected to be computationally expensive, think zk primitives, so we don't want to encourage a lot of restores

glossy garnet
#

I do think though that even with the bool flags manual bumps via a separate operation might still be needed. e.g. consider an account that receives and accumulates a reasonable amount of incoming transfers. it will only be autobumped and if the frequency of payments is lower than the autobump amount, then it will still need to do a manual bump (or a transfer...)

velvet nova
#

Is the idea with the boolean flag, that contracts call things like env.storage().unique().increaseTimeToArchive(key, N), and then a client decides if they'll pay for it or not. If they don't, they set false, and the call to increase archive time is a noop?

glossy garnet
#

no, I'm thinking of something like

fn transfer(env: Env, from: Address, to: Address,...) {
  env.storage().recreatable().hintRentBump(&DataKey::Balance(from));
  env.storage().unique().hintRentBump(&DataKey::Nonce(from));
  // no bumps for to

but I'm not a fan of that honestly, especially in case if transfer is used as a sub-contract call

molten pendant
#

This is starting to sound more like a SEP issue than something we need to solve in protocol. We define an interface where the user can specify a set of keys and bump amount, they just need to know which keys. This sounds like something we could just get a community standard on

#

The exception being creating an entry initially, which may need some special casing I'm not sure

glossy garnet
#

basically any option involving contracts managing rent bumps seems really convoluted to me and almost impossible to communicate properly to contract developers. from the experience with require_auth it seems really hard to communicate all the subtleties and here things are even worse because the contract developer doesn't even have all the necessary info to make the decision

#

(besides temp lifetime, that is, but that's really the entire point of the temp storage)

glossy garnet
velvet nova
#

It sounds like we were saying that contract developers should tell users how much to bump. Now we're saying, users should own the whole process. Or, the coordination for communicating about how much to bump should occur off chain. Is that correct?

molten pendant
#

The user knows the ammount, but the contract knows the keys I think is the issue

glossy garnet
#

yeah

#

and things are even more complicated if you consider that there might be multiple users per contract and that there might be services that would pay the tx fees, but likely wouldn't be interested to also pay the rent fees, i.e. the fee source doesn't necessarily correlate with what should be bumped.

#

I think while the issue seems really complicated in general, in reality it probably would boil down just a few meaningful use cases:

  • contract admin/instance state/contract balance - bumps managed by the contract owners (+autobumps, but that depends on how heavily it's being used). the upkeep seems reasonably straightforward - e.g. run a cron job, bump the entries below threshold. relevant keys can be exposed by the contract(s) or even hardcoded in case if owners are familiar enough with what contracts they use
  • balances/nfts - bumps ideally managed by the wallet. the UX doesn't seem too confusing - think notification every once in a while that suggests to bump by 6-12 months.
  • long-lived nonces - should probably be bumped by the wallet and probably shouldn't be encouraged due to much higher cost. actually if nonce is random (and not autoincrement), these never need to be bumped, as unique storage has to be used anyway (the downside is increased archive load)
  • long-term allowances - not that they should be encouraged anyway, but if they are used, the party that uses the allowance can take care of the bumps (think of e.g. subscription service). though they might also be ok with restoring the entries since latency is of much less concern
  • most of everything else should probably be temp storage and contracts probably can do a decently good job of correlating the minimum lifetime with the expected usage (see temp nonce example above, or e.g. a DEX offer for which the lifetime is extended respective to how much the user extends the offer, or temp allowance etc.)
urban pumice
#

If the user knows what the rent bump amount should be, why is the "hint" mechanism required? What's the use case where the user needs the contracts input?

molten pendant
#

The ammount isn't the issue, but what keys to bump. Consider a token contract that has one key-val pair for a user's balance and another for something like a nonce. It's difficult for the user to know that to bump their account, they need to bump two separate values. The initial point of the hint mechanism is for the contract to tell what keys should be bumped to the invoker, who in turn can specify the ammount via the footprint

#

Additionally, for most cases, the contract can probably define reasonable bump behavior. In most cases preflight would just use the hint values in the footprint. We just wanted a fallback for advanced users who know better than the contract as to what the lifetime should be and to protect against contracts that grief via high rent bumps

glossy garnet
#

Additionally, for most cases, the contract can probably define reasonable bump behavior.
I'm not sure if that's actually true TBH, especially if the contract is used from another contracts (e.g. token).

urban pumice
#

But the keys would be discovered during preflight right (even without the hint)? The hint is supposed to convey how long the contract thinks the entry should live for, but I would think the user usually has a better idea of what the bump should be.

#

I think this is why Dima was discussing the idea of making the hint a boolean.

glossy garnet
#

yeah, but even in boolean case it's still not clear what should be bumped, because this depends on the tx source account intentions

velvet nova
glossy garnet
#

well, depends on who is the 'user'

#

I think in a lot of cases dapp should know the amount

velvet nova
molten pendant
#

I talked some more about this with Nico, and I think the best solution is to let contracts bump lifetimes for all storage types. It's pretty clear this is a strong requirement for temp entries, so I think it makes the most sense to unify the interface. To do this, we'll drop the lifetime bump argument from the footprint and instead have a classic op that can manually bump rent in addition to contract defined bumps. Previously, our concerns with contract defined lifetime bumps were that the contract did not know how to properly bump rent and that contracts could exploit this behavior. I don't think lifetime bump exploits are a major concern since we don't refund outstanding lifetime balance on deletion and we have an upper bound for the maximum allowed lifetime of a given entry. While I agree with @glossy garnet that there are cases where it is unclear whether the contract should bump or not bump an entry, I think we should leave that up to the discretion of the contract developers. In the worst case, a contract call is more expensive than it should be because you bump entries you don't care about, but I think this is probably a fair trade off for a more straightforward and intuitive developer UX. Additionally, I think we should set up the UI such that contract defined lifetime extensions are optional and have to be explicitly invoked to avoid unnecessary bumps like Dima is worried about. What I mean by this is that the proper bump behavior should be

env.storage().type().set(key, val)
env.storage().type().bumpLifetimeTo(key, lifetime)

and not

env,storage().type().set(key, val, lifetime)

While it's a subtle difference, I think an explicit lifetime bump will help reduce over eager bumps. If a contract is too expensive because it's rent bump behavior is bad, market dynamics should take over, reducing usage of that contract until devs update it

#

A dev could also decide not to do lifetime bumps at all and leave it up to invokers to manage lifetimes manually via classic bump ops.

#

For the interface, I don't think additive bumps make much sense. Instead all bumps should be conditional based on the entry's current lifetime. This means that bumpLifetimeTo(key, 128) will bump the lifetime of the entry up to current_ledger + 128 instead of extending it to current_expiration_ledger + 128 . If a contract calls bumpLifetimeTo(key, 128) and entry_lifetime > current_ledger + 128, the bump is a no op. Does this make sense from a UX perspective @velvet nova?

glossy garnet
#

but I think this is probably a fair trade off for a more straightforward and intuitive developer UX.
I really can't agree here vs just not allowing this (where developers doesn't need to worry about this)

#

a classic op that can manually bump rent in addition to contract defined bumps
that's a host fn still, right? we don't need a classic op for that

molten pendant
# glossy garnet > but I think this is probably a fair trade off for a more straightforward and ...

It just feels weird semantically that env.storage().temp() allows users to define lifetimes while the other types don't. It also feels weird from an invoker perspective that whenever they create new entries via a contract call, those entries only last for 6 hours (the minimum lifetime for restorable types) unless they bundle a follow up op to initialize all the keys the function call created. I don't like that the contract creates the keys, but the invoker still has to do initialization work w.r.t. lifetimes

glossy garnet
molten pendant
glossy garnet
#

host fns don't need to involve VM. we do want them to use soroban metering though

velvet nova
glossy garnet
#

It just feels weird semantically that env.storage().temp() allows users to define lifetimes while the other types don't.
but as we have already discussed they are different, aren't they? the lifetime != time until archived. these are different concepts with different behavior and I'm not feeling too bad about having different API around them (like for temp entry it's not offensive to specify lifetime on creation as it's by definition a part of the entry functionality)

molten pendant
#

Sure, but I feel like this will lead to a lot of archive churn, which we really don't want from a network health perspective. If we don't allow contracts to define bumps, I imagine we'll see a lot of entries get created, immediately archived at the 6 hour mark, and need to be restored for pretty much any operation. If the contracts don't define bump behavior I don't know if we'll be able to have good, consistent lifetime hygene

velvet nova
molten pendant
#

I want to avoid a scenario where the archive is used as a key-value store, where you restore, use the entry, allow it to get archived in 6 hours, repeat. My hunch is we'll see this a lot less if contracts can define bumps

glossy garnet
#

I think this is really orthogonal to the API issue

#

if anything, splitting this responsibility between contracts and dapps might result in neither one doing the bumps

velvet nova
glossy garnet
#

well, I'm not sure it's going anywhere

molten pendant
#

cc @digital sigil

glossy garnet
#

but really, can we just take, say transfer_from token function and try adding lifetime annotations that we think are correct?

velvet nova
#

I don't think we've actually changed the responsibility, but we've made it less blurry. Before with hints and clients maybe doing the right thing, it was super blurry who was actually responsible, and what the ramifications would be if one person didn't do the right thing.

#

Iiuc what we are saying now is that contract developers have control over who is responsible.

glossy garnet
#

I'm not sure if contracts really have enough information to make a meaningful decision...

velvet nova
#

Which has the benefit that they can take control if they are able to, and if not we're leaving it open to the ecosystem to find the right way to coordinate bumps.

#

I think it depends on the contract.

#

If no contract can make a meaningful decision, we definitely shouldn't have hints imo. Hints are worse because of how blurry the responsibilities are.

glossy garnet
#

sure... so can we just try doing this to a real contract and device at least a sketch for a guideline?

molten pendant
#

I don't think it's necessarily wrong for transfer_from to define

increaseLifetimeTo(from, 3_months)
increaseLifetimeTo(to, 3_months)

You essentially allow contracts to define a minimum lifetime for the contract. Even if from has to pay a little rent on behalf of to, I think it's a reasonable convention to say "every entry you touch, you bump to a minimum amount we think is reasonable"

#

Especially since these are conditional bumps, you bump up to 3 months only if it's less than that

velvet nova
#

Running through an example sounds like a good idea. A simple transfer might not be sufficient. @glossy garnet You might need to run through an example that involves an AMM and two separate tokens to understand how the lifetimes/archivetimes interact?

glossy garnet
#

well, that's what I'm trying to get to

#

token manages lifetimes of its own entries

#

but it can then be used in arbitrary contexts. so I'm trying to think what are the implications for these contexts

#

if we think that up to 3 months of rent is going to be cheap enough for an arbitrary user to pay for (note, that neither from, nor to necessarily have to correlate with the tx source account), then why don't we define autobumps to cover rent for up to 3 months instead of just 10 ledgers we have now? if we think that a general guideline should be a couple of months, then what should be the case for setting a (significantly) lower value?

molten pendant
#

The issue is it's super use case dependent on a contract by contract case. For example, 3 months may be a reasonable autobump amount for token balances that are accessed on a weekly basis, but that would be a terrible autobump amount for a WASM blob accessed multiple times per ledger. I think the network wide autobump needs to be the minimal value that makes sense. Then via this interface, contracts can define there own defacto autobump given the specific use case or model the contract is adhering to

glossy garnet
#

but that would be a terrible autobump amount for a WASM blob accessed multiple times per ledger
how so? with 'up to' semantics for a frequently accessed entry everyone would pay minimal fees

velvet nova
#

I think the network wide autobump needs to be the minimal value that makes sense.
Presumedly if the network sees folks doing lots of bumps to a larger value, the network could always agree to increase the default autobump so as to reduce the overhead of so many manual bumps. Is that correct @molten pendant?

molten pendant
#

From a practical standpoint, we've been going back and forth on this issue for a long time, and I think there just may not be a clear answer. In the proposed interface, we have optional contract defined bumps + manual bumps. Should we find that contract defined bumps are problematic, it's easy to deprecate the feature: take it out of the SDK and turn env.storage().recreateable().bumpLifetimeTo into a no op for contracts that already have it defined. This won't break anything, because unlike temp entries, these entries are recoverable if the lifetime expires sooner than intended.

molten pendant
velvet nova
#

given the specific use case or model the contract is adhering to
This comment is really interesting, because if we went with this model, I could see bumps being a recommendation in SEPs for standard contract interfaces. E.g. the SEP for the token interface, whenever it gets written, could recommend tokens use 3 months for balance entries. And a SEP for some other type of standard interface could recommend something else.

glossy garnet
#

I'm not really trying to argue about whether this should be available or not, but rather about what are the reasonable values and if we think that large values are reasonable, then why not make them default and make everyone's lives easier?

digital sigil
molten pendant
glossy garnet
#

especially given that the contract instance and wasm bumps can't be customized IIUC

glossy garnet
digital sigil
#

There is a question above (and related SEP discussion) on how the wallet can extend the lifetime on top of whatever the contract did -- if we remove the footprint based approach, how easy/hard is it to add the extra invoke(s) to the bump host function?

glossy garnet
#

it's also interesting that in your example you suggested that we bump both from and to - is there any value in having per-key granularity if we're going to just bump all the recreatable keys?

molten pendant
#

Yeah, consider a popular DEX. An offer pair may get traded very frequently, so it's lifetimeBump should be on the scale of 100s of ledgers since it's likely the pair will be traded in that timeframe. Toke contract balances may be used on a weekly basis, so a more reasonable minimum would be in weeks/months

glossy garnet
#

but the contract writer doesn't really know if that's a 'popular' dex or not. they just write a dex or AMM or token and popularity really depends on particular instances. that probably would mean another layer of configuration on top of what we provide (e.g. consider XLM-USDC AMM vs SomeMemeCoin1-SomeMemeCoin2 AMM using exactly the same contract code)

#

also, I still don't get what's the issue with large limits given your new semantics (which I actually really like btw) - if bump up to +1 month from now, then if, say, the contract is used every 10 ledgers, then every user would bump the lifetime by + 10 ledgers (besides the creation tx). so this seems to be able to nicely adapt to traffic (unless it has a huge variance and there are weeks of gaps between usages sometimes)

molten pendant
#

What do you mean large limits?

glossy garnet
#

large bumps

#

IDK how to call this properly, the lifetime is set to X ledgers from now, but the bump itself is lower...

#

re-read your example, probably for offers you want temp entries anyway (is there ever value in 'forever' offers?)

molten pendant
#

I don't think we should have large default bumps. In the extreme case, say the autobump always bumped to the lifetime limit. For frequently used contracts, most users are only paying for say ~10 ledgers worth of rent, but the first unlucky caller has to pay MAX_LIFETIME rent, which is a function of size so it could be significant for things like WASM

#

Also for an infrequently used contract, say if only 1 user call the contract every 6 months, they have to pay a ton of rent every time just to call the contract

glossy garnet
#

that's a fair concern, but how is that different if the contract itself sets a high bump? do we have any expectations in terms of what the rent might actually cost?

molten pendant
#

If the contract sets a high bump, the contract is expensive and market dynamics will bring people to cheaper contracts with better rent bump behavior. If the network default sets a high bump, the network is expensive and market dynamics will bring people to cheaper blockchains

#

It's directly tied to BucketList size, so usage dependent. I.e., I don't know

glossy garnet
#

well, but the tradeoff we provide is to either pay more or have huge latency (up to 30 min as you've mentioned) and implementation/operational costs. it seems like a reasonable tradeoff for a contract writer to err on the side of the higher bumps (or contract not setting the bumps and the dapps manually managing the bumps)

#

we have this problem as well actually - I'm quite concerned about the default contract wasm/instance lifetimes. given the tiny autobumps and the fact that even if the contract is eventually widely used, it still would probably need some adoption time, I think that we'll either have a ton of people facing 'my contract is gone' issues or making a large manual lifetime bump as part of the deployment process (or both)

molten pendant
#

At least for a while, we won't be archiving anything but we'll have good data on expiration rates. I imagine we'll be able to use that to set reasonable defaults before turning on the archive and full state expiration

glossy garnet
#

will we make expired entries unaccessible btw?

molten pendant
#

Temp entries are deleted from day one, so those are permanently inaccessible. For other entry types, any read will fail if they are expired. Writes will succeed, but the user will be charged for increasing the lifetime to at least the minimum

glossy garnet
#

ok

#

we'll face these issues in the initial release then

molten pendant
#

They aren't charged for time that they were expired though. Like if an entry expired 6 months ago, you would not be charged for the 6 moths retroactively whenever you write or bump the entry

#

Sure, but since you don't actually have to restore, the issues will be less severe

glossy garnet
#

hmm, how do I make the contract instance accessible?

#

will the manual bump fn work?

molten pendant
#

Yeah. What I mean is say the minimum_lifetime == 4096. If an entry expired 6 months ago, you can invoke a manual rent bump (or an autobump via a write operation) to make the entry accessible again, but you'll only be charged for 4096 ledgers, not 6 months + 4096

glossy garnet
#

sure, that's fair. the main thing is that the contract will live for just 6 hours. I guess we'll get a bit of data from preview 10/testnet, though it's not quite realistic given that no real money is involved...

molten pendant
#

I was talking about data when the we launch on main net. The stakes are reasonably low since restoration will not be necessary at mainnet launch

glossy garnet
#

but I think it would be quite terrible UX if we don't do something to these initial bumps

#

at least a CLI option to make a reasonable initial bump...

molten pendant
#

Maybe this is something preflight could do, construct a rent bump op for newly created entries

glossy garnet
#

OTOH if the stakes are low enough, people might reserve to 'no bumps' strategy for mainnet

#

because restoration is reasonably cheap and fast..

molten pendant
#

Yeah I kinda suspect this to be the case at launch, but honestly I think it's a good thing that stakes are low since rent is a very novel concept to any smart contract dev

glossy garnet
#

which would mean that the initial bulk of contracts wouldn't do the right thing anyway

#

(if there is even the 'right thing', that is)

glossy garnet
#

we'll actually need to to the lifetime exercise for SAC anyway. let's see if we can figure it out.

oblique cairn