#TemporaryEntry Conversation and Open Questions

108 messages · Page 1 of 1 (latest)

slim storm
#

In the previous Soroban design discussion, we came to the following conclusions regarding TemporaryEntries and RestorableEntries:

Outstanding rent_balance of RestorableEntries should not be refunded on deletion but is burned. This is to prevent malicious gamification of the rent system.

TemporaryEntries are discounted compared to RestorableEntries. This incentivizes the use of TemporaryEntries and is better for the health of the network and keeps archives small. As an alternative, we considered adding an additional base fee to RestorableEntries that is refunded on deletion. However, it was determined that incentives at creation time are much more powerful than "cleanup" incentives on deletion. To reduce complexity and give a creation time incentive, TemporaryEntries are discounted compared to RestorableEntries and RestorableEntries do not refund anything on deletion and have no additional base cost.

The following topics are still open questions:

Should TemporaryEntries be able to change their TTL?

Should TemporaryEntries have arbitrary expiration ledgers, or should TemporaryEntries expire at set intervals determined by BucketList internals?

I think we should allow TemporaryEntries to bump their TTL's and that TemporaryEntries should not be able to define arbitrary expiration ledgers. Allowing arbitrary expiration ledgers at the protocol level allows the TemporaryEntry to be used out of the box for security related features. However, this puts more strain on downstream systems such as horizon. Additionally, it is easy for contract developers to enforce arbitrary TTL on their own without a protocol implementation. For example, if a contract needed a TemporaryEntry to only be accessible for 10 ledgers, it would be easy to include and expirationLedger field in the stored struct and check if the entry is still valid after loading it. Security concerns seem outside the scope of protocol storage and should be handled by the contract.

#

Instead of arbitrary expiration ledgers, TemporaryStorage would expire at BucketList merge boundaries, which are powers of 4. This means a TemporaryEntry could define a lifetime of 4^n where 0 < n < 10. In practice, this could be abstracted away from the user, where we define the constants TemporaryEntry::SHORT_LIFETIME, MEDIUM_LIFETIME. and LONG_LIFETIME. Short lifetime could be 128 ledgers (10 minutes), medium is 32,768 ledgers (about 2 days) and long lifetime is the maximal value of 524,288 (30 days). These exact values could be tweaked, but these are the approximate ranges available.

#

Additionally, the community seemed to want to be able to change the TTL of temporary entries. This should be doable without much additional work

uncut folio
#

On the topic of TTL extensions, can people reiterate why they think this is a necessity? @calm geode I think you were in that camp

#

(keep in mind that it's pretty easy in a smart contract to achieve a ttl bump with custom logic)

uncut folio
#

Instead of arbitrary expiration ledgers, TemporaryStorage would expire at BucketList merge boundaries, which are powers of 4.
This feels pretty arbitrary to me. Giving exact TTLs sounds like a saner contract to me. But not religious about it

slim storm
#

The lift on downstream systems is not insignificant for arbitrary TTL. My thinking with the short, medium, and long lifetime distinctions is that users know that their entry will live around 30 days, but don't care about the exact ledger boundary

honest eagle
#

Should TemporaryEntries have arbitrary expiration ledgers, or should TemporaryEntries expire at set intervals...
Do the fixed expiration TTLs serve only downstream systems, or do they benefit core and the archive in some way? I agree with @uncut folio, they seem pretty arbitrary, and I think it's difficult to know if the values we pick will be good values, and it will also become difficult to change them as contracts will ultimately end up depending on them in some way.

#

Will downstream systems like Horizon see a meta record for when temporary entries expire? If so, it's unclear to me how fixed vs arbitrary TTLs would impact it. Could someone expand on that?

#

Should TemporaryEntries be able to change their TTL?
Yes. If we don't support updating the TTL, folks will use recoverable entries for cases where temporary entries would be perfectly suitable. However, if there's significant constraints against making TTL mutable, I don't think it would be world-crushing to make it immutable.

uncut folio
#

Yes. If we don't support updating the TTL, folks will use recoverable entries for cases where temporary entries would be perfectly suitable.
I'm not sure I agree with this statement. They can always just create a new temporary entry. Let's say for example that you have an oracle contract that accepts price information that is valid for one hour. whenever it accepts new information it just creates a new temporary ledger entry and points to that

honest eagle
#

That'd create ledger churn, and ledger bloat.

uncut folio
#

what do you mean by churn? why bloat - it's all temporary and people are encouraged to use short ttls to save on $$$

#

more features more problems

honest eagle
#

@slim storm Can you delete a temporary entry?

#

@slim storm Does touching/accessing/writingto a temporary entry reset its TTL such that it then has TTL time left again from that point onwards?

slim storm
#

They can be deleted before the TTL, write does not automatically change TTL

calm geode
# uncut folio On the topic of TTL extensions, can people reiterate why they think this is a ne...

I am in the camp of TTL extensions because of two main economic factors and one blockchain factor: #1 sometimes other people don't execute on time, so we need to give them more time to do their part. #2 if you allow extensions, people will "aim low" and add time if needed instead of always overestimating. The #3 blockchain factor is that the original transaction can't be "restored" so if systems depend on the original transaction, all might be lost in that case.
TTL are critical for escrow, temporary authorizations, access controls on chain, telemetry data that gets flagged for follow up or audit, the integration of AI with blockchain and a large number of other new and existing features.

calm geode
#

Should TemporaryEntries have arbitrary expiration ledgers, or should TemporaryEntries expire at set intervals determined by BucketList internals? I think this is a question of how expensive the garbage collector is at execution. The "powers of 4 limitation" is a really bad user experience if they need to calculate it, but I wouldn't want GC running frequently nor would I want users needing to memorize their powers of 4 + current ledger.

#

A middle ground may be promising the user "at least" a certain interval and then wipe them at next GC without any secondary mechanism. It creates a consistent expectation at the better performance tradeoff. I think something similar was proposed in the call but I am new to the architecture and don't know limitations.

#

TemporaryEntry::SHORT_LIFETIME, MEDIUM_LIFETIME. and LONG_LIFETIME I think these sound great

#

It is entirely possible to get millions of transactions per month with SHORT_LIFETIME that are a syndication method for telemetry data. Instead of the sensors doing complicated networking tasks and hardware, they simply broadcast a tx whenever they can connect. Cars can do this at stoplights using 5G for example, and wifi at home. The data integrity is preserved, they don't have to do complicated tasks involving networking or add cellular subscriptions. The SHORT_LIFETIME is plenty of time for insurance, manufacturer and other agencies (with access) to read the TX data and record a permanent record off chain.

#

If there is a serious event that requires additional review, they can extend the TTL on the Telemetry data and then get a secondary validator system to respond.

#

While it is tempting to think that everything will be performed in the first 10 minute lifetime, some in-production privacy protecting ZK SNARKS have 3-8 minutes as their runtime.

delicate plaza
slim storm
#

I was thinking that on set there would be an optional field to reset TTL to one of the lifetime values (short, medium, or long)

#

What invariant does this break?

delicate plaza
#

if the ttl has any meaning beyond 'this entry is going to be removed after N ledgers', e.g. 'approve spending token for the next 10 minutes'

slim storm
#

I think the way we will need to expose the interface is to ensure that we don't give any guarantees. For example, the lifespans are approximate values, not specific ones. I the conversation yesterday, it seems that the exact TTL for security purposes is outside the scope of protocol storage. The use case for exact TTL values seemed relatively limited while the use case for TemporaryEntries that don't require exact values wer significantly larger (according to community sentiment during the chat). If a contract needs a TTL for security purposes, they should just put a TTL value in the struct they store. It seems like this special casing is outside the scope of protocol and should be at the smart contract level. This will also make temporary entries more efficient for downstream systems.

delicate plaza
#

hmm, I see. the last iteration had exact TTLs

#

but the exact TTLs aside, still not sure about non-confusing bump ux

honest eagle
#

we will need to expose the interface is to ensure that we don't give any guarantees
This is easier said than done. If the interface produces behavior that is not guaranteed, but is common or consistent in anyway, people will build around that behavior.

delicate plaza
#

yeah... there is also a very specific use case I've planned to use the temp storage for, which is nonces

#

though if we can guarantee that the entry has a lower lifetime bound, that should work fine

slim storm
#

I think the flags we expose should do this pretty well. We're not exposing exact values, and the comment for the SHORT_LIFESPAN could be something like "a temporary entry that lives between 114 and 128 ledgers"

#

The variance is relatively small, its just not exactly ledger specific

#

Each lifetime will have a guaranteed upper and lower bound, is that good enough for your use case @delicate plaza

delicate plaza
#

lower bound is good enough

#

upper bound is meaningless I guess

calm geode
delicate plaza
#

this has nothing to do with the wallets really; I'm asking about the contract-side UX

#

IIUC that's what the debate is about - whether we allow contracts to extend the ttl of their temp entries

calm geode
#

I'm not sure that the contract can or should trigger events. Web3 bots with a wallet key can do that

slim storm
#

My current thinking is read/writes do not change TTL by default. On write user's have an option of setting a new TTL. This TTL does not stack, but replaces the previous TTL. For a concrete example, say a user creates an entry with a TTL of 128 (this would be abstracted away by the lifetime arguments, no actual numbers should be used) on ledger 0. 64 ledgers later, the user writes the entry and sets the optional TTL argument again to 128. The TTL is replaced, not summed, so the entry expires on ledger 192, not ledger 256

calm geode
#

"initiating" a transaction should probably be "wallet only"

slim storm
#

The reason we replace instead of sum is because we need a strict maximum TTL value, and the value must fall on a bucket merge. If we sum, this is no longer guaranteed.

delicate plaza
#

hmm, not sure I understand. what exactly are you replacing it with?

calm geode
#

The previous value

#

So the TTL will always be "now() + SHORT" instead of "SHORT + SHORT"

delicate plaza
#

to be more precise, what is the frame of reference

slim storm
#

The current ledger, Ian above is correct where now() is the ledger in which set is called again

#

Every call to set (with a lifetime parameter) resets the expiration ledger based on the current ledger and the lifetime value supplied

delicate plaza
#

but if the lifetime is bound to the bucket list merges, then will something like doing short and then short again change anything?

#

or will this be the next merge?

slim storm
#

In the actual implementation, we write an expirationLedger in the TemporaryEntry. This expiration ledger has to fall on a merge boundary. The lifetime is not bound to a specific bucket merge, just a bucket merge in general. i.e., we don't care if an entry expires on the level 5 merge or the level 6 merge. If an entry with SHORT_LIFETIME is never written again, it will expire around level 3 (I think, the exact level doesn't matter). If the temp entry changes but the expirationLedger does not, it will still expire on the same ledger, but it won't be at the level 3 merge, but at the level 1 or 2 merge. This is because whenever the TemporaryEntry is rewritten, it gets placed back into the top level bucket. In this way, temporary entries are not tied to a specific merge event, but are tied to a ledger that occurs on a merge boundary. One interesting property of the BucketList is that whenever a given lever merges, all the levels above it also merge on the same ledger. This means that it doesn't matter how many times a TemporaryEntry is rewritten, it will expire on the same ledger no matter what level it ends up in

#

All that to say temp entries are still driven by a specific expiration ledger, it's just that the ledger must be a merge boundary. On writes, users can change this to a future merge boundary, or keep the same boundary ledger

#

So to answer your question, setting short then setting short again will extend the life of the entry because the expirationLedger is advanced to the next merge boundary that is approximately SHORT_LIFETIME ledgers away

delicate plaza
#

hmm, but sometimes it will not do anything? or will it always do something? e.g. what happens if I bump(short) in ledger N and then do the same in ledger N+1?

slim storm
#

Yeah, so because the lifetime is a range, if you call the same lifetime again and the range is the same its effectively a no op

delicate plaza
#

right, so that's pretty confusing and it's also completely opposite to how bumps work for non-temp entries

#

(which are strictly additive)

slim storm
#

Maybe the best option then is to allow TTL resets but still support exact TTL?

delicate plaza
#

I would try to come up with some use case for extending ttl and then try expressing it with the suggested api

slim storm
#

What do you mean by suggested API?

delicate plaza
#

the suggested bump api

slim storm
#

So bumps are additivie?

delicate plaza
#

no, I mean try working on the api UX in a more realistic setting. currently it's not obvious to me if this does or doesn't work for the real use cases...

#

e.g. let's say I have an offer contract and I want to use a temp entry for storing the offer state. let's say I want to save on the storage and use short ttl (not sure what's the value, let's say O(minutes)). but also if the offer clears (partially) I want to extend its lifetime, but I don't want to trade it for longer than, say, 1 hour

#

and the question is if/how we can implement something like this

slim storm
#

What if temporary entries are subject to the same rent charges as restorable entries but at a discounted rate? Because the timespan is short, even though the rent fee is variable, we could still offer a lower bound guarantee. This would also unify the interface, and we may want to do automatic bump on access too?

#

For your example the contract could just bump rent on the temp entries

#

This way they function the same as restorable entries but aren't subject to minimum_rent and are cheaper

#

This would also allow a user to have perpetual temporary entries at a discounted rate compared to restorable entries if they didn't care if the entry gets permanently deleted

delicate plaza
#

yeah, consistency would be nice...

#

still wondering about semantics, if contracts can bump the rent. like how would I design my example if I don't really know what every bump does

slim storm
#

I guess in your example above, would the user calling the function know the life needs to be extended

#

Or how does the contract know it needs to extend the temporary entry?

delicate plaza
#

yeah, that's my point - if we want to have programmable logic, then it probably would need to be pretty involved and provide ways to figure out things like the remaining lifetime and the expected bump boundaries

slim storm
#

I guess this gets back into the rent issue though. If we allow contracts to define how much to bump a temporary entry by, why can't they do the same for rent bumps on restorable entries?

#

If we expose bump_rent_balance and get_rent_balance host functions and make both temp and restorable entries subject to rent, this unifies the interface

#

The more I think about use cases the more it seems to make sense for contracts to define rent bumps

#

Both in the temp case and for restorable entries

delicate plaza
#

interesting, my point was actually that we don't need either 🙂

#

the impact is somewhat more limited for temp entries, but again, in a general case, it's hard/impossible for the contract to debate about the bump strategy, as that's the user who pays for the bumps. in case of the temp entries there is actually not that much of a user interest to do a bump at all

slim storm
#

Yeah it's just a weird, complicated system if temp entries have lifetimes managed by the contract and restorable entries have rent managed by callers

#

I think there's some UX benefit for having a unified interface with the maximal amount of feature set exposed instead of having two pretty different UX with different APIs and different features

delicate plaza
#

right, so my vote would be to have a consistent UX where we don't allow contract-specified bumps. I'm still on the side of not updating the temp entry TTLs at all, but if we feel like that's required, the same sort of API as for all the other entries would be preferable (which IIUC currently is small automatic bumps + a host fn for bumping arbitrary entries)

slim storm
#

According to the community members at the last talk extending TTL is a must have, otherwise the use case for temp entries is very limited to oracle use cases

#

I don't think having an op only host_function is good enough. Especially for short lived temp entries that only live say 10 minutes, the most common bump case is just extending the life by another 10 minutes. On this short time frame, I think it would be pretty much required to have the contract defined rent bump behavior, otherwise who would be able to issue the op in time?

delicate plaza
#

interesting, I wonder what were the use cases mentioned?

slim storm
#

In the discussion the most common use case for temp entries was as inputs to some result, where only the result is actually saved in a non-temporary entry. For this type of work, the temp entries are technical and implementation specific to the contract. I don't think we can expect users to understand the intricacies of a contract well enough to issue ops to bump the TTL of intermediate values, especially if we require giving the keys as input to the op

delicate plaza
#

otherwise who would be able to issue the op in time?
there is really no difference between any sort of transaction - it has to be built by someone and sent to the network

#

this is really vague..

slim storm
#

"TTL extensions are important for Zero Knowledge Proofs if the resulting data still needs to be used for additional transactions. For example I do a ZK SNARK that proves I have the authorization to purchase a security. The ZK SNARK is really big, so I want to expire the data. I may not have actually purchased all of the securities (stocks, bonds) that I want to buy, so I will extend the TTL on my ZK SNark rather than redo it."

#

This seems like a contract implementation level detail

delicate plaza
#

not really

#

I don't quite see how the contract-driven bump would help

#

basically we're back to the contract_bump function vs contract_bump_meta+host fn debate

#

it doesn't seem like the automated bumps help here as someone still needs to access this proof and it's not clear how frequently (if at all) that happens

#

and what should be the extension period - the example implies that the user wants to initiate the bump

#

basically for automated (including contract-driven) bumps to work there needs to be some pretty complex and bespoke logic based on the remaining time and context and even that is not enough as it relies on a dapp to call the contract 'in time' (which is even more critical for the temp entries)

slim storm
#

I think extendable TTL via op is good for now, we can always had host functions later

#

I don't know the use cases well enough to definitively comment one way or the other

#

I think we should unify though, both temp and restorable pay rent which can be bumped via a op, where TemporaryEntries pay a reduced rate for rent.

delicate plaza
#

yeah... I'm still pretty skeptical of this use case TBH, just allocating as much time as the entry is expected to live seems to cover most of the cases (you can't be too greedy of course, but it's not even clear if sending a separate 'bump' transaction is any cheaper)

delicate plaza
uncut folio