GetLedgerEntry vs. GetLedgerEntries

[civic burrow]

The soroban-rpc has a RPC endpoint called GetLedgerEntry. Using this entry allow the consumer to retrieve the ledger entry associated with the most recent ledger.

When a client needs to have multiple fetch operations, she would issue multiple GetLedgerEntry requests. To resolve this issue, we are considering replacing the GetLedgerEntry endpoint, with an GetLedgerEntries endpoint, allowing the client to retrieve multiple entries as an atomic operation.

The operation atomicity ensures that all the entries are being retrieved off the same ledger - which is a requirement for simulating the execution of a smart contract. We believe that this could be the foundation for future decouplilng of the storage and execution which are currently part of the soroban-rpc.

from technical perspective, the soroban-rpc endpoint currently has the following interface

getLedgerEntry(key: xdr.LedgerKey): {
xdr: xdr.LedgerEntryData,
lastModifiedLedgerSeq: number,
latestLedger: number,
}

which we consider replacing with

getLedgerEntries(keys: xdr.LedgerKey[]): {
entries: (null | {
xdr: xdr.LedgerEntryData
})[],
lastModifiedLedgerSeq: number,
latestLedger: number,
}

I'd like to hear your thoughts.

[twilit geyser]

I love the idea, it will reduce the amount of request made from apps (I guess it will have a limit in the amount of items per request)

[celest scaffold]

the json-rpc standard already has support for batching and our go implementation supports it so we should get request batching for free. In other words you should already be able to put multiple getLedgerEntrey requests in a single http request without this change.

The open question is around atomicity across these requests:

• (a) how important is atomicity?
• (b) Assuming atomicity is important, can we introduce this at the batch request processing level so that all batches (regardless of which handler they call) are atomic?

[sullen jasper]

@civic burrow thoughts 👆🏻 ?

[civic burrow]

I have a few separate concerns here:

1. I don't think that the rpc level ( transport level ) is the appropriate method to control the functionality. Atomicity is a key for any meaningful transaction execution. i.e. if you know that you need a set of keys to make a decision - you'd want to make sure they came from the same ledger.

2. While it might be true that we can support batching on the server, not all client can do that easily. I think that it would be wise to use a common denominator when designing these features.

3. Would that mean that any set of batched rpc requests is to be executed atomically ? I'm hesitant if we want to go that way. This might have some security and latency implications. Also, supporting that for one set of command is an "invitation" for additional tickets ( let's call them "bugs" )

4. last, the json-rpc standard batching feature was focused on reducing the round time when invoked by a single-threaded source ( i.e. javascript ). this is well-designed solution.

[worn nexus]

Conceptually, I view batched json-rpc requests in an HTTP request as a short-lived "transport". Similar to a short-lived websocket. I don't think we should try to guarantee atomicity within a "transport", only within a single method. You wouldn't expect atomicity between methods called in a single websocket session (nor would you want that).
(Now, if that matches what users expect is another question)

[sullen jasper]

not all client can do that easily
This is a really good point. It would be disappointing if a feature was gated by general json-rpc capabilities in clients that folks are using, and it would be a lot for us to take on improving that.

[worn nexus]

In this specific case. If you don't want to use getLedgerEntries, you can just pass a single-item-array for the arg, stick a bunch of those into a batched request, and pretend. 🤷 but the implementation time/effort was probably less than we've spent in this thread...

[celest scaffold]

these are all good points

[sullen jasper]

Is this the only function we think folks will call where they need atomic access to data?

[celest scaffold]

adding to @sullen jasper's question: we already support batching by virtue of implementing the json-rpc spec. Is there a reasonable expectation that batched requests work on a consistent view? It's obviously not part of the json-rpc spec but it's often expected when executing a db multiquery of sorts

[sullen jasper]

Is there any case where a user might need to call getLedgerEntry/ies and some other query function and expect data consistency?

[civic burrow]

Dima suggested that we might have getNetworkSettings or similar; I believe that it would be better to let the client request the ledger entry and decode the xdr themselves.

Separately, the user might have a suggested-fee endpoint. However, given that we don't yet know what the expected implementation would be, it's a bit hard to make concrete case here. I think that given that it's a "suggested fee based on history", a potential mismatch of a single round shouldn't be critical.

[celest scaffold]

I wonder if a downstream system that uses getEvents to index effects might use batched requests to get disjoint relevant event streams and maybe it's problematic to have an inconsistent view there?

[worn nexus]

getNetworkSettings seems similar to getAccount. which we moved the implementation into the client and it uses GetLedgerEntries behind the scenes.

[sullen jasper]

If we wanted atomic requests over different things, without relying on the transport or format providing it, maybe we'd need an endpoint for composing other endpoints, which sounds a little like graphql.

[civic burrow]

@celest scaffold could you try and articulate a sample case for that, it would be great.

[sullen jasper]

I think getLedgerEntries is fine fwiw. It just seems like this problem will occur in other places, but we don't have to solve for it today, and we can solve for it holistically later.

[worn nexus]

You’d probably do something like a batch(methods[]) that takes other methods

but yeah agree no need to solve it right now. lets do it properly later when we have a concrete use case

[celest scaffold]

Not the greatest example in the world but let's say that you build an app that's looking for arb opportunities between 2 AMMs so you batch two getEvents requests to the AMM price fees and you may assume that the latest events on both streams come from the same ledger but maybe that's inconsistent.

[civic burrow]

in the above case, you would get a ledger number ( for both the events as well as the AMM price fees ). You'd need to cross-reference these to ensure they are the same or match your application needs.

[celest scaffold]

@civic burrow why doesn't the same argument apply for getLedgerEntry? You have a latestLedger field in the response that you could use to cross check with other ledger entries

https://soroban.stellar.org/api/methods/getLedgerEntry#return

[civic burrow]

It does apply.. that's the reason it's there. Did I miss anything ?

[celest scaffold]

If we consider it acceptable for the client to cross check then why do we need a consistent getLedgerEntries? The client can just batch all the relevant getLedgerEntry requests and then cross check the latestLedger number on the responses

[civic burrow]

In that case, there might be some mismatch. Making the call again doesn't guarantee we won't have mismatch again. What you really want is an atomic call to get these values at the same time.

[worn nexus]

You can filter for events from up to 5 contracts in a single getEvents, which will give you all the ordering and atomicity you’d want. but if you wanted more, yes

[celest scaffold]

TIL!

[celest scaffold]

Why 5?

[worn nexus]

seemed like a reasonable limit. 5 contracts x 5 topic filters = 25 things to check for each event. more would risk combinatorial explosion

easier to raise it than lower

[sullen jasper]

not all client can do that easily

I think this is an interesting point that according to the emojis struck a chord with a few people.

@civic burrow Is this because client implementations of json-rpc are partial implementations of the spec?

I'm wondering how this idea that clients might not be able to actually use json-rpc in the ways we might benefit from using it over HTTP is going to play out as we try to make more use of it.

(Related conversation about if json-rpc is at #1097914661933948938 message)

cc @soft forum

[civic burrow]

@sullen jasper exactly. This is a good example for such a case : https://pkg.go.dev/github.com/go-pkgz/jrpc#section-readme. I don't know if there is a good way to demo that via command line /w curl/wget.

[celest scaffold]

Why is this an example of clients not having adequate support?

Also, wrt batching, which clients don't support it?

[civic burrow]

The mentioned package doesn't have jrpc 2.0 support. I'm not saying that it would be a dead-end for some users, but it might make it harder. I'm not opposing to thinking about composability here, but I think that this direction would not be the best use of our time. i.e. it won't make any meaningful difference for end users.