Lookup Performance (3500 Devices/ 3 Seconds)

[marsh willow]

hi @limber path and @stable spindle, just wanted to create this threat for you guys if you would continue the discussion

[limber path]

Thanks for creating the thread @marsh willow

@stable spindle, replying to your questions:

1. What would the user be wanting to do with the map of devices? You've explained what you'd like in the UI, not what the user is expected to be doing. 🙂

R: The stakeholders/users say there is value in having that map visualization. A user can click on a device in the map and navigate to it. With Permify in place, we would want to show only the devices a user can read in the map, not all the existing devices. If we assume this is a real requirement that brings value, we need to have a way to implement it.

2. The macro filter is the micro read filter. In other words, if you ask Permify for the allowed BUs and Customers, and pass those to the database (WHERE bu_id IN (...) AND customer_id IN (...)), the database will only return devices the user is mathematically guaranteed to have access to.

R: Yes, I know that. But if I filter in the database, then I'm moving the permission logic to the application instead of Permify.
For example, if one day I want to change the permission to 'OR' instead of 'AND', I now have to change business logic and deploy new code, whereas I could just do a minor change in schema if we were using Permify to answer the question 'which devices can the user read'.

[stable spindle]

Hmm... have you seen the streaming entity lookup endpoint? (I just found it myself, to be honest 😊 )

I think that is the solution. Maybe? Sounds like it from the description:

The difference between this endpoint from direct Lookup Entity is response of this entity gives the IDs’ as stream. This could be useful if you have large data set that getting all of the authorized data can take long with direct lookup entity endpoint.

[limber path]

I did see that and played a bit with it. But from my understanding, this still takes a long time to complete, it just streams the results over time.
We would have to change for example our GET devices endpoint to also work as a stream. And if we think of the map functionality, I guess this would mean the user would see devices popping up in the map over a few seconds, which would also be a bit of a weird user experience. Pagination / sorting would also not be very well suited with the streaming approach.

Maybe someone from Permify team can chime in and confirm that without something like what's being proposed in https://github.com/Permify/permify/issues/2681 (and which introduces a lot of complexity to the system too), it's just impractical to use Permify to do entity lookups at such scale and not suitable for my use case?

[stable spindle]

I was wondering, does the "map" need to show all devices at once? I mean, is there some form of fidelity you could use? Some form of area calculation, which could lower the number of devices in a given view/ zoom level?