#Data Filtering
scarlet pasture
Hi @sage pike welcome to the community and thanks for the support!
We have a data filtering API (https://docs.permify.co/api-reference/permission/lookup-entity) that lets you query 'Which resources can user perform action Y on?'. The response will return entity results in a string array format or as a streaming response, depending on your preference. It supports pagination, allowing you to check access control and filter large amounts of data.
We do not have any support for OpenSearch cases right now but would love to listen your need and navigate accordingly
sage pike
I'll take a look. OpenFGA also has some rough topics around this.
The core issue I've run into:
Say you have a bunch of e-commerce orders, like 10m
A customer can see their records. So it's cheaper to access their records (get all) from the Permify API
From there you filter where in one of the say 150 record ids that come back.
And you page that result.
--
The inverse, you have say 1 Trillion chat messages.
You want to allow searching of those messages for that user.
Let's take discord for example, you are in a chat and a user so you can scope the query. But the messages IDs you get back could be in the millions.
So you do an open search query for the channel/I'd then you pass the resulting message IDs though Permify to double check the access level?
scarlet pasture
Hi @sage pike sorry for not replying promptly we’ve discussed this with the engineering team and data filtering should work fine with millions of the data.
For the search cases, we don’t have exact solution or workflow right now. But I can suggest an approach that might help.
Assuming that you apply pagination to the search results, you can send the results to Permify one by one, as you'll only be displaying a limited number to the user. Permify is designed to handle millions of requests per second, so this approach won't cause any issues for your specific case.
While this solves the issue, sending multiple checks at once could create a problem as the number of items on the page increases, though it shouldn't be much of a problem even 500 items.
scarlet pasture
We're also working at bulk access check. Here are the issues related to that.
Do you think any of these approaches address the concern you have? Or do you have any other suggestions we can work on?
sage pike
Yeah possibly. I'm just thinking through some of these edge cases. For example, if you have a Channel & Messages. Maybe you don't bother checking messages on a per check basis (too granular) but you scope to the Channel (check) andtrust the down stream query call
It's definitely harder in search calls