Data Filtering | Permify | Page 1

lavish dagger Aug 21, 2023, 9:28 PM

#

Hello @elder tendon , I'm curious about the technical details of your infrastructure. Specifically, are you utilizing a PostgreSQL database, or are you relying on an memory database?

elder tendon Aug 22, 2023, 5:29 AM

#

Hi @lavish dagger, thanks for the answer. Currently I'm testing permify locally with PostgreSQL.

lavish dagger Aug 22, 2023, 7:23 AM

#

Thank you @elder tendon , Can you specify the version of Permify you are using and describe the exact problem you are experiencing?

elder tendon Aug 22, 2023, 7:32 AM

#

I'm using permify in version 0.4.8.

Let's say we have a relation between a user and a repository and a repository and a file.
The user has 10 repoositories and the repositories have all together around 3000 files.
If I do a permission lookup for which files the user has access the request takes around 500ms. which is already a lot. However, If I add a contextual tupel to the request the same request already takes over a second.

Since I will have much larger datasets in production I`m wondering if I'm doing something wrong or if the performance is not (yet) good enough.

lavish dagger Aug 22, 2023, 10:16 AM

#

Hi @elder tendon ,
For data filtering, we're performing a reverse lookup by finding the linked entities. For example:

entity user {}

entity document { 

    relation viewer @user
    relation editor @user 
    relation owner @user

    action view = viewer or editor or owner 
}

When we get a request like document#view - user, the linked entities we look for are document#viewer, document#editor, and document#owner.
After finding the linked entities, we then perform a reverse query on the database. Since we don't yet support pagination, it's normal for the process to take a bit longer with larger data sets. For this, we have an API that operates as a stream. Have you reviewed it?

elder tendon Aug 22, 2023, 10:20 AM

#

Yeah I have it's not really what I need. pagination would be best 😅

I expected that you do a reverse lookup. I just hoped it would be faster 😄 Maybe I have to build something on y own using a graph database.

lavish dagger Aug 22, 2023, 6:50 PM

#

Could you please share your expected performance for the reverse lookup? If you prefer, you can put it through stress tests in a non-local environment. If it’s below your expectations, we can definitely find a solution.

elder tendon Aug 23, 2023, 7:29 AM

#

Good Morning @lavish dagger

thanks for comming back to me.
The scenario is like this:
A User is part of an organization. The organization has Repositories. The Repositories have Sub Repositories. The Sub Repositories have files.
If a User has access on the organization he has also access on all files.
However: we don't know which files that are. So we want to use the reverse lookup combined with a contextual tuple to find all files of one organization a user has access too.

I would expect this to run in a couple of ms. Preferable with pagination. Is that realistic?

lavish dagger Aug 23, 2023, 2:32 PM

#

Hi @elder tendon , we're working on enhancing its performance, but it might not deliver the desired performance on heavy data. The key question here is how important is consistency for you. Some of our users handle it this way: they pull the data and map it with a snapshot daily (or at different intervals), synchronize it once, and then retrieve the data from cache where it's stored. Would such a scenario be suitable for your case?

elder tendon Aug 23, 2023, 2:45 PM

#

Yes it would be suiteable. But then it would make more sense to run a graph database to query from it and use something like open policy agent locally to evaluate permissions. I hoped to get both out of the box from your side 😉 however I will considere your suggestion with my team 🙂

#Data Filtering