#Resource limits for this query exceeded - Github App + githubOrg

9 messages · Page 1 of 1 (latest)

cyan pilot
#

I'm trying to grasp how to get teams + members in from Github using the githubOrg catalog provider. I'm using a Github App that's been added to the organisation, and I use Github Auth for logging in. This is on a pretty new Backstage version 1.44.1.

I keep getting these Resource limits for this query exceeded.-errors. Am I supposed to keep Backstage running for hours or should I be filtering the data somehow ( I do actually only need part of the organisation data in). Also, is there some way to check what data has been written to Backstage?

The org is pretty large and has about 2300 repos, 200 teams and 700 people.

mortal yoke
cyan pilot
#

Oh! So this has been a long running issue, or has Gituhb changed something recently? There's got to be a lot more bigger companies than mine that uses Backstage...

dim sand
#

I work for a company with 6000 repos, 1000+ developers, and 300+ teams.
This bug stopped almost all updates on our GitHub users and teams I ended up removing this provider and scripting my own scraper that runs inside a GitHub action then generates a catalog info file. My script only makes ~350 API calls which is far less than the limit
(~20 API calls to list users with all data that needed, 3 Calls to list teams another 300 calls for the team memberships)

cyan pilot
dim sand
#

Let me check in with my management

cyan pilot
#

Or maybe just a write up on the essentials and the bare minimum to get it work. An article like "Creating a GithubOrg provider for a larger enterprise that wont end up in RESOURCE_LIMITS_EXCEEDED" .

cyan pilot
#

In regards to the above - if we only need to retrieve a few specific teams from our organisaztion - is there some way to filter the query initally, in the same way you can filter what github repositories are to be searched for catalog-info.yaml?

cyan pilot