#JSON Data from api calls are massive [2MB-5MB range]

14 messages · Page 1 of 1 (latest)

thin hare
#

This isn't a bug, and I've been conscious of it during the last year of development. Now that I'm closing in on production and launching, I'm appreciating the scale of the problem it causes.

The issues comes from the liberal use of depth: 2-3 on collections and queries, which is especially pronounced due to the use of a ReusableContent collection pattern. The layout blocks on these ReusableCollection records add an extra level to the required depth (so that all the data required by the layout block components is available).

The size of the json data returned from the api is huge because of the side effect of the above. Some of my collections have records which contain 70 pages on their relationship field. You can see how these depth queries produce exponentially large responses.

It's not a big issue if I keep the mongo database running on the same machine as the server, and the lag here is tolerable thanks to the payload api (with a single user interacting with the site), however I was hoping to try Payload Cloud, and take advantage of the MongoDB Atlas serverless db.

As a test, I added my DB to the a new MongoDB Atlas serverless instance and connected it to my production build. The results were awful. 4-5 sec response lags on every route change, and initial load even worse 🤦‍♂️.

Optimisation advice

So with the above in mind — I believe I need to migrate all of my api calls to GraphQL, right?

I expect this to be a huge change, and I'm curious to know of others' experience in using GraphQL inside a NextJS application.

Can I incrementally adopt GraphQL? How would this work in practice?
I currently use the pages folder for the whole frontend, and wondering whether this would be a good point to embrace the app folder — If this were so, can I separate out the REST/GraphQL approach? Or does this need to be a wholistic change (all components/routes/pages need to use GraphQL)?

Any advice would be very welcome!

Thanks 🙏

hollow isle
#

You can do it incrementally, at my job we are slowly migrating our ~40 page app to graphQL page by page and testing results. We are using app router, but to be honest if you have working pages then stick to pages. Maybe app router is production ready, but many plugins like next-intl isn't ready in 100% which results in some serious problem, like whole app needs to be SSR and you can't utilize SSG.

verbal anvil
#

I second Veth in that you can incrementally migrate to using GraphQL. In many cases, I actually use both REST and GraphQL for certain applications. This is not uncommon.

As for working with GraphQL in Next.js - I want to first highlight that you should NOT think that GraphQL requires a ton of new plugins and frameworks and tech in order to work for you. In many cases, GraphQL calls are quite similar to regular REST API calls.

Basically you call your GraphQL endpoint as you would a REST endpoint:
fetch('https://mycms.com/api/graphql') - but with a few different settings. For example - all queries, mutations, subscriptions, etc. are sent to the API endpoint as POST requests. The reason for this is because GraphQL relies on you sending a body along with the API call. This body is your GraphQL query/mutation:

const query = `query {
  Pages {
    docs {
      id
      title
      slug
    }
  }
}`;

const pages = await fetch(
  'https://mycms.com/api/graphql',
  {
    method: 'post',
    body: JSON.stringify({ query }),
  }
).then(res => res.json());

const slugs = pages.docs?.map(page => page?.slug);
#

Of course - there are several tools out there that can help you to e.g. generate types based on your GraphQL Schema, have a caching layer, to reduce the amount of calls made to your CMS, passing auth credentials etc. but basically just taking your REST calls and making them GraphQL calls is relatively simple.

#

The payload team has actually been so kind as to make their entire Next.js front-end site publicly available in GitHub. So you can take a look at their repo and see how they have solved the GraphQL in Next.js task.

#

It helps with suggestions, autocompletion, error handling and can be very helpful when you want to compose more complex queries that take variables.

thin hare
#

Thanks for your thoughts and advice Veth and Corfitz. Really appreciate it. I will take a look at the way that Payload does it on their Next site and see if I can understand what is going on.
My site in next is 100% SSR, and I utilise the Payload API here extensively (only exceptions are within the _app where we must use fetch, and fetch within react-query and it's useInfiniteQuery which is hydrated with initial data from from SSR).

For GraphQL, we lose the speed of the Payload API for queries if we must use fetch...

#

I guess this would be offset with more efficient responses

thin hare
#

Looking at Payload's website repo I see they only use fetch which makes sense seeing as they split the cms and frontend into two instances, and next would need to use fetch here. My app is a monorepo, which runs next and payload on one "custom" server, so I can use the Payload Local API for SSR requests.

verbal anvil
#

If you perform the fetch requests server side, I believe the server automatically will internalise the communication as it would know the api IP is itself. So whatever the speed impact may be, it would be minor.

However, using GraphQL you can tailor you query and thus only fetch the information you actually need, which would likely even out - if not improve tremendously - the performance and speed of your queries.

rotund vapor
#

I’ve run into this issue on a smaller scale and I ended up making a few custom endpoints to trim off some unneeded data.

#

Also, the app router structure in definitely made fetching more simple in my case. I can have each component fetch what it needs (like blog feeds) and that means I don’t have to fetch a large payload at the page level.

#

GraphQL sounds like it would also be a good solution. I’ll give that a go one of these days.