Fetching paginated server data into one JSON file

[wanton willow]

I've successfully fetched one page of data but I need all of the data from the server into a single json file. The url is this: https://ticketwala.pk/api/backend/events/public?page=1

My code is:

const url = "https://ticketwala.pk/api/backend/events/public?page=1"
const data = fetch(url)
.then(response => response.json())
.then(data => {
    const fs = require('fs')
    fs.writeFile('test.json', JSON.stringify(data), function(err) {
        if(err){
            return console.log(err)
        }
        console.log("file was saved")
    })
}).catch(error => {
    console.error(error)
})

If I use a loop, how do I know where the execution should stop? Is a loop the best way to do this? I'm new to javascript.

[mystic spoke]

response.status should be >200 and <300 on success. any other value means an error. not found (404) is the most likely, but it could be something else.

the other option is to wrap .json() in an try{...}catch(e){...}, but that is not the preferred way

i would do something like this:

async function get(url) {
    let responsedata = [];
    let page = 1;
    let response;

    while (true) {
        response = await fetch(`${url}?page=${page++}`);

        if (response.status >= 200 && response.status < 300) {
            const data = await response.json();
            responsedata.push(data);
        } else break;
    }

    return responsedata;
}

[lunar raven]

You want <400

300 series codes are often redirects or just totally fine.

[wanton willow]

i solved it, the json had a totalPages item

thank you

const fs = require('fs')
const url_base = 'https://ticketwala.pk/api/backend/events/public?page='

async function getEvents(api_link){
    let page = 1
    let allData = []
    response = await fetch(api_link + page)
    res_json = await response.json()
    console.log(res_json.items)
    allData.concat(res_json.items)
    console.log('first page data acquired')
    for(;page <= res_json.totalPages; ++page){
        response = await fetch(api_link + page)
        res_json = await response.json()
        allData.push(res_json.items)
        console.log(`page ${page} data acquired`)
    }
    fs.writeFileSync('allData.json', JSON.stringify(allData, null, 2))
}

getEvents(url_base)```

[wanton willow]

I'll keep this in mind if i come across a similar problem

[lunar raven]

This might be a good place for a do-while loop

const allData = []
let maxPages = null
let page = 0
do {
    page += 1
    const response = await fetch(api_link + page)
    const { totalPages, items } = await response.json()
    maxPages = data.totalPages
    allData.push(..items)
} while (pages <= maxPages)
// import fs from "fs/promises"
await fs.writeFile('allData.json', JSON.stringify(allData, null, 2))

[mystic spoke]

response.ok does the status check automatically and returns false on error

[lunar raven]

Also a solid approach

[wanton willow]

oh yes i forgot that existed in js lol

thank you

[wanton willow]

is this guaranteed for every case in paginated server data?

[lunar raven]

It has to be per api you interact with.

[wanton willow]

I'm just wondering

[mystic spoke]

it should be, if the backend follows http spec

[wanton willow]

alright alright

thank you

[mystic spoke]

(all servers do at least 200 and 404: bare minimum, but enough for response.ok)

[lunar raven]

I'm inclined also to handle the page parameter with a URL object: https://developer.mozilla.org/en-US/docs/Web/API/URL_API

So doing something like

const url = new URL("https://ticketwala.pk/api/backend/events/public")
function getEvents() {
  do { 
    url.searchParams.set("page", pageNumber) 
    res = fetch(url)
  }
}

[wanton willow]

i also saw this but i was confused about how i would include this in my code

i mainly use python and string operations took me back so i used what i knew haha