🔒 API Limit, getting the same data

[opaque plaza]

Hello,

I am currently trying to retrieve data from an API. The rate limit is 100 requests per minute.

I am currently using the following code:

import requests
import time
import json

def get_data(url):
    headers = {
        'Accept': 'application/json'
    }
    response = requests.get(url, headers=headers)
    if response.status_code == 200:
        return response.json()
    else:
        print(f"Failed to fetch data from API. Status code: {response.status_code}")
        return None

def fetch_data_with_rate_limit(url):
    all_data = []
    while True:
        response_data = get_data(url)
        if response_data:
            all_data.append(response_data)
            time.sleep(0.6)  # Pause for 0.6 seconds between requests (100 requests per minute)
        else:
            break
    return all_data

def save_data_to_json(data, filename):
    with open(filename, 'w') as json_file:
        json.dump(data, json_file)

api_url = "https://www.api-url.com/"
all_data = fetch_data_with_rate_limit(api_url)

save_data_to_json(all_data, 'data.json')

My problem is, if I just fetch once I get only 100 dictionaries containing data. But using the provided code it looks like I get the same data over and over, which results in an infinite loop.

Can someone help me with a hint, how I can solve this issue? The data should consist of multiple thousands of data points and I want to retrieve all of them.

I hope this does not violate rules, since I want to overcome the api rate limit. But since I am currently waiting for 0.6 seconds per requests this should be fine.

[pale verge]

@opaque plaza

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

Hey @opaque plaza!

It looks like you pasted Python code without syntax highlighting.

Please use syntax highlighting to improve the legibility of your code and make it easier for us to help you.

To do this, use the following method:
```py
print('Hello, world!')
```

This will result in the following:

print('Hello, world!')```
You can **edit your original message** to correct your code block.

[opaque plaza]

import requests
import time
import json

def get_data(url):
    headers = {
        'Accept': 'application/json'
    }
    response = requests.get(url, headers=headers)
    if response.status_code == 200:
        return response.json()
    else:
        print(f"Failed to fetch data from API. Status code: {response.status_code}")
        return None

def fetch_data_with_rate_limit(url):
    all_data = []
    while True:
        response_data = get_data(url)
        if response_data:
            all_data.append(response_data)
            time.sleep(0.6)  # Pause for 0.6 seconds between requests (100 requests per minute)
        else:
            break
    return all_data

def save_data_to_json(data, filename):
    with open(filename, 'w') as json_file:
        json.dump(data, json_file)

api_url = "https://www.api-url.com/"
all_data = fetch_data_with_rate_limit(api_url)

save_data_to_json(all_data, 'data.json')

I am sorry, for some reason I could not add syntax highlighting to the original message, you can see the code with syntax highlighting above.

[crystal lichen]

you are making requests to the exact same URL, with the same query parameters and headers, every single time?..

Read the API's documentation - it probably supports some sort of pagination you are not using

[pale verge]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.