🔒 FastAPI/HTML Rendering Issues | Python | Page 1

wary nova Jun 23, 2024, 12:40 AM

#

I have a discord.py bot that has an avatar history command. When a user updates their avatar on discord, it keeps a record of it in a PostgreSQL database. The avatar history command generates a link that when clicked, takes you to my API and shows HD/Animated version of all of the avatars for that user.

For my user, I have about 130 in the database and it loads fine, but for one bot, there are about 600 pictures in the database for this bot. When you try to load the api link, it loads for about 30 seconds and then it goes to a 502 Bad Gateway page.

Where might the issue be, and how can i fix this issue?
My process works, so I want to keep the process the same if possible, but also optimize it if possible.

Here is the API code
api.py

@waywardapi.get("/avatarhistory/{user_id}", response_class=HTMLResponse)
async def get_avatar_image(user_id: int, request: Request, temp_ref_id: Optional[str] = Query(None, alias="reference_id")):
    async with pool.acquire() as conn:
        avatar_rows = await conn.fetch(
            "SELECT avatar_data, format, user_name FROM scrap_avatar_history WHERE user_id = $1 ORDER BY timestamp DESC",
            user_id
        )

    avatars = []
    for row in avatar_rows:
        avatar_data = row['avatar_data']
        file_format = row['format']
        username = row['user_name']
        is_animated = file_format == 'gif' 
        avatar = {
            'image_data': avatar_data,
            'is_animated': is_animated,
            'username': username
        }
        avatars.append(avatar)

    if avatars:
        img_bytes_value = avatars[0]['image_data']
        return templates.TemplateResponse("avatar.html", {"request": request, "avatars": avatars, "image_data": img_bytes_value})```

HTML
avatar.html
```html
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <title>{{ avatars[0].username }}'s Avatar History</title>
    {% if avatars %}
        <link rel="icon" type="image/png" href="data:image/png;base64,{{ avatars[0].image_data|b64encode }}">
    {% endif %}
    <!-- <link rel="icon" type="image/x-icon" href="/favicon.ico"> -->
    <!-- <link href="css/avs.css" rel="stylesheet"> -->
  </head>
  <body>
    <div class="myGallery">
      {% for avatar in avatars %}
        {% if not avatar.is_animated %}
          <img src="data:image/png;base64,{{ avatar.image_data|b64encode }}" width="184.5" height="184.5" class="avatar-image" alt="animated avatar">
        {% else %}
          <img src="data:image/gif;base64,{{ avatar.image_data|b64encode }}" width="184.5" height="184.5" class="avatar-image" alt="avatar">
        {% endif %}
      {% endfor %}
    </div>
  </body>
</html>

potent crystalBOT Jun 23, 2024, 12:40 AM

#

wary nova I have a discord.py bot that has an avatar history command. When a user updates ...

#

@wary nova

Python help channel opened

Remember to:

Ask your Python question, not if you can ask or if there's an expert who can help.
Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

frail tulip Jun 23, 2024, 12:44 AM

#

Have you tried adding debug prints?

wary nova Jun 23, 2024, 12:45 AM

#

frail tulip Have you tried adding debug prints?

im not sure where to add the debug prints since I know it works

frail tulip Jun 23, 2024, 12:46 AM

#

Try adding one after retrieving data from db and one after parsing it to avatars list

#

Also maybe add timestamps to it

#

And try running that one with 600 images

wary nova Jun 23, 2024, 12:53 AM

#

frail tulip And try running that one with 600 images

so here is what it printed when it went to the 502 Bad Gateway page

#

Hi
OK
Killed

#

I added debug print statements for hi and ok

#

not killed

#

@waywardapi.get("/avatarhistory/{user_id}", response_class=HTMLResponse)
async def get_avatar_image(user_id: int, request: Request, temp_ref_id: Optional[str] = Query(None, alias="reference_id")):
    async with pool.acquire() as conn:
        avatar_rows = await conn.fetch("SELECT avatar_data, format, user_name FROM scrap_avatar_history WHERE user_id = $1 ORDER BY timestamp DESC", user_id)
        print("Hi")

    avatars = []
    for row in avatar_rows:
        avatar_data = row['avatar_data']
        file_format = row['format']
        username = row['user_name']
        is_animated = file_format == 'gif' 
        avatar = {
            'image_data': avatar_data,
            'is_animated': is_animated,
            'username': username
        }
        avatars.append(avatar)
    
    print("OK")

    if avatars:
        img_bytes_value = avatars[0]['image_data']
        return templates.TemplateResponse("avatar.html", {"request": request, "avatars": avatars, "image_data": img_bytes_value})```

frail tulip Jun 23, 2024, 12:55 AM

#

@waywardapi.get("/avatarhistory/{user_id}", response_class=HTMLResponse)
async def get_avatar_image(user_id: int, request: Request, temp_ref_id: Optional[str] = Query(None, alias="reference_id")):
    async with pool.acquire() as conn:
        avatar_rows = await conn.fetch("SELECT avatar_data, format, user_name FROM scrap_avatar_history WHERE user_id = $1 ORDER BY timestamp DESC", user_id)
    
    print(len(avatar_rows))

    avatars = []
    for row in avatar_rows:
        avatar_data = row['avatar_data']
        file_format = row['format']
        username = row['user_name']
        is_animated = file_format == 'gif' 
        avatar = {
            'image_data': avatar_data,
            'is_animated': is_animated,
            'username': username
        }
        avatars.append(avatar)
    
    print(len(avatars))

    if avatars:
        img_bytes_value = avatars[0]['image_data']
        return templates.TemplateResponse("avatar.html", {"request": request, "avatars": avatars, "image_data": img_bytes_value})```

#

Do you have else for the if avatars:?

wary nova Jun 23, 2024, 12:55 AM

#

no

frail tulip Jun 23, 2024, 12:58 AM

#

You should add handling if they send invalid user_id or your db search fails, and send them error code

wary nova Jun 23, 2024, 1:00 AM

#

frail tulip You should add handling if they send invalid user_id or your db search fails, an...

i mispoke, its 2639 images

#

for now I just added this

#

@waywardapi.get("/avatarhistory/{user_id}", response_class=HTMLResponse)
async def get_avatar_image(user_id: int, request: Request, temp_ref_id: Optional[str] = Query(None, alias="reference_id")):
    async with pool.acquire() as conn:
        avatar_rows = await conn.fetch("SELECT avatar_data, format, user_name FROM scrap_avatar_history WHERE user_id = $1 ORDER BY timestamp DESC", user_id)
    
    print(len(avatar_rows))

    avatars = []
    for row in avatar_rows:
        avatar_data = row['avatar_data']
        file_format = row['format']
        username = row['user_name']
        is_animated = file_format == 'gif' 
        avatar = {
            'image_data': avatar_data,
            'is_animated': is_animated,
            'username': username
        }
        avatars.append(avatar)
    
    print(len(avatars))

    if avatars:
        img_bytes_value = avatars[0]['image_data']
        return templates.TemplateResponse("avatar.html", {"request": request, "avatars": avatars, "image_data": img_bytes_value})
    else:
        print(len(avatar_rows))```

#

INFO:     Started server process [870291]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
2639
2639
Killed```

frail tulip Jun 23, 2024, 1:06 AM

#

What is the size of one avatar?

wary nova Jun 23, 2024, 1:08 AM

#

frail tulip What is the size of one avatar?

it varies but the highest ive seen is about 350 kb

#

some of the gifs are around 850 kb

frail tulip Jun 23, 2024, 1:10 AM

#

Maybe try limiting amount of avatars

        return templates.TemplateResponse("avatar.html", {"request": request, "avatars": avatars[:500], "image_data": img_bytes_value})

#

Could be that you are hitting some kind of limit somewhere else in your program

wary nova Jun 23, 2024, 1:11 AM

#

frail tulip Maybe try limiting amount of avatars ```py return templates.TemplateResp...

I limit here to a 100 for the image in the embed, but the purpose of the html page on the api is to show a full history

#

im using nginx

#

fastapi

frail tulip Jun 23, 2024, 1:18 AM

#

That avatar.html is filled with images on server?

wary nova Jun 23, 2024, 1:26 AM

#

frail tulip That avatar.html is filled with images on server?

the avatar.html loads the picture grid

#

frail tulip Jun 23, 2024, 1:36 AM

#

What is the size of the response with all images

wary nova Jun 23, 2024, 1:41 AM

#

frail tulip What is the size of the response with all images

I am not sure, ill need to add some code to calculate

#

for that user, its less than 1gb

#

roughly

frail tulip Jun 23, 2024, 1:41 AM

#

You should probably load them with js one at time or in batches

#

Page will load faster

#

And images will load after

#

You probably don’t even need any client side script and just use <img src=“url”>

wary nova Jun 23, 2024, 1:51 AM

#

frail tulip You should probably load them with js one at time or in batches

how might i do that vs my current approach? Im still fairly new to all this lol

#

I wonder if any of the issue is with the db table itself? I feel like its fine, but then again i set it up awhile ago

#

#

frail tulip Jun 23, 2024, 1:56 AM

#

You would send small html page that loads simple JavaScript script that would make new get or post requests to your server for images from the db

frail tulip Jun 23, 2024, 2:02 AM

#

wary nova I wonder if any of the issue is with the db table itself? I feel like its fine, ...

Db looks fine, not sure what is causing it exactly but according to internet 502 is received when nginx receives invalid response from upstream server. So the problem could be somewhere else in Python part of your app

potent crystalBOT Jun 23, 2024, 3:02 AM

#

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.

#🔒 FastAPI/HTML Rendering Issues