🔒 deployment of an ai app

[harsh totem]

So i have this chatbot i built, iam using one of gemini's models and i want to deploy the thing iam confused on how i can deploy it properly since i dont want to like just share my credentials with everyone on the internet any tips and advice. the applications pretty simple just a very basic website with a chat interface, i have a database where iam storing my embeddings and chunks thats on this website called pinecone my code just takes those embeddings and runs the llm on it. Iam just getting started with llm's so i thought this would be a pretty good starting point. I was gonna deploy this on a free platform like railway or something. if theirs something better would love to know it.
Currently iam just loading my api keys from the .env file locally so i need help figuring out how to deploy my code

[torpid vapor]

@harsh totem

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[nocturne bridge]

unless the service provider has a secrets management service that you can use most people just use the .env file as far as i know, just put it somewhere where other people can't access it, as long as you trust your hosting provider that is

[potent pelican]

You should be able to pull just from the .env. That's what all of my projects always use. I just make sure things like API keys are stuck on the backend that the frontend connects to so that they don't have any access what so ever to the API keys. I don't know exactly how your stack is set up but that's how I do it. So basically you would put up your backend on something like AWS, or a custom server or whatever, and then have a website hosting service that just points to the API URL of the backend

[harsh totem]

ok cool that just works on its own, like what if i have a bunch of people using it at once. This was for a project in my class so during the final presentation a good chunk of the class would connect to the app. would that cause an issue or would it be ok?

sorry for the ping i frgot to disable it

[potent pelican]

It depends on how its set up. That's in the world of scalability and distributed systems which can be a lot more complicated. Just to use AWS as an example because it's what I specifically work with, I can set up what's basically storage on one of their computers to put my website onto (literally just the HTML/CSS/JS code that the user interacts with). Then I'd use what's called a lambda on the backend which is basically just a function that is stored on a server somewhere that would call the AI model and return the AI models response to the user. So the flow would be user -> website -> lambda -> AI call -> lambda -> website -> user.

That's a slightly overcomplicated explanation I think so if it doesn't make super good sense I apologize but essentially thats how it would work for me. But that setup would automatically scale meaning that you could have 1M+ people call it all at once and each one would receive a response in a pretty decent time depending on how its setup.

But on the other hand, say you run it from your local computer, it can only handle one request at a time because it's only 1 computer hosting the content rather than thousands having copies. Assuming that your call to the AI stops the computer from computing any other requests, then each request depends on how long the requests before it take.

[harsh totem]

do you have any resources to learn more about this. I have a bunch of free credits on google cloud so ill try their

and wow deployment really gets annoying

[potent pelican]

Yes and no lol. If you're willing to spend some money then Udemy is always decent to learn a new skill (though wait until it goes on sale because $100+ courses are insane). But at the same time, if you're doing this for a hobby project and not because you want to become a cloud architect or something, then just watching videos on how to host web applications and backends should be sufficient.

[potent pelican]

It can. A lot of it depends on the methods that you want to use, how much scalability you want/need, use cases, etc. But overall, once you get a hang of it, it becomes a lot easier and thankfully, once its set up, your done, you just have to upload the new files

[harsh totem]

dam 100 dollars wow thats insane

[potent pelican]

Yeah, but often the go on sale for $10-$20

[harsh totem]

well one of the requirements for the class is to have a deployed project so iam gonna ahve to figure it out somehow

[potent pelican]

Maybe 1-5 times a month. And theyre usually on sale for like 2-3 days when you sign up with a new account as a like new comer discount

[harsh totem]

ill check it out thanks

[potent pelican]

So if it's just the frontend you need deployed, look at something like github pages. It's super easy to set up the frontend because it's built into github. You literally just press launch. If you need a backend, then I'd look at some platforms that can run the backend and how to use them. Personally, as I've said before I work with AWS a lot which is a mix of their API gateway and Lambda, but there are hundreds of platforms that might suit you better as they are all on a case by case basis.

[harsh totem]

cool

can i drop u a dm on discord later if i get stuck or have more questions

[potent pelican]

Yeah of course.

[harsh totem]

awsm thank you

[nocturne bridge]

that isn't correct, as long as you don't build it completely wrong so that it blocks the whole server it wouldn't just serve one person just because it's "only" running on one computer

[potent pelican]

That's why I said if you build it in a blocking way. I.e not async. From what I can tell this is a coding project that comes from a class and depending on what classes that they've taken, they haven't learned async practices yet. Considering they're just getting into deployment, from what my school has taught they haven't gotten into async just yet or have just started. If I'm wrong then please correct me but that's my understanding of it?

[harsh totem]

yeh so my degree is in ml and AI so at least as of today we havent done much if any of like core stuff like i know what async is but havent implemented it in anything yet

[potent pelican]

Then yeah, unless my understanding is wrong, if you're using synchronous methods, then you should only be able to handle one request at a time.

Idk, maybe my fundamental understanding is wrong though in which case I gotta take a major refresher lol.

Ok, I just did a mini search and I was wrong but only to a certain extent. It depends on threads. If you run out of threads, THEN it blocks another request until a thread is freed. So much more hardware/low-level programming theories

[harsh totem]

ok that makes sense

[nocturne bridge]

except async there is also things like threads or even multiprocess that the server can use so that each request that it processes will not freeze up other requests, it all depends on the framework and how you use it

[harsh totem]

yeh talking to real people is deffinately better than an ai. Chatgpt just kept ranting about serverless stuff

[potent pelican]

Ah okay, well when it comes to that stuff, I'll defer to you 100%. Need to take a major refresher course in how computers handle request flows. I've mainly been doing game development recently so haven't been working a ton with backend stuff beyond minor stuff that unity interacts with.

[harsh totem]

ok so right now iam just serving the html page with flask. i had gpt write the code for that since i dont really do frontend. its all async as far as i can see

[potent pelican]

It might just be easier to host the html page with github pages. No need for flask or a server or anything and its free. I can send you a video on how to do that unless rndpkt has a better solution on how to continue using flask or another method?

[nocturne bridge]

but if you are using a async server and doing blocking calls without any speciall handling, then you are completely, you'll be in big trouble

[harsh totem]

i would deffinately like to take a look yeh

[nocturne bridge]

but the backend stuff that is accessing the backend ai service that needs the secrets needs something like Flask or similar

[potent pelican]

Yeah but if they're just hosting the html page, there isn't a reason for flask on the html page unless you wanted to do server side rendering with it for some reason which you would just use a framework for instead I believe?

[nocturne bridge]

one more potential problem with having multiple people using your service would be that they might use up all your quota for your account on the backend ai service provider that you are using or that you get runaway costs

[nocturne bridge]

that is right, as long as there isn't any server side generated content, which it would be of some kind for a backend service that talks with the ai service, even if it's just as an api

[harsh totem]

well its going to be like 20 people tops

[potent pelican]

Well I mean couldn't you use javascript to just dynamically update based on the answer from the API? I haven't done much with server side rendering but if javascript makes a call to the backend and awaits the response, once that response comes back it can just select that element that needs to be updated and set it's text to whatever the chatbot responds with? No?

[potent pelican]

You're most likely not going to run into any performance issues then as long as the server wasn't made in like 2000

[harsh totem]

ok uk what i think ill just make the thing and try deployment

and see how that goes

when things break ill just come back here

[potent pelican]

Probably the best idea yeah. Otherwise you end up doing a deep dive on one specific thing that might not even apply to you lol

[harsh totem]

exactly

because my god you guys stopped making sense a while ago

time to go back to stack overflow ig lol

[potent pelican]

Good luck lol. Come back if you have any further questions

[harsh totem]

yup ull definately be seeing more of me

[nocturne bridge]

sure, but you need to run you api somewhere, you don't want the front-end talking with the ai providers service (such as Gemini) directly as the they would have to have access to your secrets (tokens, passwords or whatever)

[nocturne bridge]

go for it!

[harsh totem]

thanks a lot for the help

[nocturne bridge]

the most important thing is to go and make stuff, the wonderful thing with software is that you can mostly fix things if you break then, so you don't need to worry too much
sure you want to have backups that are safe and you don't want any sensitive information leaking, but other then that you are most of the time pretty safe

[harsh totem]

aye aye captn

[torpid vapor]

Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.