🔒 What are the most privacy centered LLMs?

[vestal kelp]

I am looking for an LLM API that does not store any data at all, not for training or for any temporary usage at all. Sort of something like a Zero Retention Policy where no data is stored or processed beyond the immediate request. I'm doing this cuz I want to build AI Agents for business with confidential business data where I can't afford the data being anywhere outside of the confidential files that the LLM can access to get the data. Can I somehow configure the OpenAI API to get this to work? Cuz they don't use our data for training models but they do indeed temporarily store it. If I can't do that then are there any alternative LLM APIs I can use to get this functionality? It should also be available to work with LangChain for the Agentic AI functionality.

[last pagoda]

@vestal kelp

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[halcyon summit]

a local LLM, that is the only one you can truly trust to not save any data

[vestal kelp]

That's true

What are some good local LLMs that I can use?

Like an LLM that I can download off of the internet and then use

[halcyon summit]

there are many to choose from at Hugging Face (their list of models)

[vestal kelp]

Please send the link to that, I've never used Hugging Face before

[halcyon summit]

https://huggingface.co/models

[vestal kelp]

Also, how good of a PC will I need to run these models locally?

I'm not sure how much the businesses would be open to spend on a PC to run a model locally

[vestal kelp]

Thanks a bunch :))

[halcyon summit]

it totally depends on the model and a lot of other parameters such as context window and such, as well as your requirements/tolerance for latency/response time

learning how to use one of them properly will be quite a rabbit hole to dive into

[vestal kelp]

For the requirements, I think at a level of "smartness" equivalent to GPT 3 or so should be fine

And the latency and response time, something slightly slower than GPT models should be fine, but not that much slower

Essentially no super significant difference in the response time from what general everyday users are used to when compared to other LLMs

[vestal kelp]

I think we are already in one when this conversation started

[halcyon summit]

yeah, that is true

buy you will have to get way deeper into the rabbit hole if you are going to go through with this alternative

[rose cedar]

I think you can do zero retention with Azure

[golden musk]

GPT 3 was very 'dumb' and wasn't even instruction tunned, you couldn't talk with it in chat form, only complete text - are you sure you mean that one?

the very first model available as ChatGPT was 3.5

[sturdy temple]

You can't change the regular OpenAI API to have zero data storage, but you can use Microsoft's Azure OpenAI Service, which lets you apply to turn off all logging. The only other way to be 100% sure is to self-host an open-source model (like Llama or Mistral) on your own private servers, which works great with LangChain too.

[golden musk]

AWS Beckrock and Google's Vertex AI also have similar guarantees about data privacy

[halcyon summit]

i think you will have to test several of them out and see which one you are happy with
maybe try the "openai/gpt-oss-20b" version and go from there (see if it's fast enough or good enough for you)

[vestal kelp]

Woops, I meant 3.5

I used 3.5 turbo for a project and that was pretty decent so I thought GPT 3 would be fine too 😭

[vestal kelp]

True

[halcyon summit]

google vertex ai do store some data for a certain amount of time https://docs.cloud.google.com/vertex-ai/generative-ai/docs/vertex-ai-zero-data-retention

[vestal kelp]

You mean Azure AI Foundary?

[vestal kelp]

Are you really really sure that they're safe for enterprise level uses when it comes to privacy?

[vestal kelp]

"for a certain amount of time"

One line that's very hard for developers to explain and justify to non dev people 😭

[sturdy temple]

azure is a good alternativ if ur not finding it easy

[golden musk]

that's up for your legal team to decide

[golden musk]

that page literally describes each way they store it, and how you can disable it

[vestal kelp]

Well, I'm screwed (I don't have one)

[vestal kelp]

You mean Azure AI Foundary?

[golden musk]

Azure has rebranded their AI offerings like 3 times or so
but yes, the current way to do it is via Foundry

[sturdy temple]

yep

[vestal kelp]

I see I see

Which do you think is better? AWS Bedrock or Azure AI Foundary?

[golden musk]

whichever one you already have a contract with

[vestal kelp]

None, I was discussing it with someone who is at a managerial position at a company and he said the reason many companies don't use LLM APIs is precisely cuz of the privacy concern

Which is why I'm looking for the better alternative

[golden musk]

if you don't have active accounts neither on AWS nor on Azure, what do you have accounts on? or are self hosting everything?

if the later, may as well self-host it too (e.g. run some Qwen or Gemma model around 10B parameters using vLLM )

[vestal kelp]

I have my account only on OpenAI API

I've not deployed any models or agentic AI yet

[golden musk]

Do your company not have anything hosted on the cloud?

Not specific to AI - any kind of website or anything

[vestal kelp]

See, here's the thing

I am looking for companies to work with 💀

I don't want to not have answers or knowledge when I do eventually start talking to them about this very soon

[golden musk]

Learn how to set it up for both AWS and Azure and Self-Hosting then, such that you can adapt to whatever they're using

for Azure you have to create a Foundry resource, inside of it select the model(s) you want to use, then create a 'deployment' for them and set the quota
for self-hosting you just pull the vLLM docker image and copy/paste a docker run command specifying which model you want to use, be careful about the context window though
for AWS idk

either way in the end ideally you just grab an OpenAI compatible API URL + an API Key and throw it in your program's environment variables to configure which backend to use

some providers might use a different API, but LangChain, pydantic.ai and most other major frameworks have utilities that build a client with an unified interface, at most you just have to create a separate object for the backend and pass it

[sturdy temple]

For AWS, you just use their Amazon Bedrock service, which gives you one API for lots of models, and you can get a key and a special URL that makes it look just like OpenAI. You're right that for Azure you just create a "deployment" to get your endpoint, and self-hosting is just running a vLLM docker command to create your own API. In the end, you just grab that API URL and key, put them in your environment variables, and LangChain can then use any of them without a problem.

[ocean cedar]

unless you have a very beefy rig, ideally you should choose a model that excels at your specific task, rather than look for an all powerful general purpose model

[vestal kelp]

Damn, so if I understood correctly, if I use bedrock, then I get an API key that I can use for many models that I can simply use in Langchain??

And I looked it up and it seems bedorck doesn't have the main GPT models, is that true?

[vestal kelp]

That sounds crazy difficult

[steel imp]

Only MS can resell open AI models

but also the open AI models are almost never the best you can do, so that's generally not an issue

[golden musk]

it really isn't that difficult though, although you might be biting more than you can chew - maybe learn some basic DevOps first if you're not familiar with Docker and Azure/AWS

(but to be fair, there is a fair amount of things you have to learn... none of which are particularly difficult on their own, but it can get a bit complex once it piles up)

[rose cedar]

I know self-hosting is the only sure-fire way, but I imagine that's not feasible for a lot of people.

[last pagoda]

Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.