#LangChain.js and Vertex problems
crisp helm
Moving to a thread since this could be long.
Well... Gemini and LangChain.js is a bit of a specialty of mine. {:
I have a meeting in 3 minutes, but can try to help more after that.
What exactly are you having problems with?
(Just the quota / rate limit?)
slim coral
Yeah it is a better idea..
First things first we are meyou.ai and we are trying to build an AI companion that is adaptable an easy to use for everyone without prompting that connects to External APIs such as Dall-E, Google News and to some of our own APIs.
We have been using gpt-4-1106-preview model until some point but we have migrated to Gemini 1.5 due to the cost of GPT models and also we believe Gemini 1.5 was more promising in the long run.
And of course we always used Langchain from the beginning for the tools. The minute we have migrated to Gemini it forced us to use the VERTEX API because non of our tool did not work with the [ChatGoogleGenerativeAI]
VERTEX API works fine but the rate is limited to "5" request per minute.
This is how we use it..
import { ChatVertexAI } from '@langchain/google-vertexai';
const model = new ChatVertexAI({ maxConcurrency: 8, modelName: 'gemini-1.5-pro-latest', temperature: 0.8 });
crisp helm
Sounds cool!
Regarding Langchain.js:
- Are you having any issues with langchain and gemini itself?
- I'm one of the major contributors to the google-common package. So if you do see issues - I at least have a reasonable idea where to look.
Regarding the rate limits:
- The 5 is definitely meant as a "as you get started" level. I've seen people get that raised already. If it is telling you to contact Sales - you may need to, as I mentioned.
- That quota is per region. So one trick you can do (at least for testing, I wouldn't do this in production) is have different devs use different regions while working.
(hmm... but I guess that doesn't address the concurrency part()
slim coral
Hello Allen,
I hope you're well. We're still awaiting an update on the rate increase from Google Support.
On a related note, I came across a recommendation from LangChain about accessing Google's Gemini models, including gemini-pro and gemini-pro-vision, through the GoogleGenerativeAI class. However, this method requires the use of Vertex AI, which isn't compatible with our current tools.
In our previous discussions, you mentioned there were some complications with this approach. Do you have any alternative suggestions that could help us effectively test the Gemini Pro models with our existing tools without implimentin Vertex AI?
Thank you for your guidance.
crisp helm
Wait, I'm lost.
I thought you needed Vertex AI due to your other tools.
The google-genai package only works with an API Key from AI Studio.
It also has rate limits.
The google-vertex package works with Vertex AI
The google-gauth package will work with both. (the google-vertex package is a thin veneer over it forcing Vertex.)
slim coral
We've been using the Google Generative AI package via the Google API from AI Studio, configured with our server account, and there haven't been any issues on that front. However, we're encountering a problem where our tools aren't working with this setup. Interestingly, we're not receiving any error messages; it's as if the tools simply do not exist.
Due to this issue, we transitioned to using Vertex AI, although this wasn't our original plan. This move was aimed at ensuring compatibility and functionality of our tools.
Could you suggest how we might address this to either make our original setup work as intended or optimize our use of Vertex AI?
crisp helm
I'm not sure what needs optimizing. If you're talking about quota - you probably have a better quota on Vertex AI than you do on Generative AI / AI Studio.
If you want to try both with the same classes and see if one works better, you can try using the "@langchain/google-gauth" package. This uses a different library than the Google Generative AI library, so may work with your tools. You can also use it to more easily switch between the Generative AI / AI Studio platform and the Vertex AI platform.
But I think the bigger question is why it isn't working with what you had before, why you think switching to Vertex would solve that, and what issues, specifically, you're having with Vertex other than the quota.
slim coral
type": "service_account",
"project_id": "meyou-374719",
"private_key_id": "###################",
"private_key": "###########",
"client_email": "meyou-692@meyou-374719.iam.gserviceaccount.com",
"client_id": "##############",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/meyou-692%40meyou-374719.iam.gserviceaccount.com
This our setup.. the problem here is we are forced to use the Vertex AI in order to use our tools.. We can not use Geneative AI/ AI studio API alone if we want to use our tools. If google set this up like this we have no choice of course and we ll just have to wait for a quota increase. (yet it has been 4 businees days and got no response from them even though we did exactly what they asked us.)
crisp helm
You said:
Could you suggest how we might address this to either make our original setup work as intended or optimize our use of Vertex AI?
So I was trying to address that question specifically.
What you've said about using AI Studio is that the tools don't work. You haven't told me enough to help you diagnose exactly why. I suggested using a different langchain.js package because I know that package works with both Vertex and AI Studio, and I was speculating that, perhaps, it means it would work with your tools.
slim coral
It appears that the issue with the quota problem was resolved, stemming from a simple oversight on our part where we requested a quota increase for an incorrect identifier. This mistake led to unnecessary frustration and confusion.
For future reference and to assist anyone who might face a similar issue, the correct name for the quota increase is as follows:
"Generate content requests per minute per project per base model per minute per region per base_model."
Additionally, for users of LangChain, it's important to utilize Vertex AI to ensure optimal functionality of your tools.
Thank you Allen and sorry if we have taken your time unnecessarily.