Vertex and Langchain, notes and thoughts

[broken loom]

So I had a chance to dive into this a little deeper and wanted to share what I found and some thoughts.

This POV comes from a user wanting to build and deploy a Next.js app

You are correct with GOOGLE_VERTEX_AI_WEB_CREDENTIALS, and it is in the langchain documentation. It seems to take the env variable as a full .json file

  get lc_secrets(): { [key: string]: string } {
    return {
      "authOptions.credentials": "GOOGLE_VERTEX_AI_WEB_CREDENTIALS",
    };
  }

The concern with this is it becomes difficult to secure a .json file like this in a repo and for deployment. This leaves the user having to upload their service-account.json into their repo for deployments to work.

It seems like a better approach would be to encode the service-account.json file into a base64 string and store as a env variable.

This worked for me

  const credential = JSON.parse(
    Buffer.from(
      process.env.CLOUD_RUN_SERVICE_ACCOUNT as string,
      "base64"
    ).toString()
  );

  const model = new ChatGoogleVertexAI({
    authOptions: {
      credentials: {
        project_id: credential.project_id,
        client_email: credential.client_email,
        private_key: credential.private_key,
        token_uri: credential.token_uri,
      },
    },
    temperature: 0.4,
  });

I also tested using the @google-cloud/vertexai library in a Next.js app. I couldn't find a way to add credentials directly to use VertexAI in a deployment on Vercel. So I experimented with putting the Next.js app in a container and deployed it to Cloud Run.

Doing this I also needed to create a base64 string of the .json file to deploy in a GitHub workflow. This added the needed auth to be able to use the VertexAI class in a Next.js app.

  const vertexAI = new VertexAI({
    project: process.env.CLOUD_RUN_PROJECT_NAME!,
    location: "us-central1",
  });

  const generativeModel = vertexAI.preview.getGenerativeModel({
    model: `gemini-pro`,
  });

This opened up more options, I could then use video in the gemini-pro-vision model easily locally. (Using it deployed didn't work because of the length of time ended the request too early, not really a concern for this discussion, it's more of a Next.js/Cloud Run issue)

[oblique turret]

It seems like a better approach would be to encode the service-account.json file into a base64 string and store as a env variable.

Why encode it as base64, tho, and not just leave it as the contents of the json file? So set the env variable to the contents of the json file?
(Which is how I thought it was done with the "web" version of the Langchain Vertex library.)

[broken loom]

So the env variable is pointing to the json file itself?

[oblique turret]

no no no.
The env variable contains the same string that is in the json file.

(At least I think that's how it's supposed to work)

There are two versions of each class (for example, ChatGoogleVertexAI)

• One version that you get from chat_models/googlevertexai that uses the google-auth-library and assumes that GOOGLE_APPLICATION_CREDENTIALS points to the json file that has credentials
• One version that you get from chat_models/googlevertexai/web that uses the web-auth-library and assumes that GOOGLE_VERTEX_AI_WEB_CREDENTIALS contains the credentials

[broken loom]

I could try that, but it seemed like the string would be too long as a env variable?

[oblique turret]

(The one that uses google-auth-library will also sometimes magically get the credentials from using gcloud commands or other magic when running in a gcloud environment)

[oblique turret]

Well... two things

• I believe there is no limit on a single environment variable, tho I forget what the max environment space size is. But it's in the thousands of characters.
• base64 encoding it would make it even larger.

[broken loom]

very true 🤔

maybe it was the excess whitespace, /n and double quotes I was getting thrown of with?

I just tried GOOGLE_VERTEX_AI_WEB_CREDENTIALS locally and deployed with the entire JSON in a string and it seems to work fine. I'm not sure why it didn't work like this for me before which sent me down this rabbit hole, my apologies! Although I did learn how to deploy a Next app on Cloud Run 😅

[oblique turret]

could be you were using the other version of the class. Or a typo. Or who knows what. {:

Most importantly - glad you got it working!

And I had the chance to look at that code more closely so I can make some improvements in the new version.

[broken loom]

I ended up creating a blog and example repo using Next.js 😅
https://github.com/e-roy/next-cloud-run-app

I was excited to have options to not be confined to Vercel for all deployments 😃

[broken loom]

wow, that's a ton of work! To try it from a user POV, I attempted to use the packages by cloning your fork and link locally, but I couldn't figure out setting the dependency portal: properly 😅

Hopefully they fast track the blockers for you so you can get this through!

[oblique turret]

Fortunately, they're very open to incorporating it. Or at least were when I told them about it. They just have a backlog at the moment.

But I hope this goes through relatively quickly. I think there is some discussion still to be had, but I'm hopeful I addressed most of the issues.

And thank you for your support. {:

[broken loom]

Yeah, they've got a lot on their plate over there. It looks like Google is the first to add large data to the message structure for LLMs? Maybe that is what is slowing them down a little?

[oblique turret]

Not sure what you mean by "large data to the message structure".

Tho I suspect it's that I dropped 2000 lines on them and a dependency on an earlier (much much smaller) PR that they haven't pulled in yet. They had a backlog of PRs while they were getting the 0.1 release and the new project structure out, plus the holidays, so theyre catching up now. And mine is a big big change, so they're tackling smaller stuff first.

[broken loom]

oh sorry, images and video data in the message object of the array. I've only seen it separated as a prompt and the image data, but not combined into the message object.