Hi @errant warren. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
#streamText requires model, but if using lightning rag api can I use AI SDK?
misty warrenBOT
Yes, you can use the AI SDK with a custom model implementation when working with a Lightning RAG API or any other custom AI service. The AI SDK is designed to be flexible and work with various model providers.
When using a custom API like Lightning RAG, you need to create a custom model implementation rather than using one of the pre-built providers. Here's how you can modify your code:
import { streamText } from 'ai';
import { createAI } from '@ai-sdk/core';
// Create a custom model implementation for your Lightning RAG API
const lightningRag = createAI({
async doTextGeneration({ prompt }) {
// Implement your Lightning RAG API call here
const response = await fetch('your-lightning-rag-endpoint', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ query: prompt }),
});
// Return the response stream
return response.body;
}
});
export async function POST(req: Request) {
const { query } = await req.json();
return streamText({
prompt: query,
model: lightningRag('your-model-name'),
}).toDataStreamResponse();
}
The AI SDK provides a unified interface for working with different AI models through the streamText function, as shown in the Vercel documentation. You can create custom model implementations using createAI from the @ai-sdk/core package to integrate with any API, including your Lightning RAG API.
For more information on using the AI SDK with custom implementations, you can refer to the Vercel AI SDK documentation.
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
errant warren
@misty warren I found createAI with ai/rsc
but Im getting:
ts: Type 'Promise<ReactElement<unknown, string | JSXElementConstructor<any>>>' is missing the following properties from type 'LanguageModelV1': specificationVersion, provider, modelId, defaultObjectGenerationMode, and 2 more. [2740]
ts: Argument of type 'string' is not assignable to parameter of type 'AIProviderProps<any, any, {}>'.
//code
import { streamText } from 'ai';
import { createAI } from 'ai/rsc';
const lightningRag = createAI({
async doTextGeneration({ prompt }) {
const response = await fetch('your-lightning-rag-endpoint', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ query: prompt }),
});
return response.body;
}
});
export async function POST(req: Request) {
const { query } = await req.json();
return streamText({
prompt: query,
model: lightningRag('lightning-rag'),
}).toDataStreamResponse();
}
@misty warren found createAI with ai/rsc
but Im getting:
ts: Type 'Promise<ReactElement<unknown, string | JSXElementConstructor<any>>>' is missing the following properties from type 'LanguageModelV1': specificationVersion, provider, modelId, defaultObjectGenerationMode, and 2 more. [2740]
ts: Argument of type 'string' is not assignable to parameter of type 'AIProviderProps<any, any, {}>'.
//code
import { streamText } from 'ai';
import { createAI } from 'ai/rsc';
const lightningRag = createAI({
async doTextGeneration({ prompt }) {
const response = await fetch('your-lightning-rag-endpoint', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ query: prompt }),
});
return response.body;
}
});
export async function POST(req: Request) {
const { query } = await req.json();
return streamText({
prompt: query,
model: lightningRag('lightning-rag'),
}).toDataStreamResponse();
}
misty warrenBOT
Let me check the knowledge base for you again. Give me one moment! :books:
I see the issue with your code. You're mixing two different concepts in the AI SDK: the createAI function from ai/rsc is meant for creating AI components for React Server Components, not for creating custom model providers.
For creating a custom model provider to use with streamText, you should use createModel from ai instead. Here's how to fix your code:
import { streamText, createModel } from 'ai';
// Create a custom model implementation for your Lightning RAG API
const lightningRag = createModel({
provider: 'lightning-rag',
models: {
'lightning-rag': {
doTextGeneration: async ({ prompt }) => {
const response = await fetch('your-lightning-rag-endpoint', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ query: prompt }),
});
return response.body;
}
}
}
});
export async function POST(req: Request) {
const { query } = await req.json();
return streamText({
prompt: query,
model: lightningRag('lightning-rag'),
}).toDataStreamResponse();
}
The error you're seeing is because createAI from ai/rsc is designed for a different purpose - it's used for creating AI components in React Server Components, as seen in the AI SDK RAG Template where it's used to create a chatbot component.
The streamText function expects a model that implements the LanguageModelV1 interface, which is what createModel provides, not what createAI from ai/rsc provides.
For more information on streaming responses with the AI SDK in Next.js route handlers, you can refer to the Next.js documentation on streaming.