After coding the openAI and Gemini hooks, I decided to also make a hook for Claude. Important note: The openAI and Gemini hook speak to the API directly. For the Claude hook I had to use an API route like Vercel AI SDK, because of their wired CORS politics. And guess what? Same problem as with the Vercel AI SDK. Locally the streams works, when accessing the page from a client computer, streaming breaks and only the final result gets submitted.
I have to go back on step, the Vercel AI SDK is NOT the issue, but Next.js. Of course, I added the magic export const dynamic = "force-dynamic" line to the route.
I use a pretty basic Next.js setup, didn't change anything fundamental, next.config.js is empty.
If anyone more involved in Nextjs is read this, who can I address this issue/question to? @lgrammel
BTW: Happy to share the code, for anyone interested.

@tetaplaun thanks for investigating! It would be super helpful if you could share your code so I can try it out myself

Hook
Api Route

Just as a note again, I'm running this project self-hosted on node 18.17.0

Thanks for further investigating. It's good to know that the problem is in the network setup and not in the libraries.

Hi,

I was not able to get streaming working with a custom fine-tuned model using the google-vertex provider in the ai sdk. All the text appears in chunks. In general, I am not clear whether something needs to be set on the vertex AI deployment server to enable streaming or if all the magic happens in the AI sdk code? Any help would be greatly appreciated.

Since you wrote your own hooks, any chance you might be open to sharing some code on how you got this working? As noted, I have a custom model I fine tuned and then deployed to vertex AI and I'm using the google-vertex provider. Here is some simple code I tried to output to the console, but as noted, it does not stream character by character but instead appears in chunks:

import { streamText } from 'ai';
import { vertex } from '@ai-sdk/google-vertex';
import { createVertex } from '@ai-sdk/google-vertex';

const euVertex = createVertex({
//project: 'edubot-439916'
project: PROJECT_ID,
location: 'us-central1', // optional
baseURL: 'https://us-central1-aiplatform.googleapis.com/v1/',
googleAuthOptions: {
//apiKey: API_KEY
keyFilename:'edubot-439916-0950981fb9bc.json',
},
});

const model = euVertex('projects/' + PROJECT_ID + '/locations/us-central1/endpoints/' + ENDPOINT)

const result = streamText({
model: model,
prompt: 'Generate a lecture on stationary states in quantum mechanics',
});

//example: use textStream as an async iterable
for await (const textPart of result.textStream) {
console.log(textPart);
}

When you say streaming is not working, do you mean all text arrives in one chunk, or that the chunks are too large and it does not visually seem to stream?

If it's the latter, then that is the default way how vertex returns generated responses. That being said, you can rechunk and create a smoother stream with stream transformations: https://sdk.vercel.ai/docs/ai-sdk-core/generating-text#smoothing-streams

Hi @lgrammel, sorry for possible spam but I have a problem with streaming the data (all text arrives in one chunk in response). This is my route. It works perfect locally but does not work in dev/prod. Do you want me to create a separate issue for that?

"react": "18.2.0",
"next": "^14.2.4",
"@ai-sdk/openai": "^1.0.19",
"ai": "^4.0.36",

import { openai } from '@ai-sdk/openai'
import { convertToCoreMessages, streamText, createDataStreamResponse } from 'ai'

// Force the route to be dynamic and allow streaming responses up to 30 seconds
export const dynamic = 'force-dynamic'
export const maxDuration = 30

export async function POST(req: Request) {
  const { messages } = await req.json()

  return createDataStreamResponse({
    execute: dataStream => {
      const result = streamText({
        model: openai('gpt-4o-mini'),
        system: 'string',
        messages: convertToCoreMessages(messages),
        maxSteps: 5,
      })

      result.mergeIntoDataStream(dataStream)
    },
  })
}

yes and please describe what "dev/prod" means

Sure @lgrammel here is the issue - #4464