How to send OpenAI stream response from Nextjs API to client

I tried openai-streams + nextjs-openai, they only works for Node 18+, however, they failed on Node 17 and lower. I'm restricted to Node 17 and lower as Digital Oceans App Platform currently not supporting Node 18.

I also tried this method which works well on client side, but it exposes the API key. I want to implement within the NextJS API route, but I'm unable to pass the streaming response to the client.

With the code below, I can only get the first chunk of response from the API route, and not able to get the streaming response to have the ChatGPT effect. Please kindly help.

<!-- begin snippet: js hide: false console: true babel: false -->

<!-- language: lang-js -->

// /api/prompt.js

import { Configuration, OpenAIApi } from "openai";
import { Readable } from "readable-stream";

const configuration = new Configuration({
  apiKey: process.env.NEXT_PUBLIC_OPENAI_API_KEY,
});
const openai = new OpenAIApi(configuration);

export default async function handler(req, res) {
  const completion = await openai.createCompletion(
    {
      model: "text-davinci-003",
      prompt: "tell me a story",
      max_tokens: 500,
      stream: true,
    },
    { responseType: "stream" }
  );

  completion.data.on("data", async (data) => {
    const lines = data
      .toString()
      .split("\n")
      .filter((line) => line.trim() !== "");

    for (const line of lines) {
      const message = line.replace(/^data: /, "");
      if (message === "[DONE]") {
        return;
      }
      try {
        const parsed = JSON.parse(message);
        const string = parsed.choices[0].text;
        Readable.from(string).pipe(res);
      } catch (error) {
        console.error("Could not JSON parse stream message", message, error);
      }
    }
  });

<!-- end snippet -->

<!-- begin snippet: js hide: false console: true babel: false -->

<!-- language: lang-js -->

// /components/Completion.js

export default function Completion() {
  const [text, setText] = useState();

  const generate = async () => {
    const response = await fetch("/api/prompt");
    console.log("response: ", response);
    const text = await response.text();
    console.log("text: ", text);
    setText((state) => state + text);
  };
  
  // ... rest
}

<!-- end snippet -->

Well you can use StreamingTextResponse
of vercel AI sdk

Attached is some example code form their docs

import { Configuration, OpenAIApi } from 'openai-edge'
import { OpenAIStream, StreamingTextResponse } from 'ai'

const config = new Configuration({
  apiKey: process.env.OPENAI_API_KEY
})
const openai = new OpenAIApi(config)

export const runtime = 'edge'

export async function POST(req) {
  const { messages } = await req.json()
  const response = await openai.createChatCompletion({
    model: 'gpt-4',
    stream: true,
    messages
  })
  const stream = OpenAIStream(response)
  return new StreamingTextResponse(stream)
}

Notes:

1. This requires the Edge Runtime
2. The library openai-edge requires Node 18+. However, this is expected to be deprecated soon with the support for streaming in the official openai library.

You can use the openai-streams/node entrypoint on Node <18, which will return a Node.js Readable instead of a WHATWG ReadableStream. I'll update the docs to be clearer soon.

Node: Consuming streams in Next.js API Route

If you cannot use an Edge runtime or want to consume Node.js streams for another reason, use openai-streams/node:

import type { NextApiRequest, NextApiResponse } from "next";
import { OpenAI } from "openai-streams/node";

export default async function test(_: NextApiRequest, res: NextApiResponse) {
  const stream = await OpenAI("completions", {
    model: "text-davinci-003",
    prompt: "Write a happy sentence.\n\n",
    max_tokens: 25,
  });

  stream.pipe(res);
}