英文:
How to Implement Streaming API Endpoint with Next.js 13 Route Handlers Using LangChain?
问题
I am trying to create an API endpoint using Nextjs 13's new Route Handler solution.
This API uses LangChain, and streams the response back to the frontend.
When calling the OpenAI wrapper class, I am passing in the Streaming property, and supplying the callback function. This callback function then provides the stream as chunks (ie. tokens).
I want to stream these tokens to the frontend to output the AI's response as it's being generated.
I was able to get this working using the "old" API route solution with the following code:
import { OpenAI } from "langchain/llms/openai";
export default async function handler(req, res) {
const chat = new OpenAI({
modelName: "gpt-3.5-turbo",
streaming: true,
callbacks: [
{
handleLLMNewToken(token) {
res.write(token);
},
},
],
});
await chat.call("Write me a song about sparkling water.");
res.end();
}
I am trying to convert this code to the new Route Handler solution, but I haven't been able to get this working.
I have tried many different approaches to this, with no luck.
For example:
import { NextResponse } from "next/server";
import { OpenAI } from "langchain/llms/openai";
export const dynamic = "force-dynamic";
export const revalidate = true;
export async function GET(req, res) {
const chat = new OpenAI({
modelName: "gpt-3.5-turbo",
streaming: true,
callbacks: [
{
handleLLMNewToken(token) {
// res.write(token);
return new NextResponse.json(token);
},
},
],
});
await chat.call("Write me a song about sparkling water.");
}
There just seems to be no way to "write" the tokens to the response as they are streamed to the Route Handler's response.
Any assistance will be GREATLY appreciated.
英文:
I am trying to create an API endpoint using Nextjs 13's new Route Handler solution.
This API uses LangChain, and streams the response back to the frontend.
When calling the OpenAI wrapper class, I am passing in the Streaming property, and supplying the callback function. This callback function then provides the stream as chunks (ie. tokens).
I want to stream these tokens to the frontend to output the AI's response as it's being generated.
I was able to get this working using the "old" API route solution with the following code:
import { OpenAI } from "langchain/llms/openai";
export default async function handler(req, res) {
const chat = new OpenAI({
modelName: "gpt-3.5-turbo",
streaming: true,
callbacks: [
{
handleLLMNewToken(token) {
res.write(token);
},
},
],
});
await chat.call("Write me a song about sparkling water.");
res.end();
}
I am trying to convert this code to the new Route Handler solution, but I haven't been able to get this working.
I have tried many different approaches to this, with no luck.
For example:
import { NextResponse } from "next/server";
import { OpenAI } from "langchain/llms/openai";
export const dynamic = "force-dynamic";
export const revalidate = true;
export async function GET(req, res) {
const chat = new OpenAI({
modelName: "gpt-3.5-turbo",
streaming: true,
callbacks: [
{
handleLLMNewToken(token) {
// res.write(token);
return new NextResponse.json(token);
},
},
],
});
await chat.call("Write me a song about sparkling water.");
}
There just seems to be no way to "write" the tokens to the response as they are streamed to the Route Handler's response.
Any assistance will be GREATLY appreciated.
答案1
得分: 4
在路由处理程序中,我使用TransformStream类创建了一个新的流对象。然后,我将生成的令牌写入此流对象。由于流需要传输字节,我使用TextEncoder将令牌编码为Uint8Array值。
最后,我将流的可读属性作为API响应返回。这似乎可以解决问题,尽管比旧API路由方法的解决方案略复杂。
import { OpenAI } from "langchain/llms/openai";
export const dynamic = "force-dynamic";
export const revalidate = true;
async function runLLMChain() {
// 创建编码器以将令牌(字符串)转换为Uint8Array
const encoder = new TextEncoder();
// 创建一个TransformStream,用于写入生成的令牌作为响应
const stream = new TransformStream();
const writer = stream.writable.getWriter();
const chat = new OpenAI({
modelName: "gpt-3.5-turbo",
streaming: true,
callbacks: [
{
async handleLLMNewToken(token) {
await writer.ready;
await writer.write(encoder.encode(`${token}`));
},
async handleLLMEnd() {
await writer.ready;
await writer.close();
},
},
],
});
chat.call("Write me a song about sparkling water.");
// 返回可读流
return stream.readable;
}
export async function GET(req) {
const stream = runLLMChain();
return new Response(await stream);
}
这是您提供的代码的翻译部分。
英文:
I think I might have a solution.
In the Route Handler, I create a new stream object using the TransformStream class.
I then write the tokens to this stream object as they are generated.
Because the stream expects bytes to be transferred to it, I use the TextEncoder to encode the token to a Uint8Array value.
Lastly, I then return this readable property of the stream in our API response.
This seems to do the trick, although slightly more complex than the solution from the older API route approach.
import { OpenAI } from "langchain/llms/openai";
export const dynamic = "force-dynamic";
export const revalidate = true;
async function runLLMChain() {
// Create encoding to convert token (string) to Uint8Array
const encoder = new TextEncoder();
// Create a TransformStream for writing the response as the tokens as generated
const stream = new TransformStream();
const writer = stream.writable.getWriter();
const chat = new OpenAI({
modelName: "gpt-3.5-turbo",
streaming: true,
callbacks: [
{
async handleLLMNewToken(token) {
await writer.ready;
await writer.write(encoder.encode(`${token}`));
},
async handleLLMEnd() {
await writer.ready;
await writer.close();
},
},
],
});
chat.call("Write me a song about sparkling water.");
// Return the readable stream
return stream.readable;
}
export async function GET(req) {
const stream = runLLMChain();
return new Response(await stream);
}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论