AWS HTTP API 网关 + Lambda(Node/Express)内部服务器错误

huangapple go评论66阅读模式
英文:

AWS http api gateway + lambda (node/express) Internal Server Error

问题

我在运行长时间查询时遇到了“内部服务器错误”。
实际上,我需要通过API获取历史数据,有时可能需要超过30秒。这取决于查询的复杂性。它也可能需要1分钟。

不确定,但我猜测API网关的超时设置为30秒(我不能增加它),而我的查询执行时间超过30秒。所以我认为我会收到“内部服务器错误”。

我如何表达上述情况?

因为如果我在本地运行相同的查询,我的意思是在本地的node/express中运行 npm run start,即使需要1分钟,响应也会正常返回。

但是当我将node/express代码部署到lambda函数时,如果任何查询执行时间超过一定时限,它就会抛出错误。

我有以下node/express的设置:

const express = require("express");
const serverless = require("serverless-http");
const app = express();

app.use(cors());
app.use((req, res, next) => {
    res.setHeader('Connection', 'keep-alive'); // 我按照某些帖子的建议添加了这一行,但没有帮助
    res.setHeader('Keep-Alive', 'timeout=30'); // 我按照某些帖子的建议添加了这一行,但没有帮助
    res.setHeader("Access-Control-Allow-Headers", "X-Requested-With,content-type");
    res.setHeader("Access-Control-Allow-Origin", "*");
    res.setHeader("Access-Control-Allow-Methods", "GET, POST, OPTIONS, PUT, PATCH, DELETE");
    res.setHeader("Access-Control-Allow-Credentials", true);
    next();
});

app.use(express.json());
app.use(express.urlencoded({ extended: true }));

app.use(`api-end-point/user`, userRoute);
....

if (process.env.NODE_ENV !== "lambda") {

    PORT = process.env.PORT || 7000;
    const server = app.listen(PORT, () => {
        console.log(`node-express server running in ${process.env.NODE_ENV} mode on ${PORT}`);
    });
    server.timeout = 0;

}else {

    module.exports.handler = serverless(app); // 这是为lambda函数

}

我将此代码部署到AWS的lambda函数。

HTTP API网关配置了两个路由 /ANY, /{proxy+}

超时

API网关设置为默认的30秒[我不能增加此时间,因为AWS不允许]

Lambda设置为10分钟

CORS
AWS HTTP API 网关 + Lambda(Node/Express)内部服务器错误

我真的不知道如何解决这个问题

我如何增加API网关的超时或者如何保持连接活动?

英文:

I get internal server error when I have a long running query.
Actually, I have to fetch historic data through an API, which sometime can take longer than 30 seconds. It depends on the query how complex it is. It can take 1 min also.

Not sure but guessing, API gateway timeout is set to 30 seconds (and I cann't increase it) and my query execution time is more then 30 seconds. So I get internal server error I believe.

HOW can I say above statement ?

because If I run the same query locally, I mean in node/express locally by running npm run start, it works fine even if takes1 mins, response will always come back.

But when I deploy node/express code to lambda function, it throws error if any query takes longer period to execute.

I have following setup of node/express

const express = require("express");
const serverless = require("serverless-http");
const app = express();

app.use(cors());
app.use((req, res, next) => {
    res.setHeader('Connection', 'keep-alive');                   // I added this line as suggested in some post but not helping
    res.setHeader('Keep-Alive', 'timeout=30');                   // I added this line as suggested in some post but not helping
    res.setHeader("Access-Control-Allow-Headers", "X-Requested-With,content-type");
    res.setHeader("Access-Control-Allow-Origin", "*");
    res.setHeader("Access-Control-Allow-Methods", "GET, POST, OPTIONS, PUT, PATCH, DELETE");
    res.setHeader("Access-Control-Allow-Credentials", true);
    next();
});

app.use(express.json());
app.use(express.urlencoded({ extended: true }));

app.use(`api-end-point/user`, userRoute);
....

if (process.env.NODE_ENV !== "lambda") {

    PORT = process.env.PORT || 7000;
    const server = app.listen(PORT, () => {
        console.log(`node-express server running in ${process.env.NODE_ENV} mode on ${PORT}`);
    });
    server.timeout = 0;

}else {

    module.exports.handler = serverless(app);     // this is for lambda function

}

I deploy this code to AWS lambda function.

<hr>

HTTP API gateway is configured with two routes /ANY, /{proxy+}

<hr>

TIMEOUT

API gateway is set to default 30 seconds. [I can not increase this time as not allowed by AWS]

Lambda is set to 10 **mins**

<hr>

CORS
AWS HTTP API 网关 + Lambda(Node/Express)内部服务器错误

<hr>

I really have no idea how can I fix this problem ?

How can I increase API gateway timeout or How can I keep connection alive ?

答案1

得分: 1

API网关的超时时间不能增加到超过30秒,如前所述。

我目前知道的唯一解决方法是以异步方式运行Lambda函数,但这无法在Http API中完成。但如果你愿意将其更改为REST API,那么可以通过在REST API中启用Lambda代理集成,并使用X-Amz-Invocation-Type调用头来异步调用Lambda来实现。这将允许你的Lambda以异步方式运行(最多15分钟),并通过API调用来触发。

英文:

You cannot increase the API Gateway timeout to greater than 30 seconds, as has already been mentioned.

The only solution I know of at this time is to run your Lambda asynchronously, but this cannot be done in an Http API. But if you're willing to change it to a REST API, then this can be done with a combination of turning on Lambda Proxy Integration in the REST API and invoking the Lambda asynchronously utilizing an invoke header X-Amz-Invocation-Type. This will allow your Lambda to run asynchronously (up to 15 minutes) with an API call.

答案2

得分: 0

自从超时时间无法增加后,您可以考虑改为使用单个HTTP请求:

  1. 客户端发送POST请求
  2. 服务器响应一个结果的URL
  3. 客户端多次GET该URL - 当结果准备好时,将返回200 OK
  • 或者使用WebSocket

文档中提到WebSocket的"空闲连接超时"最长可达10分钟。

英文:

https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html

Since timeout cannot be incresed, you might change using single HTTP request to

  1. Client POST query
  2. Server response an url for the result
  3. Client GET the url multiple times -- it will be 200 OK when the result ready
  • or WebSocket

The document says Idle Connection Timeout for WebSocket is up to 10 minutes

答案3

得分: 0

使用Lambda意味着订阅Serverless目录/哲学中的模式。这意味着尽可能地使用异步。

据我理解,您的Lambda需要接收请求,然后调用某个未指定的内容,这个调用需要30~60秒。

API Gateway的硬编码超时限制为29秒。

为了解决这个问题,应用程序需要重新架构:

  • 从前端异步触发Lambda,使用X-Amz-Invocation-Type事件
  • Lambda调用历史API,并将结果存储在某个存储中(DynamoDB、S3、RDS等)。
  • 前端从前端查询后端,直到数据可用(或使用WebSocket)。

这样,历史API调用最多可以需要15分钟,并且可以将调用缓存在存储中以加快后续调用的速度。如果需要超过15分钟,那么我建议重新架构历史API。

英文:

Using Lambda means subscribing to patterns from the Serverless catalog/philosophy. Which means using async whenever possible.

As far as I understand your Lambda needs receives a request, does another call to something (not specified) which takes 30~60s.

The API Gateway has a hardcoded timeout of 29s (hard limit).

To solve this problem the application would need to be re-architectured:

  • Trigger the Lambda asynchronously using X-Amz-Invocation-Type Event from the Frontend.
  • The Lambda calls the history API and stores the result in some storage (DynamoDB, S3, RDS, ...).
  • The frontend queries the backend from the frontend until the data is available (or use WebSockets)

This way the historic API call can take up to 15m and the calls can be cached in the storage to speed up further calls. If it needs more than 15m then I would ask the historic API to re-architecture.

huangapple
  • 本文由 发表于 2023年1月7日 13:10:40
  • 转载请务必保留本文链接:https://go.coder-hub.com/75038243.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定