英文:
AWS http api gateway + lambda (node/express) Internal Server Error
问题
我在运行长时间查询时遇到了“内部服务器错误”。
实际上,我需要通过API获取历史数据,有时可能需要超过30秒。这取决于查询的复杂性。它也可能需要1分钟。
不确定,但我猜测API网关的超时设置为30秒(我不能增加它),而我的查询执行时间超过30秒。所以我认为我会收到“内部服务器错误”。
我如何表达上述情况?
因为如果我在本地运行相同的查询,我的意思是在本地的node/express中运行 npm run start
,即使需要1分钟,响应也会正常返回。
但是当我将node/express代码部署到lambda函数时,如果任何查询执行时间超过一定时限,它就会抛出错误。
我有以下node/express的设置:
const express = require("express");
const serverless = require("serverless-http");
const app = express();
app.use(cors());
app.use((req, res, next) => {
res.setHeader('Connection', 'keep-alive'); // 我按照某些帖子的建议添加了这一行,但没有帮助
res.setHeader('Keep-Alive', 'timeout=30'); // 我按照某些帖子的建议添加了这一行,但没有帮助
res.setHeader("Access-Control-Allow-Headers", "X-Requested-With,content-type");
res.setHeader("Access-Control-Allow-Origin", "*");
res.setHeader("Access-Control-Allow-Methods", "GET, POST, OPTIONS, PUT, PATCH, DELETE");
res.setHeader("Access-Control-Allow-Credentials", true);
next();
});
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
app.use(`api-end-point/user`, userRoute);
....
if (process.env.NODE_ENV !== "lambda") {
PORT = process.env.PORT || 7000;
const server = app.listen(PORT, () => {
console.log(`node-express server running in ${process.env.NODE_ENV} mode on ${PORT}`);
});
server.timeout = 0;
}else {
module.exports.handler = serverless(app); // 这是为lambda函数
}
我将此代码部署到AWS的lambda
函数。
HTTP API网关配置了两个路由 /ANY, /{proxy+}
超时
API网关设置为默认的30秒
。[我不能增加此时间,因为AWS不允许]
Lambda设置为10分钟
我真的不知道如何解决这个问题?
我如何增加API网关的超时或者如何保持连接活动?
英文:
I get internal server error
when I have a long running query.
Actually, I have to fetch historic data through an API, which sometime can take longer than 30 seconds. It depends on the query how complex it is. It can take 1 min also.
Not sure but guessing, API gateway timeout is set to 30 seconds (and I cann't increase it) and my query execution time is more then 30 seconds. So I get internal server error
I believe.
HOW can I say above statement ?
because If I run the same query locally, I mean in node/express locally by running npm run start
, it works fine even if takes1 mins, response will always come back.
But when I deploy node/express code to lambda function, it throws error if any query takes longer period to execute.
I have following setup of node/express
const express = require("express");
const serverless = require("serverless-http");
const app = express();
app.use(cors());
app.use((req, res, next) => {
res.setHeader('Connection', 'keep-alive'); // I added this line as suggested in some post but not helping
res.setHeader('Keep-Alive', 'timeout=30'); // I added this line as suggested in some post but not helping
res.setHeader("Access-Control-Allow-Headers", "X-Requested-With,content-type");
res.setHeader("Access-Control-Allow-Origin", "*");
res.setHeader("Access-Control-Allow-Methods", "GET, POST, OPTIONS, PUT, PATCH, DELETE");
res.setHeader("Access-Control-Allow-Credentials", true);
next();
});
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
app.use(`api-end-point/user`, userRoute);
....
if (process.env.NODE_ENV !== "lambda") {
PORT = process.env.PORT || 7000;
const server = app.listen(PORT, () => {
console.log(`node-express server running in ${process.env.NODE_ENV} mode on ${PORT}`);
});
server.timeout = 0;
}else {
module.exports.handler = serverless(app); // this is for lambda function
}
I deploy this code to AWS lambda
function.
<hr>
HTTP API gateway is configured with two routes /ANY, /{proxy+}
<hr>
TIMEOUT
API gateway is set to default 30 seconds
. [I can not increase this time as not allowed by AWS]
Lambda is set to 10 **mins**
<hr>
<hr>
I really have no idea how can I fix this problem ?
How can I increase API gateway timeout or How can I keep connection alive ?
答案1
得分: 1
API网关的超时时间不能增加到超过30秒,如前所述。
我目前知道的唯一解决方法是以异步方式运行Lambda函数,但这无法在Http API中完成。但如果你愿意将其更改为REST API,那么可以通过在REST API中启用Lambda代理集成,并使用X-Amz-Invocation-Type调用头来异步调用Lambda来实现。这将允许你的Lambda以异步方式运行(最多15分钟),并通过API调用来触发。
英文:
You cannot increase the API Gateway timeout to greater than 30 seconds, as has already been mentioned.
The only solution I know of at this time is to run your Lambda asynchronously, but this cannot be done in an Http API. But if you're willing to change it to a REST API, then this can be done with a combination of turning on Lambda Proxy Integration in the REST API and invoking the Lambda asynchronously utilizing an invoke header X-Amz-Invocation-Type. This will allow your Lambda to run asynchronously (up to 15 minutes) with an API call.
答案2
得分: 0
自从超时时间无法增加后,您可以考虑改为使用单个HTTP请求:
- 客户端发送POST请求
- 服务器响应一个结果的URL
- 客户端多次GET该URL - 当结果准备好时,将返回
200 OK
- 或者使用WebSocket
文档中提到WebSocket的"空闲连接超时"最长可达10分钟。
英文:
https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html
Since timeout cannot be incresed, you might change using single HTTP request to
- Client POST query
- Server response an url for the result
- Client GET the url multiple times -- it will be
200 OK
when the result ready
- or WebSocket
The document says Idle Connection Timeout
for WebSocket is up to 10 minutes
答案3
得分: 0
使用Lambda意味着订阅Serverless目录/哲学中的模式。这意味着尽可能地使用异步。
据我理解,您的Lambda需要接收请求,然后调用某个未指定的内容,这个调用需要30~60秒。
API Gateway的硬编码超时限制为29秒。
为了解决这个问题,应用程序需要重新架构:
- 从前端异步触发Lambda,使用X-Amz-Invocation-Type事件。
- Lambda调用历史API,并将结果存储在某个存储中(DynamoDB、S3、RDS等)。
- 前端从前端查询后端,直到数据可用(或使用WebSocket)。
这样,历史API调用最多可以需要15分钟,并且可以将调用缓存在存储中以加快后续调用的速度。如果需要超过15分钟,那么我建议重新架构历史API。
英文:
Using Lambda means subscribing to patterns from the Serverless catalog/philosophy. Which means using async whenever possible.
As far as I understand your Lambda needs receives a request, does another call to something (not specified) which takes 30~60s.
The API Gateway has a hardcoded timeout of 29s (hard limit).
To solve this problem the application would need to be re-architectured:
- Trigger the Lambda asynchronously using X-Amz-Invocation-Type Event from the Frontend.
- The Lambda calls the history API and stores the result in some storage (DynamoDB, S3, RDS, ...).
- The frontend queries the backend from the frontend until the data is available (or use WebSockets)
This way the historic API call can take up to 15m and the calls can be cached in the storage to speed up further calls. If it needs more than 15m then I would ask the historic API to re-architecture.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论