英文:
GCP deployment with nginx - uwsgi - flask fails
问题
我有一个非常简单的Flask应用程序,部署在GKE上,并通过Google外部负载均衡器公开。但是从后端服务获取随机的502响应(在后端服务和Nginx上添加了自定义标头,以确保源可以看到后端服务的标头,但看不到Nginx的标头)
设置如下:
负载均衡器(LB) -> 后端服务 -> 网络端点组(NEG) -> Pod(Nginx -> uWSGI),其中Pod是使用Flask构建并通过uWSGI和Nginx部署的应用程序。
情景是以简单安全的方式处理图像上传。发送方向我发送一个上传请求的令牌。
我的Flask应用程序
- 接收请求并通过"requests"使用另一个服务检查发送的令牌。
- 如果令牌有效,则继续处理图像并返回200
- 如果令牌无效,则停止并返回401响应。
首先,我对200和401感到怀疑。并将所有响应恢复为200。在响应的一些预期响应之后,服务器开始响应502并不断发送它。"一开始的一些消息成功了"。
Nginx错误日志包含以下行
2023/02/08 18:22:29 [error] 10#10: *145 readv() failed (104: Connection reset by peer) while reading upstream, client: 35.191.17.139, server: _, request: "POST /api/v1/imageUpload/image HTTP/1.1", upstream: "uwsgi://127.0.0.1:21270", host: "example-host.com"。
我的uwsgi.ini文件如下:
[uwsgi]
socket = 127.0.0.1:21270
master
processes = 8
threads = 1
buffer-size = 32768
stats = 127.0.0.1:21290
log-maxsize = 104857600
logdate
log-reopen
log-x-forwarded-for
uid = image_processor
gid = image_processor
need-app
chdir = /server/
wsgi-file = image_processor_application.py
callable = app
py-auto-reload = 1
pidfile = /tmp/uwsgi-imgproc-py.pid
我的nginx.conf如下
location ~ ^/api/ {
client_max_body_size 15M;
include uwsgi_params;
uwsgi_pass 127.0.0.1:21270;
}
最后,我的应用程序具有一个简单的JSON响应的健康检查方法。它不执行额外的操作,只是简单地返回。正如上面所解释的,这从不失败。
编辑:我的Pod中的Nginx访问日志显示响应为401,但客户端收到502。
英文:
I have a very simple flask app that is deployed on GKE and exposed via google external load balancer. And getting random 502 responses from the backend-service (added a custom headers on backend-service and nginx to make sure the source and I can see the backend-service's header but not nginx's)
The setup is;
LB -> backend-service -> neg -> pod (nginx -> uwsgi) where pod is the application built using flask and deployed via uwsgi and nginx.
The scenario is to handle image uploads in simple-secured way. Sender sends me a token with upload request.
My flask app
- receive request and check the sent token via another service using "requests".
- If token valid, proceed to handle the image and return 200
- If token is not valid, stop and send back a 401 response.
First, I got suspicious about the 200 and 401's. And reverted all responses to 200. Following some of the expected responses, server starts to respond 502 and keep sending it. "Some of the messages at the very beginning succeeded".
nginx error logs contains below lines
2023/02/08 18:22:29 [error] 10#10: *145 readv() failed (104: Connection reset by peer) while reading upstream, client: 35.191.17.139, server: _, request: "POST /api/v1/imageUpload/image HTTP/1.1", upstream: "uwsgi://127.0.0.1:21270", host: "example-host.com"
my uwsgi.ini file is as below;
[uwsgi]
socket = 127.0.0.1:21270
master
processes = 8
threads = 1
buffer-size = 32768
stats = 127.0.0.1:21290
log-maxsize = 104857600
logdate
log-reopen
log-x-forwarded-for
uid = image_processor
gid = image_processor
need-app
chdir = /server/
wsgi-file = image_processor_application.py
callable = app
py-auto-reload = 1
pidfile = /tmp/uwsgi-imgproc-py.pid
my nginx.conf is as below
location ~ ^/api/ {
client_max_body_size 15M;
include uwsgi_params;
uwsgi_pass 127.0.0.1:21270;
}
Lastly, my app has a healthcheck method with simple JSON response. It does no extra stuff and simply returns. This never fails as explained above.
Edit : my nginx access logs in the pod shows the response as 401 while the client receives 502.
答案1
得分: 1
为了那些将要面临相同问题的人,问题是关于POST数据的读取(或未读取)。
nginx 期望由代理(在我们的情况下是uwsgi应用程序)读取POST数据。但根据我的逻辑,在某些情况下我没有进行读取并返回响应。
设置uwsgi的post-buffering 解决了这个问题。
post-buffering = %(16 * 1024 * 1024)
这导致我找到了这个解决方案:
https://stackoverflow.com/a/26765936/631965
https://stackoverflow.com/questions/22697584/nginx-uwsgi-104-connection-reset-by-peer-while-reading-response-header-from-u
英文:
for those who gonna face with the same issue, the problem was post data reading (or not reading).
nginx was expecting to get post data read by the proxied, in our case uwsgi, app. But according to my logic I was not reading it in some cases and returning back the response.
Setting uwsgi post-buffering solved the issue.
post-buffering = %(16 * 1024 * 1024)
Which led me to this solution;
https://stackoverflow.com/a/26765936/631965
https://stackoverflow.com/questions/22697584/nginx-uwsgi-104-connection-reset-by-peer-while-reading-response-header-from-u
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论