Python Lambda AWS参数和秘密Lambda扩展超时随机发生。

huangapple
go评论80阅读模式
英文:

Python Lambda AWS Parameters and Secrets Lambda Extension timeout randomly

问题

我已设置一个 Python 3.10 Lambda，已附加 Secret Layer，并且未附加 VPC。

  • 超时时间为20秒
  • 执行通常需要1-3秒
  • 所有 Secret Layer 变量均未设置，因此使用默认设置

我注意到随机地（大约1次每10次），执行会超时。

大多数情况下，第一次执行会失败。

我已启用 PARAMETERS_SECRETS_EXTENSION_LOG_LEVEL 为 DEBUG 并在 CloudWatch 日志中检查：

通常在 DEBUG Caching secret with key /secretsmanager... 行之后，源代码会执行并且函数成功执行。
但在某些情况下，源代码甚至不会启动，因此我无法记录任何日志，因为它根本没有执行。

任何帮助都将受到欢迎，

编辑：
将超时时间增加到1分钟后，似乎对 Secret Manager 的请求速度太慢了。
问题在于 ApiGateway 超时时间为30秒，所以它会失败。

英文:

I have set up a Python 3.10 Lambda that has attached Secret Layer and NO VPC attached.

  • The timeout time is 20 seconds
  • The execution normally takes 1-3seconds
  • All Secret Layer variables aren't set, so the default

I have noticed that randomly (1 in ~10) the execution timeout.

It's mostly the first execution the one that fails.

I have enabled the PARAMETERS_SECRETS_EXTENSION_LOG_LEVEL to DEBUG and checked in CloudWatch logs:

[AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:15:27 PARAMETERS_SECRETS_EXTENSION_LOG_LEVEL is debug. Log level set to debug.
[AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:15:27 DEBUG PARAMETERS_SECRETS_EXTENSION_CACHE_ENABLED is not present. Cache is enabled by default.
[AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:15:27 DEBUG PARAMETERS_SECRETS_EXTENSION_CACHE_SIZE is not present. Using default cache size: 1000 objects.
[AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:15:27 DEBUG SECRETS_MANAGER_TTL is not present. Setting default time-to-live: 5m0s.
[AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:15:27 DEBUG SSM_PARAMETER_STORE_TTL is not present. Setting default time-to-live: 5m0s.
[AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:15:27 DEBUG SECRETS_MANAGER_TIMEOUT_MILLIS is not present. Setting default timeout: 0s.
[AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:15:27 DEBUG SSM_PARAMETER_STORE_TIMEOUT_MILLIS is not present. Setting default timeout: 0s.
[AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:15:27 DEBUG PARAMETERS_SECRETS_EXTENSION_MAX_CONNECTIONS is not present. Setting default value: 3.
[AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:15:27 DEBUG PARAMETERS_SECRETS_EXTENSION_HTTP_PORT is not present. Setting default port: 2773.
[AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:15:27 INFO Systems Manager Parameter Store and Secrets Manager Lambda Extension 1.0.103
[AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:15:27 DEBUG Creating a new cache with size 1000
[AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:15:27 INFO Serving on port 2773
EXTENSION	Name: AWSParametersAndSecretsLambdaExtension	State: Ready	Events: [INVOKE, SHUTDOWN]
START RequestId: 2bdcca16-5b8b-449d---- Version: $LATEST
[AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:15:51 INFO ready to serve traffic
[AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:15:51 DEBUG Caching secret with key /secretsmanager/get?secretId=arn:aws:secretsmanager:.....
2023-06-13T14:15:57.299Z 2bdcca16-5b8b-449d---- Task timed out after 30.08 seconds

END RequestId: 2bdcca16-5b8b-449d----
REPORT RequestId: 2bdcca16-5b8b-449d----	Duration: 30075.25 ms	Billed Duration: 30000 ms	Memory Size: 160 MB	Max Memory Used: 122 MB	
XRAY TraceId: 1-648879ff-4cf918900c6e2df----	SegmentId: 2584b0c77f85---	Sampled: true

Normally after the DEBUG Caching secret with key /secretsmanager... line, the source code is executed and the function succeds.
But in some cases the source code doesn't even start , so I can't write any logs inside because it doesn't get there

Any help is appreacited,

EDIT:
Increased time to 1m and looks like the request to secret manager is just too slow.
The problem is that ApiGateway timeouts at 30s, so it fails

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                                                                                                                                                                                                                     
|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| 2023-06-13T11:23:36.533-03:00 | INIT_START Runtime Version: python:3.9.v23 Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:bf3c2cf12ad277455c37                                                                                                                              
| 2023-06-13T11:23:36.533-03:00 | [AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:23:36 PARAMETERS_SECRETS_EXTENSION_LOG_LEVEL is debug. Log level set to debug.                                                                                                          
| 2023-06-13T11:23:36.533-03:00 | [AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:23:36 DEBUG PARAMETERS_SECRETS_EXTENSION_CACHE_ENABLED is not present. Cache is enabled by default.                                                                                     
| 2023-06-13T11:23:36.533-03:00 | [AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:23:36 DEBUG PARAMETERS_SECRETS_EXTENSION_CACHE_SIZE is not present. Using default cache size: 1000 objects.                                                                             
| 2023-06-13T11:23:36.533-03:00 | [AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:23:36 DEBUG SECRETS_MANAGER_TTL is not present. Setting default time-to-live: 5m0s.                                                                                                     
| 2023-06-13T11:23:36.533-03:00 | [AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:23:36 DEBUG SSM_PARAMETER_STORE_TTL is not present. Setting default time-to-live: 5m0s.                                                                                                 
| 2023-06-13T11:23:36.533-03:00 | [AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:23:36 DEBUG SECRETS_MANAGER_TIMEOUT_MILLIS is not present. Setting default timeout: 0s.                                                                                                 
| 2023-06-13T11:23:36.533-03:00 | [AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:23:36 DEBUG SSM_PARAMETER_STORE_TIMEOUT_MILLIS is not present. Setting default timeout: 0s.                                                                                             
| 2023-06-13T11:23:36.533-03:00 | [AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:23:36 DEBUG PARAMETERS_SECRETS_EXTENSION_MAX_CONNECTIONS is not present. Setting default value: 3.                                                                                      
| 2023-06-13T11:23:36.533-03:00 | [AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:23:36 DEBUG PARAMETERS_SECRETS_EXTENSION_HTTP_PORT is not present. Setting default port: 2773.                                                                                          
| 2023-06-13T11:23:36.533-03:00 | [AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:23:36 INFO Systems Manager Parameter Store and Secrets Manager Lambda Extension 1.0.103                                                                                                 
| 2023-06-13T11:23:36.533-03:00 | [AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:23:36 DEBUG Creating a new cache with size 1000                                                                                                                                         
| 2023-06-13T11:23:38.776-03:00 | [AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:23:36 INFO Serving on port 2773                                                                                                                                                         
| 2023-06-13T11:23:38.778-03:00 | EXTENSION Name: AWSParametersAndSecretsLambdaExtension State: Ready Events: [INVOKE, SHUTDOWN]                                                                                                                                                      
| 2023-06-13T11:23:38.799-03:00 | START RequestId: b1e85fec-aef3-45c1-8486- Version: $LATEST                                                                                                                                                                                          
| 2023-06-13T11:23:39.249-03:00 | [AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:23:38 INFO ready to serve traffic                                                                                                                                                       
| 2023-06-13T11:23:39.249-03:00 | [AWS Parameters and Secrets Lambda Extension] 2023/06/13 14:23:39 DEBUG Caching secret with key /secretsmanager...                                                                                                                                  
| 2023-06-13T11:24:13.788-03:00 | ISSUE: TESTTW-76.....
| 2023-06-13T11:24:13.788-03:00 | ISSUE: TESTTW-76.....
| 2023-06-13T11:24:13.788-03:00 | ISSUE: TESTTW-76.....
| 2023-06-13T11:24:13.788-03:00 | ISSUE: TESTTW-76.....
| 2023-06-13T11:24:13.788-03:00 | ISSUE: TESTTW-76.....
| 2023-06-13T11:24:15.560-03:00 | END RequestId: b1e85fec-aef3-45c1-8486-                                                                                                                                                                                                             
| 2023-06-13T11:24:15.560-03:00 | REPORT RequestId: b1e85fec-aef3-45c1-8486- Duration: 36782.12 ms Billed Duration: 36783 ms Memory Size: 160 MB Max Memory Used: 157 MB Init Duration: 2243.76 ms  XRAY TraceId: 1-64887bdd-6e0c2f25704568f90d6 SegmentId: 502be5bc3c57 Sampled: true
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

答案1

得分: 0

经过许多故障排除后，发现问题的原因是源代码导致了超时。

对https://api.ipify.org的调用是罪魁祸首。

显然，这个API并不总是正常工作，随机花费1秒到大于1分钟的时间来回应IP。

英文:

So after much troubleshooting, found out that the reason was source code giving the timeout

A call to https://api.ipify.org was the culprit

Apparently the api isn't working all the time and randomly takes 1s to >1m to answer with the IP

huangapple
  • 本文由 发表于 2023年6月12日 23:55:26
  • 转载请务必保留本文链接：https://go.coder-hub.com/76458344.html
  • amazon-web-services
  • aws-lambda
  • aws-secrets-manager
  • python-3.x
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定