实施与阶段相关的S3存储桶。

huangapple
go评论109阅读模式
英文:

Implement s3 bucket depending on stage

问题

我想在我的Python Lambda中实现一个依赖于部署阶段的变量S3存储桶。
以下是我的代码：

  1. def count_objects(year, last_month):
  2.     amount_s3objects = 0
  3.     response = boto3.client('s3').list_objects_v2(Bucket='my-bucket_name', Prefix=f'xxx/xxx/year={year}/month={last_month}')
  4.     for object in response['Contents']:
  5.         amount_s3objects += 1
  6.     days_of_last_month = monthrange(year, last_month)[1]
  7.     return amount_s3objects, days_of_last_month

现在，当我部署到dev阶段时，存储桶应该是"my-bucket_name-test"，当部署到prod阶段时，存储桶应该是"my-bucket_name-prod"。
我已经尝试使用事件来实现它，但我不知道应该触发哪个事件，因为该函数只计算S3前缀中的对象。
这是我的serverless.yaml，如果有帮助的话：

  1. service: name
  3. frameworkVersion: '3'
  4. configValidationMode: error
  6. custom:
  7.   pythonRequirements:
  8.     dockerizePip: true
  9.   stage: ${opt:stage, self:provider.stage}
  10.   profile:
  11.     dev: dev
  12.     prod: prod 
  13.   deploymentBucket:
  14.     dev: deployment-bucket-test
  15.     prod: deployment-bucket
  16.   serverlessIfElse:
  17.     - If: '"${self:custom.stage}" == "prod"'
  18.       Set:
  19.         provider.s3.athena-result-bucket.name: athena-results
  20.         provider.s3.my-bucket_name.name: my-bucket_name-prod
  22. provider:
  23.   name: aws
  24.   runtime: python3.9
  25.   region: eu-central-1
  26.   stage: dev
  27.   profile: ${self:custom.profile.${self:custom.stage}}
  28.   deploymentMethod: direct
  29.   deploymentBucket:
  30.     name: ${self:custom.deploymentBucket.${self:custom.stage}}
  31.   s3:
  32.     athena-result-bucket:
  33.       name: test-athena-results
  34.     my-bucket_name-bucket:   
  35.       name: my-bucket_name-test
  37. functions:
  38.   lambda_handler:
  39.     handler: handler.lambda_handler
  40.     timeout: 300
  41.     events:
  42.       - eventBridge:
  43.           schedule: cron(0 0 ? * 4#1 *)
  44.       - s3:
  45.           bucket: athena-result-bucket
  46.           event: s3:ObjectCreated:*
  47.       - s3:
  48.           bucket: my-bucket_name
  49.           event: s3:ObjectCreated:*
  50.           
  51. package:
  52.   exclude:
  53.     - node_modules/**
  55. plugins:
  56.   - serverless-python-requirements
  57.   - serverless-plugin-ifelse
  58.   - serverless-better-credentials

请注意，我已经更新了存储桶的命名以反映您的要求。

英文:

I want to implement a variable s3 bucket which is depending on the deploying stage in my Python Lambda.
This is my code here:

  1. def count_objects(year, last_month):
  2.     amount_s3objects = 0
  3.     response = boto3.client('s3').list_objects_v2(Bucket='my-bucket_name',Prefix=f'xxx/xxx/year={year}/month={last_month}')
  4.     for object in response['Contents']:
  5.         amount_s3objects+=1
  6.     days_of_last_month = monthrange(year, last_month)[1]
  7.     return amount_s3objects, days_of_last_month

Now when I deploy to dev the bucket should be "my-bucket_name-test" and when deploying to prod the bucket should be "my-bucket_name-prod".
I've already tried to do it with an event, but I don't know which event should trigger that because the function only counts the objects in the s3 prefix.
This is my serverless.yaml, if that helps:

  1. service: name
  3. frameworkVersion: '3'
  4. configValidationMode: error
  6. custom:
  7.   pythonRequirements:
  8.     dockerizePip: true
  9.   stage: ${opt:stage, self:provider.stage}
  10.   profile:
  11.     dev: dev
  12.     prod: prod 
  13.   deploymentBucket:
  14.     dev: deployment-bucket-test
  15.     prod: deployment-bucket
  16.   serverlessIfElse:
  17.     - If: '"${self:custom.stage}" == "prod"'
  18.       Set:
  19.         provider.s3.athena-result-bucket.name: athena-results
  20.         provider.s3.my-bucket_name.name: my-bucket_name-prod
  22. provider:
  23.   name: aws
  24.   runtime: python3.9
  25.   region: eu-central-1
  26.   stage: dev
  27.   profile: ${self:custom.profile.${self:custom.stage}}
  28.   deploymentMethod: direct
  29.   deploymentBucket:
  30.     name: ${self:custom.deploymentBucket.${self:custom.stage}}
  31.   s3:
  32.     athena-result-bucket:
  33.       name: test-athena-results
  34.     my-bucket_name-bucket:   
  35.       name: my-bucket_name-test
  37.   # glue:
  38.   #   reporting-databse: 
  42. functions:
  43.   lambda_handler:
  44.     handler: handler.lambda_handler
  45.     timeout: 300
  46.     events:
  47.       - eventBridge:
  48.           schedule: cron(0 0 ? * 4#1 *)
  49.       - s3:
  50.           bucket: athena-result-bucket
  51.           event: s3:ObjectCreated:*
  52.           # rules:
  53.           #   - prefix: "D_CSV/"
  54.           #   - suffix: ".7z"
  55.       - s3:
  56.           bucket: my-bucket_name
  57.           event: s3:ObjectCreated:*
  58.           # rules:
  59.           #   - prefix: "D_CSV/"
  60.           #   - suffix: ".7z 
  61.           
  62. package:
  63.   exclude:
  64.     - node_modules/**
  66. plugins:
  67.   - serverless-python-requirements
  68.   - serverless-plugin-ifelse
  69.   - serverless-better-credentials

答案1

得分: 2

这将解决您的Lambda关注点：

  1. import os
  3. def lambda_handler(event, context):
  4.     bucket_name = os.environ.get('BUCKET_NAME')
  5.     # 您的其他代码在这里...
  6.     # 使用bucket_name变量调用count_objects函数：
  7.     result = count_objects(bucket_name, year, last_month)
  8.     # 您的其余代码...

使用serverless-dotenv-plugin来处理环境变量。

现在关于如何动态处理阶段：在部署代码时，向CLI命令添加--stage参数：

  1. sls deploy --stage dev
  2. sls deploy --stage prod

此外，您可以像这样命名您的存储桶：

  1. BucketName: yourbucketname-${opt:stage, self:custom.defaultStage}
英文:

This above will resolve your lambda concern:

  1. import os
  3. def lambda_handler(event, context):
  4.     bucket_name = os.environ.get('BUCKET_NAME')
  5.     # Your other code here...
  6.     # Call count_objects function with the bucket_name variable:
  7.     result = count_objects(bucket_name, year, last_month)
  8.     # Rest of your code...

Use serverless-dotenv-plugin to work with environment variables.

Now regarding how to handle the stage dynamically: when deploying the code, add the --stage parameter to the CLI command:

  1. sls deploy --stage dev
  2. sls deploy --stage prod

Also you would name your bucket like this:

  1. BucketName: yourbucketname-${opt:stage, self:custom.defaultStage}

huangapple
  • 本文由 发表于 2023年7月24日 17:23:21
  • 转载请务必保留本文链接：https://go.coder-hub.com/76753061.html
  • amazon-s3
  • amazon-web-services
  • aws-lambda
  • python
  • serverless
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定