Set Subnet ID and EC2 Key Name in EMR Cluster Config via Step Functions

As of November 2019 AWS Step Function has native support for orchestrating EMR Clusters. Hence we are trying to configure a Cluster and run some jobs on it.

We could not find any documentation on how to set the SubnetId as well as the Key Name used for the EC2 instances in the cluster. Is there any such possibility?

As of now our create cluster step looks as following:

"States": {
          "Create an EMR cluster": {
            "Type": "Task",
            "Resource": "arn:aws:states:::elasticmapreduce:createCluster.sync",
            "Parameters": {
              "Name": "TestCluster",
              "VisibleToAllUsers": true,
              "ReleaseLabel": "emr-5.26.0",
              "Applications": [
                { "Name": "spark" }
              ],
              "ServiceRole": "SomeRole",
              "JobFlowRole": "SomeInstanceProfile",
              "LogUri": "s3://some-logs-bucket/logs",
              "Instances": {
                "KeepJobFlowAliveWhenNoSteps": true,
                "InstanceFleets": [
                  {
                    "Name": "MasterFleet",
                    "InstanceFleetType": "MASTER",
                    "TargetOnDemandCapacity": 1,
                    "InstanceTypeConfigs": [
                      {
                        "InstanceType": "m3.2xlarge"
                      }
                    ]
                  },
                  {
                    "Name": "CoreFleet",
                    "InstanceFleetType": "CORE",
                    "TargetSpotCapacity": 2,
                    "InstanceTypeConfigs": [
                      {
                        "InstanceType": "m3.2xlarge",
                        "BidPriceAsPercentageOfOnDemandPrice": 100                         }
                    ]
                  }
                ]
              }
            },
            "ResultPath": "$.cluster",
            "End": "true"
          }
}

As soon as we try to add "SubnetId" key in any of the subobjects in Parameters, or in Parameter itself we get the error:

Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The field "SubnetId" is not supported by Step Functions at /States/Create an EMR cluster/Parameters' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition;

Referring to the SF docs on the emr integration we can see that createCluster.sync uses the emr API RunJobFlow. In RunJobFlow we can specify the Ec2KeyName and Ec2SubnetId located at the paths $.Instances.Ec2KeyName and $.Instances.Ec2SubnetId.

With that said I managed to create a State Machine with the following definition (on a side note, your definition had a syntax error with "End": "true", which should be "End": true)

{
"Comment": "A Hello World example of the Amazon States Language using Pass states",
"StartAt": "Create an EMR cluster",
"States": {
    "Create an EMR cluster": {
        "Type": "Task",
        "Resource": "arn:aws:states:::elasticmapreduce:createCluster.sync",
        "Parameters": {
            "Name": "TestCluster",
            "VisibleToAllUsers": true,
            "ReleaseLabel": "emr-5.26.0",
            "Applications": [
                {
                    "Name": "spark"
                }
            ],
            "ServiceRole": "SomeRole",
            "JobFlowRole": "SomeInstanceProfile",
            "LogUri": "s3://some-logs-bucket/logs",
            "Instances": {
                "Ec2KeyName": "ENTER_EC2KEYNAME_HERE",
                "Ec2SubnetId": "ENTER_EC2SUBNETID_HERE",
                "KeepJobFlowAliveWhenNoSteps": true,
                "InstanceFleets": [
                    {
                        "Name": "MasterFleet",
                        "InstanceFleetType": "MASTER",
                        "TargetOnDemandCapacity": 1,
                        "InstanceTypeConfigs": [
                            {
                                "InstanceType": "m3.2xlarge"
                            }
                        ]
                    },
                    {
                        "Name": "CoreFleet",
                        "InstanceFleetType": "CORE",
                        "TargetSpotCapacity": 2,
                        "InstanceTypeConfigs": [
                            {
                                "InstanceType": "m3.2xlarge",
                                "BidPriceAsPercentageOfOnDemandPrice": 100
                            }
                        ]
                    }
                ]
            }
        },
        "ResultPath": "$.cluster",
        "End": true
    }
}
}