How do I set special parameters in an AWS Glue job using AWS CloudFormation?

Short description

To set special parameters for your job in AWS Glue, you must supply a key-value pair for the DefaultArguments property of the AWS::Glue::Job resource in CloudFormation. If you supply a key only in your job definition, then CloudFormation returns a validation error.

Resolution

1.    In your CloudFormation template, set the value of your special parameter to an empty string for the DefaultArguments property of your job definition.

JSON:

"MyJob": {
  "Type": "AWS::Glue::Job",
  "Properties": {
    "Command": {
      "Name": "glueetl",
      "ScriptLocation": "s3://my-test//test-job1"
    },
    "DefaultArguments": {
      "--job-bookmark-option": "job-bookmark-enable",
      "--enable-metrics": ""
    },
    "ExecutionProperty": {
      "MaxConcurrentRuns": 2
    },
    "MaxRetries": 0,
    "Name": "cf-job3",
    "Role": {
      "Ref": "MyJobRole"
    }
  }
}

YAML:

MyJob:
  Type: 'AWS::Glue::Job'
  Properties:
    Command:
      Name: glueetl
      ScriptLocation: 's3://my-test//test-job1'
    DefaultArguments:
      '--job-bookmark-option': job-bookmark-enable
      '--enable-metrics': ''
    ExecutionProperty:
      MaxConcurrentRuns: 
    MaxRetries: 0
    Name: cf-job3
    Role: !Ref MyJobRole

Note: In the preceding example JSON and YAML templates, the value of --enable-metrics is set to an empty string. The empty string validates the template and launches the resource that's configured with the special parameter.

2.    To activate your special parameter, run your job.