Please subscribe to the product from AWS Marketplace and Activate the Glue connector from AWS Glue Studio .

Pre-requisite

• A SFTP environment.
• AWS Secrets Manager.

Create a new secret for SFTP Storage in AWS Secrets Manager

We create a secret in AWS Secrets Manager to store the SFTP Storage credentials.

1. Please prepare SFTP Storage and its credentials in advance.
2. On the Secrets Manager console, choose Store a new secret.
3. For Secret type, select Other type of secret.
4. Enter key as host for SFTP storage host.
5. Enter keys as username and password(Optional) for SFTP credential.
6. Enter key as keyS3Uri for SFTP secret key file. Value is s3 full path where key file uploaded, like s3://myBucket/myKeyFile.
7. Leave the rest of the options at their default.
8. Choose Next.
9. Give a name to the secret sftp_credentials.
10. Follow through the rest of the steps to store the secret.

Create a custom connection

Select the created Secret name, sftp_credentials, in connection edit page. Then save it.

Create a Glue job and set connection options

Create a Glue Job and specify details:

1. Create a job of data source or target from this connector, select custom connection. Then input options and values.
2. File format connection options, we support 5 different formats, csv/parquet/json/orc/text. The basic options are path, the SFTP storage cloud storage URI, e.g. /input/covid-csv-data/. fileFormat, input or output file format, e.g. csv/parquet/json/orc/text
3. For each format, there are different connection options supported. CSV: Option name, header, delimiter, compression, Option Value corresponding,true/false, any delimiter char, none/uncompressed/snappy/gzip/lzo/lz4/brotli/zstd, default value is false, , none. PARQUET, ORC, TEXT and JSON: Option name, compression, Option Value, none/uncompressed/snappy/gzip/lzo/lz4/brotli/zstd, default value is none.
4. Remember to set the Glue version to be Glue 3.0 on job detail tab.