Please subscribe to the product from AWS Marketplace and Activate the Glue connector from AWS Glue Studio .

Pre-requisite

• An account in Google Cloud, specifically a service account that has permissions to Google BigQuery
• GCP credentials (service_account_json_file)
• GCS bucket (only for writes)
• BigQuery dataset (only for writes)
• AWS Secrets Manager secret (you can create the secret in following steps)

Create a new secret for Google BigQuery in AWS Secrets Manager

We create a secret in AWS Secrets Manager to store the Google service account file contents as a base64-encoded string.

1.Download the service account credentials JSON file from Google Cloud.

• For base64 encoding, you can use one of the online utilities or system commands to do that. For Linux and Mac, you can use base64 [service_account_json_file] to print the file contents as a base64-encoded string.

2. On the Secrets Manager console, choose Store a new secret.
3. For Secret type, select Other type of secret.
4. Enter your key as credentials and the value as the base64-encoded string.
5. Leave the rest of the options at their default.
6. Choose Next.
7. Give a name to the secret bigquery_credentials.
8. Follow through the rest of the steps to store the secret.

Connection options

You can pass the following options to the connector.

• parentProject (required): The Google Cloud Project ID of the table
• dataset(optional unless omitted in table): The BigQuery dataset containing the table.
• table (required): The BigQuery table in the format [[project:]dataset.]table
• temporaryGcsBucket (optional. required for writes):

You can see other available options here: https://github.com/GoogleCloudDataproc/spark-bigquery-connector/tree/0.24.2 

Spark configurations

Following Spark configurations are required only for writes into BigQuery.

• spark.hadoop.fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem
• spark.hadoop.google.cloud.auth.service.account.json.keyfile=true

You also need to configure credentials in one of following set of configurations.

Credential file

• spark.hadoop.fs.gs.auth.service.account.json.keyfile=credentials.json

You need to upload credentials.json to your S3 bucket, and set the file path in Referenced files path.

Private key

• spark.hadoop.fs.gs.auth.service.account.email= [your-email-extracted-from-service_account_json_file]
• spark.hadoop.fs.gs.auth.service.account.private.key.id= [your-private-key-id-extracted-from-service_account_json_file]
• spark.hadoop.fs.gs.auth.service.account.private.key= [your-private-key-body-extracted-from-service_account_json_file]

You can set these Spark configurations in one of following ways.

• The param --conf of Glue job parameters
• The job script using SparkConf

from pyspark.conf import SparkConf conf = SparkConf() conf.set("spark.hadoop.fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem") conf.set("spark.hadoop.fs.gs.auth.service.account.enable", "true") conf.set("spark.hadoop.google.cloud.auth.service.account.json.keyfile", "credentials.json")