Automatically Create Machine Learning Models

To start a new SageMaker Autopilot experiment, in the Launcher window, scroll down to ML tasks and components, click the + icon in the tab for New Autopilot experiment.

Alternatively, you can also click on the File from the top menu and select New and then New experiment.

Leave the manifest file option set to Off. In the Target dropdown, select y as the target feature which our model will attempt to predict.

For the Output data, you can optionally choose your own S3 bucket by toggling the flag for Auto create output data location and specifying the S3 bucket in the Output data (S3 bucket) section. We are going to leave it as default and let Sagemaker Studio create an output bucket. Then select Next: Training method

For the training method, you can keep default option selected Auto (or optionally select Ensembling or Hyperparameter optimization(HPO). Then select Next: Deployment and advanced settings

For Deployment and advanced settings, you can keep everything as default. (Optionally, you specify the Auto deploy endpoint name and Select the machine learning problem from the drop down.) Then select Next: Review and create

You can review the summary about the experiment. If you wish to modify any option you can click on the previous button and modify it. Click the Create Experiment button to start the first stage of the SageMaker Autopilot experiment. SageMaker Autopilot will begin to run through the phases of an experiment. In the experiment window, you can track progress through the phases of preprocessing, candidate definitions, feature engineering, model tuning, explainability, and insights.

Once the SageMaker Autopilot job is complete, you can access a report that shows the candidate models, candidate model status, objective value, F1 score, and accuracy. SageMaker Autopilot will automatically deploy the endpoint.

Now that the SageMaker Autopilot experiment is complete, you can open up the top ranking model to obtain more details on the model’s performance and metadata. From the list of models, highlight the first one and right click to bring up model options. Click on Open in model details to review the model’s performance statistics.

In the new window, click on Explainability. The first view you see is called Feature Importance and represents the aggregated SHAP value for each feature across each instance in the dataset. The feature importance score is an important part of model explainability because it shows what features tend to influence the predictions the most in the dataset. In this use case, the customer duration or tenure and employment variation rate are the top two fields for driving the model's outcome.

Now, click on the tab Performance. You will find detailed information on the model’s performance, including recall, precision, and accuracy. You can also interpret model performance and decide if additional model tuning is needed.

Next, visualizations are provided to further illustrate model performance. First, look at the confusion matrix. The confusion matrix is commonly used to understand how the model labels are divided among the predicted and true classes. In this case, the diagonal elements show the number of correctly predicted labels and the off-diagonal elements show the misclassified records. A confusion matix is useful for analyzing misclassifications due to false positives and false negatives.

Next, look at the precision versus recall curve. This curve interprets the label as a probability threshold and shows the trade-off that occurs at various probability thresholds for model precision and recall. SageMaker Autopilot automatically optimizes these two parameters to provide the best model.

Next, look at the curve labelled Receiver Operating Characteristic (ROC). This curve shows the relationship between the true positive rate and the false positive rate over a variety of potential probability thresholds. A diagonal line represents a hypothetical model based on random guessing. The more this curve pulls to the upper left of the chart, the better the model will perform.

The dashed line represents a model with 0 predictive value, which is often called the null model. The null model would randomly assign a 0/1 label, and its area under the ROC curve would be 0.5, representing that it would be accurate 50% of the time.

Next, click on the tab Artifacts. You can find the SageMaker Autopilot experiment’s supporting assets, including feature engineering code, input data locations, and explainability artifacts.

Finally, click the Network tab. You will find information about network isolation and container traffic encryption.

Click the + icon to bring up a new Python notebook. Select Python3 as the kernel.

To know where to send the request, look up the model endpoint’s name. On the left pane, click the SageMaker Resources icon. In the SageMaker resources pane, select Endpoints. Click on the endpoint associated with the experiment name you created at the start of this tutorial. This will bring up the Endpoint Details window. Record the endpoint name and navigate back to the Python 3 notebook.

Copy and paste the following code snippet into a cell in the notebook, and press Shift+Enter to run the current cell. This code sets the environment variable ENDPOINT_NAME and runs inference. As the code completes, you will see a result consisting of the model label and associated probability score.

import os
import io
import boto3
import json
import csv

#: Define the endpoint's name.
ENDPOINT_NAME = 'autopilot-experiment-6d00f17b55464fc49c45d74362f284ce'
runtime = boto3.client('runtime.sagemaker')

#: Define a test payload to send to your endpoint.
payload = {
    "data":{
        "features": {
            "values": [45,"blue-collar","married","basic.9y",'unknown',"yes","no","telephone","may","mon",461,1,999,0,"nonexistent",1.1,93.994,-36.4,4.857,5191.0]
        }
    }
}

#: Submit an API request and capture the response object.
response = runtime.invoke_endpoint( 
EndpointName=ENDPOINT_NAME, 
ContentType='application/json', 
Body=json.dumps(payload)
)

#: Print the model endpoint's output.
print(response['Body'].read().decode())

Congratulations! You have learned how to use SageMaker Autopilot to automatically train and deploy a machine learning model.

If you ran the CloudFormation template in Step 1 to create a new SageMaker Studio domain, continue with the following steps to delete the domain, user, and the resources created by the CloudFormation template.

To open the CloudFormation console, enter CloudFormation into the AWS console search bar, and choose CloudFormation from the search results.

In the CloudFormation pane, choose Stacks. From the status drop down list, select Active. Under Stack name, choose CFN-SM-IM-Lambda-catalog to open the stack details page.

On the CFN-SM-IM-Lambda-catalog stack details page, choose Delete to delete the stack along with the resources it created in Step 1.