Short-term Electric Load Forecasting with Amazon Forecast

Today, the majority of electric residential, commercial, and industrial (RCI) customers receive their electricity over electric distribution circuits called “feeders.” Electric distribution feeders originate at circuit breakers within a distribution substation, and serve many customers, as shown in the following graphic. To increase grid reliability, feeders are often configured to receive electricity from multiple substations, but only one of these connection switches is active at any one time. Should a wire break occur at one end of the feeder, switches are opened and closed in a prescribed manner (“switching”), to restore power to the feeder from the other end.

Figure One: Image Showing Substation, Feeder, and Customers

The electric load of a feeder, measured in kilowatts (kW), is dynamically fluctuating and has a nonlinear hourly, daily, and seasonal pattern. Accurate short-term electric load forecasting (ST-ELF) of feeder and substation demand is an integral part of electric utilities’ operation and planning, as supply and demand on the grid must be kept electrically balanced at all times to avoid grid negative impacts, such as brown-outs and service outages. Many factors complicate ST-ELF, including the high penetration of intermittent distributed generation (for example, RCI solar, energy storage, and wind generation systems) connected to the grid.

The main goal of ST-ELF is to predict the near future net feeder load in next 1-14 days. The net feeder load is the load measured at the feeder circuit breaker, which includes the aggregate of all types of customer’s electric consumption and generation connected to the grid plus the electric losses of the lines and transformers. Utility control center operators and grid operational planning engineers are typically the end users of ST-ELF. There are many use cases requiring ST-ELF, including:

• Operational planning to effectively perform distribution operations such as circuit switching to allow maintenance or to restore power and dynamic control the load. This is done to ensure that the new distribution circuit arrangement configured by switching, can adequately handle the customer loads.
• Facilitating Non-Wire Alternatives (NWA) that are intended to defer or remove the need to construct or upgrade overloaded (or marginally overloaded) distribution system components such as lines or transformers. In some cases, to enable NWA, utilities are procuring power from Distributed Energy Resources (DER) owners to provide load relief to an overloaded asset (such as power transformer, gateway cable) of a feeder/substation. Using accurate ST-ELF utilities can identify and enable potential NWA needs. Enabling NWA solutions addresses grid infrastructure needs at a lower cost with higher customer and environmental benefits.
• Identifying and enabling which feeder circuits, and in-turn which customers, should be called upon to reduce their energy consumption during periods of peak demand, to avoid distribution circuit overloads. These are known as Demand Response programs, and rely upon ST-ELF as key inputs.
• Optimizing utility owned Energy Storage (ES) charge and discharge patterns to optimize energy supply and demand on the feeder, during both normal operation and also for emergency load relief.

Conventional approaches have been used to forecast feeder/substation electric demand manually with the help of spreadsheets, charts, and tables, but these methods cannot provide accurate and reliable forecasts. With the advent of artificial intelligence and machine learning (AI/ML) techniques, researchers are now improving the modeling of ST-ELF. These approaches typically involve AI/ML and data science expertise to build and maintain ST-ELF models. Today, these skillsets typically fall outside of the areas of expertise of many utilities.

ST-ELF on AWS

The purpose of this blog is to demonstrate one method of how effortlessly you can automate the data extraction, transformation, and building of an accurate ST-ELF forecasting pipeline with AWS services such as Amazon Forecast. Amazon Forecast is a fully managed service that uses advanced machine learning techniques to deliver highly accurate forecasts. This methodology paves the way for utilities to generate accurate ST-ELF models without incurring significant investment in AI/ML and data scientists.

Amazon Forecast puts the power of Amazon’s extensive forecasting experience into the hands of all developers, without requiring machine learning expertise. It is a fully managed service that delivers highly accurate forecasts, up to 50% more accurate than traditional methods. As shown in the following diagram, historical electric demand data can be sent to Amazon Forecast. The service then automatically sets up a data pipeline, ingests data, trains a model based on historical data, provides accuracy metrics, and generates forecasts. It identifies features, applies the most appropriate algorithm for the data, and automatically tunes hyperparameters. Amazon Forecast will then host the models so they can be easily queried when needed. With all of this work done behind the scenes, utilities can save time and effort by not building their own machine learning expert team or resources to maintain in-house models. In addition, Amazon Forecast Weather Index is a built-in feature that incorporates historical and projected weather information into the model (for US, Canada, and Europe locations; with additional ones planned over time). When the Weather Index is enabled, Amazon Forecast automatically applies the weather feature only to time series where it finds improvements in accuracy during predictor training, which is especially helpful when considering distributed generation systems like solar.

Figure Two: Overview of Solution Approach

Solution Architecture

The following diagram illustrates an architecture of a solution for feeder-level forecasting for a utility ST-ELF use case. This solution is an automated solution, created in early 2021, that was verified on publicly available data of a utility located in North-East USA, on approximately 10 feeders with at least two years hourly historical data. (Note: This approach may be generalized even further using AWS’ latest data preparation tools such as AWS Glue Data Brew or Sagemaker Data Wrangler). Accuracy of the forecast results was calculated using Mean Absolute Percentage Error (MAPE) index and averaged 5–7%. This solution requires zero AI/ML expertise, as it is leveraging AWS Managed Services.

Figure Three: Architecture Overview

The solution approach includes these broad steps, which are explained further:

• Gold Module 1 – Ingest and Transform data from the on-premises OSIsoft PI (PI) system
• Blue Module 2 – Forecast: Calling series of APIs to Amazon Forecast
• Green Module 3 – Monitoring and Notification
• Red Module 4 – Evaluation and Visualization

The left box on previous diagram (labeled “On-Prem”) is an example of field data ingestion that a typical utility has established on-premises. In this case, data ingestion from field sensors (feeder head meter) is already in place through the PI system. Therefore, the starting point is raw feeder circuit demand data extracted in excel format and stored on-premises. In order for Amazon Forecast to use this data, we first need to upload it to Amazon Simple Storage Service (Amazon S3), which can be performed periodically by a scheduled script.

Next (in Module 1), as the raw data is available in AWS S3 in utility custom-built formats, we need to massage the data. This involves extracting target data (real power in kW), cleaning the data, and reformatting it. To do this, we use the services outlined in the box labeled “ETL & Data Lake” as follows:

• To store raw and formatted data, Amazon S3 is used. This highly durable and highly available object storage is where the ST-ELF input datasets will be consumed by Amazon Forecast.
• To massage and transform the raw data, AWS Lambda is used. This serverless compute service runs your code in response to events (in this case, new raw file uploads) and automatically manages the computing resources required by the code.
• The Amazon Simple Queue Service (Amazon SQS), a fully managed message queuing service, decouples these transformation AWS Lambda functions. The queue acts as a buffer and can help smooth out traffic for the systems when consuming a multitude of events from many field sensors.
• For data cataloging, AWS Glue is used. This service can run a crawler to update tables in your Glue Data Catalog, and as configured here runs on a daily schedule.

With data wrangling complete, the ST-ELF model is now ready for training. To train the model, we use Amazon Forecast (in Module 2). Using Amazon Forecast requires the import of training data, creation of a predictor (the ST-ELF model in this case), and creation of a forecast using the model, and finally exporting of the forecast and model accuracy metrics to Amazon S3.

To streamline the process of ingesting, modeling and forecasting multiple ST-ELF models, the Improving Forecast Accuracy with Machine Learning solution is leveraged. This best practice AWS solution streamlines the process of ingesting, modeling, and forecasting using Amazon Forecast by providing AWS CloudFormation templates and a workflow around the Amazon Forecast service.

Since forecasting might take some time to complete, the solution uses the Amazon Simple Notification Service (Amazon SNS) to send an email when the forecast is ready. Also, all AWS Lambda function logs are captured in Amazon CloudWatch (Module 3).

Once the forecast is ready, the solution ensures your forecast data is ready and stored in Amazon S3. This fulfills 14-day ahead forecasting need. The solution also provides a mechanism to query the forecast input and output data with SQL using Amazon Athena (Module 4). Additionally, the solution can automatically create a visualization dashboard in the AWS Business Intelligence (BI) service Amazon QuickSight, which allows you to visualize the data interactively in a dashboard:

Figure Four: Example Feeder Forecast Dashboard in Amazon QuickSight

In order to assess the accuracy of the forecast results, you can calculate MAPE index (a standard measure of prediction accuracy of a forecasting method.) of actual test data against generated forecast using an Amazon SageMaker Jupyter notebook. This provides a mechanism to evaluate forecast performance. In the following notebook screen shot, you can see MAPE results for a feeder that is approximately 6% (which is a remarkably low forecast error for a dynamic feeder power profile, as it is typically 20% to 50%, when done using conventional non-ML methods).

Figure Five: Jupyter notebook screen shot showing MAPE calculation

Summary

This post discussed the importance and use cases of Short-term Electric Load Forecasting (ST-ELF) for electric utilities, then proposed a solution architecture to leverage AWS Managed AI/ML service called Amazon Forecast to generate an accurate forecast with no AI/ML expertise required. This solution is fully automated and was tested and verified on multiple actual feeders. Forecast performance measured in MAPE with the average of 5-7% among randomly tested feeders.

Learn more about the latest ways AWS is helping Utilities by browsing our blog or AWS for Power and Utilities.