Published on the 23/03/2018 | Written by Empired

AT A GLANCE

• Australian Energy Market Operator (AEMO)

INDUSTRY

• Energy regulator

BUSINESS OBJECTIVE

• Let users analyse datasets at an unprecedented scale, using tools they are familiar with (such as R)
• Use the data in the solution to plan how the network will look in the future and what infrastructure is required to ensure continued high levels of service to Australian energy consumers
• Compare and contrast grids or groups of grids in their own right or against each other in order to uncover hidden trends in the data
• Create unit testable highly scalable forecasting models and able to automate their executions (this is a highly manual labour intensive process at the moment)
• Allow their analysts to focus on their strengths in model creation and algorithms instead of focusing on data engineering elements

SOLUTION

• Microsoft Azure

BUSINESS BENEFITS

• Ability to access remote data from energy grids across Australia every 2 minutes
• Ingest, store and consolidate data from across Australia in one environment
• Analyse and investigate energy consumptions across Australia to prevent power disruption
• Uncover hidden trends in the data to improve service levels to Australians
• Forecast energy consumption every 30 minutes, for the next 20 years
• Investigate the historic energy consumption of any grid across Australia.

FOR MORE INFORMATION

Empired
W: https://www.empired.com/
E: contact@empired.com
T:
Perth – 08 6333 2200
Sydney – 02 8355 8100
Melbourne – 03 8658 5300
Brisbane – 07 3067 5400
Adelaide – 08 7333 4200

The situation
The Australian Energy Market Operator’s (AEMO) vision is to deliver energy security for all Australians. It administers and operates the country’s wholesale national energy markets, with around 18 terabyte of data stored across a myriad of different on-premises systems, including some high-volume transactional systems.

With energy consumption on the rise around the world, recent heatwaves saw Australians use more electricity than ever before. This anomaly highlighted how difficult it is to gain insights into how electricity grids cope with the ebb and flow of energy consumption across all consumer and commercial meters.

The opportunity
Historically, AEMO’s data was used in isolation across its many systems. Different business units acted in silos, generating their own reports without a consolidated, single source of truth, meaning energy grids couldn’t be compared and increases or declines were not noticed. An internal initiative to aggregate data in a pseudo data warehouse was carried out, however the volume of data proved too much as certain queries were taking days to execute and ministerials were taking weeks or even months to answer.

Empired helped AEMO use big data to drive smarter ways to save energy consumption and improve forecasting accuracy across Australia. This solution is key to planning how the network will look in the future and the infrastructure required to ensure continued high levels of service to energy consumers.

The technology
Using Microsoft Azure’s best-of-breed data platform with rich ecosystems of data components, catering for AEMO’s evolving needs, Empired assisted AEMO in tapping into their energy consumption data. This data intelligence solution allows data to be analysed right across the country and across grids every two minutes in a flexible way.

Energy data, population data and weather data can all be factored in and assessed to forecast a future approximate for the next 20 years in 30 minute intervals. Furthermore, Azure provides the elasticity to be scalable depending on AEMO’s needs, achieving cost savings by scaling up only when required, then pausing or powering down when not in use, such as weekends. This is one of the most complicated and sophisticated uses of data intelligence in the world.

Empired designed and built big data warehouses using Azure SQL Data Warehouse, Azure Data Lake, Power BI, machine learning, Python, Spark, R and Spark-R, Data Factory and HDInsight. The data flows were designed to load data from AEMO’s on-premises systems to the Azure-based data warehouses, automating the build of the data flows from more than 700 tables (some larger than one terabyte) from onpremises systems in Azure. Australian-based data centres ensured all information was stored in-country.

“It is magic. But it is a true story and an enormous success story.”
Ben Johnson, National Business Manager, Data Insights & Integration, Empired

Source: This article was originally sourced from Empired

FURTHER READING