What Do We Mean By 'Big Data'

The concept of data originated 7,000 years ago in Mesopotamia when accounting was introduced. We have come a long way since, having major data projects implemented by government for voter registration, tax returns and hospital records to name just a few.

In 2005, the term ‘Big Data’ was coined to refer to a large set of data that is almost impossible to manage and process using traditional business intelligence tools.

It can also be seen as a concept because as the technological capacity to store, process and manage data evolves we open up new dimensions of data insights that were not previously available.

How can we use Big Data in the energy industry?

Big data has become a hot topic as a powerful tool.  In the energy industry, this is also the case.

Rapidly increasing amounts of data are being collected from new sensor technologies, the Internet of Things, network communications technology, cloud computing and smart mobile devices.

This presents huge opportunities for community energy and the wider energy market in general.

Big data insights will enable;

  • Development of more accurate forecasting which will better enable fully renewable energy powered networks.
  • Innovation of customer based (demand side management) solutions that will reduce energy costs whilst supporting the transition to low carbon renewable energy.

What are the four V's of Big Data?

There is a general consensus that big data has characteristics based around the 4 V’s – volume, variety, velocity and value. 

Volume – refers to the magnitude of data which can often be measured in terabytes but more recently in petabytes. It is estimated that 1 million SMART meters recording information every 30 mins will require 1.46 petabytes. (1 petabyte is enough to store 13 billion pictures on Facebook).

Variety – refers to the different types of data. These are structured (spreadsheet), semi-structured (weather data) or unstructured (consumer behaviour data).

Velocity –  refers to the rate at which new data is required to be processed. Data in energy systems can currently be transmitted as fast as 15 minute to sub second intervals.

Value –  Data can be described as having a ‘low value density’ which means the value of data in its original form is low compared to the volume. High value is generated through the analysis of the high volumes of data.

How Big Data can improve the efficiency and reliability of renewable energy

Big data in energy will come in many shapes and sizes with data on production, consumption, weather, geographical location and consumer behaviour expected to play crucial roles in the development of smart-grids, distributed generation and flexibility markets.

One of the hurdles that renewable energy is quickly surpassing is it’s reliance on weather conditions.

By intelligent analysis of consumption, production and forecasting data, renewable energy generators can improve their reliability, promoting increased confidence and encouraging consumer demand. ‘Big Energy Data’ will provide a valuable boost to performance management. 

Big Data Community Energy
Big Data and advanced AI algorithms are transforming all aspects of our modern economy, including community energy. Image: UnSplash

The convergence of the internet and intelligent devices has created the opportunity to collect previously unimaginable quantities of data due to the influx of advanced monitoring sensors, such as smart meters and data loggers that monitor the energy consumption of appliances.

The volume of this data is expanding rapidly with the introduction of smart meters. It is expected that a large utility firm will see a rise from 24 million readings per year to 220 million readings per day.

Data on weather, wind speeds and temperature will play an ever increasingly valuable role as we make the important transition to a zero carbon economy. Weather data is the next generation infrastructure platform including maps and location data that are fundamental to the operation of so many other applications.

Geographic Information Systems (GIS) data also maintains a crucial role in energy data transformation by providing the geographical features of specific regions.

Who can you trust with your data?

Since the rise of Big Data we have seen abundant examples of data misuse and theft.

The documentary ‘The Great Hack’, which looks specifically into the Cambridge Analytica scandal and data misuse in the UK and US political processes, has elevated social debate regarding data, who we trust with it and how it is used.

Energy companies will have to request your permission before collecting and sharing your smart meter data, but knowing exactly who is handling your data, and for what purpose can be extremely difficult to discern. 

Take, for example, the ‘Nest’ thermostat, developed by Google.  A new customer has to agree to over 1,000 pages of data contracts in order to access the full spectrum of the device’s benefits. 

Whilst they argue that this will lead to more dynamic pricing for consumers, there are understandable concerns amongst privacy advocates that it’s use will not be clear and obvious.

Typical objections may include:

  • What is the consumer giving permission for?
  • Who is the consumer giving access to their data?
  • How is the consumer’s data is being used?
Big Data Community Energy Neon Computer
The massive increase of Big Data collection has raised concerns over issues of privacy and trust. Photo: UnSplash
Where community energy groups can make a valuable contribution

As part of a network of community energy groups across the UK and Europe, BHESCo is a consumer co-operative which means we are owned by our members and our community.

For this reason, we are 100% transparent and accountable to all of our customers and our members.

Our customers are our neighbours, community centres, schools and churches.  BHESCo is committed to developing long term relationships built on trust. We believe that trust will become a central feature for consumers across all industries, which goes beyond how data is handled.

Already BHESCo use remote monitoring technologies to assess the performance of our various solar power systems across Sussex. We use this data only to maximise systems’ performance on behalf of our customers and to address any issues that may arise.

We hope that as technologies improve we will be able to gain greater insight into the energy use of people in communities, enabling us to identify the most cost-effective solutions to reduce their energy bills and carbon emissions.

With community energy groups, consumers can be sure that that their data is safe and secure and will not be passed on or sold to third parties for marketing purposes, profit or other exploitative gain.


The Age of Surveillance Capitalism – Sushan Zuboff