Tag Archives: big data

An introduction to the Big Data Landscape

What is Big Data?

Data has always been an important asset in every industry. Since the early days of the information age, business intelligence and descriptive statistics have been used as the standard tools for extracting information and make important decisions from all kinds of collected data. However, as the cost of collecting, storing, and processing data has been dropping exponentially, the amount and the diversity of the data has reached the point where traditional approaches are no longer feasible. In fact, the term Big Data is often used to refer to any data that requires new techniques and tools in order for it to be processed and analyzed.

A more formal definition of Big Data was introduced by Gartner in 2012, in which the well-known 3Vs – Volume, Velocity, and Variety – were used to characterize Big Data. Since then, the 3Vs model has been expanded to several other characteristics, including a fourth V for Veracity and more recently a fifth for Value. Without going into the details of each V, and resisting the temptation to look for a sixth V, we could also look at Big Data from the point of view of the new set of technologies that are helping to solve the challenges in collecting, managing, and analyzing Big Data. These technologies include cloud computing and cluster computing (with Hadoop MapReduce as the most well-known example) for data storage and manipulation, and machine learning for data analysis.

What value does Big Data bring?

The value of Big Data comes from two main use cases: as a source of analytics, and as an enabler of new products and services. In the first use case, big data analytics is used to improve an existing business model by revealing insights from data which was previously too costly to store or process. Amazon’s recommendation system, USPS’s preventive maintenance system, or Walmart’s demand forecasting system are all very good examples. These companies track, collect, and store all available data, from customer transactions to social data, from GPS trails to geographical and meteorological data, then combine them together and use big data analytics to produce high value actionable insights. This would not be possible without the new big data technologies such as cluster computing and machine learning.

In the second use case, big data technologies open up completely new business models and introduce new products and services. Many recent so-called unicorn startups, such as Airbnb, Uber, or Snapdeal, are founded on and enabled by big data analytics. Their products have unique features thanks to the new technologies that they are using. Without following the big data approach, their products would not be able to compete against all the traditional business models.

Big data at eSmart Systems

Since the early of the company, big data has always been considered as our core value. Big data analytics and machine learning are involved in many of our products.  They are used to forecast electricity demand at substation level, segment customers based on their power consumption patterns, and implement demand response strategies. We have also started to use machine learning to automate the analysis of power lines imaging surveys autonomously conducted by drones with the aim of assisting failure detection and preventive maintenance.

Written by:
Dang Ha The Hien
Dang Ha The Hien2
PhD-student, eSmart Systems

Azure Machine Learning helping the smart grid become smarter

A few weeks ago Microsoft published a blog post on their Machine Learning Blog mentioning eSmart Systems. Last week they posted a case study of eSmart Systems.

"We just published a new case study where partner eSmart Systems helps Utilities keep the lights on with an automated energy management system based on Microsoft cloud services, including Azure Machine Learning (ML). What does this mean for Utilities?  Utility companies can now connect sensors, smart meters, and software to forecast consumption, reduce outages, and monitor assets to improve efficiencies. It helps Utilities avoid costly investment in both grid and IT infrastructure while serving an ever-growing and evolving  smart energy ecosystem through smarter solutions that leverage leverage the enormous capabilities of the Microsoft cloud", writes Microsoft in their Power and Utilities Blog. 
Smarter Grids

Energy companies can meet complex challenges—including massive population growth—more easily and affordably by using eSmart and Microsoft technologies to monitor assets and improve efficiencies.

– We need a cloud solution that really scales, and we also need the computing ability to perform big data analysis and predictions, says Knut Johansen, CEO in eSmart Systems, in the case study. Kontor11 v2

– The fact that Microsoft chooses to make a case study about eSmart Systems is a distinction for our work around predictions and Big Data, says Erik Åsberg, Head of Development, eSmart Systems.

Read the full blog post here.
Read the case study here.

Webinar sammen med Microsoft

Fredag 9.januar holdt eSmart Systems og Microsoft et webinar om Big Data-løsninger og skyteknologi.

Big Data blir gjerne omtalt som et buzzword, men faktum er at mange bransjer og virksomheter vil bli påvirket av de teknologiske endringene som kommer. Energibransjen er blant dem. I fremtiden vil vi hente data fra mange ulike kilder, blant annet smarte strømmålere, sensorer, smarte komponenter i nettet, og eksterne kilder som sosiale nettverk. Utviklingen innen smarte nett skyter fart både i Norge og internasjonalt, og mengden data som må håndteres økes hurtig. De nærmeste årene skal rundt 2,5 millioner strømmålere i Norge byttes ut med smarte målere hvor forbruket blir meldt inn automatisk. I Europa skal 80% av el-målerne i alle EU-land være byttet ut med smarte målere innen 2020. I tillegg er smarte boliger og bygg på full fart inn i energisystemet, der både energiforbruk, lagring og lokal produksjon trenger nye systemer for styring og informasjon.

Big Data benyttes for å kunne ta bedre beslutninger basert på informasjon man har i egne systemer. Stadig flere står overfor en uhåndterlig mengde data som øker i voldsom takt. Overgangen til smarte målere vil gi en rekke fordeler for energi- og nettselskap, men for å dra fordeler av dette er det nødvendig med nye IT-systemer som er i stand til å bearbeide betydelig større datamengder og gjennomføre tilhørende analyser i sann tid.

Det finnes nå systemer som benytter seg av det siste innenfor teknologi for datahåndtering. Med skytjenesten Microsoft Azure som grunnmur har eSmart Systems designet et system som håndterer mengden og variasjonen av data, og oversetter den til innsikt. Ved bruk av skyteknologi har de klart å kombinere ulike lagringsmetodikker for å håndtere store datamengder som både kan være strukturerte og ustrukturerte, sette opp prediksjoner og avanserte maskinlæringsalgoritmer. Big Data-løsninger kombinert med skyteknologi gir brukerne muligheten til å få oversikt over dataene som produseres i nettet. Dette gir en stor merverdi i en bransje som er vant med mange silobaserte systemer som ikke snakker sammen, og ofte har begrensede analyse- og visualiseringsmuligheter.

Big Data – slett ikke et buzzword.

Strengthening our R&D team

eSmart Systems possesses great expertise in IT and energy and the company is growing rapidly. The latest addition is Dang Ha The Hien, in the R&D team.

Dang Ha The Hien2
Dang is 26 years old, he earned his Bachelor Degree in Vietnam before he moved to Norway in 2011. He graduated with the degree Master in Computer Science from Østfold University College in 2014.

This fall Dang started an industrial phd hosted by eSmart. The phd is a collaboration between eSmart, University of Oslo and the Norwegian Research Council.

In his research Dang is taking advantage of Big Data technology to develop a platform for real-time analysis. By utilizing this technology, he can develop smart energy models that will provide critical decision support including optimal network operations for utility companies. The models will combine both structured and unstructured information from social media with smart energy data and provide utility companies with valuable insights. These insights can be used to improve the efficiency of the companies own operations, as well as better utilization of the energy market in its entirety.

Dang is involved in various projects at eSmart. He works closely with Microsoft in a project that focus on Machine Learning and prediction. Currently he is designing a model to be implemented and integrated in Microsofts cloud platform, Azure. The project it’s about building a complete model which can forecast long-term and short-term future consumption at any given node on the grid. This information can be used to optimize the grid power flows, overload early detection, or reschedule flexible loads. The project will be completed in December.

The Nexus of Forces

After working with relational and transactional systems for nearly two decades, the limitations of traditional technology became more and more obvious for us. It became very clear that this is not the future. A system built on such principles cannot handle the constant increase of data, both in volume, velocity and variety. We were already struggling with performance in our exchange settlement systems and trading and risk management systems. Therefore, when the chance to start over came along, using all kinds of technology now available, it was not a doubt in our minds that this is what we must do.
Gartner has defined the change now happening as “The Nexus of Forces”. Four trends, The Nexus of Forceseach one of them innovative and disruptive on their own, are now happening at once: Social, Mobile, Cloud and Information. Together they are revolutionizing business and society, disrupting old business models and creating new leaders. The Nexus is the basis of the technology platform of the future.

The Nexus of Forces
The Nexus of Forces
Source: Gartner, 2013

For an energy company, social media might not be obvious. However, it is easy to picture customer portals that incorporate social media. In addition, what if utilities could tap into social media to get near real-time information about what is going on in their area? Information that is not possible to reach through press or other official channels. Our system has a wide spread of social media

As everyone becomes constantly online, new possibilities arises. Wherever you are, you want to be able to check the status of your energy trading portfolio, whether your grid is operating as it should, and if there is a problem you should know. Our system provides client agnostic services to support all kind of client devices and receiving systems.

The cost of having on premise data centers are high and it is expensive to meet the availability, security and scalability of today’s cloud services. Our system is native cloud based.

A huge amount of data is collected daily, but the full potential of all this data is rarely reached. With new technologies like noSQL databases and analytic platforms like Hadoop, the possibility to reach the full potential is closer than ever. With the guidance of industry leaders we have established a data platform that lets you handle huge amounts of data in near real-time and do deep analytics and predictions to unlock hidden information and take right decisions.

By utilizing the Nexus of Forces, we are building the next generation IT systems for tomorrow’s energy companies.