An introduction to the Big Data Landscape

What is Big Data?

Data has always been an important asset in every industry. Since the early days of the information age, business intelligence and descriptive statistics have been used as the standard tools for extracting information and make important decisions from all kinds of collected data. However, as the cost of collecting, storing, and processing data has been dropping exponentially, the amount and the diversity of the data has reached the point where traditional approaches are no longer feasible. In fact, the term Big Data is often used to refer to any data that requires new techniques and tools in order for it to be processed and analyzed.

A more formal definition of Big Data was introduced by Gartner in 2012, in which the well-known 3Vs – Volume, Velocity, and Variety – were used to characterize Big Data. Since then, the 3Vs model has been expanded to several other characteristics, including a fourth V for Veracity and more recently a fifth for Value. Without going into the details of each V, and resisting the temptation to look for a sixth V, we could also look at Big Data from the point of view of the new set of technologies that are helping to solve the challenges in collecting, managing, and analyzing Big Data. These technologies include cloud computing and cluster computing (with Hadoop MapReduce as the most well-known example) for data storage and manipulation, and machine learning for data analysis.

What value does Big Data bring?

The value of Big Data comes from two main use cases: as a source of analytics, and as an enabler of new products and services. In the first use case, big data analytics is used to improve an existing business model by revealing insights from data which was previously too costly to store or process. Amazon’s recommendation system, USPS’s preventive maintenance system, or Walmart’s demand forecasting system are all very good examples. These companies track, collect, and store all available data, from customer transactions to social data, from GPS trails to geographical and meteorological data, then combine them together and use big data analytics to produce high value actionable insights. This would not be possible without the new big data technologies such as cluster computing and machine learning.

In the second use case, big data technologies open up completely new business models and introduce new products and services. Many recent so-called unicorn startups, such as Airbnb, Uber, or Snapdeal, are founded on and enabled by big data analytics. Their products have unique features thanks to the new technologies that they are using. Without following the big data approach, their products would not be able to compete against all the traditional business models.

Big data at eSmart Systems

Since the early of the company, big data has always been considered as our core value. Big data analytics and machine learning are involved in many of our products.  They are used to forecast electricity demand at substation level, segment customers based on their power consumption patterns, and implement demand response strategies. We have also started to use machine learning to automate the analysis of power lines imaging surveys autonomously conducted by drones with the aim of assisting failure detection and preventive maintenance.

Written by:
Dang Ha The Hien
Dang Ha The Hien2
PhD-student, eSmart Systems

Study Big Data Analytics at Østfold University College

Being able to extract insights from massive amount of data has become a key to success in many industries. This Big Data Analytics (BDA) course will survey state-of-the-art topics in big data, focusing on data analysis and machine learning technologies. Participants will gain an understanding of what opportunities big data can bring to their business through hands-on experience with real applications. The topic will be approached from both the practical and theoretical sides.

Østfold University College
Østfold University College

From the practical side:

Participants will have a chance to practice with state-of-the-art big data toolkits and platforms, such as R, Hadoop, Spark,… There will be hands-on assignments which let you build your first spam classifier, recommendation system, and anomaly detector. At the end of the course, participants will be able to answer these following questions:

  • What is big data and why the big data era has come to be?
  • What are the main challenges of big data? How to tackle those challenges using state-of-the-art big data technologies?
  • What is the difference between data and “information”?
  • What is the grammar of data manipulation and how to manipulate data?
  • What is the grammar of graphics and how to visualize things?
  • What are the common statistical machine learning models used in big data? What techniques are suitable in different

From the theoretical side:

Besides introducing state-of-the-art big data toolkits and technologies, this course will also pay a lot of attention on the theory of learning from data, from both the statistical and the machine learning perspectives. To provide a solid foundation for your future big data project, the following concepts will be carefully covered during the course:

  • Sample bias problem
  • Occam’s razor and the inherently imperfect definition of simplicity
  • Bias and Variance Tradeoff
  • Curse of Dimensionality
  • Overfitting and Regularization

Who should participate?

This is the second time we run this course. Thus, many materials and assignments have been adapted to allow participants with different backgrounds to take the most out of the course. No prior programming experience is needed, although there will be challenging programming exercises for those who are serious about becoming a data scientist. Participants may include:

  • Engineers who need to learn the data analysis skills and the new Big Data technologies to apply to their work.
  • Technical managers who need to familiarize themselves with these emerging technologies.
  • Computer science students who want to become a data scientist.

Apply within May 15th 2016! 

Østfold University College and NCE Smart Energy Markets also offer a short program in Big Data Analytics. Read about it here (in Norwegian).

Written by:
Dang Ha The Hien
Dang Ha The Hien2
PhD-student, eSmart Systems

Prosumer bidding and scheduling in electricity markets

Stig Ødegaard Ottesen, Head of R&D in eSmart Systems, recently got accepted for publication an article in the international research journal Energy.


The title of the article is “Prosumer bidding and scheduling in electricity markets” and is about how a smart grid aggregator should take optimal bidding decisions when trading in day-ahead markets and further how to operate flexible energy appliances.

The main part of the article is the mathematical formulation of the decision problems, also taking into consideration the fact that some information (like prices and loads) is not known with certainty at the decision time. It also contains a case study based on data from Norske Skog Saugbrugs and the municipality of Hvaler. Co-authors are Asgeir Tomasgard and Stein-Erik Fleten, both professors at NTNU, Department of Industrial Economics and Technology Management.

Download and read the article here

Et tilbakeblikk på 2015

Et begivenhetsrikt år nærmer seg slutten. Før vi tar fatt på 2016 tar vi et lite tilbakeblikk på de viktigste begivenhetene for eSmart Systems i året som har gått.

Som oppstartselskap er vi nødt til å jobbe hardt. Vi må yte litt mer, gi det lille ekstra og hele tiden være i forkant. Vårt mål er å ha høy integritet i arbeidet. I eSmart Systems er vi åpne og ærlige, vi støtter hverandre, og deler ideer og kompetanse. Vi er stolte av jobben vi gjør og streber alltid etter å prestere bedre enn forventet. Vi er opptatt av å omfavne endring, oppmuntre og støtte nye ideer. Vi tror energibransjen trenger innovative aktører!

Frisk start

I vårt tredje driftsår har vi posisjonert oss i det norske markedet, vi har doblet staben, og vi har fått nye kunder og samarbeidspartnere.

Vi startet friskt i januar med å være medarrangør til konferansen Smart Energy 2015. Konferansen samlet 140 aktører fra energibransjen til to dager med faglig påfyll i Halden. På samme tid ble EB Nett med som partner i vårt utviklingsprosjekt.

Utover våren deltok vi på flere konferanser. Vi var blant annet hovedsponsor på TU Smart Grid Summit. Vi deltok også på Fremtidskonferansen hvor vi fikk fremtidsprisen for årets innovatør innen energi og IT.

To sentrale prosjekter for oss i år har vært deltakelsen i Elhub, som underleverandør til Accenture, og Smartliv. I Elhub-prosjektet har vi bidratt med lokal markedskunnskap og erfaring fra å levere kritiske IT-løsninger til sentrale markedsaktører.

I samarbeid med Ringeriks-Kraft lanserte vi SMARTliv-appen i mai. Her kan strømkunder få full oversikt over forbruket, og muligheten til større bevissthet rundt eget forbruk.

Et stort øyeblikk

En av de aller største begivenhetene for eSmart Systems i år er vår deltakelse og utviklingssjef Erik Åsberg’s presentasjon på Build 2015, Microsofts største utviklerkonferanse. Det å få muligheten til å fortelle hva vi jobber med til en million mennesker over hele verden er stort for et lite oppstartsselskap fra Halden.



Rett før sommeren ble eSmart Systems plukket ut som en Norges fremtidige ledestjerner av nettsiden Omtrent samtidig hadde teknologimagasinet CIO Review en kåring av de 100 mest lovende løsningsleverandørene i verden, hvor vi var med på listen.

En fredag midt i fellesferien publiserte Dagens Næringsliv en to-siders artikkel om oss hvor de skrev at eSmart Systems tar kraftmarkedet med storm.

2015 07 17 bilde i DN

Vi inngikk et samarbeid med Kamstrup og startet et prosjekt med smarte vannmålere i Hvaler kommune sammen med næringsklyngen NCE Smart Energy Markets.

Fokus på droner

I september åpnet Fredrikstad Energi Norges mest moderne driftssenter for nett og AMS. eSmart Systems har levert systemløsningen Connected Grid til driftssenteret og bidrar til at nettvirksomheten er optimalt rigget med funksjoner som forbedrer nettdriften og muliggjør god forbrukerfleksibilitet.

Vi var hovedsponsor på Smartgridkonferansen i Trondheim. Med et stort fokus på droner var det en fin oppladning til kickoff i prosjektet Connected Drone et par uker senere.

Vi var også blant de nominerte til å vinne Norwegian Tech Award, noe som resulterte i denne artikkelen i Teknisk Ukeblad. I oktober inngikk vi også en avtale med Hurum Energiverk.

Microsoft kåret eSmart Systems til Årets Programvarehus 2015, foran 700 andre programvarehus i Norge.

årets programvarehus

Vi satser internasjonalt og deltar i to program i regi av Innovasjon Norge, FRAM Smart Cities og FRAM IoT. I desember signerte vi en avtale med EDF Trading for å ferdigstille neste generasjons plattform for tradingsystem.

Øker staben i 2016

Når vi nå går inn i 2016 er det med stor motivasjon og energi til å fortsette å innovere energibransjen.

Vi har flere spennende prosjekter på trappene, vi intensiverer vår internasjonale satsing, og vi skal fortsette å ansette nye eSmartere!

Riktig god jul ønskes til alle våre kunder, samarbeidspartnere og støttespillere!

Esmartsamling03 (1)

Microsoft Student Community HIOF with Enigma, eSmart Systems and Smart Simulation

The newly formed Microsoft Student Community at HIOF (Høgskolen i Østfold) held their first Hackathon on Friday 13th of November 2015. In support of this event, eSmart Systems took the opportunity to offer their assistance. eSmart Systems was represented on the day by Nicklas Eltvik, Morten Sørby, who acted as judges for the event, and presenter, Ole Amundsen. The Hackathon started with several presentations, where eSmart Systems own Ole Amundsen was the first to present. Ole gave an exciting presentation on how eSmart Systems develops mobile applications and how developers can use Xamarin and C# to develop apps that provide the developer with one place to manage code and their User Interfaces.

Ole Amundsen, Product Manager Mobile in eSmart Systems.

Followed by Ole, Anders Gill (Technical Evangelist) presented the power and speed of Azure cloud services, and Universal Windows Platform development. Prior to the main event, a final presentation was held and presented by Kristoffer Møgster; the leader of MS student community, HIOF. Kristoffer Møgster demonstrated how to develop games quickly and efficiently with the Unity engine and the Microsoft Kinect sensor.

The main event comprised of an 18-hour long coding session, where participants of the events became direct competitors after created groups primarily consisting of 2-4 members. From an initially large number of competitors at the start of the day, by the time the clock struck 12.00am the following day after a night of coding, there were only eight groups left. Only one group would stand victorious.

The judges selected the victors for their outstanding teamwork, excellent use of the planning phase and solid project architecture in addition to their already great code. Their solution was built with Universal Windows Platform Application as their frontend, Web API as a service, Entity Framework and Azure SQL Server as their backend. The winning application had a simple and elegant look. The group named their product “Brackets”, a name that reflects their application (an administration system for tournaments) very well.

The winners. From left, Michael Glomnes, Øyvind Christoffer Aronsen, Jens Petter Grimstad Andersen, Stephan Askel Birkeland.
The judges. From left, Morten Sørby(eSmart Systems), Berzi Wasfy(HIOF) , Per Edvard Volla(HIOF), Kristoffer Møgster(HIOF), Nicklas Eltvik(eSmart Systems), Anders Gill (Microsoft).
The judges. From left, Morten Sørby(eSmart Systems), Berzi Wasfy(HIOF) , Per Edvard Volla(HIOF), Kristoffer Møgster(HIOF), Nicklas Eltvik(eSmart Systems), Anders Gill (Microsoft).

Want to learn more about a Hackathon?
Check out

Written by:
Nicklas Eltvik
til blogg
System Engineer, eSmart Systems