Both Serverless and containers offer huge cost advantages compared to what went before them. Before the era of cost-effective, scalable public cloud platforms (it’s easy to forget AWS only launched in 2006 and took several years to become ubiquitous), the insights afforded by Big Data analysis were practically limited to large enterprises because of the infrastructure overheads entailed. There was also a lot less data in general with the connected devices that now produce so much of it also a consequence of the cloud computing revolution.

Cloud development gave rise to containers, which allowed applications to be broken down into the smallest independent modules that made sense and improved speed, efficiency, reliability. In big data projects, the advantage of containers running in VMs over the containerless VMs previously relied on was also that they allowed Big Data applications to be hardware and GPU agnostic and improved GPU-sharing efficiency.

Serverless takes things one step further and provides Containers-as-a-Service, removing the job of setting up and managing the underlying VM infrastructure. The software development team only needs to interact with the container.

K&C’s experts believe that if a Serverless architecture is not yet always the right solution for Big Data-powered AI applications, the technology is moving in that direction.

In this article, we examine why Serverless is so well suited to Big Data processing. And why Serverless architecture is becoming such a popular approach to app development more generally.

In 2012, Forbes published a guest post on the rise of Big Data, written by John Bantleman, CEO of database software company Rainstor. The ‘age of Big Data’ was announced. Bantleman wrote:

“We’ve entered the age of Big Data where new business opportunities are discovered every day because innovative data management technologies now enable organizations to analyze all types of data”. 

However, Bantleman quickly moved on to warn that the business opportunities Big Data was opening up would come with costs not yet appreciated. Collecting, storing, processing and using AI/machine learning algorithms to analyse the huge volumes of semi and unstructured data being generated requires a huge computing resource. 

Big Data architecture challenges

The infrastructure of any application built to process Big Data must answer the following questions and meet the  challenges:

Before the rise of Cloud and Serverless offered storage and computing resource as a utility service, processing Big Data meant building and maintaining the server infrastructure to do so. That had to be large enough to accommodate peaks of data flow even if they were only occasional.

It is precisely the anomalies such as peaks and troughs in data flow that often offer the most valuable scientific or commercial insights. However, an application being able to handle occasional peaks meant paying for and maintaining expensive infrastructure that spent most of its time redundant.

First Hadoop and then Cloud computing changed that. By distributing Big Data sets across many cheap ‘commodity server’ nodes, which combine into a computational resource capable of storing and handling huge data sets, Hadoop significantly lowered the cost of scalable bare metal infrastructure.

That 2012 Bantleman Forbes article estimated a Hadoop cluster and distribution facility for Big Data cost around $1 million compared to the $10 million to $100s of millions for enterprise data warehouses. But of course, $1 million is still not pocket change and maintained a barrier to entry that kept most out of Big Data applications.

Next came Cloud computing and containers. Cloud providers such as AWS turned computing power into a service – removing the requirement for major upfront investment in hardware infrastructure. The pay-as-you-go and fluidly scalable model of public cloud platforms like AWS opened the door for the experimentation and innovation that led to a rich open source development ecosystem.

It has also allowed many young companies using Big Data to grow and flourish that would otherwise have had to contend with much tougher barriers to entry.

Cloud Computing meant no upfront investment and only paying for the processing power needed for irregular data flow peaks when they occurred.

Cloud computing’s democratisation of Big Data can be credited as the catalyst for a new technology revolution. One that spans digital technology and biotechnology. Revolutions gathering pace in medicine, pharmaceuticals, finance, commerce, agriculture, food technology and pretty much any other sector you may care to mention are happening because start-ups and SMEs can now afford to build and run Big Data applications.

The Machine Learning zeroing in on patterns previously undetectable is suddenly turbo charging new discoveries.

Within a few short decades the world we live in will be unrecognisable. Yes, technology has advanced quickly over the decades before. But what Cloud-powered Big Data and AI will achieve over the next several will be a paradigm shift.

We’ll be able to cure quite possible a majority of previously incurable diseases and conditions. The human genome and those of other forms of life will be mapped. Autonomous vehicles will reshape the economy and our lifestyles more than most imagine today. Ecommerce will be truly a truly personalised experience. The list goes on.

But as much as cloud computing has knocked down barriers to entry for Big Data and the AI that feeds on it, there is still a bottleneck. Cloud has hugely cut costs and container orchestrators such as Kubernetes have helped make apps more efficient and flexible.

But setting up and maintaining the cloud infrastructure that containerised Big Data applications run on is still very difficult. The main gains are in velocity and time required to maintain architecture. But containers require specialists with a very specific skill set that isn’t easy to acquire. Those specialists are expensive, either to hire ‘off-the-shelf’ or as an investment in further training, and in short supply.

The explosion of IoT across pretty much every sector imaginable means huge demand for the cloud architects and DevOps engineers able to set up container infrastructure. Everyone is fishing in the same shallow pool and the difficulty and expense of hiring the skills needed to manage container infrastructure has become a strategic problem.

Serverless reduces the complexity and cost of Big Data apps

The main advantage of Serverless computing over traditional cloud-based or server-centric infrastructure is that freeing developers from the need to deal with purchasing, provisioning and managing backend servers can lead to quicker time to release and less ongoing maintenance. That reduces development overheads and Serverless can also often, under the right circumstances, also lower cloud costs because charging is entirely based on resources actually used with no overheads for maintaining unused capacity.

K&C – can we help you build your next Serverless app?

An early adopter of the Serverless approach, K&C (Krusche & Company) has established a reputation as one of Munich, Germany and Europe’s most trusted IT services providers over more than 20 years. We operate tech talent centres in Krakow, Poland, Kyiv, Ukraine, and Minsk in Belarus, combining German management and company presence with cost-efficient access to nearshore IT talent.

If you are considering a Serverless approach for your next Big Data application and would like an expert opinion on the suitability of a Serverless architecture or a dedicated team to build it end-to-end please do get in touch!