Lambda, Azure Functions and Google Cloud Functions? It’s a question any company or DevOps team opting for a Serverless stack has to ask and one K&C’s Serverless consultants also has to analyse on a general and project-by-project basis.
But the answer is not always immediately apparent. AWS Lamda, Azure Functions and Google Cloud Functions all offer equivalent Serverless components, or functions. And they all offer pay-as-you-use pricing models that, on the face of it are similar.
So, once you’ve already assessed the pros and cons of Serverless and decided to go with it as an architecture, does the choice of provider simply boil down to who you may already use as your Cloud vendor? Or are there genuine distinctions between their resource management, performance isolation and pricing models that mean one might be better or worse than the others for a particular Serverless project?
A 2018 study ‘Peeking Behind the Curtains of Serverless Platforms’, applies academic rigour to a comparative analysis of AWS Lambda, Azure Functions and Google Functions. Here we summarise the paper’s findings.
The paper, joint-authored by Liang Wang, Mengyuan Li, Yinqian Zhang, Thomas Ristenpart and Michael Swift, crucially compares the strengths of AWS Lambda, Azure Functions and Google Functions from the point of view of a Serverless customer. An academic methodology to comparing the main Serverless providers is all very well but of little real value unless within the real life context of what is important to customers.
The team’s data set was built by launching over 50,000 function instances across the 3 Serverless providers with a view to characterising variations in their architectures, performance, resource management efficiency and security:
“We explain how the platforms isolate the functions of different accounts, using either virtual machines or containers, which has important security implications. We characterize performance in terms of scalability, coldstart latency, and resource efficiency, with highlight including that AWS Lambda adopts a bin-packing-like strategy to maximize VM memory utilization, that severe contention between functions can arise in AWS and Azure.”
So how did the researchers approach the tricky task of trying to objectively benchmark AWS Lambda, Microsoft Azure Functions and Google Cloud Functions against each other as Serverless services?
‘Vantage points’ from which functions were managed and invoked from official APIs were set up in the same Cloud region. The vantage points were VMs of a minimum of 4 GB RAM and 2 vCPUs. The regions were ‘US-east-1’ in AWS, ‘US-central-1’ in Google and ‘East US’ in Azure.
The same functions were invoked under settings that were varied by adjusting functions configurations and workloads.
All the functionalities and subroutines used were integrated into a single control ‘measurement function’ designed to perform to core tasks that provided objective measurables:
(1) collect invocation timing and function instance runtime information, and
(2) run specified subroutines (e.g., measuring local disk I/O throughput, network throughput) based on received messages. The measurement function collects runtime information via the proc filesystem on Linux (procfs), environment variables, and system commands. It also reports on execution start and end time, invocation ID (a random 16-byte ASCII string generated by the function that uniquely identify an invocation), and function configurations to facilitate further analysis.
Provider-recommended software was used and time synchronisation in the vantage points configured according to official instructions of the respective providers.
Most measurement function experiments used Python 2.7 and Nodejs 6 as the language runtime. These were selected as the statistically most popular in AWS. Experiments were carried out over H2 of 2017.
The testing across the three Serverless providers assessed two coldstart variations. The first was the launch of a function instance on a freshly introduced VM. And the second the same launch on an existing VM. The assumption was that coldstart latency would be higher on the new VM that may need to be initiated.
AWS Lambda again returned very good results. The median coldstart latency on a newly introduced VM was just 39ms longer than on an existing VM across all settings. There was also evidence that in some instances new VMs were launched prior to invocation based on the shortest VM kernel uptime coming in at just 132 seconds. The conclusion was that AWS sets aside a pool of ready VMs ready to spring into action when needed.
One of Azure Functions’ marketed strengths is that instances are always assigned 1.5GB memory. However, the launch of a function instance showed a median coldstart latency of 3640 ms. Azure’s latency also had by far the highest volatility of the three providers.
While relatively stable on AWS Lambda and Google Functions (there were a few spikes), Azure’s network variation ranged between 1.5 and 16 seconds.The study’s findings around scalability and instance placement were particularly interesting due to significant divergence.
AWS Lambda was found to demonstrate notably superior support during concurrent execution. AWS Lambda VMs allocated a maximum aggregate memory of 3328MB across all function instances, with placement treated as a ‘bin-packing problem’. The attempt was always made to maximise VM memory utilisation rates by placing new function instances on already active VMs.
On the other hand, the controls found that Azure is able to run less function instances concurrently for a single function than claimed by official documentation. The provider says that a single Nodejs-based function can scale to up to 200 instances, with one new instance able to be launched every 10 seconds. However, based on the test results it was not possible to have more than 10 function instances running concurrently for a single function regardless of the time lag between invocations.
Google Functions also disappointed on scalability. The provider’s documentation claims HTTP-triggered functions can scale quickly to the necessary invocation rate. However, this was not found to be the case over the benchmark tests conducted:“In general, only about half of the expected number of instances, even for a low concurrency level (e.g., 10), could be launched at the same time, while the remainder of the requests were queued.”So benchmarked for scalability and instance placement, AWS Lambda came out as the clear winner between the three.
Google Functions’ performance across this measurable was an improvement on that of Azure but still inferior to AWS. Despite Google allocating CPU proportionally to memory, memory allocation was demonstrated to have a greater impact on coldstart latency when compared to AWS Lambda. Google Functions’ median coldstart latency fell within a 100 ms t0 493 ms range.
One comparable where Azure Functions’ did show the best performance of the three was across average instance lifetime. This is important to Serverless customers because the longer an instance is maintained the better the Serverless application will hold in-memory states such as database connections. This also reduces cold start latency.
Instance lifetime was estimated through the set-up of an array of functions spanning different memory sizes and languages invoked at 5, 30 and 60 second frequencies with at least 50 lifetime measurements gathered per setting.
How long an instance stays idle before the Serverless provider shuts it down is a balance. The longer an idle instance is maintained, the more VM memory resource wastage. However, once an instance is shut down it will have a coldstart when invoked again.
AWS Lambda was found to generally shut down idle instances after 26 minutes, with 80% conforming to this norm. The longest idle period recorded was 27 minutes. However, if an idle instance was hosted on the same VM as active instances, it was observed that they could be held open for up to 3 hours.
Google Functions maintained idle instances for longer, presumably to compensate for its poorer coldstart latency. 18% of tests showed idle instances maintained for over 120 minutes. Azure Functions test demonstrated a somewhat baffling lack of consistency and no apparent standard. On different days, idle instances were seen to be shut down after 22, 40 or more than 120 minutes with no obvious pattern explaining the variation.
It can be deduced from the findings that the combination of AWS Lambda’s impressive coldstart speed and shorter average time to idle instance shutdown results in cost efficiencies for customers without a negative impact on application performance.
Serverless customers expect the new function code to handle any requests that follow a function update. This is especially the case for function updates with security implications.
However, tests showed that AWS Lambda sometimes (across 2% of tests) used the old version of function. This is referred to by the paper as ‘inconsistent function usage’. However, within seconds of a function update inconsistencies dropped to 0%, leading to the conclusion the issue lay with AWS’s instance scheduler being unable to manage an ‘atomic update’.
“AWS’s bin-packing placement may bring security issues to an application, depending on its design. When a multi-tenant application in Lambda uses IAM roles to isolate its tenants, function instances from different application tenants still share the same VMs”.
Performance isolation assessment looked at CPU utilisation, I/O & network and resource accounting.
The CPU Utilisation rate of AWS Lambda, Azure Functions and Google Functions was defined as “the fraction of the 1,000 ms for which a timestamp was recorded”.
AWS Lambda assigns more CPU cycles to instances with higher memory. However, it was found that potential inconsistencies in application performance could result from significant variations in the CPU utilisation rate when there is ‘no contention from other co-resident instances’.
Azure and Google Functions showed variance in CPU utilisation rates that varied between 14.1% and 90% and 11.1% and 100% respectively.
“To measure I/O throughput, our measurement functions in AWS and Google used the dd command to write 512 KB of data to the local disk 1,000 times (with fdatasync and dsync flags to ensure the data is written to disk). In Azure, we performed the same operations using a Python script (which used os.fsync to ensure data is written to disk).
For network throughput measurement, the function used iperf 3.13 with default configurations to run the throughput test for 10 seconds with different same-region iperf servers, so that iperf server-side bandwidth was not a bottleneck. The iperf servers used the same types of VMs as the vantage points”.
In AWS the aggregate I/O and network throughput held at fairly consistent rates. However, the tests found that as co-level increases, instances are allocated a smaller share of I/O and network resources. Instances co-resident in one VM received a smaller share of the network when contention was higher.
A similar pattern was observable in the case of Azure, where instances also saw I/O and network throughput drop as co-level increased. Throughput fluctuated when there was contention between co-resident instances.
In Google Functions a lack of consistency was again apparent. I/O and network throughout increased in relation to function memory allocation but network throughout levels varied significantly across instances with identical memory size. This suggests that there is a high likelihood Google Cloud Functions tenants “actually share hosts and suffer from resource contention”.
AWS Lambda achieved the best scalability and the lowest coldstart latency (the time to provision a new function instance), followed by GCF. But the lack of performance isolation in AWS between function instances from the same account caused up to a 19x decrease in I/O, networking, or coldstart performance.
Other than performance, cost is obviously the next most important factor when making a decision on a Serverless provider. Is one of AWS Lambda, Azure Functions and Google Cloud Functions significantly cheaper or more expensive than the other? Let’s take a quick tour of how the three’s pricing models work.
AWS Lambda Serverless Pricing Model: the first 1 million requests and 400,000 GB-seconds of compute time used per month fall into a free-tier. Additions requests are then charged at $0.00001667/GB-s. Memory and CPU are charged as one item as their increases are proportional and other AWS services used within a Lambda function will also incur additional services. AWS’s price-per-request is currently the cheapest on the market.
Azure Functions has the same billing model as Lambda as well as an identical free tier allocation. Cost-per-request is more or less the same.
Azure also offers lower pricing for Windows and SQL usage, which is pretty logical. So, the choice between the two depends on the environment you use more than the costs you incur. Because Azure bills for used rather than allocated memory, heavy-load costs tend to come in slightly cheaper than AWS. For average loads, the costs of AWS and Azure are almost identical.
However, as Microsoft products, Azure’s pricing for Windows and SQL usage is lower. All-in-all, one a cost basis, the choice between the two depends would usually be based more on the environment you use more than the costs running an app will incur.
Google Cloud Functions will generally work out as the most expensive option of the three because networking traffic is included in requests billed and functions run for longer. Allocated memory and CPU usage are also billed separately. A free tier of 2 million requests and 400,000 GB-seconds of compute time are offered.
It is important to note that pricing models do change and there are calculators available that can give you a pretty accurate cost comparison between the major Serverless providers. The A Cloud Guru Team have put together a very nice Serverless costs comparison calculator that is currently in beta. Image below.
The providers themselves also all offer calculators you can use independently:
As outlined by the academic study, there are some differences between the infrastructure services provided by AWS Lambda, Azure Functions and Google Cloud Functions. However, they are minor enough to not be obvious in the majority of customer use cases. It can probably be fairly said that AWS Lambda and Azure offer fuller and more diverse services than Google Cloud Functions. They are also slightly cheaper.
Google itself has recognised the market edge its two major competitors have and has at the time of writing announced the strategic decision to reposition its Cloud service. It has followed IBM in recognising that AWS Lambda and Microsoft’s Azure have secured dominant market positions and will no longer try to compete directly.
Google Cloud is moving to a multi-cloud strategy and is shifting focus to its new Anthos software that facilitates the movement of workloads between Cloud providers. The good news is this looks to be a very positive step in the road towards neutralising the vendor lock-in issue.
For now at least, the Serverless provider market is moving towards the duopoly of AWS Lambda and Azure Functions. Both are great services and a choice between the two currently really boils down to preferred environment.
Krusche & Company (K&C) are a Munich, Germany-based DevOps, web development and cloud services consultant and development agency. We offer dedicated developments teams or team extensions as well as consultancy across all major contemporary tech stacks.
If you have any current or future needs across Serverless architecture or in any of our other areas of expertise, please do get in touch. We’d love to hear from you.