benchmarks

Benchmarks for Bref running on AWS Lambda

brefphp
18
1
PHP

Benchmarks for Bref running on AWS Lambda.

Scenarios

  • PHP function: a simple PHP function, see function/index.php
  • HTTP application: a simple PHP web page that returns Hello world, see fpm/index.php

Median (warm) execution time

Average execution time for a lambda that doesn’t do anything.

Number of samples: 900

Bref 2.x (PHP 8.3)

Memory 128 512 1024 1769
PHP function 135ms 27ms 14ms 14ms
PHP function (BREF_LOOP_MAX) 1ms 1ms
HTTP application 1ms 1ms 1ms 1ms
Laravel 7ms

Bref 2.x ARM (PHP 8.3)

Memory 128 512 1024 1769
PHP function 135ms 30ms 14ms 14ms
PHP function (BREF_LOOP_MAX) 1ms 1ms
HTTP application 1ms 1ms 1ms 1ms
Laravel 8ms

For comparison on a 512M Digital Ocean droplet we get 1ms for “HTTP application” and 6ms for Symfony.

CPU performance

The more RAM, the more CPU power is allocated to the lambda. This is clearly visible when running PHP’s official bench.php script.

Memory 128 512 1024 1769
bench.php 5.7s 1.4s 0.65s 0.33s

For comparison bench.php runs in 1.3s on a 512M Digital Ocean droplet, in 0.8s on a 2.8Ghz i7 and in 0.6s on a 3.2Ghz i5.

Cold starts

Number of samples: 20

High level learnings:

  • Don’t use less than 512M of memory
  • Cold starts are not faster on ARM, they are same or slightly slower
  • Containers have the fastest cold starts
  • When using containers, you need to warm up the first requests following a deployment

Comparison (1024M):

x86 container x86 layer ARM layer
HTTP (duration) 180ms 300ms 300ms
HTTP (latency) 350ms 540ms 530ms
Laravel (duration) 850ms 1190ms 1160ms
Function 160ms 230ms 220ms

Bref 2.x containers (PHP 8.3)

Note: the first cold starts are much slower (see https://aaronstuyvenberg.com/posts/containers-on-lambda), e.g. 5s, because the container cache is warming up. In production warm up Lambda after deploying.

Function duration:

Memory 1024
PHP function 160ms
HTTP application 180ms
Laravel 850ms

Total latency (measured from API Gateway or X-Ray):

Memory 1024
PHP function
HTTP application 350ms
Laravel

Bref 2.x (PHP 8.3)

Function duration:

Memory 128 512 1024 1769
PHP function 500ms 260ms 230ms 230ms
HTTP application 430ms 330ms 300ms 310ms
Laravel 1190ms

Total latency (measured from API Gateway or X-Ray):

Memory 128 512 1024 1769
PHP function 700ms 430ms 440ms 410ms
HTTP application 580ms 450ms 460ms 450ms
Laravel 1400ms

Bref 2.x ARM (PHP 8.3)

Function duration:

Memory 128 512 1024 1769
PHP function 450ms 240ms 220ms 210ms
HTTP application 410ms 310ms 300ms 300ms
Laravel 1160ms

Total latency (measured from API Gateway or X-Ray):

Memory 128 512 1024 1769
PHP function 670ms 470ms 430ms 430ms
HTTP application 620ms 510ms 540ms 510ms
Laravel 1460ms

Measuring cold starts in CloudWatch Logs Insights:

filter @type = “REPORT” and @initDuration
| stats
 count(@type) as count,
 min(@billedDuration) as min,
 avg(@billedDuration) as avg,
 pct(@billedDuration, 50) as p50,
 max(@billedDuration) as max
by @log
| sort @log

Reproducing

You will need to install dependencies of Bref. Then:

  • clone the repository
  • make setup
  • make deploy

Then run the make bench-* commands.