In this series, I will be investigating throughput tuning for a Lambda that receives SQS events, reads data from S3 object, and blasts the data into DynamoDB. While I'm at it, I'll do a performance shootout between Rust and Typescript versions, attempting to optimize each as much as possible to create a fair comparison.
Initialize the Project
Create a new Rust project:
cargo new sqs_ddb_rust --bin
Initialize Typescript and Webpack:
npm init
npm install -g typescript
npm install -g webpack webpack-cli
npm install --save-dev @tsconfig/recommended
Add Lambda Dependencies
Typescript
Adding Lambda event type definitions:
npm install --save-dev @types/aws-lambda
Add a Webpack config so I can minimize my TS Lambda size (webpack.config.js):
const path = require('path');
module.exports = {
mode: 'production',
target: "node",
entry: './target/js/index.js',
output: {
library: {"name": "blaster", "type": "this"},
filename: 'index.js',
path: path.resolve(__dirname, 'target/js'),
}
};
Add tsconfig.json:
{
"extends": "@tsconfig/node14/tsconfig.json",
"include": ["src/ts/*"],
"compilerOptions": {
"outDir": "target/js"
}
}
Rust
Cargo.toml:
[package]
name = "sqs_to_ddb"
version = "0.1.0"
edition = "2021"
[dependencies]
lambda_runtime = "0.4.1"
tokio = { version = "1.0", features = ["macros", "io-util", "sync", "rt-multi-thread"] }
serde_json = "^1"
Add Basic Handlers
Rust
I want to set things up so I can have multiple handlers and have the binaries output under the name of the handler file, so I first:
mkdir src/bin
mv src/main.rs src/bin/blaster_handler.rs
Then in src/bin/blaster_handler.rs:
use lambda_runtime::{handler_fn, Context, Error as LambdaError};
use serde_json::{Value};
#[tokio::main]
async fn main() -> Result<(), LambdaError> {
let func = handler_fn(func);
lambda_runtime::run(func).await?;
Ok(())
}
async fn func(event: Value, _: Context) -> Result<(), LambdaError> {
println!("Hello Event: {}", serde_json::to_string(&event).unwrap());
Ok(())
}
Typescript:
src/ts/index.ts:
import { SQSEvent, SQSHandler } from "aws-lambda";
export const handler: SQSHandler = async (event: SQSEvent) => {
console.log(`Hello Event: ${JSON.stringify(event)}`);
}
Add Makefile
I want to be able to build everything with just the 'sam build' command, so I use a Makefile to do this.
Makefile:
build-BlasterLambdaTS:
tsc
webpack
cp ./target/js/index.js $(ARTIFACTS_DIR)
build-BlasterLambdaRust:
docker run --platform linux/arm64 \
--rm --user "$(id -u)":"$(id -g)" \
-v "$(PWD)":/usr/src/myapp -w /usr/src/myapp rust:latest \
cargo build --release --target aarch64-unknown-linux-gnu
cp ./target/aarch64-unknown-linux-gnu/release/blaster_handler $(ARTIFACTS_DIR)/bootstrap
Add SAM Template
template.yml:
AWSTemplateFormatVersion: "2010-09-09"
Transform:
- "AWS::Serverless-2016-10-31"
Resources:
BlasterLambdaRust:
Type: AWS::Serverless::Function
Properties:
Architectures:
- arm64
Handler: none
Runtime: provided.al2
CodeUri: .
Timeout: 30
MemorySize: 512
Policies:
- AWSLambdaBasicExecutionRole
Metadata:
BuildMethod: makefile
BlasterLambdaTS:
Type: AWS::Serverless::Function
Properties:
Architectures:
- arm64
Handler: index.blaster.handler
Runtime: nodejs14.x
CodeUri: .
Timeout: 30
MemorySize: 512
Policies:
- AWSLambdaBasicExecutionRole
Metadata:
BuildMethod: makefile
One interesting thing to note here is that I'm using the 'arm64' architecture. I've found that Lambdas running on this not only run faster than on x86, but are also less expensive. More information here . I'll perhaps do some comparisons later in the series.
Sam Local Testing
Adding a test-event.json (copied from here ):
{
"Records": [
{
"messageId": "059f36b4-87a3-44ab-83d2-661975830a7d",
"receiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...",
"body": "Test message.",
"attributes": {
"ApproximateReceiveCount": "1",
"SentTimestamp": "1545082649183",
"SenderId": "AIDAIENQZJOLO23YVJ4VO",
"ApproximateFirstReceiveTimestamp": "1545082649185"
},
"messageAttributes": {},
"md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
"eventSource": "aws:sqs",
"eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue",
"awsRegion": "us-east-2"
},
{
"messageId": "2e1424d4-f796-459a-8184-9c92662be6da",
"receiptHandle": "AQEBzWwaftRI0KuVm4tP+/7q1rGgNqicHq...",
"body": "Test message.",
"attributes": {
"ApproximateReceiveCount": "1",
"SentTimestamp": "1545082650636",
"SenderId": "AIDAIENQZJOLO23YVJ4VO",
"ApproximateFirstReceiveTimestamp": "1545082650649"
},
"messageAttributes": {},
"md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
"eventSource": "aws:sqs",
"eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue",
"awsRegion": "us-east-2"
}
]
}
Now I can run SAM Local:
sam build
sam local invoke BlasterLambdaRust -e test-event.json
sam local invoke BlasterLambdaTS -e test-event.json
Both functions succeed and print out the input event. Lets deploy and test:
sam deploy
Testing the TS version gives this the two invocations:
Duration: 3.80 ms Billed Duration: 4 ms Memory Size: 512 MB Max Memory Used: 55 MB Init Duration: 161.25 ms
Duration: 10.60 ms Billed Duration: 11 ms Memory Size: 512 MB Max Memory Used: 56 MB
Testing the Rust version gives this for the first two invocations:
Duration: 0.93 ms Billed Duration: 18 ms Memory Size: 512 MB Max Memory Used: 13 MB Init Duration: 16.90 ms
Duration: 0.71 ms Billed Duration: 1 ms Memory Size: 512 MB Max Memory Used: 13 MB
We'll gather more samples next time to get a better idea of the comparison, but it is remarkable how much lower the init durations, overall durations, and memory usage are with Rust. Another interesting thing is how the node version doesn't include the init duration in the billed duration, I wonder if it's a bug or a bonus for using Node.
Next time we'll also start adding code to read from an S3 object and send it line-by-line into DynamoDB. We'll gather some metrics and do some comparisons between the two runtimes.
