SQS To DynamoDB Tuning - Setting Up Rust And Typescript Lambdas

SQS To DynamoDB Tuning - Setting Up Rust And Typescript Lambdas

Larry Christensen's photo
Larry Christensen
·

4 min read

In this series, I will be investigating throughput tuning for a Lambda that receives SQS events, reads data from S3 object, and blasts the data into DynamoDB. While I'm at it, I'll do a performance shootout between Rust and Typescript versions, attempting to optimize each as much as possible to create a fair comparison.

Initialize the Project

Create a new Rust project:

cargo new sqs_ddb_rust --bin

Initialize Typescript and Webpack:

npm init
npm install -g typescript
npm install -g webpack webpack-cli
npm install --save-dev @tsconfig/recommended

Add Lambda Dependencies

Typescript

Adding Lambda event type definitions:

npm install --save-dev @types/aws-lambda

Add a Webpack config so I can minimize my TS Lambda size (webpack.config.js):

const path = require('path');

module.exports = {
  mode: 'production',
  target: "node",
  entry: './target/js/index.js',
  output: {
    library: {"name": "blaster", "type": "this"},
    filename: 'index.js',
    path: path.resolve(__dirname, 'target/js'),
  }
};

Add tsconfig.json:

{
  "extends": "@tsconfig/node14/tsconfig.json",
  "include": ["src/ts/*"],
  "compilerOptions": {
    "outDir": "target/js"
  }
}

Rust

Cargo.toml:

[package]
name = "sqs_to_ddb"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
lambda_runtime = "0.4.1"
tokio = { version = "1.0", features = ["macros", "io-util", "sync", "rt-multi-thread"] }
serde_json = "^1"

Add Basic Handlers

Rust

I want to set things up so I can have multiple handlers and have the binaries output under the name of the handler file, so I first:

mkdir src/bin
mv src/main.rs src/bin/blaster_handler.rs

Then in src/bin/blaster_handler.rs:

use lambda_runtime::{handler_fn, Context, Error as LambdaError};
use serde_json::{Value};

#[tokio::main]
async fn main() -> Result<(), LambdaError> {
    let func = handler_fn(func);
    lambda_runtime::run(func).await?;
    Ok(())
}

async fn func(event: Value, _: Context) -> Result<(), LambdaError> {
    println!("Hello Event: {}", serde_json::to_string(&event).unwrap());
    Ok(())
}

Typescript:

src/ts/index.ts:

import { SQSEvent, SQSHandler } from "aws-lambda";

export const handler: SQSHandler = async (event: SQSEvent) => {
    console.log(`Hello Event: ${JSON.stringify(event)}`);
}

Add Makefile

I want to be able to build everything with just the 'sam build' command, so I use a Makefile to do this.

Makefile:

build-BlasterLambdaTS:
    tsc
    webpack
    cp ./target/js/index.js $(ARTIFACTS_DIR)

build-BlasterLambdaRust:
    docker run --platform linux/arm64 \
    --rm --user "$(id -u)":"$(id -g)" \
    -v "$(PWD)":/usr/src/myapp -w /usr/src/myapp rust:latest \
    cargo build --release --target aarch64-unknown-linux-gnu
    cp ./target/aarch64-unknown-linux-gnu/release/blaster_handler $(ARTIFACTS_DIR)/bootstrap

Add SAM Template

template.yml:

AWSTemplateFormatVersion: "2010-09-09"

Transform:
- "AWS::Serverless-2016-10-31"

Resources:

  BlasterLambdaRust:
    Type: AWS::Serverless::Function
    Properties:
      Architectures:
        - arm64
      Handler: none
      Runtime: provided.al2
      CodeUri: .
      Timeout: 30
      MemorySize: 512
      Policies:
        - AWSLambdaBasicExecutionRole
    Metadata:
      BuildMethod: makefile

  BlasterLambdaTS:
    Type: AWS::Serverless::Function
    Properties:
      Architectures:
        - arm64
      Handler: index.blaster.handler
      Runtime: nodejs14.x
      CodeUri: .
      Timeout: 30
      MemorySize: 512
      Policies:
        - AWSLambdaBasicExecutionRole
    Metadata:
      BuildMethod: makefile

One interesting thing to note here is that I'm using the 'arm64' architecture. I've found that Lambdas running on this not only run faster than on x86, but are also less expensive. More information here . I'll perhaps do some comparisons later in the series.

Sam Local Testing

Adding a test-event.json (copied from here ):

{
    "Records": [
        {
            "messageId": "059f36b4-87a3-44ab-83d2-661975830a7d",
            "receiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...",
            "body": "Test message.",
            "attributes": {
                "ApproximateReceiveCount": "1",
                "SentTimestamp": "1545082649183",
                "SenderId": "AIDAIENQZJOLO23YVJ4VO",
                "ApproximateFirstReceiveTimestamp": "1545082649185"
            },
            "messageAttributes": {},
            "md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
            "eventSource": "aws:sqs",
            "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue",
            "awsRegion": "us-east-2"
        },
        {
            "messageId": "2e1424d4-f796-459a-8184-9c92662be6da",
            "receiptHandle": "AQEBzWwaftRI0KuVm4tP+/7q1rGgNqicHq...",
            "body": "Test message.",
            "attributes": {
                "ApproximateReceiveCount": "1",
                "SentTimestamp": "1545082650636",
                "SenderId": "AIDAIENQZJOLO23YVJ4VO",
                "ApproximateFirstReceiveTimestamp": "1545082650649"
            },
            "messageAttributes": {},
            "md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
            "eventSource": "aws:sqs",
            "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue",
            "awsRegion": "us-east-2"
        }
    ]
}

Now I can run SAM Local:

sam build
sam local invoke BlasterLambdaRust -e test-event.json
sam local invoke BlasterLambdaTS -e test-event.json

Both functions succeed and print out the input event. Lets deploy and test:

sam deploy

Testing the TS version gives this the two invocations:

Duration: 3.80 ms    Billed Duration: 4 ms    Memory Size: 512 MB    Max Memory Used: 55 MB    Init Duration: 161.25 ms

Duration: 10.60 ms    Billed Duration: 11 ms    Memory Size: 512 MB    Max Memory Used: 56 MB

Testing the Rust version gives this for the first two invocations:

Duration: 0.93 ms    Billed Duration: 18 ms    Memory Size: 512 MB    Max Memory Used: 13 MB    Init Duration: 16.90 ms

Duration: 0.71 ms    Billed Duration: 1 ms    Memory Size: 512 MB    Max Memory Used: 13 MB

We'll gather more samples next time to get a better idea of the comparison, but it is remarkable how much lower the init durations, overall durations, and memory usage are with Rust. Another interesting thing is how the node version doesn't include the init duration in the billed duration, I wonder if it's a bug or a bonus for using Node.

Next time we'll also start adding code to read from an S3 object and send it line-by-line into DynamoDB. We'll gather some metrics and do some comparisons between the two runtimes.

aws lambdaRustTypeScript