# SQS To DynamoDB Tuning - Setting Up Rust And Typescript Lambdas In this series, I will be investigating throughput tuning for a Lambda that receives SQS events, reads data from S3 object, and blasts the data into DynamoDB. While I'm at it, I'll do a performance shootout between Rust and Typescript versions, attempting to optimize each as much as possible to create a fair comparison. ### Initialize the Project **Create a new Rust project:** ```bash cargo new sqs_ddb_rust --bin ``` **Initialize Typescript and Webpack:** ```bash npm init npm install -g typescript npm install -g webpack webpack-cli npm install --save-dev @tsconfig/recommended ``` ### Add Lambda Dependencies **Typescript** Adding Lambda event type definitions: ```bash npm install --save-dev @types/aws-lambda ``` Add a Webpack config so I can minimize my TS Lambda size (webpack.config.js): ```javascript const path = require('path'); module.exports = { mode: 'production', target: "node", entry: './target/js/index.js', output: { library: {"name": "blaster", "type": "this"}, filename: 'index.js', path: path.resolve(__dirname, 'target/js'), } }; ``` Add tsconfig.json: ```json { "extends": "@tsconfig/node14/tsconfig.json", "include": ["src/ts/*"], "compilerOptions": { "outDir": "target/js" } } ``` **Rust** Cargo.toml: ```toml [package] name = "sqs_to_ddb" version = "0.1.0" edition = "2021" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] lambda_runtime = "0.4.1" tokio = { version = "1.0", features = ["macros", "io-util", "sync", "rt-multi-thread"] } serde_json = "^1" ``` ### Add Basic Handlers **Rust** I want to set things up so I can have multiple handlers and have the binaries output under the name of the handler file, so I first: ```bash mkdir src/bin mv src/main.rs src/bin/blaster_handler.rs ``` Then in src/bin/blaster_handler.rs: ```rust use lambda_runtime::{handler_fn, Context, Error as LambdaError}; use serde_json::{Value}; #[tokio::main] async fn main() -> Result<(), LambdaError> { let func = handler_fn(func); lambda_runtime::run(func).await?; Ok(()) } async fn func(event: Value, _: Context) -> Result<(), LambdaError> { println!("Hello Event: {}", serde_json::to_string(&event).unwrap()); Ok(()) } ``` **Typescript:** src/ts/index.ts: ```typescript import { SQSEvent, SQSHandler } from "aws-lambda"; export const handler: SQSHandler = async (event: SQSEvent) => { console.log(`Hello Event: ${JSON.stringify(event)}`); } ``` ### Add Makefile I want to be able to build everything with just the 'sam build' command, so I use a Makefile to do this. Makefile: ```makefile build-BlasterLambdaTS: tsc webpack cp ./target/js/index.js $(ARTIFACTS_DIR) build-BlasterLambdaRust: docker run --platform linux/arm64 \ --rm --user "$(id -u)":"$(id -g)" \ -v "$(PWD)":/usr/src/myapp -w /usr/src/myapp rust:latest \ cargo build --release --target aarch64-unknown-linux-gnu cp ./target/aarch64-unknown-linux-gnu/release/blaster_handler $(ARTIFACTS_DIR)/bootstrap ``` ### Add SAM Template template.yml: ```yaml AWSTemplateFormatVersion: "2010-09-09" Transform: - "AWS::Serverless-2016-10-31" Resources: BlasterLambdaRust: Type: AWS::Serverless::Function Properties: Architectures: - arm64 Handler: none Runtime: provided.al2 CodeUri: . Timeout: 30 MemorySize: 512 Policies: - AWSLambdaBasicExecutionRole Metadata: BuildMethod: makefile BlasterLambdaTS: Type: AWS::Serverless::Function Properties: Architectures: - arm64 Handler: index.blaster.handler Runtime: nodejs14.x CodeUri: . Timeout: 30 MemorySize: 512 Policies: - AWSLambdaBasicExecutionRole Metadata: BuildMethod: makefile ``` One interesting thing to note here is that I'm using the 'arm64' architecture. I've found that Lambdas running on this not only run faster than on x86, but are also less expensive. More information [here](https://aws.amazon.com/blogs/aws/aws-lambda-functions-powered-by-aws-graviton2-processor-run-your-functions-on-arm-and-get-up-to-34-better-price-performance/) . I'll perhaps do some comparisons later in the series. ### Sam Local Testing Adding a test-event.json (copied from [here](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html) ): ```json { "Records": [ { "messageId": "059f36b4-87a3-44ab-83d2-661975830a7d", "receiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...", "body": "Test message.", "attributes": { "ApproximateReceiveCount": "1", "SentTimestamp": "1545082649183", "SenderId": "AIDAIENQZJOLO23YVJ4VO", "ApproximateFirstReceiveTimestamp": "1545082649185" }, "messageAttributes": {}, "md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3", "eventSource": "aws:sqs", "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue", "awsRegion": "us-east-2" }, { "messageId": "2e1424d4-f796-459a-8184-9c92662be6da", "receiptHandle": "AQEBzWwaftRI0KuVm4tP+/7q1rGgNqicHq...", "body": "Test message.", "attributes": { "ApproximateReceiveCount": "1", "SentTimestamp": "1545082650636", "SenderId": "AIDAIENQZJOLO23YVJ4VO", "ApproximateFirstReceiveTimestamp": "1545082650649" }, "messageAttributes": {}, "md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3", "eventSource": "aws:sqs", "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue", "awsRegion": "us-east-2" } ] } ``` Now I can run SAM Local: ```bash sam build sam local invoke BlasterLambdaRust -e test-event.json sam local invoke BlasterLambdaTS -e test-event.json ``` Both functions succeed and print out the input event. Lets deploy and test: ```bash sam deploy ``` Testing the TS version gives this the two invocations: ``` Duration: 3.80 ms Billed Duration: 4 ms Memory Size: 512 MB Max Memory Used: 55 MB Init Duration: 161.25 ms Duration: 10.60 ms Billed Duration: 11 ms Memory Size: 512 MB Max Memory Used: 56 MB ``` Testing the Rust version gives this for the first two invocations: ``` Duration: 0.93 ms Billed Duration: 18 ms Memory Size: 512 MB Max Memory Used: 13 MB Init Duration: 16.90 ms Duration: 0.71 ms Billed Duration: 1 ms Memory Size: 512 MB Max Memory Used: 13 MB ``` We'll gather more samples next time to get a better idea of the comparison, but it is remarkable how much lower the init durations, overall durations, and memory usage are with Rust. Another interesting thing is how the node version doesn't include the init duration in the billed duration, I wonder if it's a bug or a bonus for using Node. Next time we'll also start adding code to read from an S3 object and send it line-by-line into DynamoDB. We'll gather some metrics and do some comparisons between the two runtimes.