Setting Up Dynamodb For Local Development

Posted on

The finished project will be a go binary that will live in an AWS EC2 instance and speak to Dynamodb. I will be running my end to end tests locally and for this I want my code to speak to a locally running instance of Dynamodb; instead of speaking to AWS servers. I will accomplish this by using the amazon/dynamodb-local docker image.

My application code will live in a scratch image, and will call into the amazon/dynamodb-local container. These will be managed by docker-compose.

The Data

I will be building an app to display football scores. The “partition key” will be the league id. The sort id will be a mix of the entry type (team, match or league info), and a unique identifier (uuid).

Below is a csv of the data I will be adding. See source here

To give you a better idea of how we will access this data. There will be three routes;

* /league/{leagueID} // league/english-premier-league
* /league/{leagueID}/match/{matchID} // league/english-premier-league/match/54daa00f-323b-4502-9d64-74236ff2e52e
* /league/{leagueID}/team/{teamID} // league/english-premier-league/team/5f8f8424-b3b3-4c87-aa49-9de1fp5c81de

Setting up

Start by adding the amazon/dynamodb-local image to docker-compose.yml.

docker-compose.yml

version: "2"

services:
  dynamodb-example-db:
    container_name: dynamodb-example-db
    image: amazon/dynamodb-local
    ports:
      - "8000:8000"

Creating and running migrations

In production, the Dynamodb table will have been created via Terraform, so our code will expect a table to be there already. Locally I will run a shell script once the container has been brought up.

Check out the directory structure for all this below;

  • migration
    • 01-create-table.json
    • 02-load-data.json
    • init.sh
  1. 01-create-table.json

This file will contain the logic to create the Dynamodb table;

We will create a dynamo table called “football-results” which will have a self-titled partition_key and sort_key

{
  "TableName": "football-results",
  "KeySchema": [
    { "AttributeName": "partition_key", "KeyType": "HASH" },
    { "AttributeName": "sort_key", "KeyType": "RANGE" }
  ],
  "AttributeDefinitions": [
    { "AttributeName": "partition_key", "AttributeType": "S" },
    { "AttributeName": "sort_key", "AttributeType": "S" }
  ],
  "ProvisionedThroughput": {
    "ReadCapacityUnits": 5,
    "WriteCapacityUnits": 5
  }
}

02-load-data.json

This will load the data into Dynamodb

Note: this is not the full file, hence the “…”

{
  "football-results": [
    {
      "PutRequest": {
        "Item": {
          "partition_key": {
            "S": "english-premier-league"
          },
          "sort_key": {
            "S": "team-bcdd89aa-6785-4570-b4e7-605f667367c7"
          },
          "name": {
            "S": "Newcastle United"
          }
        }
      }
    },
    {
      "PutRequest": {
        "Item": {
          "partition_key": {
            "S": "english-premier-league"
          },
          "sort_key": {
            "S": "54daa00f-323b-3916-9d64-74236ff2e32z"
          },
          "name": {
            "S": "Manchester United"
          }
        }
      }
    },
    {
      "PutRequest": {
        "Item": {
          "partition_key": {
            "S": "english-premier-league"
          },
          "sort_key": {
            "S": "team-5f8f8424-b3b3-4c87-aa49-9de1cd0c88ff"
          },
          "name": {
            "S": "Tottenham Hotspurs"
          }
        }
      }
    },
    ...

init.sh

This will run some AWS CLI commands and use the above files as input.

A Few points about the below;

The AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars are set for every command. This is because I want to ensure that my local global version of these env vars (which are pointed at my own private AWS account) are not used.

#!/bin/sh

echo "Deleting any pre-existing tables\n"
AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY \
aws dynamodb delete-table \
	--table-name football-results \
	--endpoint-url http://0.0.0.0:8000 \
	--region us-west-2

echo "Creating table football-results \n"
AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY \
aws dynamodb create-table \
	--cli-input-json "$(cat 01-create-table.json)" \
	--endpoint-url http://0.0.0.0:8000 \
	--region us-west-2

echo "Loading data into football-results table\n"
AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY \
aws dynamodb batch-write-item \
	--request-items "$(cat 02-load-data.json)" \
	--endpoint-url http://0.0.0.0:8000 \
	--region us-west-2

echo "Querying the data in football-results table, this is just to ensure that everything has worked as expected\n"
AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY \
aws dynamodb query \
	--table-name football-results \
	--key-condition-expression "partition_key = :league" \
	--expression-attribute-values "{\":league\": {\"S\": \"english-premier-league\"}}" \
	--endpoint-url=http://0.0.0.0:8000 --region us-west-2

Finally, I need to ensure that this script is ran as soon as my DB container is brought up. I do this via a Makefile ‘make init’

app-container = dynamodb-example-app
db-container = dynamodb-example-db

init:
	$(MAKE) build
	$(MAKE) up

build:
	docker-compose rm -vsf
	docker-compose down -v --remove-orphans
	docker-compose build

up:
	docker-compose up -d ${db-container}
	$(MAKE) migrate
	docker-compose up -d ${app-container}

migrate:
	cd ./maintenance && \
	./init.sh

Which should give us;

        Name                      Command               State           Ports
--------------------------------------------------------------------------------------
dynamodb-example-db    java -jar DynamoDBLocal.ja ...   Up      0.0.0.0:8000->8000/tcp

Success! A local Dynamodb image with a table and test data.

Running some queries

To play with the query results here are a few queries to fire off;

Get me the basic league information

AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY \
aws dynamodb query \
	--table-name football-results \
	--key-condition-expression "partition_key = :league and begins_with(sort_key, :prefix)" \
	--expression-attribute-values "{\":league\": {\"S\": \"english-premier-league\"}, \":prefix\": {\"S\":\"info\"}}" \
	--endpoint-url=http://0.0.0.0:8000 --region us-west-2

Get me all the teams in the english-premier-league

AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY \
aws dynamodb query \
	--table-name football-results \
	--key-condition-expression "partition_key = :league and begins_with(sort_key, :prefix)" \
	--expression-attribute-values "{\":league\": {\"S\": \"english-premier-league\"}, \":prefix\": {\"S\":\"team\"}}" \
	--endpoint-url=http://0.0.0.0:8000 --region us-west-2

Get me a certain match result

AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY \
aws dynamodb query \
	--table-name football-results \
	--key-condition-expression "partition_key = :league and sort_key = :matchID" \
	--expression-attribute-values "{\":league\": {\"S\": \"english-premier-league\"}, \":matchID\": {\"S\":\"match-2fed9de2-96da-4ade-b6cb-22c06499498t\"}}" \
	--endpoint-url=http://0.0.0.0:8000 --region us-west-2