The finished project will be a go binary that will live in an AWS EC2 instance and speak to Dynamodb. I will be running my end to end tests locally and for this I want my code to speak to a locally running instance of Dynamodb; instead of speaking to AWS servers. I will accomplish this by using the amazon/dynamodb-local docker image.
My application code will live in a scratch image, and will call into the amazon/dynamodb-local container. These will be managed by docker-compose.
The Data
I will be building an app to display football scores. The “partition key” will be the league id. The sort id will be a mix of the entry type (team, match or league info), and a unique identifier (uuid).
Below is a csv of the data I will be adding. See source here
To give you a better idea of how we will access this data. There will be three routes;
* /league/{leagueID} // league/english-premier-league
* /league/{leagueID}/match/{matchID} // league/english-premier-league/match/54daa00f-323b-4502-9d64-74236ff2e52e
* /league/{leagueID}/team/{teamID} // league/english-premier-league/team/5f8f8424-b3b3-4c87-aa49-9de1fp5c81de
Setting up
Start by adding the amazon/dynamodb-local image to docker-compose.yml.
docker-compose.yml
version: "2"
services:
dynamodb-example-db:
container_name: dynamodb-example-db
image: amazon/dynamodb-local
ports:
- "8000:8000"
Creating and running migrations
In production, the Dynamodb table will have been created via Terraform, so our code will expect a table to be there already. Locally I will run a shell script once the container has been brought up.
Check out the directory structure for all this below;
- migration
- 01-create-table.json
- 02-load-data.json
- init.sh
- 01-create-table.json
This file will contain the logic to create the Dynamodb table;
We will create a dynamo table called “football-results” which will have a self-titled partition_key and sort_key
{
"TableName": "football-results",
"KeySchema": [
{ "AttributeName": "partition_key", "KeyType": "HASH" },
{ "AttributeName": "sort_key", "KeyType": "RANGE" }
],
"AttributeDefinitions": [
{ "AttributeName": "partition_key", "AttributeType": "S" },
{ "AttributeName": "sort_key", "AttributeType": "S" }
],
"ProvisionedThroughput": {
"ReadCapacityUnits": 5,
"WriteCapacityUnits": 5
}
}
02-load-data.json
This will load the data into Dynamodb
Note: this is not the full file, hence the “…”
{
"football-results": [
{
"PutRequest": {
"Item": {
"partition_key": {
"S": "english-premier-league"
},
"sort_key": {
"S": "team-bcdd89aa-6785-4570-b4e7-605f667367c7"
},
"name": {
"S": "Newcastle United"
}
}
}
},
{
"PutRequest": {
"Item": {
"partition_key": {
"S": "english-premier-league"
},
"sort_key": {
"S": "54daa00f-323b-3916-9d64-74236ff2e32z"
},
"name": {
"S": "Manchester United"
}
}
}
},
{
"PutRequest": {
"Item": {
"partition_key": {
"S": "english-premier-league"
},
"sort_key": {
"S": "team-5f8f8424-b3b3-4c87-aa49-9de1cd0c88ff"
},
"name": {
"S": "Tottenham Hotspurs"
}
}
}
},
...
init.sh
This will run some AWS CLI commands and use the above files as input.
A Few points about the below;
The AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars are set for every command. This is because I want to ensure that my local global version of these env vars (which are pointed at my own private AWS account) are not used.
#!/bin/sh
echo "Deleting any pre-existing tables\n"
AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY \
aws dynamodb delete-table \
--table-name football-results \
--endpoint-url http://0.0.0.0:8000 \
--region us-west-2
echo "Creating table football-results \n"
AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY \
aws dynamodb create-table \
--cli-input-json "$(cat 01-create-table.json)" \
--endpoint-url http://0.0.0.0:8000 \
--region us-west-2
echo "Loading data into football-results table\n"
AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY \
aws dynamodb batch-write-item \
--request-items "$(cat 02-load-data.json)" \
--endpoint-url http://0.0.0.0:8000 \
--region us-west-2
echo "Querying the data in football-results table, this is just to ensure that everything has worked as expected\n"
AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY \
aws dynamodb query \
--table-name football-results \
--key-condition-expression "partition_key = :league" \
--expression-attribute-values "{\":league\": {\"S\": \"english-premier-league\"}}" \
--endpoint-url=http://0.0.0.0:8000 --region us-west-2
Finally, I need to ensure that this script is ran as soon as my DB container is brought up. I do this via a Makefile ‘make init’
app-container = dynamodb-example-app
db-container = dynamodb-example-db
init:
$(MAKE) build
$(MAKE) up
build:
docker-compose rm -vsf
docker-compose down -v --remove-orphans
docker-compose build
up:
docker-compose up -d ${db-container}
$(MAKE) migrate
docker-compose up -d ${app-container}
migrate:
cd ./maintenance && \
./init.sh
Which should give us;
Name Command State Ports
--------------------------------------------------------------------------------------
dynamodb-example-db java -jar DynamoDBLocal.ja ... Up 0.0.0.0:8000->8000/tcp
Success! A local Dynamodb image with a table and test data.
Running some queries
To play with the query results here are a few queries to fire off;
Get me the basic league information
AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY \
aws dynamodb query \
--table-name football-results \
--key-condition-expression "partition_key = :league and begins_with(sort_key, :prefix)" \
--expression-attribute-values "{\":league\": {\"S\": \"english-premier-league\"}, \":prefix\": {\"S\":\"info\"}}" \
--endpoint-url=http://0.0.0.0:8000 --region us-west-2
Get me all the teams in the english-premier-league
AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY \
aws dynamodb query \
--table-name football-results \
--key-condition-expression "partition_key = :league and begins_with(sort_key, :prefix)" \
--expression-attribute-values "{\":league\": {\"S\": \"english-premier-league\"}, \":prefix\": {\"S\":\"team\"}}" \
--endpoint-url=http://0.0.0.0:8000 --region us-west-2
Get me a certain match result
AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY \
aws dynamodb query \
--table-name football-results \
--key-condition-expression "partition_key = :league and sort_key = :matchID" \
--expression-attribute-values "{\":league\": {\"S\": \"english-premier-league\"}, \":matchID\": {\"S\":\"match-2fed9de2-96da-4ade-b6cb-22c06499498t\"}}" \
--endpoint-url=http://0.0.0.0:8000 --region us-west-2