TinyDevCRM Update #11: Foundational Learnings

This is a summary of TinyDevCRM development for the week of April 18th, 2020 to April 25th, 2020.

Goals from last week

  • [❓] Lay out the full infrastructure definition for TinyDevCRM, as described by repository tinydecrm-infra:
    • [❓] Reproduce prior week's success with PostgreSQL installation on ECS, EC2 + EBS, and CloudFormation
    • [❓] Copy over the docker-compose setup I used for tinydevcrm-api, and load up Django + gunicorn + NGINX + PostgreSQL + static files
    • [❓] Create a CloudFormation IAM + Secrets Management definition to see whether IAM user setup from Chapter 3 of Docker on AWS can be automated, and secrets management for app secret keys, app superuser, and database passwords can be pre-created and hooked in beforehand
    • [❓] Create CloudFormation templates for ECR repositories, and get the PostgreSQL + pg_cron image pushed up to ECR as part of that effort
    • [❓] Set up a CloudFormation EBS volume to scale out the data layer
    • [❓] Set up a CloudFormation EFS volume for static files to be served
    • [❓] Set up a CloudFormation EC2 definition for the compute layer, with NAT traversal, autoscaling groups, and load balancing
    • [❓] Set up a CloudFormation ECS definition for the container orchestration layer, with service / task / cluster definitions, and auto-pulling from ECR
    • [❓] Set up CI/CD pipelines for test and production deploys with AWS CodeBuild and AWS CodePipeline
    • [❓] Template the template standup process with dynamically loaded environment variables and Makefiles
    • [❓] Document each step (esp. CloudFormation resources and quirks encountered) in the YAML templates and the README

What I got done this week

  • [👉] Lay out the full infrastructure definition for TinyDevCRM, as described by repository tinydecrm-infra:
    • [✔] Reproduce prior week's success with PostgreSQL installation on ECS, EC2 + EBS, and CloudFormation

    • [✔] Create a CloudFormation IAM + Secrets Management definition to see whether IAM user setup from Chapter 3 of Docker on AWS can be automated, and secrets management for app secret keys, app superuser, and database passwords can be pre-created and hooked in beforehand

    • [✔] NEW Create a CloudFormation EC2 networking definition for VPC + subnets.

    • [❓] Set up a CloudFormation EC2 definition for the compute layer, with NAT traversal, autoscaling groups, and load balancing

    • [❓] Set up a CloudFormation EBS volume to scale out the data layer

    • [❓] Set up a CloudFormation EFS volume for static files to be served

    • [❓] Set up a CloudFormation ECS definition for the container orchestration layer, with service / task / cluster definitions, and auto-pulling from ECR

    • [❓] Copy over the docker-compose setup I used for tinydevcrm-api, and load up Django + gunicorn + NGINX + PostgreSQL + static files

    • [❓] Create CloudFormation templates for ECR repositories, and get the PostgreSQL + pg_cron image pushed up to ECR as part of that effort

    • [❓] Set up CI/CD pipelines for test and production deploys with AWS CodeBuild and AWS CodePipeline

    • [✔] Template the template standup process with dynamically loaded environment variables and Makefiles

    • [✔] Document each step (esp. CloudFormation resources and quirks encountered) in the YAML templates and the README

Metrics

  • Weeks to launch (primary KPI): 3 (7 weeks after declared KPI of 1 week)
  • Users talked to total: 1

RescueTime statistics

  • 65h 12m (50% productive)
    • 18h 45m “software development”
    • 13h 47m “entertainment”
    • 10h 43m “utilities”
    • 9h 50m “communication & scheduling”
    • 4h 32m “news & opinion”

iPhone screen time (assumed all unproductive)

  • Total: 31m 28m
  • Average: 4h 29m
  • Performance: Equal to last week

Hourly journal

https://hourly-journal.yingw787.com

Goals for next week

In tinydevcrm-infra:

  • [❓] Set up a CloudFormation autoscaling definition for the compute layer
  • [❓] Set up a CloudFormation persist definition for EBS + EFS data layer

In a new repository:

  • [❓] Set up a CloudFormation ECS definition for the container orchestration layer, with service / task / cluster definitions, and auto-pulling from ECR
  • [❓] Copy over the docker-compose setup I used for tinydevcrm-api, and load up Django + gunicorn + NGINX + PostgreSQL + static files
  • [❓] Create CloudFormation templates for ECR repositories, and get the PostgreSQL + pg_cron image pushed up to ECR as part of that effort
  • [❓] Set up CI/CD pipelines for test and production deploys with AWS CodeBuild and AWS CodePipeline

Things I've learned this week

  • Secrets lifecycle management may prove difficult. Secrets are complicated. You can't delete them through a CloudFormation stack without having a recovery period, which means you can't use the same name. I don't think you get secret aliases like you get key aliases. This means you need to use the AWS CLI for certain tasks. Secrets rotation also ties into client support, which I'm guessing is why there's an AWS Lambda custom resource embedded into the secret rotation logic. I don't have pre-configured templates for how PostgreSQL secret rotation might work. I think this may be one major gap between deploying my own database, and using something off the shelf like AWS RDS.

    You also can't create an EC2 keypair via CloudFormation, I'm guessing because you need to download the file to your own persist and because a key pair grants sudo access to any EC2 instance you might wish.

    Yeah, not touching this part of the deployment lifecycle often looks like a good idea right about now.

  • It doesn't end with CloudFormation. This Hacker News thread about Pulumi debates between using bare CloudFormation, and I guess procedures in order to generate CloudFormation (another forms of infrastructure-as-code) templates. It's interesting, especially since CloudFormation is not pure (esp. YAML), it has a bunch of intrinsic functions creating its own AST, where the limitations are clear chaining a bunch of functions together. I'm sticking to CloudFormation for the time being though.

Subscribe to my mailing list