TinyDevCRM Update #10: The End of the Beginning

This is a summary of TinyDevCRM development for the week of April 11th, 2020 to April 18th, 2020.

Goals from last week

  • [❓] Read through the Packt book “Docker on AWS”
  • [❓] Read through instructions given by AWS Support for deploying PostgreSQL to AWS using Elastic Container Service, with EC2 Task Definitions and an Elastic Block Store volume, all defined using AWS CloudFormation, and create a blog post and GitHub repo to lock in knowledge.
  • [❓] Add AWS CodeBuild and CodePipeline for CI/CD and automated, rolling deployments before beginning to iterate on the Django backend.
  • [❓] Update TinyDevCRM landing page according to feedback from three weeks ago.

What I got done this week

  • [✔] Read through the Packt book “Docker on AWS” (blog post of my thoughts on the book here)
  • [✔] Read through instructions given by AWS Support for deploying PostgreSQL to AWS using Elastic Container Service, with EC2 Task Definitions and an Elastic Block Store volume, all defined using AWS CloudFormation, and create a blog post and GitHub repo to lock in knowledge (GitHub repo here)
  • [❌] Add AWS CodeBuild and CodePipeline for CI/CD and automated, rolling deployments before beginning to iterate on the Django backend.
  • [❌] Update TinyDevCRM landing page according to feedback from three weeks ago.

I think I finally see a light at the end of the tunnel here, especially after finishing this book. I've been digging and digging and digging for so long I kind of don't know what sunlight feels like, but now I think you don't ever forget that feeling of hope, it's kind of just there waiting for you. It isn't the end, and it isn't the beginning of the end, but it feels like the end of the beginning.

Metrics

  • Weeks to launch (primary KPI): 3 (6 weeks after initially declared KPI of 1 week)
  • Users talked to total: 1

RescueTime statistics

  • 57h 4m (61% productive)
    • 15h 52m “software development
    • 8h 16m “communication and scheduling”
    • 7h 33m “utilities”
    • 6h 9m “entertainment”
    • 6h 2m “news and opinion”

iPhone Screen Time (assumed all unproductive)

  • Total: 32h 40m
  • Average: 4h 40m
  • Performance: 20% decrease from last week

Hourly journal

https://hourly-journal.yingw787.com

Goals for next week

  • [❓] Lay out the full infrastructure definition for TinyDevCRM, as described by repository tinydecrm-infra:
    • [❓] Reproduce prior week's success with PostgreSQL installation on ECS, EC2 + EBS, and CloudFormation
    • [❓] Copy over the docker-compose setup I used for tinydevcrm-api, and load up Django + gunicorn + NGINX + PostgreSQL + static files
    • [❓] Create a CloudFormation IAM + Secrets Management definition to see whether IAM user setup from Chapter 3 of Docker on AWS can be automated, and secrets management for app secret keys, app superuser, and database passwords can be pre-created and hooked in beforehand
    • [❓] Create CloudFormation templates for ECR repositories, and get the PostgreSQL + pg_cron image pushed up to ECR as part of that effort
    • [❓] Set up a CloudFormation EBS volume to scale out the data layer
    • [❓] Set up a CloudFormation EFS volume for static files to be served
    • [❓] Set up a CloudFormation EC2 definition for the compute layer, with NAT traversal, autoscaling groups, and load balancing
    • [❓] Set up a CloudFormation ECS definition for the container orchestration layer, with service / task / cluster definitions, and auto-pulling from ECR
    • [❓] Set up CI/CD pipelines for test and production deploys with AWS CodeBuild and AWS CodePipeline
    • [❓] Template the template standup process with dynamically loaded environment variables and Makefiles
    • [❓] Document each step (esp. CloudFormation resources and quirks encountered) in the YAML templates and the README

If I finish these, which is a tall order, I'll likely have more things I would need to add but am blanking on at the moment.

Things I've learned this week

  • Asynchronicity makes for poor scripting behavior. Default awscli behavior is to return a response from AWS when the action is initiated, not when it has been completed. So if you do a aws cloudformation deploy or aws cloudformation create-stack action, you get a JSON response with the Stack or the ChangeSet metadata, while the stack is being set up. You need to add aws cloudformation wait $STATUS_CODE in order to enforce synchronicity.

    I think this is probably the right system design overall, but it's something to be aware of since I'm not personally familiar with this level of sophistication. It also means keeping as much logic in CloudFormation templates and AMI definitions as possible, and avoiding bare bash scripting when possible.

  • Rolling deployments are quite slow. I'm surprised by the amount of time it takes to drain a particular container instance on ECS, especially if there's no load coming in. I thought it would linearly correlate with the number of active connections, since I figured the process would effectively re-route new connections after closing old ones. That might be happening underneath the hood, but there may be other complexities I don't yet fully understand.

  • I'm probably not going to go with a custom-defined AMI. The largest problem I see in creating a custom AMI is defining the EBS setup with an EC2 instance together, tightly coupling them. I want to scale my data and compute layers separately (because why not), and I think it's possible given the exercise I did with the PostgreSQL database (where you can run aws ec2 create-volume and delete the EC2 instance without deleting the EC2 volume (that should be templated by CloudFormation). Given the difficulties I had even reproducing the AMI setup after updating the base AMI from Amazon Linux 1 to Amazon Linux 2, and the custom scripting in order to get the EBS volume properly loaded, and the various services I would need to know to include (AWS Time Sync Service is important for databases but also really out there), I think going with CloudFormation will have better results.

I need to stop writing these reports at 1:00AM in the morning. Also, maybe I should buy a bottle of wine for all this. Until then, here's some virtual wine for the week ahead 🍷

Subscribe to my mailing list