TinyDevCRM Update #6: More DevOps Disappointment

This is a summary of TinyDevCRM development for the week of March 14th, 2020 to March 21st, 2020.

Goals from last week

  • [❓] Finish a tutorial on shipping a webapp backend to AWS Fargate + AWS Elastic Container Service (ECS)
  • [❓] Apply learnings from that tutorial to original Docker Compose tutorial completed last week
  • [❓] Apply learnings from both tutorials to TinyDevCRM API and launch something to AWS ECS
  • [❓] Finalize basic authentication workflows and configuration work between backend / frontend and dev / production
  • [❓] Update front-end dashboard and auth pages with necessary pages + mobile responsiveness for MVP
  • [❓] Soft launch MVP by talking to one person from my early adopters list
  • [❓] Update landing page using https://landing.ant.design

What I got done this week

  • [❌] Finish a tutorial on shipping a webapp backend to AWS Fargate + AWS Elastic Container Service (ECS)
  • [❌] Apply learnings from that tutorial to original Docker Compose tutorial completed last week
  • [❌] Apply learnings from both tutorials to TinyDevCRM API and launch something to AWS ECS
  • [❌] Finalize basic authentication workflows and configuration work between backend / frontend and dev / production
  • [❌] Update front-end dashboard and auth pages with necessary pages + mobile responsiveness for MVP
  • [❌] Soft launch MVP by talking to one person from my early adopters list
  • [✔] Update landing page using https://landing.ant.design

Metrics

  • Weeks to launch (primary KPI): 1 (2 weeks after declared KPI)
  • Users talked to: 1

RescueTime statistics

  • Total: 75h 41m (58% productive)
    • 18h 56m “software development”
    • 13h 54m “communication & scheduling”

iPhone Screen Time (assumed all unproductive)

  • Total: 31h 30m
  • Average: 4h 30m / day
  • 19% decrease from last week

Hourly journal

March 15th, 2020

  • 07:00AM-08:00AM: N/A

  • 08:00AM-09:00AM: N/A

  • 09:00AM-10:00AM: Commitment from 9:30AM to 10:30AM.

  • 10:00AM-11:00AM: Commitment from 9:30AM to 10:30AM.

  • 11:00AM-12:00PM: Created this blog post, working to ship last week's blog post to Bytes by Ying.

  • 12:00AM-01:00PM: N/A, not quite sure where this time went.

  • 01:00PM-02:00PM: N/A, not quite sure where this time went.

  • 02:00PM-03:00PM: N/A, not quite sure where this time went.

  • 03:00PM-04:00PM: Chores.

  • 04:00PM-05:00PM: Chores.

  • 05:00PM-06:00PM: N/A

  • 06:00PM-07:00PM: Commitment until 8:00PM.

  • 07:00PM-08:00PM: Commitment until 8:00PM.

  • 08:00PM-09:00PM: N/A

  • 09:00PM-10:00PM: N/A

  • 10:00PM-11:00PM: Restarting work to ship last week's blog post to Bytes by Ying. Launched blog post to Bytes by Ying and finished Pioneer.app weekly update by 10:45PM ET. Doing household chores again.

  • 11:00PM-12:00PM: N/A

March 16th, 2020

  • 07:00AM-08:00AM: N/A

  • 08:00AM-09:00AM: N/A

  • 09:00AM-10:00AM: N/A

  • 10:00AM-11:00AM: Starting off the day. Posted on Indie Hackers Daily Standup. Posting on Pioneer.app voting.

  • 11:00AM-12:00PM: Finished up Pioneer.app voting around 11:14AM, offline commitment until 11:32PM, lunch until 12:00PM.

  • 12:00AM-01:00PM: Finished lunch @ 12:00PM, read “The Black Swan” for about 20 minutes, working on chores until 1PM.

  • 01:00PM-02:00PM: Chores stretched to 1:15PM, mediation for ten minutes, emails, N/A, and chores until 1:54PM. Preparing for commitment @ 2:00PM.

  • 02:00PM-03:00PM: Commitment until 3:00PM.

  • 03:00PM-04:00PM: Commitment stretched until 3:17PM. News until 3:49PM. Working through tutorial on getting set up with AWS Elastic Container Service.

  • 04:00PM-05:00PM: Working through tutorial to launch an AWS Fargate instance w/ AWS ECS task definition. Pausing in order to get to commitment @ 5PM on time.

  • 05:00PM-06:00PM: Commitment at 5PM.

  • 06:00PM-07:00PM: Commitment from 5PM stretching until 6:45PM. Chores + N/A responsible until 7:30PM.

  • 07:00PM-08:00PM: Dinner, chores, N/A.

  • 08:00PM-09:00PM: Commitment at 8PM lasting until 10PM.

  • 09:00PM-10:00PM: Commitment at 8PM lasting until 10PM.

  • 10:00PM-11:00PM: N/A (not unreasonable)

  • 11:00PM-12:00PM: Updating this Bytes by Ying weekly journal.

March 17th, 2020

  • 07:00AM-08:00AM: Woke up around 7:30AM, N/A (productive) delayed until 9:00AM

  • 08:00AM-09:00AM: N/A (unproductive) delayed until 9:00AM

  • 09:00AM-10:00AM: Preparing for water shutoff for building between 10:00AM - 2:00PM, morning routine.

  • 10:00AM-11:00AM: Beginning the work day. Wrote up some community management copy for commitment at 6:30PM and sent it to another organizer for proofing. Read through emails.

  • 11:00AM-12:00PM: Posted on Indie Hackers Daily Standup. Turned off website access for the day using /etc/hosts. Answering Pioneer.app feedback. Creating ideal WFH calendar. Updated financial ledger.

  • 12:00AM-01:00PM: Lunch, reading book, joined MicroConf Connect on Slack, N/A productive otherwise.

  • 01:00PM-02:00PM: Continuing to work on ECS tutorial. Didn't really find anything useful. Tempted to pay money for a Udemy tutorial, then looking at the large catalog of tutorials that I haven't been able to work through, and second guessing myself. Apparently, the RescueTime blockers blocks medium.com by default, so I can only screenshot some tutorials for later use. I added medium.com to FocusTime exceptions. Firefox Screenshots for full HTML bodies redact images for some reason, even through images by default have read privileges. Not sure why it does that.

  • 02:00PM-03:00PM: More bad news. According to this Stack Overflow post, AWS Fargate (serverless container orchestration) does not persist Docker volumes, and AWS RDS is recommended. I don't think AWS credits cover any kind of person support. I'm thinking about whether to rethink my development model, get something onto AWS Fargate + RDS, and look at job scheduling + pub/sub using AWS Batch and AWS Simple Notification Service. I don't like where this is going. How would you replicate Batch + SNS locally? It also seems very heavyweight (I would then be using Fargate + ECS + Batch + SNS for an ostensibly free and open source application), not to mention completely locked in.

    Looking at the Kubernetes documentation for persistent volumes, which revealed a variable called awsElasticBlockStore referencing AWS Elastic Block Store. When I searched ecs ebs on DuckDuckGo, this AWS article came up, that demonstrates how to integrate ECS and CloudFormation to support your own custom database needs. AWS is infinitely flexible, and I'm really grateful for that.

    I might start on building features locally, finalize the build pipeline for binaries and artifacts, and accelerate the timeline for creating custom package archives. That way, I can cleanly separate out the build pipeline vs. the ops pipeline, which has turned out to be a nightmare.

    I'll stay focused on Docker composing my existing application according to the django-on-docker tutorial, and push those images to AWS ECR before resuming feature development on local.

  • 03:00PM-04:00PM: Went out for a walk.

  • 04:00PM-05:00PM: Came back and took a shower, otherwise N/A unproductive.

  • 05:00PM-06:00PM: N/A unproductive until 5:33PM. Apparently Netlify charges $1000 / month for static sites. That's…impressive and messed up at the same time, given how I'm charged about 10 cents a month. I wonder how much interest there is in managed CloudFormation templates as a service. Heaven knows I need to get better at CloudFormation, after realizing what it can do.

  • 06:00PM-07:00PM: Commitment starting at 6:15PM ET.

  • 07:00PM-08:00PM: Commitment from 6:15PM ET runs until 8:00PM ET.

  • 08:00PM-09:00PM: N/A unproductive (was going to eat dinner but did other stuff). Going out for a walk.

  • 09:00PM-10:00PM: N/A unproductive.

  • 10:00PM-11:00PM: N/A unproductive.

  • 11:00PM-12:00PM: N/A unproductive.

March 18th, 2020

  • 07:00AM-08:00AM: Woke up @ 7:30AM, but didn't get much sleep last night. Slept until 8:30AM.

  • 08:00AM-09:00AM: Morning Zoom call @ 8:30AM with friends until 9:00AM. This will be taking place every day at the same time, and I'm really excited for what this might mean in whipping me into shape.

  • 09:00AM-10:00AM: N/A, breakfast, chores, unproductive otherwise

  • 10:00AM-11:00AM: Wrapping up some chores and getting started with the day. Keep waffling between desire to Dockerize application to avoid days-long troubleshooting, or package archives and shell scripts on EC2. I'll start with Dockerizing the TinyDevCRM API on localhost in order to get some commits out today, which will likely mean ECS + EBS + ECR stack on AWS in production if I don't want to replicate work for EC2.

  • 11:00AM-12:00PM: Writing my own tutorial in order to keep track of progress, and ensure that I know about the topic enough to teach others.

  • 12:00AM-01:00PM: Working to Dockerize TinyDevCRM API. Lunch break + chores.

  • 01:00PM-02:00PM: Reading through Docker Compose v3 documentation. Mein Gott, there's literally a Django + PostgreSQL tutorial on Docker Compose. I wish Docker's documentation had better SEO, especially for these specific long-tail queries.

  • 02:00PM-03:00PM: Working on copying files over from django-on-docker to have a base to intro my blog post.

  • 03:00PM-04:00PM: Call from 3:15PM to 4:00PM.

  • 04:00PM-05:00PM: I believe I got development environments working-ish, or at least Django doesn't crash when I hit ‘localhost:8000’ after executing command docker-compose -f $FILE up -d --build.

  • 05:00PM-06:00PM: Almost got production up and running. Time around 7:15PM. N/A unproductive, reading coronavirus news.

  • 06:00PM-07:00PM: N/A unproductive, reading coronavirus news. Thinking about volunteering for a software project part-time, and looking for ways to help.

  • 07:00PM-08:00PM: N/A unproductive, joined a Zoom call for volunteer coordination, submitted form to help with one project, added contact details to help with another. Honestly a bit disappointed with the quality of talent that showed up and with the committee-like organization (I think great software usually has a core developer when starting off). If I hear back and think there's a chance of success I'll pitch in.

    Taking a break from the computer, going to eat dinner and maybe go for a walk. Can't really focus right now.

  • 08:00PM-09:00PM: N/A unproductive, Skyped into a friend's meeting due to me being sick and self-quarantining. Updated Indie Hackers Daily Standup thread. Don't think I'll be much productive for the rest of the evening.

  • 09:00PM-10:00PM: N/A unproductive.

  • 10:00PM-11:00PM: N/A unproductive.

  • 11:00PM-12:00PM: N/A unproductive.

March 19th, 2020

  • 07:00AM-08:00AM: N/A

  • 08:00AM-09:00AM: N/A unproductive until 8:36AM, then joined a call until 9:00AM.

  • 09:00AM-10:00AM: N/A moderately productive.

  • 10:00AM-11:00AM: N/A breakfast until 10:15AM, continuing work on properly Dockerizing development environment locally. Got things running, need to confirm by checking API endpoints, then will tag and start working on properly Dockerizing the production environment.

  • 11:00AM-12:00PM: Confirmed one API endpoint.

  • 12:00AM-01:00PM: Confirmed rest of the API endpoints work. Heroku $5k credits came through, might look at what I can do with Heroku. Reading about Heroku Schedulers and Heroku Clock Processes, and a bit amazed at the complexity needed. Statements like “Use a job scheduler only to queue background jobs, not to execute them. Background workers then independently receive and execute the queued jobs”, or “Dynos are restarted at least once a day, some logic will need to exist on startup of the clock process to ensure a job interval wasn't skipped during the dyno restart” seem a little scary in how complex they might grow, whereas pg_cron just uses a custom PostgreSQL table cron.schedule.

  • 01:00PM-02:00PM: Lunch break @ 1:00PM, listening to Rob and Sherry Walling's livestream on staying mentally healthy during the coronavirus pandemic commitment @ 1:30PM.

  • 02:00PM-03:00PM: Commitment @ 1:30PM lasts until 2:30PM. I need to rethink my approach to this. Also I should apply to YC.

  • 03:00PM-04:00PM: Applying to YC.

  • 04:00PM-05:00PM: Finished applying to YC on a whim around 4:20PM. Going out for a walk.

  • 05:00PM-06:00PM: Came back from walk around 5:30PM. N/A unproductive otherwise.

  • 06:00PM-07:00PM: N/A unproductive.

  • 07:00PM-08:00PM: N/A unproductive.

  • 08:00PM-09:00PM: N/A unproductive.

  • 09:00PM-10:00PM: N/A unproductive.

  • 10:00PM-11:00PM: N/A unproductive.

  • 11:00PM-12:00PM: N/A unproductive.

March 20th, 2020

  • 06:00AM-07:00AM: Got up @ 6:45AM, pretty giddy about this because it's the first time I've been able to get up so early. Commitment @ 7:00AM, arrived @ 7:06AM.

  • 07:00AM-08:00AM: Commitment @ 7:00AM lasted until 8:00 (realistically maybe 7:45AM). Morning routine until commitment @ 8:30AM.

  • 08:00AM-09:00AM: Commitment @ 8:30AM.

  • 09:00AM-10:00AM: Commitment @ 8:30AM lasted until 9:07AM. Going out to Trader Joe's to purchase some groceries.

  • 10:00AM-11:00AM: Mein Gott the line is long. Got back around 10:45PM. N/A unproductive otherwise.

  • 11:00AM-12:00PM: N/A unproductive.

  • 12:00AM-01:00PM: Commitment @ 12:00PM started @ 12:30PM.

  • 01:00PM-02:00PM: Commitment @ 12:30PM ended @ 2:00PM. Preparing for another commitment @ 2:00PM.

  • 02:00PM-03:00PM: Call @ 2:00PM - 2:30PM. Lunch until 3:00PM.

  • 03:00PM-04:00PM: Figuring out Dockerizing production environment. NGINX is not giving me any feedback or logs using docker logs service_nginx_1. NGINX is quite new to me, like I understand it's important but I'm not sure why. Having a hard time reconciling development and production environments, even though docker run contexts are the same. I'm tempted to just run the development version of docker-compose in production, and forget about NGINX and gunicorn.

  • 04:00PM-05:00PM: Continuing to dick around with gunicorn settings. Screw it, I'll ship the development version into production and ask for help later.

    Apparently ecs-cli has ordering in its CLI flag, or at least ecs-cli compose -f $FILE create $PARAMS works while ecs-cli compose create $PARAMS -f $FILE does not. Also, images must be pre-built. Built an image for the web container, created a repository on AWS Elastic Container Registry, and pushed up an image.

  • 05:00PM-06:00PM: Encountering error from AWS command:

    $ ecs-cli compose -f services/docker-compose.production.yaml create --region us-east-1 --aws-profile admin
    WARN[0000] Skipping unsupported YAML option for service...  option name=depends_on service name=web
    ERRO[0000] Error registering task definition             error="ClientException: Fargate only supports network mode ‘awsvpc’." family=services
    ERRO[0000] Create task definition failed                 error="ClientException: Fargate only supports network mode ‘awsvpc’."
    FATA[0000] ClientException: Fargate only supports network mode ‘awsvpc’.
    

    With some degree of desperation, trying to use an AWS EC2 instance in order to run a development version of the git repo. Then tried to access $PUBLIC_IPV4_ADDRESS:8000, got a network timeout. People are unsubscribing from my product updates because I haven't been able to release something. I think I'm going to go for a run and try to calm my racing mind.

  • 06:00PM-07:00PM: N/A productive until 6:15PM. N/A unproductive otherwise.

  • 07:00PM-08:00PM: N/A productive.

  • 08:00PM-09:00PM: Not sure what happened to this time.

  • 09:00PM-10:00PM: Not sure what happened to this time.

  • 10:00PM-11:00PM: N/A unproductive.

  • 11:00PM-12:00PM: N/A unproductive.

March 21st, 2020

  • 07:00AM-08:00AM: N/A

  • 08:00AM-09:00AM: N/A unproductive until 8:45AM, got up late for commitment @ 8:30AM.

  • 09:00AM-10:00AM: Commitment @ 8:30AM until 9:00AM, N/A semi-productive until 9:30AM, breakfast.

  • 10:00AM-11:00AM: Setting up a new landing page. This is a pretty wrenching decision for me, because I had wanted to update the landing page after I had finished building out the actual MVP, which has taken two months and not gotten anywhere. But I don't think I can afford to put it off any longer, because I can't get any feedback if I don't have a landing page and people aren't interested in the product in the first place.

    I settled on using Gatsby.js instead of hugo for this. I chose Gatsby.js because it's React.js based, which is a JavaScript UI framework I'm familiar with, and because since this landing page is more or less final, I need the ability to tune layouts and components without tearing down the entire thing. I might also need to have some degree of interactivity. I'd rather start at React.js, and move towards templating, rather than start with a template and add in state management tools post-hoc.

  • 11:00AM-12:00PM: Got deployment pipeline set up as usual. I think unlike the numerous hugo sites I've built out, I should probably Dockerize the development site. I've encountered build errors out the bat, as well as linting warnings, and I need some way to lock this all down, and I don't trust a language-specific package management framework like yarn to do it for me.

  • 12:00AM-01:00PM: Making lunch.

  • 01:00PM-02:00PM: Working on landing page. Man, is it so much easier to get started with something when you have already done something similar half a dozen times. Hopefully this will be the case for when

  • 02:00PM-03:00PM: Working on landing page.

  • 03:00PM-04:00PM: Working on landing page.

  • 04:00PM-05:00PM: Finished creating the new landing page @ https://new.tinydevcrm.com. Working on writing up this weekly report. Hope to get it out before the end of the day today.

  • 05:00PM-06:00PM: N/A unproductive.

  • 06:00PM-07:00PM: N/A unproductive.

  • 07:00PM-08:00PM: N/A unproductive.

  • 08:00PM-09:00PM: N/A unproductive.

  • 09:00PM-10:00PM: N/A unproductive.

  • 10:00PM-11:00PM: N/A unproductive.

  • 11:00PM-12:00PM: N/A unproductive.

Goals for next week

  • [❓] Finish the MVP. Whatever finish means at this point.

Things I've learned this week

  • Let's talk about the elephant in the room. The coronavirus pandemic officially kicked into high gear where I am sometime last week. I believe from sources I've found we're around two weeks behind Italy, where currently the entire country is locked down besides essential service providers. France declared a nationwide lockdown for two weeks yesterday. Here in the U.S. we don't have enough coronavirus tests and real numbers are at least two, likely three, maybe four or more orders of magnitude from what they really are.

    I'm really fortunate to have a support network where I am, and have the ability to work at home and have an extremely flexible schedule due to being on sabbatical, but even so, I feel as if a wrench had been thrown into my plans. The stock market isn't handling this well, per usual, and the forced shutdown will almost certainly cause a recession. I think this will make finding the job I want much more difficult. On the other hand, I think this will make other opportunities, such as starting a company or going to grad school, more attractive since salaries likely won't be as high. What matters now is great execution on the priorities I've laid out for myself.

    I don't hope that the virus ends soon, because I don't expect it to. I also don't expect this pandemic to be the last one I'll encounter, but rather the first of many. I'm focusing on developing coping mechanisms and productivity strategies for alleviating cabin fever and the overall doom and gloom feeling. I'm really lucky in that regards, because failure is nothing new to me, and therefore I already have plenty of experience in terms of dealing with pain and suffering. It definitely helps in building out an antifragile mindset that becomes stronger with each shock, which is always a work in progress.

    As JFK said, do not pray for easy lives. Pray to be a stronger person.

  • My God, DevOps is never-ending. Quite literally, to do anything I think you either need to pay for a managed solution, or you need to know literally half of the underlying AWS infrastructure. There's very little in between, something like a TypeForm for CloudFormation templates that might bridge technical and non-technical.

    I appreciate devops a great deal more than I did just a few short weeks ago. There's really no end to it. It's not as attractive as AI/ML or front-end engineering, and maybe not as interesting to me as data engineering or platform / infrastructure engineering and engineering optimization. But man, none of that matters if you can't ship your stuff.

  • I do like AWS. For example, I wasn't entirely sure whether AWS Fargate naturally persisted Docker volumes using EFS. This GitHub Issue is extraordinarily popular. Looking at the Kubernetes documentation for persistent volumes, revealed a variable called awsElasticBlockStore referencing AWS Elastic Block Store. When I searched ecs ebs on DuckDuckGo, this AWS article came up, that demonstrates how to integrate ECS and CloudFormation to support your own custom database needs. AWS is infinitely flexible.

    For my MVP, I might use a janky EC2 + ECR setup, and run Docker Compose locally, because I care more about getting to the feature parity I want and because I intend to reduce my ops overhead to a bunch of binaries in a package archive and some shell scripts. But it's great to know CloudFormation can do these things!

Here's to the week ahead 🍷


Subscribe to my mailing list