TinyDevCRM Update #5: DevOps Tutorials and Grunging

Table of Contents

This is a summary of TinyDevCRM development for the week of March 7th, 2020 to March 14th, 2020.

Goals from last week

  • [❓] Ship some form of backend to AWS Elastic Beanstalk to get a full-stack workflow going in production
  • [❓] Figure out how to use PostgreSQL instead of SQLite for Django while self-deploying instance (i.e. don't use RDS)
  • [❓] Talk to at least 1 person on my early adopters list in order to build accountability

What I got done this week

  • [❌] Ship some form of backend to AWS Elastic Beanstalk to get a full-stack workflow going in production
  • [✔] Figure out how to use PostgreSQL instead of SQLite for Django while self-deploying instance (i.e. don't use RDS)
  • [❌] Talk to at least 1 person on my early adopters list in order to build accountability

Metrics

  • Weeks to launch (primary KPI): 2 (has been at 2 for the past 4 weeks or so + 1 week after declared KPI)
  • Users talked to: 0

Hour by hour journal

March 8th, 2020

  • 07:00AM-08:00AM: N/A

  • 08:00AM-09:00AM: N/A

  • 09:00AM-10:00AM: N/A

  • 10:00AM-11:00AM: N/A

  • 11:00AM-12:00PM: N/A

  • 12:00AM-01:00PM: N/A

  • 01:00PM-02:00PM: N/A

  • 02:00PM-03:00PM: N/A

  • 03:00PM-04:00PM: N/A

  • 04:00PM-05:00PM: N/A

  • 05:00PM-06:00PM: N/A

  • 06:00PM-07:00PM: N/A

  • 07:00PM-08:00PM: N/A

  • 08:00PM-09:00PM: N/A

  • 09:00PM-10:00PM: N/A

  • 10:00PM-11:00PM: N/A

  • 11:00PM-12:00PM: N/A

March 9th, 2020

  • 07:00AM-08:00AM: N/A (This is something I should probably change, my ideal day would start working at around 7AM and be getting up at around 6AM.)

  • 08:00AM-09:00AM: N/A (Just waking up around now)

  • 09:00AM-10:00AM: Morning routine, working on submitting feedback for Pioneer.app (around 5-10 minutes).

  • 10:00AM-11:00AM: Morning routine, working on submitting feedback for Pioneer.app (around 30 minutes).

  • 11:00AM-12:00PM: Finished up submitting feedback for Pioneer.app (around 30 minutes), published post to the Indie Hackers Daily Standup group,

  • 12:00AM-01:00PM: Took a walk to re-focus (40 minutes), prep for commitment at 1PM.

  • 01:00PM-02:00PM: Commitment at 1PM for 23 minutes, prepare for commitment at 2PM, took care of some chores that I didn't do during the weekend (flushing notes to spreadsheets)

  • 02:00PM-03:00PM: Commitment for an hour.

  • 03:00PM-04:00PM: Commitment for an hour, stretched to 45 minutes past. Doing some email work for YC SUS on the side, took another 10 minutes. Prepping for an asynchronous commitment at 4PM (online coffee and code with a friend).

  • 04:00PM-05:00PM: Looking at AWS Elastic Beanstalk documentation for deploying a Django application. Installed the AWS Elastic Beanstalk CLI. Ran eb init -p python-3.8 tinydevcrm-api but was denied automatic creation because no IAM profiles grant EB or EC2 access. Logged into AWS IAM console and discovered some extremely old IAM users with some extremely old access keys (one thing I wish to address with TinyDevCRM is an access key rotation reminder client). Rotated keys for Elastic Beanstalk user, and added them to local AWS profile using aws configure --profile $PROFILE_NAME. Re-ran eb init command with --profile flag. Apparently AWS Elastic Beanstalk only supports Python 3.6, and not Python 3.7 or Python 3.8; this may be an argument for creating a custom AMI down the line if I find there's some features for higher versions of Python that I just absolutely need for some reason (only one I can think of is async support or bugfixes, or if the C/C++ FFI gets updated and I use Cython for whatever reason). Ran eb create and I'm not sure why it's creating an AWS S3 bucket for EB assets in us-west-2 instead of us-east-1 where my stated preferred IAM region is. Deploy failed, getting an HTTP 404.

    Also a bit concerned about the PostgreSQL on EC2 thing, but apparently AWS marketplace should take care of it, through PostgreSQL managed by Bitnami. I'll worry about custom PostgreSQL EC2 AMIs at a later date.

    Also apparently, you can create Dockerfile bases from other local Dockerfiles. This may be helpful when looking at how to Dockerize a deployment to AWS using AWS Elastic Beanstalk or other.

  • 05:00PM-06:00PM: Added WSGI config to .ebextensions/django.config and re-ran deployment using eb deploy. Got another HTTP 404. Updated WSGI path in order to match path for source project. Got another HTTP 404. Also AWS EB instance is not showing up at all (not as unhealthy, but not at all) in AWS EB GUI console, while AWS EB CLI console demonstrates correct information. Not sure whether this is because I am using an IAM user to deploy items, but I thought that AWS root should be able to access data from all IAM users anyways.

    Apparently the AWS EB GUI console did not show any applications running because I was looking at us-east-1 instead of us-west-2. I changed the region and the AWS EB app is showing up properly. Error is my WSGI path points to a file that does not exist. Looking at eb logs gives much better insight into why things are failing; it may be due to me calling the directory src instead of core or something else.

    Updated AWS EB instance, and deployment is still failing with HTTP 4xx errors (better than HTTP 5xx errors). Still not sure what is going on.

  • 06:00PM-07:00PM: Attempting to run src/core/wsgi.py files independently of AWS Elastic Beanstalk, ran into a number of errors. Added some ipdb traces within Django dependency source code and application code, and came to realize that Django wasn't parsing the environment variable DJANGO_SETTINGS_MODULE correctly. moved wsgi.py and asgi.py from src/core to src/ and Python files ran fine afterwards. Published eb deploy but still encountered failures, for some reason eb config is showing an older .ebextensions/django.config setting (did it get detached, or not named correctly? I'm not sure).

    Continuing to get errors, and logs do not indicate whether source code refresh is to blame (error looks quite different). Pausing work in order to eat dinner, and head to commitment at 8PM in the city.

    Considering running through the introductory AWS Elastic Beanstalk tutorial just to verify saneness, and then copying settings to existing project, or adding SSH ability into the underlying EC2 instance, because this is quite ridiculous 😠

  • 07:00PM-08:00PM: Traveling to city for commitment at 8PM.

  • 08:00PM-09:00PM: Two hour offline commitment.

  • 09:00PM-10:00PM: Two hour offline commitment.

  • 10:00PM-11:00PM: Traveling back home.

  • 11:00PM-12:00PM: Wrapping up the day.

March 10th, 2020

  • 07:00AM-08:00AM: N/A

  • 08:00AM-09:00AM: N/A

  • 09:00AM-10:00AM: Morning routine.

  • 10:00AM-11:00AM: Beginning day's work. Starting off working on tutorial for Django application deployment to Elastic Beanstalk, and then configuring SSH access for underlying AWS EC2 instance. Finished working through tutorial up to deployment phase (up to but not including “Update your application”). While I encountered HTTP 5xx errors, they were due to some issues around Django env config ALLOWED_HOSTS and having a requirements.txt in a specific location of the application. GitHub repository here. Now there's a Django app template that I know works correctly. Terminated AWS Elastic Beanstalk instance and heading back to api.tinydevcrm.com in order to address broken AWS Elastic Beanstalk deployment there.

  • 11:00AM-12:00PM: Taking a closer look between the template AWS Elastic Beanstalk starter repo, and the tinydevcrm-api project, I think the big difference between the projects is how the latter used django-admin startapp to create different Django apps to separate concerns.

    WSGI issues continue to plague deployment efforts; however, it does appear that after tearing down and re-creating AWS Elastic Beanstalk instances, CLI deployment does reflect within the GUI, and I get different errors. One thing I'm wishing for is knowing how to validate WSGI application objects. When I run wsgi.py using python3.6, I don't get any kind of stdout.

    Also annoying is how eb health will show a green health status even though the server is throwing off HTTP 5xx errors. This means I need to look at eb logs, which aren't reflective of the development environment (which appears nominal with both the Django development server and gunicorn.)

  • 12:00AM-01:00PM: Looking at SSHing into the server. If I remember correctly, I did go ahead and create an SSH keypair for this instance.

    Well, I found one error staring me in the face when looking over the log. [Tue Mar 10 16:11:28.406381 2020] [:error] [pid 3686] [remote 127.0.0.1:0] django.core.exceptions.ImproperlyConfigured: SQLite 3.8.3 or later is required (found 3.7.17). I took a closer look at why I would have such an old version of SQLite (since my version of Python has sqlite3 3.31.x), and this may be an issue with the version of Python AWS Elastic Beanstalk ships with. The Django / AWS Elastic Beanstalk tutorial uses Django 2.x, while I am using Django 3.x, and I don't want to downgrade since Django 3.x is the first one to be Python 3.x exclusive.

    This is absolutely infuriating. I get why this happened (I mean, Python 3.6 is really good for a managed platform, many companies are still on Python 2.x and can never switch to Python 3.x due to lack of automated tooling), but environment mismatches shouldn't happen in a world of software virtualization. I think when others have mentioned using something like docker compose or Kubernetes, and I dismissed it out of hand due to concerns over premature optimization, I also assumed that dependencies would work properly for base case scenarios. I might be forced to reimagine that assumption.

    Currently, a beta AWS Elastic Beanstalk Python version with Python 3.7 and Amazon Linux 2 v0.1.0 is out. I'll see whether I can tear down my existing instance and try the beta AMI.

    Preparing for a commitment at 1PM and at 2PM.

  • 01:00PM-02:00PM: Commitment at 1PM (15 minutes), taking a midday break. Preparing for commitment at 2PM.

  • 02:00PM-03:00PM: Commitment at 2PM (45 minutes), geographic relocation.

  • 03:00PM-04:00PM: Conversation with friend for about thirty minutes, taking care of another commitment (15 minutes).

  • 04:00PM-05:00PM: Other commitment stretched until 30 minutes past. Beginning to answer Pioneer.app feedback.

  • 05:00PM-06:00PM: Answered feedback, stretched to 30 minutes past. Talking with a friend for about 10 minutes, also sending emails for AWS Activate.

  • 06:00PM-07:00PM: Preparing for a commitment @ 6:30PM. Also preparing for an offline commitment @ 7:30PM.

  • 07:00PM-08:00PM: Walking towards 7:30PM commitment, commitment until 9PM.

  • 08:00PM-09:00PM: Commitment until 9PM.

  • 09:00PM-10:00PM: (Not sure where this time went, but I could have used it to continue to work.)

  • 10:00PM-11:00PM: (Not sure where this time went, but I could have used it to continue to work.)

  • 11:00PM-12:00PM: Wrapping up the day.

March 11th, 2020

  • 07:00AM-08:00AM: N/A

  • 08:00AM-09:00AM: N/A, delayed.

  • 09:00AM-10:00AM: N/A, delayed.

  • 10:00AM-11:00AM: Morning routine, ready by 10:30AM. Seeing whether I can get an initial version of the backend deployed into production before switching gears and learning the Docker compose version of doing the same thing.

    I have no idea what's going on. After deploying my stack to AWS Elastic Beanstalk, I got a jumble of corrupted Unicode errors after downloading logs. eb logs produced a Python error, saying invalid continuation byte. As one last hurrah, downgraded Django 3.0.3 to Django 2.2.6, and redeployed to see whether that would fix things. No luck.

    Making the jump early to switch deployments to Docker and Docker Compose. I can't afford to play this game any longer. Taking a look at this tutorial in order to get a handle on how to deploy Django / PostgreSQL apps on Docker Compose and AWS Elastic Container Service (ECS).

    Checked when exactly I committed to people that I would be starting serious work on TinyDevCRM. Turns out it was on February 14th. It hasn't even been a month yet. Maybe I'm just freaking out unnecessarily about the arbitrary deadline I want to meet, but talking with people on Pioneer.app indicates MVP development for SaaS products takes around 1-2 months, with a minimal commitment of around 40 hours a week. This is also my first time building a SaaS from scratch, too.

  • 11:00AM-12:00PM: Working through tutorial. Getting an error from docker-compose build saying that Version in "./docker-compose.yml" is unsupported. docker-compose --version returns 1.21.0, docker --version returns 19.03.6. Looked online for a solution to get something going, this answer on GitHub issues did it for me:

    sudo apt-get remove docker-compose
    sudo curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
    sudo chmod +x /usr/local/bin/docker-compose
    sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
    
  • 12:00AM-01:00PM: Need to handle one small chore (8 minutes). Continuing to work on tutorial. Encountered error django.db.utils.OperationalError: FATAL: database "hello_django_dev" does not exist. Turns out it was a spelling error, I referenced hello_django_env for the environment variable plugged into the Django web container and hello_django_dev in the Docker Compose configuration file, which was the actual database name. Continuing with tutorial.

  • 01:00PM-02:00PM: After applying the migrations automatically with entrypoint.sh script, and re-running the Docker deployment process, checking URL http://localhost:8000 doesn't work because it resets the connection. Apparently, this part of the entrypoint script causes the script to hang:

    if [ "$DATABASE" = "postgres" ]
    then
        echo "Waiting for postgres..."
    
        while ! nc -z $SQL_HOST $SQL_HOST; do
            sleep 0.1
        done
    
        echo "PostgreSQL started"
    fi
    

    Having some difficulty debugging this issue, I'm not exactly sure why $SQL_HOST is denoted as db instead of something like localhost. When I do psql -h db, an error saying name resolution failed indicates to me db may be an alias, and not an actual host. Looking at /etc/hosts and taking the one 172.x internal IP address and && echo "success" also did not indicate nc -z exited with a successful exit code.

    Going to take a mental break, since I don't think I'm going to be able to solve this at the moment. Preparing for commitment at 2PM.

  • 02:00PM-03:00PM: Commitment at 2PM, until 4PM. Not expecting to do too much work concurrently.

  • 03:00PM-04:00PM: Left commitment early at 3:30PM, heading out for a walk and geographic relocation. Thinking about removing nc -z, but the whole point of having an entrypoint script instead of having a one-liner inlined in docker-compose.yaml is to have room for validation. Downloaded the original GitHub repo and seeing whether I can replicate behavior.

  • 04:00PM-05:00PM: Finished walking around outside (26m), geographic relocation, starting the hour @ 4:44PM.

  • 05:00PM-06:00PM: Just discovered I had $SQL_HOST **$SQL_HOST** instead of $SQL_HOST **$SQL_PORT** in my entrypoint.sh file, by using meld in order to diff two directories. Updating entrypoint.sh and seeing whether validation works properly. Continuing with tutorial.

  • 06:00PM-07:00PM: Learning about NGINX configurations in production, and having some difficulty understanding it all. Also running into an error with standing up the web server, where I'm got getting an HTTP response back from querying URL http://localhost:1337. Also can't remove file /etc/nginx/conf.d/default.conf, because it apparently doesn't exist, which is weird because I'm using the same ngnx image as the tutorial. Diagnosing the issue.

    Apparently, I spelled nginx wrong (as ngnix) within my Dockerfile, and set the incorrect upstream for nginx.conf as hello_world instead of hello_django. Not sure how to avoid duplicating certain configuration variables (like can the PostgreSQL db and Django webapp inherit the same credentials to avoid dealing with password / username mismatches? I shouldn't need to care what the credentials are..), but I'm glad debugging (using a difftool like meld) is getting faster and faster, from hours to minutes. This won't last, since this is a tutorial and there's no spellcheckers against sources of truth for actual source code.

    Updated files, requesting http://localhost:1337 still results in an error.

  • 07:00PM-08:00PM: This NGINX issue seems to be quite tricky. I've made no headway with this bug, and why the webpage says “page not found”, while the NGINX error logs are empty. I wonder whether this is an issue with the requirements.txt file, and having a slightly newer version of gunicorn. Pausing to go take a break.

  • 08:00PM-09:00PM: N/A

  • 09:00PM-10:00PM: N/A

  • 10:00PM-11:00PM: N/A

  • 11:00PM-12:00PM: N/A

March 12th, 2020

  • 07:00AM-08:00AM: N/A

  • 08:00AM-09:00AM: N/A

  • 09:00AM-10:00AM: N/A

  • 10:00AM-11:00AM: N/A

  • 11:00AM-12:00PM: Restarting work on figuring out the NGINX issue (30 minutes in). docker logs $CONTAINER is extraordinarily useful in terms of debugging.

  • 12:00AM-01:00PM: Apparently, URL http://localhost:1337/admin works fine…and after comparing it with the tutorial and adding, diffing, and modifying files further, I arrived at exactly the point I was at yesterday evening, except now I'm convinced that nothing is wrong in the first place.

    Continuing with tutorial.

  • 01:00PM-02:00PM: Added static files and created upload/ application. Commitment for Pioneer.app weekly update video at 1:30PM ET.

  • 02:00PM-03:00PM: Commitment from 1:30PM ET until 2:30PM ET, lunch.

  • 03:00PM-04:00PM: Lunch until 3:30PM ET, answering emails.

  • 04:00PM-05:00PM: Not entirely sure where this hour went, but part of it was walking around outside.

  • 05:00PM-06:00PM: Walked around outside until 5:30PM, continuing work on tutorial.

    Very interesting note: running command docker-compose exec web python manage.py startapp upload will create the files, but will use the docker user instead of the host user, and the files may not be owned by the host, but by root, which will cause issues when saving changes as the host user. In order to get around this issue, from host and as host user, run chown -R $USER:$GROUP . within that directory.

  • 06:00PM-07:00PM: Finished tutorial around 6:30PM, working on deploying tutorial application to AWS Elastic Container Service (ECS).

    According to the AWS documentation, while Docker Compose v3 file formats are supported, minor versions are not:

    At this time, the latest version of the Amazon ECS CLI only supports the major versions of Docker Compose file syntax versions 1, 2, and 3. The version specified in the compose file must be the string “1”, “1.0”, “2”, “2.0”, “3”, or “3.0”. Docker Compose minor versions are not supported.

    Need to check that Docker Compose v3.0 file format works fine, by restarting the Docker service. It does. Continuing walkthrough of AWS ECS tutorial.

    Finished installing ecs-cli. I do like how AWS signs its executables with its own PGP key, and provides a copy-and-paste example of how to check your executable after download. I want something like this, after I start shipping my own executables. Continuing walkthrough to configure AWS ECS CLI.

  • 07:00PM-08:00PM: Configured AWS ECS CLI with ecs-cli configure profile. I had went ahead and created an IAM user with full programmatic access to AWS ECS only, and stored the credentials locally using aws configure. Given that, it seems weird to me how there's “ECS profiles” and “AWS profiles”. I think both should be IAM profiles of some sort, and why I can't use the --profile-name flag to reference an IAM user stored in ~/.aws/credentials. Yes, it generates an ~/.ecs/credentials file instead. I have a bad feeling that splitting authentication logic can result in bad things down the line, but ultimately this is a client-side thing and shouldn't affect things on AWS's side.

    Apparently from this tutorial, I learned that you can tee a task.json for a given docker run command:

    docker run -i --rm micahhausler/container-transform << EOF | tee task.json
    

    Also, um, wow. There's an underwhelming number of existing tutorials on deploying a Docker Compose application to AWS ECS, and those that exist have an overwhelming amount of information. This is very weird, it's like Docker Compose is in this weird place between a developer cramming everything into one container, and full-blown infrastructure orchestration using Docker Swarm and Kubernetes. Also, most tutorials are with organizations that have their own stack from top to bottom and use it exclusively, but the most popular toolchains are composed of a variety of different frameworks from different vendors.

    I'll see whether I can run through a separate tutorial for just one or two examples, and then apply those learnings to this original tutorial, then apply everything to TinyDevCRM API. Taking a break @ 7:22PM.

  • 08:00PM-09:00PM: Dinner ready @ 8:30PM. Didn't make great use of time for the rest of the hour.

  • 09:00PM-10:00PM: N/A, didn't really do much, could use some improvement here.

  • 10:00PM-11:00PM: N/A, didn't really do much, could use some improvement here.

  • 11:00PM-12:00PM: N/A, didn't really do much, could use some improvement here.

March 13th, 2020

  • 07:00AM-08:00AM: N/A

  • 08:00AM-09:00AM: N/A

  • 09:00AM-10:00AM: Started the day around 9:32AM, ready by 10:00AM.

  • 10:00AM-11:00AM: Starting the working day, trying to find and execute a tutorial on shipping a Docker Compose application to AWS. Really, it's hard to find a proper tutorial on the “best” way to deploy an AWS application in 2020. Amplify? CloudFormation? Fargate? Suffering from analysis paralysis.

  • 11:00AM-12:00PM: Discussed with another engineer over unrelated commitment until 11:30PM ET, discussed ops workload. He suggested AWS Fargate + AWS ECS as probably my best bet for single-node hosting, otherwise AWS CloudFormation is if you want to automatically re-connect IPs, DNS lookups, zero downtime, and generally more ops flexibility.

    30 minute commitment stretched for the entire hour. Preparing for a commitment at 12:00PM.

  • 12:00AM-01:00PM: Commitment @ 12:00PM.

  • 01:00PM-02:00PM: Commitment @ 12:00PM, stretched until 1:20PM. Took a look at sending out networking emails for YC SUS W2020, ended up not doing so because signal to noise ratio is too low, and the more serious companies I recognized from Pioneer.app.

  • 02:00PM-03:00PM: Working on applying for YC Startup School credits (last day of YC SUS W2020), ended up applying for Stripe, Heroku, and Retool in addition to AWS. Taking a midday break around 2:40PM.

  • 03:00PM-04:00PM: Finished lunched around 3:40PM. Continuing work on deploying a Docker Compose instance to AWS ECS using AWS Fargate. Taking a look at AWS Fargate's Getting Started documentation, and working through the Mythical Misfits tutorial.

  • 04:00PM-05:00PM: I think the Mythical Misfits tutorial is geared moreso towards conference attendees, since it looks like it will take around two days to finish. I found this other tutorial on configuring automatic deployments to AWS Fargate, AWS Elastic Container Registry (ECR), and AWS ECS, which might be more within my realm of expertise at the moment. Read through the tutorial once, going to take a walk to cool down my brain.

  • 05:00PM-06:00PM: Finished walking around 5:32PM. Meditation, chores, and miscellaneous tasks until 5:57PM.

  • 06:00PM-07:00PM: Starting work on latter tutorial. Creating an IAM user in the AWS console. Encountering gcc build issues with uWSGI, not sure why as I have latest gcc version. Hoping installing locally isn't necessary. Was a little confused over the IAM account ID using command aws sts get-caller-identity --profile AmazonECSTaskExecutionRolePolicy. Realized docker-compose build should take care of the gcc issue. Script config_ecr.sh appears to be a janky way in order to convert docker-compose.yml script into an ECS-compatible docker-compose.yml script; I'm glad it exists, but I'm skeptical that it will replace something like AWS CloudFormation or Terraform. Didn't have jq command-line JSON parsing dependency installed, resulting in login_ecr.sh failing to run properly, and improper success code thrown. Ran sudo apt install jq and re-ran bash script successfully.

  • 07:00PM-08:00PM: Had a weird moment when I was wondering why config_ecr.py wasn't working properly (after installing Python dependencies). Turns out, I opened this in a new tmux pane, which meant that the bash context wasn't there. Ran source scripts/.env then ran python scripts/config_ecr.py, then Docker images were pushed up to AWS ECR. Command bash scripts/setup_ecs.sh rainbowtext didn't immediately work due to a blank value for CLI argument --role-name. Ran bash scripts/setup_ecs.sh rainbowtext $AWS_ROLE and script started properly. Script started hanging and I killed it using Ctrl + C. Will tear down partially created resources using the scripts/destroy.sh, or failing that, will go into the AWS console and delete resources one by one. Ran docker-compose build --build-arg build_env="production" to build Docker containers for production.

    It looks like the scripts/deploy.sh calls login_ecr.sh, config_ecr.py, and setup_ecs.sh so they don't need to be called individually. Called scripts/destroy.sh to delete resources and re-deploy.

    Got an error attempting to destroy a resource: API: ec2:DeleteInternetGateway You are not authorized to perform this operation. I'll look at adding full EC2 access to the IAM user in the AWS console, and then retrying the destroy operation. Added policy AmazonEC2FullAccess to group that IAM user composes. Deleting the AWS CloudFormation stack, and resources appear to be deleting correctly. Taking quite a while though (>20 minutes, wondering whether if this is normal, but then again things like AWS CloudFront take quite a while too). Taking this opportunity to make dinner (7:45PM).

  • 08:00PM-09:00PM: Coming back to this tutorial at around 8:18PM. Got an error:

    INFO[0000] Waiting for your cluster resources to be deleted...
    INFO[0000] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
    INFO[0061] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
    INFO[0121] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
    INFO[0182] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
    INFO[0242] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
    INFO[0303] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
    INFO[0363] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
    INFO[0423] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
    INFO[0484] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
    INFO[0544] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
    INFO[0605] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
    INFO[0665] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
    INFO[0726] Cloudformation stack status                   stackStatus=DELETE_IN_PROGRESS
    FATA[0756] Error executing 'down': Timeout waiting for stack operation to complete
    
    An error occurred (NoSuchEntity) when calling the DetachRolePolicy operation: The role with name AmazonE
    CSTaskExecutionRolePolicy cannot be found.
    
    An error occurred (NoSuchEntity) when calling the DeleteRole operation: The role with name AmazonECSTask
    ExecutionRolePolicy cannot be found.
    

    This seems to be weird, and AWS CloudFormation now has a template that has status DELETE_IN_PROGRESS, with greyed out “Delete” and other options. Around 8:21PM ET, this magically went away. Retrying the deployment.

    Got another error during re-run of command scripts/deploy.sh:

    INFO[0016] (service rainbowtext) failed to launch a task with (error ECS was unable to assume the role 'arn:aws:iam::267131297086:role/ecsRainbowtextTaskExecutionRole' that was provided for this task. Please verify that the role being passed has the proper trust relationship and permissions and that your IAM user has permissions to pass this role.).  timestamp="2020-03-14 00:26:10 +0000 UTC"
    

    Manually running the commands in setup_ecs.sh, where some commands were failing. When I got to ecs-cli up --azs ${AWS_REGION}a,${AWS_REGION}b --force --instance-role ${role_name} --cluster ${project_name}, the command appeared to freeze, but apparently CloudFormation will commit to DELETE_IN_PROGRESS before adding commit CREATE_IN_PROGRESS.

    Had to manually parse out subnet_a, subnet_b, vpc_id, and subnet_ids after I failed to assign log output of ecs-cli up to a variable. subnet_ids isn't used.

    Running command ecs-cli compose. Command failed after timing out for 5 minutes. I think I'm going to give up on this tutorial, I think all I really need is ecs-cli compose, and I'll try AWS's tutorials which may be of higher quality.

  • 09:00PM-10:00PM: N/A, unrelated work.

  • 10:00PM-11:00PM: N/A

  • 11:00PM-12:00PM: N/A

March 14th, 2020

  • 07:00AM-08:00AM: N/A

  • 08:00AM-09:00AM: N/A

  • 09:00AM-10:00AM: N/A

  • 10:00AM-11:00AM: N/A

  • 11:00AM-12:00PM: N/A

  • 12:00AM-01:00PM: N/A

  • 01:00PM-02:00PM: N/A

  • 02:00PM-03:00PM: N/A

  • 03:00PM-04:00PM: N/A

  • 04:00PM-05:00PM: N/A

  • 05:00PM-06:00PM: N/A

  • 06:00PM-07:00PM: N/A

  • 07:00PM-08:00PM: N/A

  • 08:00PM-09:00PM: N/A

  • 09:00PM-10:00PM: N/A

  • 10:00PM-11:00PM: N/A

  • 11:00PM-12:00PM: N/A

RescueTime statistics

56h 11m (64% productive); 18h 17m “software development”, 8h 24m “communications and scheduling”

Goals for next week

  • Finish a tutorial on shipping a webapp backend to AWS Fargate + AWS Elastic Container Service (ECS)
  • Apply learnings from that tutorial to original Docker Compose tutorial completed last week
  • Apply learnings from both tutorials to TinyDevCRM API and launch something to AWS ECS
  • Finalize basic authentication workflows and configuration work between backend / frontend and dev / production
  • Update front-end dashboard and auth pages with necessary pages + mobile responsiveness for MVP
  • Soft launch MVP by talking to one person from my early adopters list
  • Update landing page using https://landing.ant.design

Things I've learned this week

  • I really like just-in-time learning. Here's one definition of just-in-time learning:

    Just-in-time learning systems deliver training to workers when and where they need it. Rather than sitting through hours of traditional classroom training, users can tap into Web-based tutorials, interactive CD-ROMs and other tools to zero in on just the information they need to solve problems, perform specific tasks or quickly update their skills.

    I think when I'm looking back to my learnings with Haskell, or all the other books I've reviewed in the past, a big, big problem is how I don't really remember the information because I haven't applied it. With “The Haskell Book” or any book with exercises it's a bit better, but without having a long-term, core application it's difficult if not impossible to remember things. I think this means I burn a lot of hours learning things that I may quickly forget, such as how to execute on a user story map, where I have to dig the book back out of my archives. I can afford to do this now since I'm not too time-constrained (though I should artificially constrain myself to make my perception of time more valuable), but it severely reduces my productivity and resource utilization numbers, which impacts my personal efficacy. It's also not sustainable, not only for when this sabbatical period ends, but also when my free time is taken up by a wife and kids (which is my ideal usage of free time).

  • Don't use AWS Elastic Beanstalk. I thought it'd be friendlier to beginners and newbies like me, but after using it, I soured on the concept of having others manage your application backend toolchains. You don't control your dependencies, which means if you have a different version of Django than the one AWS EB ships with, you might encounter conflicts that prevent your app from shipping. The infrastructure isn't idempotent, so every eb deploy builds on the existing state of the application without having a clear rollback strategy. There's also the issue of integrating AWS EB with databases other than AWS RDS. Unless you're building something for a hackathon or a proof of concept, or you are AWS-native and want to integrate AWS EB with your own AMIs (maybe using something like EC2 image builder), Elastic Beanstalk introduces a high level of vendor lock-in and an even higher level of ops overhead.

    Docker Compose is pretty nice. It's containerized, but still targeted towards applications that are single-host, as opposed to multi-host orchestration frameworks like Docker Swarm or the very popular Kubernetes. You can do the same thing as Docker Compose as the Docker CLI, it's just YAML files and syntactic sugar. I messed up initially at certain points (chronicled in my hourly journal), but mistakes took hours to fix vs. days, and there ultimately was a fix, and that fix worked reliably and as expected. It's a world of difference. You can use ecs-cli compose with AWS-specific keys in your YAML configuration, and template that using a script, as a waypoint towards infrastruture-as-code using AWS CloudFormation.

  • Configuration management is such a PITA. Every framework, literally every framework, has a set of configurations that need to be managed. Django has a settings.py. docker has Dockerfile, docker-compose has a docker-compose.{development|test|staging|production}.yaml, ecs-cli compose has a modified docker-compose.yml, and dependent AWS services also have their own configurations. I could have a backend without all this configuration overhead (say use Heroku or Render), but I'd pay a premium for such services and there's the issue of possible vendor lock-in, I'd burn time trying to understand what's going on for vendor-specific config options instead of say nginx.conf, and I wouldn't really know what's out there per-se.

    If I'm building a product and looking to monetize immediately off the bat, then hell no this isn't worth it. It's a lot of pain for very little gain. But if I'm going to build a tool for myself to last the next 50 years, and I want to trust my stack and what it does, I think this could count as time well spent. I think for me, this is the hard thing about hard things; it costs time and energy where little of either is to be found or spared, and ultimately it is the belief this project will succeed and should succeed that will be the deciding factor in whether I prevail.

Here's to the week ahead 🍷


Subscribe to my mailing list