TinyDevCRM Update #17: MVP development Dockerization complete

This is a summary of TinyDevCRM development for the week of May 30th, 2020 to June 6th, 2020.

Goals from last week

  • [❓] Update /channels//create/ in order to create the SQL stored procedures for creating trigger, dropping trigger, creating function, and dropping function to issue a notification on a global PostgreSQL channel (wrap within django.db.transaction if possible)
  • [❓] Create Python process to read in psycopg2 asynchronous notifications from global PostgreSQL channel, and upon notify event, send event to Pushpin channel that a refresh has taken place.
  • [❓] Map Python process to a global subprocess that starts when the Django app is started (do NOT create one subprocess per API call).
  • [❓] Finish development-side Dockerization
  • [❓] Begin production Dockerization; pip install daphne and remove uwsgi, and finalize Dockerfile definitions for docker-compose.aws.yaml.
  • [❓] Publish Dockerfile images to Elastic Container Registry.
  • [❓] Update app.yaml CloudFormation stack definition in order to integrate Pushpin reverse proxy.
  • [❓] Re-deploy CloudFormation stack to release Pushpin into production.
  • [❓] Connect AWS application to AWS database, and add migration task definition.
  • [❓] Add back security group restrictions (policies and security group ingress/egress rules) for all CloudFormation resources
  • [❓] Deploy database cluster underneath private subnets + NAT instead of public subnet, and turn off direct SSH access and mapping of EC2 instances to IPV4 addresses
  • [❓] Add HTTP redirect to HTTPS using ACM-defined *.tinydevcrm.com SSL certificate.
  • [❓] Alias the DNS-qualified name of the app load balancer to api.tinydevcrm.com.

What I got done this week

  • [✔] Update /channels//create/ in order to create the SQL stored procedures for creating trigger, dropping trigger, creating function, and dropping function to issue a notification on a global PostgreSQL channel (wrap within django.db.transaction if possible) (Used a different approach instead where a global function translates the job ID into channel UUIDs to be broadcast on and starts via django-admin command)
  • [✔] Create Python process to read in psycopg2 asynchronous notifications from global PostgreSQL channel, and upon notify event, send event to Pushpin channel that a refresh has taken place.
  • [✔] Map Python process to a global subprocess that starts when the Django app is started (do NOT create one subprocess per API call).
  • [✔] Finish development-side Dockerization
  • [❌] Begin production Dockerization; pip install daphne and remove uwsgi, and finalize Dockerfile definitions for docker-compose.aws.yaml.
  • [❌] Publish Dockerfile images to Elastic Container Registry.
  • [❌] Update app.yaml CloudFormation stack definition in order to integrate Pushpin reverse proxy.
  • [❌] Re-deploy CloudFormation stack to release Pushpin into production.
  • [❌] Connect AWS application to AWS database, and add migration task definition.
  • [❌] Add back security group restrictions (policies and security group ingress/egress rules) for all CloudFormation resources
  • [❌] Deploy database cluster underneath private subnets + NAT instead of public subnet, and turn off direct SSH access and mapping of EC2 instances to IPV4 addresses
  • [❌] Add HTTP redirect to HTTPS using ACM-defined *.tinydevcrm.com SSL certificate.
  • [❌] Alias the DNS-qualified name of the app load balancer to api.tinydevcrm.com.

A bit slower than I thought it would be, but really glad that I have the development side mostly done. Now it's just packaging and deployment.

Metrics

  • Weeks to launch (primary KPI): 1 (13 weeks after declared KPI of 1 week)
  • Users talked to total: 1

RescueTime statistics

  • 75h 39m (51% productive)
    • 25h 26m “software development”
    • 21h 32m “utilities”
    • 8h 51m “entertainment”
    • 7h 20m “communication & scheduling”
    • 3h 58m “news and opinion”

iPhone screen time (assumed all unproductive)

  • Total: 27h 18m
  • Average: 3h 54m
  • Performance: ~0% change from last week

Hourly journal

https://hourly-journal.yingw787.com

Goals for next week

  • [❓] Write up README and specification documentation.
  • [❓] Write up and publish docs.tinydevcrm.com.
  • [❓] Begin production Dockerization; pip install daphne and remove uwsgi, and finalize Dockerfile definitions for docker-compose.aws.yaml.
  • [❓] Publish Dockerfile images to Elastic Container Registry.
  • [❓] Update app.yaml CloudFormation stack definition in order to integrate Pushpin reverse proxy.
  • [❓] Re-deploy CloudFormation stack to release Pushpin into production.
  • [❓] Connect AWS application to AWS database, and add migration task definition.
  • [❓] Add back security group restrictions (policies and security group ingress/egress rules) for all CloudFormation resources
  • [❓] Deploy database cluster underneath private subnets + NAT instead of public subnet, and turn off direct SSH access and mapping of EC2 instances to IPV4 addresses
  • [❓] Add HTTP redirect to HTTPS using ACM-defined *.tinydevcrm.com SSL certificate.
  • [❓] Alias the DNS-qualified name of the app load balancer to api.tinydevcrm.com.

Things I've learned this week

  • Data model definitions are difficult to think through beforehand. I had cron jobs and materialized views defined as separate entities. While cron jobs can do many things, including things unrelated to materialized views, right now they're just used to refresh a specific materialized view. So I had a cron job Django model, with a view foreign key. The tricky thing is, having multiple cron jobs to refresh a single materialized view doesn't make much sense, because the state of the view can get out of whack between multiple, unintentional refreshes. So I added a uniqueness constraint to the cron job view foreign key field. Instead of doing that, I could have added a crontab definition for refreshing the materialized view by itself, and made that an optional field, and gotten rid of the job model entirely for the MVP.

    I'm not terribly sure whether it's better to create separate models, more than you need, or create a small number and then migrate to different models after the fact. I'm not familiar with running migrations in production and I wanted to get things right beforehand.

  • It's surprisingly difficult to run SQL files from within SQL. For PostgreSQL at least, all documentation points to using psql or a similar command line utility to do so, which means SELECT cron.schedule($CRONTAB, $RUNCMD) cannot be used to run a SQL file either. This forces either a chaining of different jobs, or scheduling multiple jobs together. Initially I went for the former, then realized that I needed to know what the job ID is in order to send that out in the event refreshes table, which means I need to create and commit one job before another.

    So instead, I created an ArrayField that took in integers vs. an IntegerField. ArrayFields are only available via django.contrib.postgres. I'm very lucky to have chosen PostgreSQL.

  • I wish Django had additional lifecycle methods. Right now, you can take an AppConfig and override method ready() in order to run Django code once on startup. Which is nice, except that I wanted to run an SQL file on startup in order to create functions and triggers dependent on tables created by the Django ORM, and I couldn't because the migration command python manage.py migrate runs through ready(). It works out in the end, I think (I hope), because I can check whether there remain any existing migration plans programmatically. Then I can stuff the SQL file into an if/else tree, and run migrate and startup as two separate steps. But I do wish there was a DatabaseIsSynchronized() lifecycle method in the AppConfig, or additional lifecycles with a higher degree of granularity.

  • Message brokers / message queues aren't that complicated. From extensive trial and error, I think it's basically some form of backing store, and some top-level orchestration process. Where some might use RabbitMQ or Celery or Redis, which are all appropriate choices at scale (given the investment into routing logic and scale-out and such), for smaller tasks it seems a bit overkill. Instead, I can use PostgreSQL as my backing store, and a Django process as my top-level orchestration method.

    I like this way of doing things, because it's much simpler. PostgreSQL's whole existence is to write files in a reasonable manner. That makes things like hibernate-to-disk much easier, as per this Linux Weekly News article, which is a lot easier in order to secure properly, because you can flush RAM and encrypt disk with PostgreSQL.

    One separate process might not seem like a lot, but since cron only has per-minute granularity, it gives a full 60 seconds for processing of jobs to finish, which should be plenty of time for small workloads. This is made much easier because there are only so many jobs that need to be run at this particular minute, and because the payload size is controlled and extremely small. One process is also much easier to debug, and much easier to Dockerize and consider scaling.

    Also, no added dependencies! I'd rather not keep tabs on developments of RabbitMQ or Redis or Celery for 50 years if I can help it, especially since realtime streaming is still an evolving field.

    I'm sure I'm missing quite a good deal behind pub/sub (e.g. I can't imagine supporting more protocols with this arrangement), but for unitary dataflow, HTTP-based streaming, it's really lightweight and simple, which fits my bill.

  • External circumstances affect my productivity more than I thought it would. Living in the metro D.C. area, it's difficult to concentrate when you're half expecting the city to “go hot” and a Tiananmen Square style massacre to occur. I was polling the news on an hourly basis because I don't think the typical city alert system would issue notifications if martial law was to occur and the National Capital Region would lock down the highways to prevent people fleeing. Such a situation may also force emergency preparations in case water / electrictiy / sewage / Internet services were shut off.

    Sigh. I'm unhappy with how my federal tax dollars are being used, but I am grateful at how calmer minds appear to be winning out, and the situation appears to be de-escalating. I really do hope I'm just being melodramatic.


    On a personal note, my experience with police services hasn't been terribly great. My impression of police officers from six robberies is, they're not there to help you. They're there to enforce the will of the state, and the state rarely cares about you, your property, and your feelings. The only thing the state cares about is amassing as much power for itself as it can.

    That makes sense to me, laws of power and all, and it didn't seem worth fighting head-on if everybody else was fine with this arrangement. Seeing the events of the past few months, and the events to come, in my dreams for years on end is a big reason why I have a hard time relaxing, and why my life strategy appears a good deal different from that of others.

    I'm not even a black person living in America. From the handful of stories that make it on video and then make it on the news, they have it so, so, so much worse. I'm not going to sleep expecting to be shot in my own bed.

    So I definitely emphasize with the Black Lives Matter movement. I really do hope that consistent pressure will result in some lasting reforms. I also hope I'm very much wrong in my current worldview, and that a fairer world where we have much higher expectations of each other can indeed prevail.

    Since I'm too afraid to go out to the protests (because of the ongoing pandemic, and if you get shot, you have to deal with the American healthcare system), I'm putting my money where my mouth is and supporting people better than I am. I don't have an income at the moment since I'm on sabbatical, and money is a bit tighter than I thought it would be, but I'm keeping my donations to the ACLU, and shifted my tithing away from my community church to St. John's Episcopal Church of Washington D.C. as a better representation of my principles.

    It's not much, I know, but I hope after I get another job, I'll be in a better place to increase my regular donations, and help turn consistent pressure into public policy.


    Anyways. Last week proved I haven't been investing enough in my own stress management practices. I'll need to do so if I'm going to be effective during the remainder of my sabbatical.

    Here's to another week 🍷

Subscribe to my mailing list