Running in Production

In this episode of Running in Production, Filipe Névola goes over building a hosting platform for Meteor apps. It’s hosted on AWS with ECS and has been running in production since 2015. Filipe talks about building critical services with Go, using Meteor to build front-end web dashboards, the importance of monitoring, using Recurly for subscription payments, multi-region AWS hosting and overall providing a highly available platform for thousands of clients. Topics Include 1:45 – High level overview of the hosting platform 7:01 – Splitting up compute resources for enterprise clients 8:44 – Motivation for choosing Go for a few of the back-end services 11:24 – Feeding data to the web dashboard with MongoDB, ECS, a load balancer and Go 15:53 – The Go proxy service was built using the standard library (no 3rd party libraries) 17:46 – Differences between the free and paid plans 22:49 – Displaying a custom page if your Meteor app happens to be down 26:28 – Going over a few services, starting with the Docker image builder 31:18 – The Go services are in a mono repo but they can be individually deployed 33:36 – The next service is the scheduler which is custom built 41:49 – How the web dashboard gets updated from events on the back-end 47:08 – The last service we’ll cover is for registering SSL certificates with Let’s Encrypt 50:21 – Monitoring is very important and they’re using Datadog, plus being on-call 54:26 – Postmark and SendGrid are both used to send emails 56:23 – Payments are handled through Recurly 1:00:28 – Loggly is used for logging and a bit of analytics 1:04:08 – Handling a lot of incoming notifications and making sense out of alerts 1:08:05 – Choosing AWS for hosting everything and using ECS over EKS 1:11:20 – It’s hosted across multiple AWS regions (Virginia, Ireland and Sydney) 1:15:55 – The open source side of Meteor is very very important 1:17:49 – How Terraform is being used to manage their infrastructure 1:20:31 – ScaleGrid is used to host their MongoDB instances 1:22:29 – How clients store their secrets 1:24:15 – How deployments are handled from development to production 1:32:34 – All data is backed up on a regular basis with lots of redundancy 1:36:01 – Handling big traffic spikes with little warning 1:38:38 – Best tips? Monitor everything and avoid premature optimization 1:41:43 – Check out Filipe on Twitter, his coding education platform and YouTube channel Links 📄 Referenc

In this episode of Running in Production, Chad Wilken goes over building a service to help contractors document their job and communicate with their crew. It’s been up since 2014. Chad talks about handling ~800k photo uploads per day, building a Rails API driven app, creating a great mobile experience with React Native, handling millions of daily Sidekiq (pro) jobs, transitioning to AWS ECS with Fargate, deploying with Capistrano and so much more. Topics Include 3:04 – Motivation for switching from .NET to Rails 5:59 – It’s a Rails API with a React front-end, first class mobile support is important 9:34 – What type of screens does this app have? 11:58 – As the CTO, Chad doesn’t get a chance to code as much as he used to 13:15 – A few Rails features and gems being used (interactor, wisper and more) 16:35 – Third party sites that they integrate with 18:11 – Contracting companies are a lot more technical than you might think 22:39 – Every photo upload hits the Rails back-end but it’s not processed there 25:30 – Millions of jobs are being run through Sidekiq per day and payments 28:46 – Spending time on the company admin back-end vs user features 33:20 – Postgres (RDS), MongoDB, Elasticsearch and Redis are being used 36:13 – The Flipper gem is being used for feature flags 37:44 – Docker is being used in development with a transition to using them in production 39:30 – How things are currently deployed and where they’re going with AWS Fargate 41:03 – From Chef to Ansible with Capistrano handling app deployments 43:35 – Ubuntu is their distro of choice 44:00 – Moving over to ECS with Fargate and hiring dedicated ops folks 48:28 – A run down of which AWS resources they’re using 49:25 – The deployment process from development to production 54:59 – All developers on the team can use whichever OS they prefer 55:37 – Handling backups of all user data 58:22 – Logging, error monitoring and alerts 1:01:40 – Best tips? Design patterns are important, so is caching and saying no 1:03:48 – Their AWS bill is about $38,000 a month 1:05:03 – Check out https://companycam.com, they’re hiring! Links 📄 References https://readme.com/ https://buildbot.net/ ⚙️ Tech Stack

In this episode of Running in Production, Michael Lynch goes over building a hardware device that lets you remote control your server without installing any software. It’s been available since mid-2020. Michael talks about how it works, using Ansible to provision a Raspberry Pi, Using Flask with SocketIO, rendering 30 frames per second at 1080p with under 200ms latency, using web components, selling his devices on Shopify, hiring quality freelance developers and more. Topics Include 7:13 – The process to build a custom piece of hardware 12:33 – 3D printing a custom case 15:16 – The assembly process and selling about 150-200ish devices a month 16:20 – Ansible and a Bash script ensure everything gets installed on the device 20:16 – Everything including the OS can run on about 1.5GB of memory 22:01 – Motivation for using Flask and Python 25:29 – Using Python’s built in testing library and a few useful 3rd party packages 26:47 – Web components are being used quite heavily 31:44 – Getting started with web components 34:34 – Rendering everything at 30 frames per second at 1080p with SocketIO 38:34 – The differences between the regular and pro version 40:57 – There’s no bundling for JS and CSS 43:11 – Docker was considered but wasn’t included in the end 46:44 – Purchasing the pro version and ancillary services around the hardware 50:52 – Shopify is used as a store front and they make about 20-30k a month 54:26 – The deploy process and reviewing code from their freelance devs 1:00:09 – Hiring quality developers by making it a great place to work for 1:02:05 – Getting new versions of the code on user’s devices 1:07:35 – Is it possible to brick your own device? It hasn’t happened yet 1:11:40 – Best tips? Incrementally build and release your product as you go 1:16:26 – How Michael hired a few initial freelance developers 1:21:41 – Check out TinyPilotKVM, Michael has a blog and is on Twitter too Links 📄 References https://en.wikipedia.org/wiki/Raspberry_Pi https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface https://css-tricks.com/an-introduction-to-web-components/ https://www.talkyard.io/use-cases <a href="https://runninginproduction.com/podcast/39-place-card-me-lets-you

In this episode of Running in Production, Nick Savrov goes over building a platform to help content creators build a business with Ruby on Rails. It’s hosted on Heroku and has been up and running in production since 2014. Nick talks about supporting 6.5 million users, using Turbolinks, having a 19 developer team working on a monolithic app, sending millions of weekly emails, storing billions of weekly events, using ShapeUp to help manage the project and much more. Topics Include 2:49 – They support 6.5 million users on the platform which helped creators make $100m+ 4:19 – There’s 2 parts to the system, the admin for creators and the user experience 8:49 – BREAKING NEWS: Rails can scale and it’s working out nicely for them as a monolith 10:47 – Turbolinks and Liquid helped their app a lot 14:10 – It feels like a Rails app but uses VueJS and InertiaJS on the front-end 18:47 – A couple of interesting gems in their Gemfile, including using Fastly 21:38 – They have a great relationship with Mux for streaming video 25:25 – Approaching the latest stable version of Rails 27:05 – 19 Rails engineers are working on the monolithic code base 28:33 – Payments are supported with Stripe, PayPal and Authorize.net 31:42 – About 10 million emails are sent out every week through SendGrid 34:29 – Custom domains for customers and using Heroku with nginx / Let’s Encrypt 39:51 – Postgres is the main database along with ClickHouse (billions of events a week) 42:33 – What types of events are being logged and how can it be viewed? 46:20 – A custom nginx router that sits in front of Heroku 50:11 – Reasons for using Heroku and setting up an auto-scaler 54:51 – A couple of Heroku add-ons and using New Relic (NOTE: 10 billion rows = 250GB) 57:38 – The deployment process from dev to prod (CI, code reviews, GitHub discussions) 1:07:57 – What is ShapeUp which is Basecamp’s methodology around project management 1:09:06 – Backing up user data and more info about ClickHouse 1:14:51 – There’s value in performing soft deletes as long as you really delete it later 1:16:03 – OpsGenie, New Relic and Uptime Robot are used for alerting and being on-call 1:22:34 – Best tips? Follow standards when you can such as using built in Rails features 1:26:03 – You can find Nick on GitHub and Telegram Links 📄 References http://railscasts.com/ https://youtu.be/KJVTM7mE1Cc?t=600 (RailsConf 2015 “Backpack” reference) <a href="https://runninginproduction.

In this episode of Running in Production, PJ Murray goes over building a customer research app with Ruby on Rails. It’s hosted on Heroku and has been up and running since mid-2020. PJ talks about using feature flags, integrating Stripe with Jumpstart Pro, building out a React front-end, the value of having business metrics, taking data security very seriously, having a pragmatic approach around test coverage and tons more. Topics Include 4:16 – It took about a month to get the MVP out and a few months after for it to be sellable 6:52 – Motivation for using Rails 8:54 – How the app works and going over some of its screens 13:21 – ActiveJob is such a good abstraction it’s easy to forget what job library you use 14:14 – A bunch of useful gems that are being used 18:14 – The user experience can often impact the technical complexity of what you’re building 21:53 – Feature flags and swinging back to JSONB columns 26:21 – Stripe handles all of the payments 28:24 – The app is pretty much one big monolith and it’s a good thing 29:55 – About 25k lines of Ruby code and 40k lines of Typescript on the front-end 33:04 – If Turbo were around for years would you have used it over React? 36:37 – You shouldn’t be afraid to touch code in your codebase 38:42 – Getting a decent amount of things planned out before implementing the code 44:00 – Postgres, Redis, Mixpanel and Datadog for app and business alerts / logging 49:19 – Limiting access to production data from developers 51:17 – Heroku helped them get to market faster and they had YCombinator credits 54:36 – The deploy process from development to production 1:01:20 – Limiting access at the GitHub repo level and Heroku 1:04:48 – In general backups are handled by the providers they use (Heroku, S3, etc.) 1:07:21 – Heroku will send out alerts if something unexpected is happening with the site 1:09:31 – Best tips? Be pragmatic about testing and code coverage 1:11:58 – User design and UX is handled by a specific team member 1:14:09 – Check out https://greatquestion.co and they’re hiring too Links 📄 References https://github.com/excid3/jumpstart DHH’s YouTube video using esbuild and Tailwind’s CLI https://github.com/nickjj/docker-rails-example https://en.wikipedia.org/wiki/System_and_Organization_Controls</

In this episode of Running in Production, Josh Kinabrew goes over building an AI driven invoice categorization system using Ruby on Rails. It’s hosted on Heroku and AWS and has been up and running since 2013. Josh talks about training an AI system to scan and break down pictures of invoices, managing thousands of clients, using the latest stable version of Rails, using Sidekiq Pro, interfacing with QuickBooks and more. Topics Include 1:52 – A little bit about how the service works 5:32 – Training an AI to process invoice items 8:33 – Thousands of clients and the motivation to use Ruby on Rails 10:28 – Using AWS Rekognition’s Textract and SageMaker for AI and ML decisions 14:55 – It takes about 2 minutes to get an initial result after uploading an invoice 18:12 – The mobile version is served through a responsive web application 21:36 – They’re using the latest stable Rails release with good test coverage 25:31 – Going over a few types of Sidekiq Pro driven background jobs that are running 29:28 – Action Mailbox was a huge win for them, it’s also a monolithic code base 31:54 – They’re using Heroku’s CI service 32:52 – Good old Sprockets is used for the asset pipeline at the moment 35:31 – Josh really likes Postgres, Redis is also being used quite a bit 40:22 – What it’s like to work with QuickBook’s imports and exports 46:30 – Most things are hosted on Heroku (2 worker and web Dynos) 49:05 – Papertrail, Sentry and Mailgun are being used 51:42 – Before sending emails most agriculture folks were faxing paperwork 52:50 – The deploy process from development to production 54:58 – Terraform is used to spin up the infrastructure, even on Heroku 59:48 – Avoiding real customer data when copying data from prod to dev 1:05:42 – Currently there’s no external monitoring checking the site for up-time 1:07:49 – Best tips? Don’t worry about being perfect, get something out there 1:09:43 – They’re hiring, you can email Josh and they’re on Twitter too Links 📄 References https://en.wikipedia.org/wiki/UAN DHH’s YouTube video using esbuild and Tailwind’s CLI https://www.terraform.io/cloud https://en.wikipedia.org/wiki/Hero%27s_journey ⚙️ Tech Stack

In this episode of Running in Production, CJ Avilla goes over building a fund raising platform with Ruby on Rails. It’s hosted on Heroku and has been running in production since 2015. CJ talks about rewriting an app with Rails, processing $30 million dollars of donations, using Stripe, maintaining a Rails 4.2 app, carefully sending out bulk emails, ensuring good tests are written, keeping things simple and more. Topics Include 2:26 – Rebuilding a similar app with a new tech stack is easier the 2nd time around 6:14 – Motivation for using Rails and live coding features right next to his client 9:33 – What the app does at a high level and what some of the screens do 12:20 – The app is running Rails 4.2 and it’s humming along with minimal maintenance 13:29 – Stripe handles accepting donations and keeping things as simple as possible 17:33 – Processing over $30 million dollars since the site went live 22:35 – It’s worth checking out which payment providers are available in different countries 24:51 – Using an older version of Stripe’s API but it’s super stable 27:22 – A couple of gems from 2015 which helped build this app 28:50 – (1) standard Heroku Dyno for the web app and (1) 2x size Dyno for the worker 32:15 – Postgres, Redis and Resque are being used along with Heroku’s cron scheduler 35:48 – Sending bulk emails out can be scary 39:42 – Using Rbenv locally to manage things in development 42:42 – The reasons for picking Heroku were mainly to avoid any type of ops work 46:03 – What it’s like to develop a new feature and push it up to production 47:58 – Not doing traditional TDD but tests are still written 51:18 – Heroku is in charge of performing daily database backups 55:40 – Handling background worker spikes with popular donation pages 57:42 – Tagging the current user in Rollbar errors and then emailing customers ASAP 1:00:27 – Best tips? Keeping it simple really drives down your maintenance 1:02:38 – CJ’s has a YouTube channel and a personal site at https://cjav.dev Links 📄 References https://www.youtube.com/CJAvilla https://stripe.com/payments/checkout https://stripe.com/docs/payments/payment-element https://faye.jcoglan.com/node.html ⚙️ Tech Stack

In this episode of Running in Production, Nick Janetakis goes over building a podcast site with Jekyll and Ruby. It’s hosted on a single DigitalOcean server and has been running in production since October 2019. Nick talks about what it takes to release an episode, keeping things simple, developing a custom audio player, hosting a bunch of sites on a single DigitalOcean server with nginx, using shell scripts to help reduce human errors and more. Topics Include 2:06 – The podcast is not sponsored and it’s done in Nick’s spare time after hours 2:30 – What’s involved end to end to put together an episode 8:50 – Each episode gets about 400 downloads or listens but it’s hard to track 10:41 – Motivation for using Jekyll and Ruby 13:03 – A couple of custom Jekyll plugins to help building a podcast site 16:55 – So many static generators to choose from, just pick one 18:09 – Use the tools that you like and don’t constantly second guess yourself 19:34 – What is Liquid (Jekyll’s templating language)? 21:01 – The custom audio player is the only real amount of JavaScript on the site 25:42 – Jekyll-Assets is being used to MD5 tag static file names for cache busting 26:34 – nginx is serving the site with Let’s Encrypt handling the SSL certificates 28:12 – The only SAAS tool being used is Google Analytics but I don’t use it for much 29:20 – A few static sites are all hosted on a $5 / month DigitalOcean server 32:06 – The server load is at 2-3% CPU with 60,000+ monthly visitors 33:06 – Debian Bullseye is running on the server 34:38 – Ansible is used to provision the server 38:47 – Deploying a new podcast episode to the site 41:59 – Making sure all tags are properly filled out 43:46 – Another shell script to auto-inject an episode’s length and file size in bytes 46:55 – There’s no secret management because there’s no secrets 47:29 – The code is backed up on GitHub and an external USB HDD 49:54 – DigitalOcean’s built in monitoring and alerting as well as Uptime Robot is being used 52:34 – Best tips? Find tools that you’re genuinely happy using and stick with them 54:39 – You can find Nick on his site, @nickjanetakis on Twitter and nickjj on GitHub Links 📄 References https://buildasaasappwithflask.com https://diveintodocker.com <a href="https://www.youtube.com/c/Ni

In this episode of Running in Production, Jason Swett goes over building an internal medical record system with Ruby on Rails. It’s hosted on AWS using Kubernetes and it’s been up and running since 2019. Jason talks about replacing a few 3rd party services with 1 custom solution, using custom generators, embracing PORO, transitioning from Ansible and individual servers to Kubernetes, making safe decisions while learning as you deploy new things and much more. Topics Include 1:14 – The dream is to replace 9 separate systems with 1 custom solution 3:02 – Deploying on day 1 and what exactly is an EMR? 6:20 – Motivation for choosing Ruby on Rails to build a medical system 10:39 – Infrastructure as code is important 13:36 – The app does scheduling and soon emails will be sent to patients 15:29 – Specific things the app does and how Rails helps Jason accomplish them 21:56 – Custom generators are used for different modules 25:11 – Every 5 seconds certain forms are auto-saved and a few gems that are being used 27:56 – Rails can only help you so much, then you’re kind of on your own 34:10 – Making a distinction between imperative and declarative code 36:36 – Server rendered templates with sprinkles of Javascript (StimulusJS) 43:53 – Looking into using Hotwire Turbo in the distant future 45:41 – Migrating to use Elasticsearch and using it over PostgreSQL’s full text search 47:49 – Using AWS for hosting and landing on using Kubernetes with EKS 52:57 – 5% of Jason’s brain is taken up at all times knowing he’s on-call 55:22 – Switching from individual servers with Ansible to Kubernetes 59:37 – Using eksctl and Helm to move towards infrastructure as code 1:06:12 – The deploy process from development to production with Kubernetes 1:09:53 – Using feature flags and code reviews 1:11:35 – Handling database migrations with a load balanced application 1:15:13 – A strategy for writing Jira tickets 1:17:53 – RDS snapshots are handling database backups 1:21:14 – Monitoring, alerting and the idea of “sharpening the saw” 1:25:47 – Best tips? Make safe decisions and improve skills such as reading documentation 1:30:44 – Check out Jason’s site at https://www.codewithjason.com/ Links 📄 References https://twitter.com/dhh (Creator of Rails) https://en.wikipedia.org/wiki/Electronic_data_interchange <a href="https://www.codewithjason.com/code-with-jason-podcast/episodes/111-dockerizing-development-and-production-with-nick-jan

In this episode of Running in Production, Vlad Radulescu goes over creating a game directory site with Ruby on Rails. It’s hosted on AWS and has been up and running since 2014. Vlad talks about having thousands of active users, interfacing with a few game platform APIs, running millions of Sidekiq jobs, storing 10+ billion database records, keeping things as a monolithic app, deploying the web app to 1 server and lots more. Topics Include 6:30 – Motivation for switching from PHP to Ruby on Rails 7:07 – Figuring out which gaming API to implement first and how their APIs are 10:27 – Reverse engineering undocumented API calls from the Windows Steam client 12:18 – Using a separate database for each gaming platform provider 15:05 – A few useful features of Rails that’s being used and Stimulus Reflex 20:19 – Switching from Webpacker to using Vite 23:14 – Using Slim instead of ERB and a few other gems being used 25:29 – It’s a single monolithic Rails app that’s using namespaces 28:36 – A lot of the time spent developing this app is working with the game APIs 31:00 – Using Sidekiq, a billion (!) gamer activities stored and millions of Sidekiq jobs 34:18 – It’s hosted on a single AWS T2.large instance (4 vCPUs / 16 GB memory) 36:13 – The database is on an R6g.large with 10+ billion records 39:14 – Ansible was used to set up the EC2 instance and it’s running Ubuntu 18.04 LTS 41:13 – A couple of AWS resources being used 43:01 – The process to ship a feature from development to production 44:16 – Switching to using TailwindCSS in a nicely controlled way 47:57 – At the moment the site is free with no desire to make it pay to win 49:55 – Hosting is $1,500 pounds a month and it’s been running for free for 7+ years 51:43 – There’s 7 terabytes of S3 storage and daily database backups 55:59 – 50ish mini servers spread across Heroku’s free tier to bypass API rate limits 59:45 – Being dialed into the ops side of things but there’s room for improvement 1:00:54 – Keeping up with a full date job and a long running passion project 1:02:25 – Best tips? Don’t forget to plan things out 1:05:56 – You can find Vlad on Twitter and Games Directory will be open source soon Links 📄 References https://www.giantbomb.com/api/ https://www.telerik.com/fiddler https://games.directory/u/pacmakaveli (Vlad’s profile) <h6 id="️-tech-stac

In this episode of Running in Production, Dieter Lunn goes over building a job board site for school bus drivers and aides using Ruby on Rails. It’s hosted on DigitalOcean with HatchBox. Dieter talks about using a bit of StimulusJS to add pins to a map, keeping things simple with a monolithic app, working on the code base with another developer, upgrading to the latest versions on a regular basis and using HatchBox to manage the servers. Topics Include 3:13 – What the site does and examples of what types of pages it has 5:21 – Motivation for using Ruby on Rails 7:02 – Specific features of Rails and gems being used 11:04 – Using StimulusJS for placing pins on an embedded 2D map 14:02 – One feature lets you get emailed when new positions open at a company 15:40 – It’s a server rendered monolithic Rails app with sprinkles of JS 19:14 – Keeping your gems up to date 23:15 – Using Pundit to manage authorization 25:27 – Keeping up with the latest versions and having an app starter project 27:09 – It’s hosted with HatchBox on DigitalOcean 32:02 – Walking us through developing a feature and deploying it to production 38:07 – Payments are handled outside of the Rails app 39:39 – Automation feels pretty good! 43:45 – Currently Honeybadger sends notifications if the site is down 45:47 – Best tips? Take your time and trust what others have built 48:08 – You can find Dieter on GitHub and on Twitter Links 📄 References https://gorails.com https://runninginproduction.com/podcast/12-learn-ruby-on-rails-through-screencast-tutorials-on-gorails https://github.com/coder2000/campestral (Rails template starter project) https://www.hatchbox.io/ ⚙️ Tech Stack

In this episode of Running in Production, Matthias Viehweger goes over building a service to help organize groups of people. It’s built with Ruby on Rails and is hosted on OpenShift with Kubernetes. It’s been running in production since 2012. Matthias talks about building a Rails Engine abstraction, creating a multi-repo monolith, using Sphinx for full text search, making the most of Kubernetes / OpenShift and lots more. Topics Include 6:23 – There’s an open source version of it along with a hosted SAAS app 9:10 – Motivation for using Ruby on Rails and updating from v4 to v6 over time 12:18 – Using Action Mailer and Delayed Job to send and receive emails 16:40 – A few other gems being used in the project 19:25 – 86 models in the core app and 275 extra for a specific Scout site (~70k LOC too) 22:49 – How billing is handled for the hosted SAAS app (it’s through invoices) 24:54 – Handling integrations by exporting data 26:31 – It’s a multi-repo monolithic application, what the “core” and “wagon” are 32:02 – There’s mostly server rendered templates with a bit of JS on the front-end 35:00 – Full text search is handled with the Thinking Sphinx gem 37:09 – There’s a separate MySQL database for each tenant 42:35 – Tech stack run down so far and how memcached is being used 44:58 – The app is set up to use Docker Compose in dev but Matthias doesn’t use it 48:05 – It’s hosted on APPUiO which is a hosted version of OpenShift 50:20 – We’re all YAML engineers, configuring Kubernetes and DB migrations 1:03:04 – Kustomize is being used instead of Helm for templating YAML files 1:05:20 – The Kubernetes related code is in its own git repo 1:08:01 – From developing a feature locally to pushing it to production 1:19:38 – Resource limits are defined in the Kubernetes config files 1:27:40 – Backing up the database with daily Kubernetes cron jobs 1:31:02 – Developers are treated like grown ups when it comes to customer data 1:34:48 – Handling logging and alerting with Prometheus, Grafana and Kubernetes 1:40:51 – Handling DNS with DNSimple and SSL certificates with Let’s Encrypt 1:44:05 – Best tips? Start with a specific thing instead of making a generic thing 1:46:45 – Check out https://hitobito.com/ and their GitHub account, Matthias is also on GitHub, Twitter and he has a site at http://kronn.de/, also Puzzle is the company he works for Links <h6 id="-referen

In this episode of Running in Production, Kristian Dupont goes over building a SAAS app to manage access control to your existing SAAS subscriptions. It’s built with Koa, Node and React and is hosted on Heroku. It’s been running in production since 2018. Kristian talks about validating his idea before coding it, really leveraging code linting tools, the challenges of adding a bunch of different SAAS app integrations, using ElephantSQL to host his PostgreSQL database and more. Topics Include 3:20 – Starting with an empty folder as a sole developer looking to ship an MVP 4:29 – Motivation for using Node 9:19 – Reasons for choosing Koa and Knex with a custom library Kristian wrote 14:57 – Using eslint and the overall power of going all-in with linting 17:50 – The back-end is a RESTful API with React on the front-end 21:43 – Using interesting PostgreSQL features such as triggers 23:30 – Managing the front-end assets with Parcel and using TailwindCSS 30:27 – Fighting for your abstractions and creating React components 32:29 – Adding a bunch of SAAS app integrations was challenging at times 37:57 – Using Jest to run tests but not super happy about it 42:34 – 10k+ lines of code on the front-end and back-end 46:43 – Taxes in the US are fun (not) 49:00 – Mailchimp, Freshping, Datadog and Sentry are being used for various things 52:05 – Redis isn’t being used but the database is quite optimized 55:08 – The web app servers are hosted on Heroku and PostgreSQL is on ElephantSQL 58:45 – Running a Linux VM inside of an M1 Mac for local development 1:02:45 – The deployment process from development to production 1:07:49 – The DB is backed up every hour 1:09:43 – Freshping, Datadog and Slack provide good monitoring and alerting 1:12:09 – Best tips? Using Markdown to store test fixtures 1:15:24 – You can find Kristian on Twitter and his personal site lists all of his socials Links 📄 References https://nodejs.org/en/about/releases/ https://www.milesconsultinggroup.com/blog/2021/06/01/what-to-know-about-the-taxability-of-saas-in-18-key-states/ https://blog.serverfault.com/post/stack-exchanges-architecture-in-bullet-points/ <a href="https://kristiandupont.medium.com/and-naming-things-

In this episode of Running in Production, Andy Ide goes over building a SAAS app for horses with Django and Python. It’s hosted on a single Linode server and has been running in production since May 2021. Andy talks about learning Django while building his app, using Django Unicorn, creating a Django monolith that’s broken up with Django apps, outsourcing setting up a server, the importance of having a good testing team and more. Andy was kind enough to give listeners of this podcast 20% off for life if you use promo code RUNNING20 when signing up, that includes signing up for free since it carries over to a paid plan. Topics Include 6:11 – About 60ish users signed up about a month after launching 8:34 – The process of learning Django while building this app 11:29 – Using the Django admin and a bunch of other useful packages 14:34 – Django Unicorn is being used for interactive bits w/o having to write a lot of JS 18:34 – It’s a Django monolithic with a bunch of Django apps 22:27 – How Django Rest Framework is being used for the reporting part of the app 28:46 – The app is mostly Django rendered templates with sprinkles of JS and Unicorn 34:00 – Stripe is handling payments, their API and docs are great 41:02 – (1) $10 / month Linode server hosts everything with 1 CPU core and 2GB of memory 45:25 – Picking the most recent version of Ubuntu and keeping deploys simple 52:05 – Using Cloudflare for DNS and using their paid plan 56:54 – Running tests locally before deploying it 59:12 – Handling backups with Django DB Backup and Linode’s backup service 1:02:12 – Using Sentry for exception reporting, viewing logs and Uptime Robot 1:08:50 – Best tips? Install Sentry and have a good testing team 1:13:37 – Check out HorseRecords on Facebook and Andy’s Django Blog Links 📄 References https://en.wikipedia.org/wiki/Stud_farm https://djangochat.com/ https://djangoandy.com/2021/05/14/integrating-django-rest-framework-and-retool/ https://djangoandy.com/2021/05/02/what-i-learned-in-the-first-2-days-of-launching-my-saas-startup/ ⚙️ Tech Stack

In this episode of Running in Production, Karl Roos goes over building a B2B news and data platform with Rails, Node and Python. It’s hosted on AWS using Elastic Beanstalk and has been up and running since 2014. Karl talks about writing a Rails API back-end, scraping 400+ sites, executing 500k+ daily jobs, using a bunch of AWS resources, what it’s like dealing with a ~500 GB MySQL database, the importance on taking action and more. Topics Include 1:55 – What type of application we are talking about here? 4:41 – Switching from PHP to a combo of Rails and Node 8:02 – A few useful Ruby gems that were used to help build the app 9:41 – The Vue front-end is for a customer facing dashboard 14:48 – Where D3.js is being used to render charts and the data pipeline 17:43 – Scraping data from 400+ sites and dealing with edge cases 21:38 – The scraper runs on 10-16 EC2 instances through Elastic Beanstalk 26:04 – Each separate service lives in its own git repository and a bit of Serverless 31:06 – Sticking with the latest stable version of Rails and updating dependencies 33:10 – Sprinkles of Python to glue together a few AWS services and translating languages 36:11 – What it’s like using Elastic Beanstalk and executing 500k+ jobs a day 40:04 – Initial AWS credits helped sway the decision to try out AWS initially 43:47 – How CloudFormation and Terraform are being used 48:59 – All devs can push to production, code reviews, linting and the deploy process 54:21 – Dealing with database migrations and a ~400-500 GB data set 1:01:17 – Getting a local dump of the DB in development, seeding data and secrets 1:06:03 – Database backups are done with the built in RDS snapshots 1:10:19 – Best tips? Less thinking, more doing and learn from your experiments 1:11:35 – Karl is on GitHub and Twitter Links 📄 References https://aws.amazon.com/elasticbeanstalk/ https://github.com/localstack/localstack ⚙️ Tech Stack

In this episode of Running in Production, Matt Oliver goes over building a chat platform with Akka, Scala and React Native. It’s hosted on DigitalOcean with Kubernetes and has been running in production since 2013. Matt talks about rewriting his app a few times, handling tens of thousands of nightly TV shows, going all-in with Kubernetes, using Terraform, understanding it’ll take a while to learn things along the way and more. Topics Include 4:55 – Going from PHP to Scala to Node and back to Scala for the back-end 15:09 – Tens of thousands of channels are created every night 16:39 – What exactly is Akka and the actor model 19:55 – Websockets are being used quite heavily 23:04 – A few Scala libraries that were useful for building this platform 27:31 – There’s a mobile front-end using React Native for Android and iOS 31:30 – Storing and caching TV listings with Gracenote 34:53 – Roughly ~25k lines of Scala and ~11k on the front-end in 2 repos 39:41 – Motivation for using React Native instead of using native languages 45:55 – How to deal with large show TVs with a massive audience 50:28 – The app is basically 3 screens 54:34 – Handling user uploaded gifs and link submissions 1:00:09 – What it was like building out the React Native front-end 1:05:54 – Reasons for picking DigitalOcean and using Kubernetes 1:14:44 – Most of the infrastructure is managed by Terraform 1:16:57 – What it was like to go from not using Kubernetes to going all-in with it 1:24:21 – (3) 2 GB of memory / 2 CPU core servers are running the cluster 1:26:12 – What it’s like developing a new feature and deploying it to production 1:34:28 – The database is backed up on a schedule and before schema changes 1:39:21 – Prometheus, Grafana, Kamon and Sentry are used for metrics and monitoring 1:43:41 – Pingdom is used for an external site monitor 1:44:31 – It took about a month to get confident in using Kubernetes 1:49:29 – Best tips? It’s going to take a while, hang in there 1:53:06 – Writing comments for your future self while learning 1:54:34 – Matt is on Twitter and most socials as @halfmatthalfcat and @couchmatehq Links 📄 References https://en.wikipedia.org/wiki/Electronic_program_guide https://www.gracenote.com/on-entertainment-tv-listings/ https://en.wikipedia.org/wiki/Language_Integrat

In this episode of Running in Production, Maciej Delmanowski talks about building out a 40+ server / 200+ VM data center with Ansible. It’s hosted on premises, he’s been working on it all since 2007 and started using Ansible in 2013. Maciej talks about automating everything with Ansible, being a sysadmin for over a decade, how he picked Debian, splitting up a project into 120+ git repos and then back to 1, writing 60k+ lines of YAML, using Linux Containers, maintaining an open source project and more. Topics Include 2:35 – There’s about 40 physical servers and 200+ virtual machines / containers 4:50 – Transitioning to using Ansible over time 7:40 – A 10+ year friendship stemming from Debian and open source 8:23 – How Ansible’s role and inventory abstractions help manage a lot of VMs 10:35 – How DebOps as a name came into existence and its philosophy on being stable 14:24 – Motivation for choosing and staying with Debian 15:31 – Figuring out what new Ansible roles and playbooks to work on 19:14 – Going from a mono repo to 120+ repos and then back to a mono repo 24:48 – 67,000+ lines of YAML and 40,000+ lines of documentation 26:28 – Setting up a brand new server with Ansible and DebOps hands free in 15 minutes 28:29 – Automatically generating random passwords for services 31:22 – Not having to deal with HIPAA compliance laws and handling student emails 34:01 – Let’s Encrypt is being used on specific publicly accessible servers 34:26 – Breaking down the process of creating a new role from scratch 36:39 – Using Linux Containers in development and rolling things out to production 41:41 – Using dnsmasq in development for fully qualified domain names 43:07 – Dealing with backing everything up 46:02 – Being a maintainer of an open source project that’s extracted from work 49:19 – Best tips? Have 1 role for 1 service and find ways to connect them 51:26 – Check out DebOps, it’s on GitHub, #debops on Libera and Maciej is on Twitter Links 📄 References https://en.wikipedia.org/wiki/Orson_Scott_Card_bibliography https://github.com/nickjj/ansigenome https://github.com/nickjj/rolespec https://linuxcontainers.org/ https://en.wikipedia.org/wiki/Dnsmasq <h6 id="️-tech

In this episode of Running in Production, Adrian Marin goes over building a Ruby on Rails admin framework. It’s hosted on Heroku and has been available since late 2020. Adrian talks about building a Rails engine, using Stripe Checkout, building the admin out with Hotwire Turbo, using View Component, creating a very automated CI / CD pipeline to publish the gem and much more. Topics Include 6:38 – Motivation for using Ruby on Rails 8:57 – The gem does phone home on a 1 hour interval 11:33 – The gem is using Hotwire Turbo but the product site is not 13:10 – Handling billing with Stripe’s Checkout page and a few Rails gems 16:43 – Publishing a Rails engine with its own assets 21:10 – Using Hotwire Turbo to build the admin dashboard 26:30 – What it was like migrating the app from Vue to Turbo 31:49 – Using View Component to help improve Rails partial performance 39:21 – Postgres is being used as the database and Redis for caching 42:30 – What is Hotjar, how it’s being used and extracting features 46:36 – From Notion to Linear to using GitHub to help manage the project 49:44 – Using Heroku for hosting and what delayed job is being used for 54:55 – What it’s like deploying the site and the gem 1:01:19 – Performing database backups? Maybe, maybe not! 1:03:57 – Best tips? Start building and launch as soon as possible 1:06:06 – The gem has an open source version that’s very usable 1:07:03 – Check out https://avohq.io/ and Adrian is on Twitter Links 📄 References https://guides.rubyonrails.org/engines.html https://github.com/avo-hq/avo https://world.hey.com/dhh/building-basecamp-4-405a347f https://viewcomponent.org/ https://www.notion.so/ ⚙️ Tech Stack

In this episode of Running in Production, Przemek Matylla talks about building an image processing and optimization API with mostly C, Python and Node. It’s hosted on bare metal servers in a data center and has been running in production since 2019. Przemek talks about handling 20-50 million+ daily API calls, how they’re using C, image detection techniques, using Lua scripting with nginx, building their own servers in a data center, using boring technology and much more. Topics Include 3:17 – An average day has about 20 million API calls, busy days have 50m+ 4:11 – Breaking down where C, Node and other languages are being used 6:46 – What happens when you upload an image to their API 9:06 – Really figuring out the file type of something that’s been uploaded 11:54 – Dealing with edge cases as they come up but preparing a bit ahead of time 14:45 – Switching from Core ML on Apple hardware to Tensorflow on AMD hardware 19:43 – There’s no framework sitting on top of the Node API server 22:28 – The customer facing web dashboard is using Express, the marketing site is Jekyll 24:41 – They’re mostly B2B so feature requests end up being 1 on 1 calls 25:23 – Handling payments with Stripe and using a Node / Angular app for it 28:04 – Using Lua with nginx for rate limiting, also nginx is their load balancer 31:05 – You can’t go wrong with boring and predictable technology 31:47 – MongoDB, Redis and Elasticsearch are all running on 3 nodes each 32:18 – Having nearly instant access to a ton of data helps figure things out 34:52 – What it was like finding a freelance C developer 35:55 – Sending webhooks out is controlled by a separate Node Bull driven app 38:41 – Dealing with GDPR compliance and storing images on GlusterFS for 1 hour 40:50 – Going with bare metal servers in their own data center over the cloud 44:51 – The servers have 32-256GB of memory and a range of different CPUs 46:34 – Having spare parts and dealing with hardware failures 49:15 – About 50 servers run the latest Ubuntu LTS and are managed with Puppet 51:22 – The deployment process for a number of different services 54:19 – It takes ~30min to replace a drive and every service is tripled up 56:48 – The database servers are replicated and there’s alarms and alerts set up 58:56 – Rate limiting was put in place for limiting API calls to customers 1:01:12 – There’s custom payment rates depending on each customer’s requirements 1:03:06 – Best tips? Over provision like crazy and monitoring lets you sleep at night 1:03:31 – Do what works for you, don’t copy another company because it works for them 1:05:48 – Check out <a href="https://optidash.a

In this episode of Running in Production, Chris Samiullah goes over building a video course hosting platform with FastAPI, Strapi and Gatsby. It’s hosted on ECS along with a bunch of AWS resources. It’s been running in production since early 2021. Chris talks about combining a Gatsby static site with FastAPI, using AWS Fargate, streaming videos with Cloudflare, supporting both Stripe and Paddle, using a bit of Serverless Lambdas and tons more. Topics Include 3:19 – Hiring outside contractors to tackle setting up the infrastructure 5:05 – Motivation for choosing Strapi, FastAPI, React, Gatsby and more 9:37 – Strapi is being used as a headless CMS for the student watching front-end 10:33 – Going over how the student watching experience is set up 14:20 – Creating a course preview with Fargate using sub-domains and DNS entries 16:35 – A couple Gatsby plugins that helped build the site 18:57 – A couple of Python libraries to help build out the FastAPI back-end 20:54 – Using Cloudflare for streaming the videos and Shaka as the video player 26:39 – Stripe and Paddle are being used to handle payments (including VAT) 37:35 – About 20k lines of Python and 10k lines of front-end / React code 39:53 – Everything is Dockerized, there’s PostgreSQL and an ALB too 43:30 – Using Let’s Encrypt and letting course owners hook up custom domain names 45:05 – Reasons for choosing AWS and ECS (hosting is about $100 to $150 a month) 51:45 – Having a dev (staging) server and production in different regions 53:27 – What it’s like coding things up in development and pushing them to production 58:42 – The dev experience for using AWS Lambdas and creating internal tooling 1:05:59 – On-boarding new authors (doing something that doesn’t scale but it works now) 1:10:33 – A public service announcement about database migrations 1:13:13 – FastAPI has been working very nicely 1:16:41 – Backing up the database and user generated files 1:19:56 – Using Sentry for tracking errors and getting notified over Slack 1:25:35 – Best tips? Pick a hosting infrastructure that fits your project 1:27:44 – Picking between ECS and EKS was left to the contractor 1:30:03 – Check out https://coursemaker.org/, Chris is on Twitter and LinkedIn too Links 📄 References https://www.gatsbyjs.com/docs/conceptual/how-shadowing-works/ https://github.com/encode/sta

In this episode of Running in Production, Eduardo Messuti talks about building a status page and monitoring service with Phoenix and Elixir. It’s hosted on DigitalOcean and Heroku. It’s been up and running since 2017. Eduardo talks about upgrading their code base to use Phoenix contexts, using Oban, creating a notification abstraction, going all-in with EEx rendered templates and a bit of jQuery, using Docker Compose in production and more. Topics Include 1:59 – What it was like using Elixir back in 2017 4:52 – Elixir handles 500k requests a day and motivation to use Elixir / Phoenix 8:17 – The process to upgrade the code base to use Phoenix contexts 10:02 – Oban is being used to handle background jobs 14:39 – Sending out notifications and webhooks with custom code 16:39 – Using Mix format as part of a CI pipeline with GitHub Actions 19:00 – Breaking out the mail service into its own service 21:03 – About 28k lines of Elixir and 2k lines of JS 23:48 – Server rendered app with EEx vs going the SPA / API route vs Live View 28:45 – Supplying a status page embed widget for customers and custom domains 33:08 – Using nginx as a reverse proxy, ETS for caching along with Cloudflare 35:03 – Hugo is being used to serve the static marketing site 37:11 – Docker is being used in staging and production 38:17 – The mailer is hosted on Heroku, everything else is on DigitalOcean 39:58 – The hardware specs of the 4-5 DigitalOcean servers and what they do 42:27 – They’re running Ubuntu 20.04 LTS and upgraded the servers on the fly 44:13 – The servers are manually provisioned and run Docker Compose 50:52 – Deploying a feature from development to production 53:37 – User logos get uploaded and saved to disk with a Docker volume 54:52 – Using BackupPC for backing up the database, certificates and file uploads 56:40 – DigitalOcean’s monitoring and Pingdom are being used for various health checks 57:38 – Honeybadger and Papertrail for errors + logging, and the Elixir experience 1:00:38 – Best tips? Pick a technology you’re comfortable with and start building 1:01:31 – What it was like hiring Elixir contract workers 1:02:40 – Check out https://statuspal.io/ with promo code PRODUCTION to save 10% Links 📄 References https://turbo.hotwire.dev/ https://github.com/asdf-vm/asdf https://elixirjobs.net/ ⚙️ Tech Stack

In this episode of Running in Production, Jon Perl goes over building a service that automates creating browser tests for your web apps. It’s built with Node and React and is hosted on Netlify, Vercel, Azure Containers and DigitalOcean. It’s been running in production since 2019. Jon talks about creating an open source tool and turning it into a SAAS app, using VNC and websockets to stream test runner data to the browser, keeping things as a mono repo, moving from Azure Containers to Kubernetes and much more. Topics Include 3:11 – Going from a self hosted version to self hosted + hosted SAAS app 7:20 – Motivation for using Node and React 8:33 – Docker is being used for the test runners 11:24 – A mono repo with Next.js, GraphQL and Apollo 15:33 – Balancing your unit tests with end to end tests 18:25 – Using noVNC and websockets to link the test runner output to the browser 22:58 – Spinning up separate Docker containers for each test runner 24:52 – Next.js, Webpack, React and Typescript make up the front-end JS 28:37 – Efficiently sending VNC data over websockets 32:21 – Handling usage limits with the API server 35:55 – PostgreSQL is their primary database and they’re using Pipedream 38:01 – Stripe is being used to handle payments and SendGrid to send emails 42:00 – Everything is hosted on a combo of Netlify, Vercel, Azure and DigitalOcean 47:11 – Scaling up and down Azure containers with custom Node scripts 49:06 – Moving to Kubernetes in the future but only after it was deemed worth it 57:08 – The current deployment strategy pre-Kubernetes 1:00:03 – Jest tests are written and the code is linted with ESLint 1:04:06 – Database backups and the front-end is deployed to 2 hosts for redundancy 1:06:11 – Datadog handles monitoring and logging with Slack for notifications 1:09:30 – Keeping your assets fresh in a SPA or a long lived app in a browser tab 1:14:44 – Best tips? Focus on getting something out and learning from it 1:16:32 – Toggling feature flags with ENV variables and database fields 1:18:03 – Building the selector generator was a tricky problem to solve 1:23:06 – Check out https://www.qawolf.com/, the app is on GitHub and you can email Jon Links 📄 References https://code.visualstudio.com/docs/remote/containers https://github.com/yjs/yjs ⚙️ Tech Stack

In this episode of Running in Production, David Parker talks about building a service to subscribe to people with Ruby on Rails and Sapper. It’s hosted on Heroku ($30 / month) and has been up and running since November 2020. David talks about creating an API back-end with Rails, trying out Svelte and Sapper, the challenges of scraping names from websites, figuring out and fixing PostgreSQL performance issues, finding a good work / life balance and more. Topics Include 3:27 – Motivation for using Ruby on Rails and Sapper / Svelte 6:18 – Writing raw SQL queries before translating them to ActiveRecord 7:36 – A few Ruby libraries that helped build this project 9:37 – The challenges around scraping human names from various websites 13:17 – The API Rails back-end is a monolithic app with about 100 endpoints 15:44 – David pulls from 94 different sources to find new content 19:06 – The Engtagger gem helps figure out stop words 20:25 – Why did you go with an API back-end? 21:43 – What Svelte and Sapper are and how they help build UIs 25:34 – How the back-end and front-end communicate with each other 28:29 – ~11k LOC on the back-end and ~5 LOC on the front-end 30:49 – At the moment it’s a free service and there’s no ads too 33:23 – What Sidekiq is being used for in terms of background jobs 35:30 – Mailgun is being used to send all transactional emails 37:34 – How the service might be monetized in the future (it’s $30 / month for hosting now) 40:31 – Break down of Heroku Dynos and add-ons 43:17 – Figuring out how to add new people and also using Cloudflare 45:40 – The deployment process from development to production 49:16 – The database is being backed up through Heroku 52:00 – A features table is in the database to enable features for himself 54:37 – Logentries will send out a notification if something goes down 57:50 – What to look out for and how to fix certain performance issues with PostgreSQL 1:01:09 – An example of when to consider breaking up a wide table into multiple tables 1:05:22 – Best tips? Figure out your work / life balance and try the back-end / front-end split 1:11:21 – Find David on Twitter, https://www.programmingtil.com/ and his YouTube channel Links 📄 References https://www.mturk.com/ https://github.com/evanw/esbuild https://www.snowpack.dev/ <a href="h

In this episode of Running in Production, Ege Ersoz goes over building a project management service using Phoenix and Elixir. It runs on Gigalixir along with Heroku and has been up and running since early 2019. Ege talks about using Elixir for the last few years, upgrading to use Phoenix contexts, building a monolithic app with an API back-end / VueJS front-end, building his own Stripe billing module, what it’s been like using Gigalixir and more. Topics Include 1:24 – What it was like using Elixir back in early 2019 4:55 – Motivation for using Phoenix and Elixir 6:37 – Phoenix Channels and Elixir GenServers are being used a bit 10:14 – It’s a monolithic app with mostly an API back-end using VueJS on the front-end 13:34 – Going through the process of upgrading to use Phoenix contexts 18:19 – Interesting packages in Ege’s mix file that helped him build his app 23:00 – The front-end assets are managed with Webpack, Vuetify is being used too 26:47 – Separation of concerns was the main reason for making an API back-end 29:43 – Stripe is being used to accept payments with no 3rd party Stripe library 32:27 – PostgreSQL is the primary database and it’s all hosted on Gigalixir 36:38 – The Gigalixir bill is about $200 a month including a staging environment 39:49 – The deployment process from development to production 42:07 – Offloading PDF generation to a micro-service 43:38 – Gigalixir is handling automated database backups 45:44 – UptimeRobot monitors the app and reports back if it’s down 47:05 – Getting rid of subdomains for each user account / tenant and multi-tenancy 51:34 – Moving from Timber to Logflare for application logs 54:26 – Transactional emails are sent using SendGrid 55:57 – Best tips? Having a trustworthy co-founder has been incredibly important 59:12 – Check out https://pricetable.io/ and you can email Ege directly too Links 📄 References https://en.wikipedia.org/wiki/BEAM_(Erlang_virtual_machine) ⚙️ Tech Stack

In this episode of Running in Production, Jake Lazaroff talks about building a video rendering service with Express and Node. It runs on a few DigitalOcean servers and has been up and running in production since February 2019. Jake goes over rendering ~30k+ videos over 2+ years, executing background tasks, sharing lots of code between the back-end and front-end, using both Stripe and PayPal, not using an ORM with PostgreSQL, setting up Blue / Green deploys with Terraform, Packer and nginx, plus tons more. Topics Include 4:07 – Shipping an MVP in a bit under 6 months working on it nights and weekends 6:54 – About 125 videos are rendered per day (~30k since the 2+ years it’s been up) 9:09 – Motivation for using Express and Node and a few libs he’s using server side 12:02 – The app is mostly monolithic and Bull handles processing background tasks 13:10 – A background task spawns DigitalOcean servers for video rendering 16:14 – Both the Node back-end and JS front-end are in the same repo and share code 17:49 – The shared code lets you get real-time video previews straight in the browser 20:06 – Create React App was used as a base to build the front-end 22:16 – Hugo powers the marketing site and is hosted on Netlify 24:00 – There’s no custom theme, it’s all hand crafted CSS using SASS 25:30 – Both Stripe and PayPal are used to handle 1 time and subscription payments 31:03 – The admin back-end is a bunch of API endpoints protected by an admin user flag 32:33 – PostgreSQL is the primary database and no ORM is being used 34:56 – TypeScript is used through out the code base / nginx + Cloudflare is being used 36:31 – Docker isn’t being used in development / Postmark is used for sending emails out 38:43 – Blue / Green deploys with Terraform, Packer and nginx on DigitalOcean 41:56 – Ubuntu LTS is running on all of the servers 42:41 – 1 CPU / 1 GB for the API server & load balancer, 1 CPU / 3 GB for the renderers 43:04 – Paid videos get a CPU optimized render server with 4 CPUs / 8 GB of memory 44:41 – Videos use up a lot of storage and they’re stored on DigitalOcean Spaces 47:35 – Ansible was used in the past but now everything is baked into the Packer image 48:53 – The deploy process from development to production 54:31 – Database migrations are run out of band with a tool called Dbmate 55:28 – Backups are handled with DigitalOcean’s automated backups 58:16 – Alarms, alerts, monitoring and error reporting 1:01:57 – The paid plan has unlimited video uploads and overall hosting is ~$125 / month 1:07:05 – Best tips? You can’t go too wrong with choosing boring technology 1:09:27 – Check out <a href="https://songrende

In this episode of Running in Production, Utsav Shah goes over building Dropbox with Pylons, Python, Rust and Go. It’s mostly hosted on their own data centers across 1,000+ servers. It’s been available since 2008. They have 700+ million users and handle 100k+ requests per second. Utsav talks about working with hundreds of engineers on a multi-million line Python based monolithic app, handling payments without Stripe, storing exabytes of files, using Rust in the desktop client, having remote dev boxes, leveraging open source tools and tons more. Topics Include 1:32 – A couple of hundred of developers committing to a monolithic app 2:30 – Motivation for choosing Pylons and Python 4:04 – Their service architecture can be described as a solar system 6:03 – Building their own NoSQL database called Edgestore before MongoDB existed 8:41 – Pylons is a micro-framework for Python 10:29 – Developing a custom payment handling system before Stripe existed 13:43 – How Dropbox stores your files on disk and who can access those files 17:32 – Back in 2014 Dropbox moved away from S3 to their own infrastructure 18:50 – Dealing with exabytes of storage 20:44 – A “multi mono repo” set up with millions of lines of Python 21:57 – The desktop client is a combination of Python and Rust 23:34 – Updating the desktop client code base from Python 2 to Python 3 26:09 – Pytest is being used for tests and Black for code formatting 29:04 – A developer can spin up their own Dropbox stack on a hosted dev server 32:34 – The web UI is mostly server rendered templates and a mix of React 36:46 – There’s a lot going on with MySQL, plus memcached and nginx / Envoy 39:06 – Using open source libraries created by Facebook and YouTube 41:39 – The open source version of nginx is being used a bunch 43:33 – Most things are hosted on their own data centers running Ubuntu 45:32 – There’s dedicated teams that focus on the infrastructure 46:59 – How code gets from a developer’s dev environment to passing in CI 53:46 – Once everything passes, it’s rolled out internally and then incrementally to users 55:13 – Feature flags are being used with a home grown solution 56:06 – Dealing with secrets using something they’ve developed in house 58:01 – Sometimes you end up building out a lot more than your core product 59:03 – Your files are very safely and securely backed up 1:01:16 – Monitoring, alerting, logging and error handling 1:04:09 – Rate limiting is handled at the app level along with memcached 1:05:13 – Best tips? Monoliths aren’t bad if you invest in them 1:06:30 – Think deeply about what you’re developing and focus on the archite

In this episode of Running in Production, Nate Rush goes over building a JupyterLab extension with Python. It runs locally in a Jupyter Notebook but they also host JupyterHub instances on a Kubernetes cluster running in EKS. It’s been running in production since late 2020. Nate talks about building a cross platform Python installer, building and publishing a JupyterLab extension, leveraging Pandas, building the front-end with React / Typescript, hosting a version of their product on a Kubernetes cluster, the value in automating the hard stuff and more. Topics Include 1:15 – What is a Jupyter Notebook and how do JupyterLab extensions work? 3:35 – 2 developers are currently building it 6:50 – You can run it in an offline notebook or through their hosted service 8:48 – Handling local installations across Windows, macOS and Linux 15:58 – How users launch their Jupyter Notebook locally with this extension 17:16 – How do you create a JupyterLab extension? 21:10 – The back-end is built upon Pandas 23:31 – How do front-end assets make their way into the extension? 28:50 – Webpack is being used to bundle up the front-end assets 29:36 – Using GitHub Actions to handle the deploy process for the JupyterLab extension 35:20 – Front-end packages to help build a spreadsheet driven UI 37:37 – Discovering front-end performance issues and errors through logging 45:12 – Hosting JupyterHub instances on Kubernetes with EKS on AWS 47:32 – The hosted version is free, but it’s on its way to being phased out 51:57 – The workflow for deploying the Kubernetes set up and running 3-10 worker nodes 54:56 – Backups of the Kubernetes cluster are handled with Velero 56:13 – Route53 is hosting their DNS records 57:29 – Best tips? They really love Typescript and also automate as much as possible 1:00:58 – Don’t hesitate to ask for help and focus on what your users want 1:06:34 – Check out https://trymito.io Links 📄 References http://paulgraham.com/ds.html https://docs.anaconda.com/anaconda/navigator/ https://jupyter.org/hub The Phoenix Project book ⚙️ Tech Stack

In this episode of Running in Production, Tigran Nazaryan covers building a WordPress hosting platform with Laravel, Python and Node. It’s hosted on GCP for their clients’ sites and OVH for their core services. It’s been up and running as a hosting platform since late 2020. Tigran talks about working with a team of 34 engineers, using both MongoDB and MySQL, creating a bunch of services with a very well thought out architecture while keeping it simple enough for it to all run on a developer’s laptop. He covered a lot of ground. Topics Include 1:49 – Thousands of customers have their site hosted there 3:00 – A break down of their stack (there’s a lot going on) 4:38 – Motivation for choosing a mixture of programming languages and frameworks 7:10 – All of the services are independent and live in their own git repo 9:23 – The image optimizer service is written with Laravel 12:34 – MongoDB is quite heavily used by a number of services 15:40 – How to optimize CSS and JS bundles with many WordPress plugins 21:13 – The hosting service is written in Python with MySQL for a DB (hosted on GCP) 23:52 – It’s mostly Linux containers being used but Docker is used in some spots 25:11 – End customers don’t need to worry about things like nginx and SSL certs 30:30 – Other tech components that’s in their stack 32:00 – Payments are handled monthly or annually with Stripe and PayPal 33:12 – Client sites are hosted on Google Cloud and their services are on OVH 37:33 – Some of the servers are pretty beefy with 32GB of memory and 8+ CPU cores 39:56 – Developers can run everything on their own laptops 44:27 – The developers who build a service are responsible for deploying it 45:26 – The process to set up a server is documented and automated when possible 48:46 – Customers can access their hosted sites very quickly after signing up 51:13 – The deployment process from development to production 56:55 – Databases are backed up and core services are load balanced 59:04 – Keeping track of errors, monitoring and alerting 1:03:45 – Cloudflare is sitting in front of the core services 1:06:42 – Best tips? Really focus on getting a good architecture designed early on 1:10:36 – Check out https://10web.io/, Tigran’s LinkedIn and their Engineer blog Links 📄 References https://developers.google.com/speed/pagespeed/insights/ ⚙️ Tech Stack

In this episode of Running in Production, Ian Eyberg goes over creating a Unikernel with C as well as host a few sites supporting his tool with Go. It’s hosted on Google Cloud and their own data center. Nanos has been available since 2020. Ian talks about what a Unikernel is, their open source tools and how they manage their own services. This episode has a healthy mix between background knowledge on Unikernels and how they (as a company) set up their infrastructure. It’s worth pointing out you can run your existing applications in a Unikernel without having to change how it’s written and they support running them on most major hosting providers (AWS, GCP, Azure, DigitalOcean, your own hardware, etc.). Topics Include 1:44 – What is a Unikernel? How is it different than a traditional VM or container? 7:58 – There’s a free and open source tool and an optional SAAS offering 10:07 – How it’s possible to build a new deployable golden image in 2 minutes 12:12 – Motivation to use Go for building the surrounding sites and services 16:51 – Certain organizations are pushing decent traffic through their Unikernel driven apps 19:02 – How you can run a multi-service app with Nanos (web + worker + db + cache, etc.) 22:59 – ops.city and nanos.org are a single Go binary / 1 Unikernel driven app 25:37 – The nanovms.com site is a bit more involved and has Stripe integration 28:08 – I never heard of the term Unikernel until today 30:20 – nginx isn’t sitting in front of the Go app and how Unikernels can be so fast 40:29 – With a Unikernel approach you can easily move between hosting providers 44:23 – SSL certs are handled directly by the Go app for their sites 49:56 – nanos.org and ops.city use GCP and nanovms.com is on their own hardware 54:26 – Why they went with their own data center for hosting and their server specs / costs 1:02:02 – Terraform, Ansible and similar tools aren’t being used to set up anything 1:04:21 – What the deployment process looks like for their services 1:10:40 – You can run all of this on a Raspberry Pi 4 1:13:15 – What does the development process look like with a Unikernel driven app? 1:16:21 – Dealing with secrets in production 1:17:55 – Databases are backed up regularly and how logs are handled 1:23:52 – Getting notified of errors and up-time reports from updown.io 1:25:52 – Mailgun is used for sending out transactional emails 1:26:45 – Best tips? Keep it simple (seriously) 1:30:05 – Thoughts on the Plan9 operating system 1:34:06 – You don’t need to change how you write your apps to run them in a Unikernel 1:40:07 – The code for Nanos is <a href="https://github.com/nanovms/na

In this episode of Running in Production, Cole Simmons goes over building an AI driven service for commercial real estate professionals with Flask and Python. It’s hosted on Google Cloud using GKE (Kubernetes) and has been up and running since 2017. Cole talks about rewriting the initial prototype, the dangers of not using an ORM, splitting out the front-end from the back-end, using Google Functions, the importance in making sure you’re building what your users want and so much more. Topics Include 1:05 – The inefficiencies when dealing with commercial real estate 6:51 – From an MVP in 2016 to the main app in 2017 to the first paying customer in 2019 16:48 – Motivation for choosing Flask and Python and backing out from using Go 21:59 – Losing (recoverable) data by accidentally running a bad raw SQL query 24:14 – Using openpyxl to handle Excel files and Weasy print to export PDFs 25:31 – The API is a separate service in its own repo and the AI stuff runs on App Engine 30:47 – The API is built with Flask straight up, no external packages 32:19 – The front-end is built using React and Redux 34:24 – Creating an amazing data tables solution is still not a generic solved problem 39:43 – Payments are handled in a custom way (ACH, check, etc.) 44:06 – Going with Google Cloud, Google Functions and GKE (Kubernetes Engine) 49:18 – A DNS related GCP bug caused ~80% of their requests to 404 for 2-3 days 54:00 – Developing local functions and what it’s like to deploy everything 1:03:04 – Making sure the back-end and front-end deployments are sync’d up 1:05:15 – Dealing with secret management using Google’s key management service 1:07:48 – Currently using Mailgun but looking to switch over to Postmark 1:13:43 – Database and file backups are happening regularly 1:19:31 – GCP’s logging and error reporting has been good so far, using LogRocket too 1:21:25 – The TL;DR on what Webflow is and how they use it for their landing page 1:27:10 – Best tips? Listen to advice but make and follow your own path 1:29:41 – Customer discovery is important as well as asking the right questions 1:32:15 – It’s probably going to take you longer to build your app than you initially think 1:38:30 – Cole’s on Twitter and you can email him if you’re an engineer that is interested in this problem space Links 📄 References https://phoenixframework.org/ https://blog.samaltman.com/advice-for-ambitious-19-year-olds <

In this episode of Running in Production, Jason Charnes goes over building a video course hosting platform. It’s built with Ruby on Rails and is hosted on Heroku. It’s been running in production since 2014. Jason talks about using StimulusReflex / Action Cable, upgrading to Sidekiq Enterprise, writing tons of tests, setting up auto scaling on Heroku, tips to run zero down database migrations at scale, ways to reduce 3rd party API calls and more. Topics Include 2:39 – Motivation for using Ruby on Rails (and yes Rails does scale!) 3:50 – Using interactors and Action Cable with StimulusReflex 6:06 – Things to think about for using Hotwire Turbo Streams vs other solutions 10:56 – Action Cable is embedded into the main web server 12:25 – The workflow gem and how payments are handled with Stripe and PayPal 16:15 – Recently upgrading to Sidekiq Enterprise and also using flipper for feature flags 19:40 – Using Knapsack Pro to help speed up tests in CI 21:18 – Code to test radio is about 1 to 1.5 and there’s about 38k lines of non-test code 24:03 – Examples of when they’re using Sidekiq’s batching feature 26:38 – Mostly using the asset pipeline but also using Webpacker and using StimulusJS 30:07 – Most developers at Podia touch most areas of the code base 31:37 – The run down on how the site editor works and using Trix plus Active Storage 34:41 – It’s 2021 and uploads still aren’t a fully solved problem 36:57 – Handling bulk video uploads from Podia to Wistia 38:38 – PostgreSQL is the main database, Docker isn’t being used and on-boarding new devs 40:53 – How they’re using Caddy and Let’s Encrypt to link custom domains to customers 42:48 – Using Postmark for transactional emails and SparkPost for email campaigns 44:18 – Full text search with PostgreSQL and building the Podia admin dashboard 47:38 – It’s hosted on Heroku and they use the Rails Autoscale add-on 49:43 – Surviving Black Friday without any problems 51:25 – The end to end process and time to deploy code from development to production 53:01 – Gems and strategies to help run database migrations at scale 56:44 – Using Diffend to audit gems and switching from Heroku’s scheduler to Sidekiq 1:01:42 – The DB is backed up on Heroku’s schedule and everything else is backed up too 1:03:21 – Avoiding tons of third party API calls by storing certain things locally 1:05:16 – Their payment code builds upon the official Ruby SDKs from Stripe and PayPal 1:06:11 – Using Uptime Robot, Cronitor, OpsGenie and Slack for monitoring and alerts 1:09:13 – Best tips? Build a friendly team of co-workers and automate as much as you can 1:11:03 – You can find Jason on

In this episode of Running in Production, Sean Callaway talks about building an app to help manage REI’s server infrastructure. It was built with Django and it’s mostly hosted on premises in their own data center. Sean covers keeping the app mostly server rendered and goes into detail about using Ansible, Kubernetes and Rancher to help manage things. Topics Include 4:16 – Motivation for using Django and Python 6:27 – The Django admin is not being used 8:20 – Using Ansible to automate spinning up VMware based VMs 10:17 – It’s a monolithic application and it does use Django apps 12:14 – A high level overview of what the app does to help visualize it 15:14 – It’s using Django templates and they tuned a few database queries 18:19 – PostgreSQL is used for some data and maybe Celery will be used soon 21:07 – nginx is serving static files and Kubernetes’ ingress controller handles SSL certs 23:43 – Docker Compose is being used in dev and what Kubernetes has been like so far 28:42 – Pretty much all of their servers are hosted on premises 31:17 – Using Grafana, Splunk and Prometheus for logging and monitoring 34:26 – For the Linux servers, they’re running CentOS with Rancher 38:50 – The entire flow for pushing code from development to production 45:37 – Database backups and planning for disasters 51:50 – Best tips? Containerize your app quickly 55:05 – You can find Sean on GitHub, Twitter and he has a blog at https://callaway.dev Links 📄 References https://en.wikipedia.org/wiki/Active_Directory ⚙️ Tech Stack

In this episode of Running in Production, Nick Hnatiw goes over building a password manager with Django and Python. It’s hosted on Heroku and DigitalOcean. Nick talks about using Django Rest Framework because there’s a web app along with native mobile apps, the value in charging for a product to avoid having your data sold, using HashiCorp Vault to manage user’s private keys and more. Topics Include 5:05 – Motivation for using Django and Python 7:15 – The Django admin isn’t perfect but it helps save a lot of time 9:16 – Using Django Guardian and Django Rest Framework 11:05 – It’s a monolithic API back-end with 1 extra small service to handle private keys 13:12 – The back-end is about 1,000 or 2,000 lines of code 14:03 – It’ll be a small amount per month to use and all of your data will be private 16:42 – Using React on the front-end and why the back-end is an API 18:26 – The back-end will likely be open source in the future 20:35 – Developing the API back-end alongside the React + mobile app front-ends 22:04 – The site is hosted on Heroku’s free tier (including PostgreSQL too) 26:43 – The process to develop a feature and then have it be live on the site 29:55 – The database and encrypted keys are being backed up 30:45 – The Vault service is running on a $5 / month DigitalOcean server 34:01 – The server is running Ubuntu and we share a fun trip down memory lane with Gentoo 37:04 – Best tips? Discovering Vault and being able to use it to secure user data 42:54 – Check out http://passwordspace.com/ Links 📄 References https://en.wikipedia.org/wiki/Don%27t_repeat_yourself https://www.vaultproject.io/ ⚙️ Tech Stack

In this episode of Running in Production, Mubashar Iqbal talks about building a site to find podcast episodes with Laravel and PHP. It’s hosted on DigitalOcean using Forge and has been up and running since mid 2019. Mubs talks about using a bit of Vue as needed, not upgrading to the latest Laravel immediately, interesting problems around parsing RSS feeds, how Forge lets him easily host his site, understanding the nuances of your app’s domain and more. Topics Include 4:50 – Motivation for using Laravel and PHP over a bunch of other choices 8:20 – It’s using Laravel 6 with VueJS on the front-end for some parts 11:30 – The app will get updated to the TALL stack but not quite yet 13:04 – It’s a monolithic app in 1 git repo 16:07 – It’s mostly using Blade templates with Vue thrown in where needed 19:12 – MySQL is powering the database which is managed by RDS 21:09 – Reading in RSS feeds in background jobs and processing Open Graph images 26:07 – You need to get approved by Mubs before you can submit episodes 28:35 – Redis is powering the background jobs 33:11 – nginx / Let’s Encrypt and it’s hosted on DigitalOcean using Forge 39:06 – The deployment process from development to production 45:49 – There’s about 3k lines of PHP code and a few hundred lines of JS 47:16 – Secrets are managed through an env file that you put on your server 48:41 – Database backups happen every day and everything else is backed up too 51:38 – Sentry is in charge of exception handling and Uptime Robot for site monitoring 55:26 – Best tips? Understand the nuances of the app you’re building 56:30 – Web tech is moving fast, will HTML over the wire be main stream soon? 59:06 – Mubs has his own site at https://mubs.me and he’s on Twitter too Links 📄 References https://transistor.fm/ https://twitter.com/taylorotwell (creator of Laravel) https://twitter.com/calebporzio/ (creator of Livewire) https://twitter.com/jeffrey_way (creator of Mix / Laracasts) https://forge.laravel.com/ https://podcastindex.org/ ⚙️ Tech Stack

In this episode of Running in Production, Jonathan Adly goes over building a gift giving service with Django and Python. It’s hosted on Heroku’s hobby tier and has been up and running since November 2020. Jonathan talks about shipping his first app, using Celery for background jobs, handling non-standard payment patterns with Stripe, not getting caught up with what everyone is saying to use for tech choices and trying to develop features based on using the app as a customer. Topics Include 3:55 – With no prior programming experience an MVP was shipped in 6 part time months 6:25 – Motivation for choosing Django and Python 9:02 – Gift selection involves using a 3rd party API to provide this feature 10:02 – A lot of work is done with Celery, such as scheduling future gifts 12:09 – It’s mostly using Django templates with sprinkles of vanilla JavaScript 13:44 – Dealing with non-standard payment use cases with Stripe was a challenge 17:34 – It really is vanilla JavaScript with Halfmoon for the CSS 18:32 – A few Python and Django libraries that helped move things along 20:28 – PostgreSQL is the main database and Docker is being used too 23:52 – Switching from SendGrid to AWS’ SES for sending emails out 25:10 – Why Heroku was picked to host the application and using it with Docker 28:31 – Using Sentry for error reporting and Uptime Robot for health checks 30:57 – The deployment process from development to production 32:18 – How Stripe is being tested in development and database backups 36:14 – Best tips? Keep things simple and develop your app through your customer’s eyes 39:37 – You can find Jonathan on Twitter and check out https://joyful.gifts Links 📄 References https://openai.com/ https://www.gethalfmoon.com/docs/introduction/ https://diveintodocker.com ⚙️ Tech Stack

This episode is no longer available at the moment because it’s being reviewed by 1 or more companies that own this service. It may or may not come back. It’s up to them. If you’re wondering if something very secret was accidentally leaked, it wasn’t. We had a conversation similar to every other episode on this site. However, as a podcast host (this is Nick writing all of this) I very much respect the decision of a company requesting that their episode gets removed until they have a chance to review it. Support the Show This episode does not have a sponsor and this podcast is a labor of love. If you want to support the show, the best way to do it is to purchase one of my courses or suggest one to a friend. Dive into Docker is a video course that takes you from not knowing what Docker is to being able to confidently use Docker and Docker Compose for your own apps. Long gone are the days of "but it works on my machine!". A bunch of follow along labs are included. Build a SAAS App with Flask is a video course where we build a real world SAAS app that accepts payments, has a custom admin, includes high test coverage and goes over how to implement and apply 50+ common web app features. There's over 20+ hours of video.

Anders Hovmöller goes over building a reconciliation service with Django. It's hosted on bare metal servers in multiple data centers.

In this episode of Running in Production, Anthony Eden covers building a domain management service with Rails, Go and Erlang. It’s hosted on a combination of 70+ bare metal servers, AWS and Heroku. It’s been been up and running since 2010. Anthony talks about handling millions of API queries per day, ~5 billion monthly DNS queries, spikes of up to 10,000 requests per second, sticking with a Rails monolith for the web dashboard, scaling PostgreSQL, building multiple data centers, feature flags and tons more. Topics Include 3:58 – Millions of API queries per day and 2-5 billion DNS queries per month 6:40 – How Rails, Go and Erlang are being used along with why they were chosen 12:54 – How DNS lookups happen and the importance of DDoS protection 16:40 – The Erlang service has ~10k LOC and was written before Elixir existed 21:23 – Go is responsible for a lot of glue services 24:50 – A monolithic Rails app (server rendered templates) powers the web dashboard 28:08 – Sidekiq (Enterprise), Redis, PostgreSQL and all services run on Ubuntu LTS 29:41 – For cloud hosted services they end up on AWS or Heroku depending on what it is 31:45 – There’s 2 PostgreSQL instances and only the Rails app writes to it 34:55 – nginx is sitting in front of the Rails app 35:43 – Topping out at 5-10k requests per second through the Erlang service 42:44 – You can spin things up locally with or without Docker 47:15 – Datadog is used to help view metrics and logs to detect potential issues 50:10 – What exactly is Anycast? 52:07 – Picking out hardware for their data centers (roughly 70 physical servers) 59:29 – Chef is being used to configure all of the servers 1:02:23 – What the process is like to develop something and then deploy it to production 1:07:59 – Toggling feature flags, managing database migrations at scale and API versioning 1:16:15 – How developers add new features through pull requests and then deploy code 1:21:53 – Stripe handles all of the payments for each subscription tier 1:24:26 – Handling database backups with snapshots and streaming the data offsite 1:26:45 – Bugsnag is used for error handling and logs get written to Datadog as well 1:29:16 – Everyone’s been working remotely from day 1 and there is no centralized office 1:31:39 – Best tips? Have good processes in place as you grow in size 1:34:07 – Check out https://dnsimple.com/ and you can find Anthony on Twitter Links 📄 References https://en.wikipedia.org/wiki/Anycast <a href="https://www

In this episode of Running in Production, Justin Duke goes over building an email newsletter service. It’s mostly hosted on Heroku and has has been up and running since late 2016. Justin talks about how he handles sending out 100k+ emails, having a mix of Django templates and DRF + Vue, using rq to schedule emails, scaling with Heroku, balancing out what events to keep track of, how to figure out which features to develop and so much more. Topics Include 3:50 – It’s a weeknights / weekend project and dealing with 100k+ email spikes 11:42 – Motivation for using Django, choosing “boring tech” and email validation 20:08 – It’s a monolithic app in a mono repo but broken up into ~8 Django apps 27:48 – It used to be mostly a DRF API back-end with Vue on the front-end 30:04 – Storybook is a tool for developing UI components 32:06 – Useful Python / Django libraries that were helpful for this project 35:39 – Spending an innovation token on using websockets for link checking 41:10 – rq is a powerhouse for scheduling and sending emails in the background 47:35 – Dealing with issues and errors when sending emails out at scale 50:43 – Sentry and Datadog make for a great duo of services 54:19 – Using Stripe and avoiding dark patterns around subscription pausing 59:35 – A few strategies to prevent bad actors from sending email spam 1:03:09 – Reasons for going with Heroku to host most things (roughly 12 dynos) 1:12:13 – A few Heroku add-ons and plugins that are being used 1:16:37 – The deployment process from development to production 1:21:26 – Dealing with database and front-end changes with zero down time deploys 1:28:10 – Performing database backups and backing up other user uploads 1:34:45 – Getting notified of errors if something goes wrong 1:37:53 – Best tips? Framing out when and where to record things (ie. events, etc.) 1:43:19 – Using Notion to track bugs, roadmaps and collect user feature requests 1:47:23 – Justin has a personal site and check out Buttondown Links 📄 References https://en.wikipedia.org/wiki/Click-through_rate (CTR) https://mcfunley.com/choose-boring-technology https://storybook.js.org/ https://tailwindcss.com/ https://github.com/heroku/heroku-pg-extras https://github.com

In this episode of Running in Production, Denis Stepanenko talks about building an internal app to help manage finances around selling season tickets. It’s hosted on DigitalOcean and has been up and running since mid 2020. Denis covers what the process was like to rebuild an old PHP app with Django, coding everything with 6 months of experience, using DRF and React, keeping deployments simple and not being afraid to read other people’s source code. Topics Include 2:59 – Converting an older PHP app into Django and motivation for picking Django 6:13 – Switching from Django templates to a DRF API back-end with a React front-end 10:58 – Webpack is being used to bundle all of the assets 12:34 – Hosted on DigitalOcean, gunicorn and nginx is being used, plus cron jobs too 15:09 – Switching from PostgreSQL back to SQLite and being productive 17:47 – Useful Python and Django libraries that helped get things going 19:56 – nginx is serving SSL certs created by Let’s Encrypt and picking DigitalOcean 26:03 – The server was set up by hand and he also uses WSL on Windows for local testing 27:30 – What it’s like to deploy a new feature from development to production 32:37 – Database backups are done manually and also by DigitalOcean’s backup feature 36:34 – Database migrations are kind of a pain 39:09 – Best tips? Don’t be afraid to read other people’s source code 42:08 – You can find Denis on GitHub at https://github.com/Denis-Step/ Links 📄 References https://material-ui.com/ https://nickjanetakis.com/blog/a-recycled-ip-address-caused-me-to-pirate-390000-books-by-accident ⚙️ Tech Stack

In this episode of Running in Production, Ian Davidson goes over building a data sharing service using Phoenix and Elixir. It’s using Live View too. At its peak the site received a spike of 10k+ requests in a day and it’s hosted on a $20 / month DigitalOcean server. Ian talks about quickly building the app, reacting quickly to add user requested features, using DigitalOcean for the first time, some pitfalls of using Live View / websockets when it comes to configuring nginx and more. Topics Include 1:35 – Motivation for using Phoenix, Elixir and Live View 4:14 – How the site works, it saves data to a combination of PostgreSQL and the file system 5:48 – The experience of developing the app with Phoenix Live View 8:42 – Adding features quickly based on user feedback 11:13 – It’s an open source monolithic code base sitting in 1 git repo on GitHub 14:43 – nginx is sitting in front of the cowboy server and terminals SSL 16:57 – It’s hosted on DigitalOcean for $20 / month (2 vCPUs and 4 GB of RAM) 18:56 – The OS is Ubuntu 20.04 LTS and DigitalOcean’s automated backups are on 20:17 – The server was set up and configured manually without using Elixir releases too 22:36 – The deployment process from development to production 26:41 – The BEAM takes about 5 seconds to restart which is a bit of down time 28:50 – DigitalOcean’s Cloud Firewall is being used along with their monitoring tools 32:00 – nginx is taking care of basic rate limiting by IP address 34:20 – Best tips? Phoenix is a great choice for side projects but beware of websockets 36:11 – Not having end to end tests before launching was kind of a mistake 37:34 – Check out the site at https://textdb.dev/ Links 📄 References https://news.ycombinator.com/item?id=23948234 https://www.youtube.com/watch?v=MZvmYaFkNJI (Real-time Twitter clone Live View demo) https://applitools.com/wp-content/uploads/2019/05/pasted-image-0.png https://en.wikipedia.org/wiki/BEAM_(Erlang_virtual_machine) https://github.com/artilleryio/artillery ⚙️ Tech Stack

In this episode of Running in Production, Dash Winterson talks about building a new type of chat app for iOS using Django and Python on the back-end. It’s hosted on AWS using Elastic Beanstalk and has been up and running since mid 2020. Dash covers creating async compatible code in Django, developing a monolithic back-end that a few developers develop against, investing in the AWS ecosystem, end to end message encryption and the Zen of Python. Topics Include 3:05 – Rewriting an existing Django app into a new Django app with 90%+ test coverage 8:49 – Async features in Django and not using Celery 12:07 – A breakdown of gRPG and Protobuff 15:13 – The platform has about 300 daily active users, dead locks and monitoring 17:47 – A monolithic Dockerized app with 4 developers working on the project 20:05 – There’s only an API back-end in Django, the front-end is a native iOS app 22:51 – The SimpleJWT and Redis / ARedis libraries were very helpful in Python 27:04 – The phonenumbers library showed to be very useful too 30:02 – Using Docker Compose in development with a multi-stage Dockerfile 32:22 – Using Elastic Beanstalk in production and a number of other tech choices 38:06 – There’s no configuration management tools being used, but that’s next 42:09 – Most AWS instance types they are using are medium size 44:00 – The process to deploy code from development to production 49:20 – Dealing with backing up user generated data and encrypting messages 54:06 – Getting alerted of errors through Slack and email 58:01 – Best tips? When in doubt stick to the Zen of Python 1:00:53 – You can find Dash on GitHub and Twitter Links 📄 References https://en.wikipedia.org/wiki/Protocol_Buffers https://www.elastic.co/observability https://python-poetry.org/ https://www.python.org/dev/peps/pep-0020/ (The Zen of Python) ⚙️ Tech Stack

In this episode of Running in Production, Nidal Alhariri goes over building a sales and CRM tool with Flask and Python. It’s hosted on a few cloud providers such as AWS and Linode. It was created in mid-2019. Nidal talks about using MongoDB, splitting tenants out with their own individual servers, mixing in Vue where needed, using Ansible and he also drops great advice around moving forward and shipping code without chasing perfection. Topics Include 3:35 – Motivation for using Flask and Python 5:38 – Vue, MongoDB, Celery and other libraries / tools being used 8:28 – Dealing with a schemaless database 10:04 – It’s a monolithic app that uses Flask blueprints and dealing with permissions 14:58 – Each tenant / customer of the app gets their own virtual private server 17:24 – Vue is mixed in where needed, the app is not a dedicated API only back-end 22:29 – Parcel is being used to bundle assets and SASS is being used too 24:26 – Docker isn’t used yet but nginx + Let’s Encrypt and Ansible is being used 27:23 – Multiple hosting providers are used (AWS, Linode) with varied server specs 30:03 – uwsgi is being used mainly due to certain app features using websockets 32:39 – Ubuntu LTS is happily being used along with configuring the servers with Ansible 36:05 – The deploy process from development to production 41:16 – Environment variables for secrets and rate limiting with nginx 45:18 – Handling database backups for MongoDB 47:28 – Getting notified of system health issues with alarms and alerts 51:00 – Best tips? Keep moving forward because failure is always better than regret 53:50 – Nidal is on Twitter, GitHub and also has an agency at https://www.level09.com/ Links 📄 References https://mitogen.networkgenomics.com/ansible_detailed.html ⚙️ Tech Stack

In this episode of Running in Production, Jared Chung talks about building a community platform with Django and Python. It’s hosted on AWS and has been up and running in production since 2011. The beta version was intially built during a 4 day hackaton. Jared talks about taking advantage of Django’s batteries included, heavily using Celery / Redis and hosting the main web app on a single EC2 instance. Their AWS bill is roughly $2,500 a month. Topics Include 3:59 – Building a beta version of the site during a 4 day hackathon 5:38 – Motivation for picking Django and Python 10:11 – Leaning heavily on the built in Django admin and other libraries 12:24 – Mostly server rendered Django templates with a touch of React 16:03 – It’s mostly a monolithic app but it has a few services pulled out 20:54 – PostgreSQL is the main database and Celery / Redis is heavily used 24:58 – gunicorn + Caddy, Route53 for DNS and a load balancer, Docker end to end 27:45 – Tens of thousands of lines of code, ~3,500 files and good test coverage 31:11 – Webpack is being used to handle the front-end assets (SASS and ES6 JS) 34:36 – Static files are cached on CloudFront and in general development is pretty hard 39:07 – RDS is being used and the production web app has 4 vCPUs & 16GB of RAM on EC2 42:46 – New Relic, PagerDuty and Sentry are being used for monitoring and alerts 46:11 – There’s no configuration management tools being used to set up the EC2 server 48:13 – Going from developing a feature to deploying it into production 53:56 – RDS is configured for backups and there’s a lot of backing up in general 55:39 – Storing ~200k questions each having 10-20 DB rows + 120k registered users 56:32 – Their AWS bill fluctuates between $2,000 and $3,000 USD per month 58:29 – Best tips? There’s a lot to learn out there but focus on keeping your users happy 1:00:28 – Want to volunteer? Contact them at [email protected] 1:01:04 – CareerVillage.org is on Twitter, facebook and Instagram - so is Jared Links 📄 References https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow https://en.wikipedia.org/wiki/Chaos_engineering#Chaos_Monkey ⚙️ Tech Stac

In this episode of Running in Production, Sascha Wolf walks us through an internal tool he built to help deal with microservices. It’s written in Phoenix and Elixir and is hosted within a Kubernetes cluster on AWS using EKS. It’s been running in production since mid 2019. Sascha talks about rewriting the prototype from Java to Elixir, taking advantage of Phoenix Channels, creating event driven systems, moving from Heroku to AWS and really making the most of OTP features such as using stateful processes. Topics Include 4:26 – Motivation to rewrite the prototype from Java to Phoenix and Elixir 6:45 – Live View likely would have been used if it were around when they started 9:17 – 30+ microservices in 1 git repo and deployment isn’t really a pain 12:46 – This Stacker service is pretty big with about 90 Elixir modules 18:06 – On the front-end it’s just a little bit of vanilla Javascript 21:44 – Useful libraries in their mix.exs file 23:59 – Most of the other services are written in Ruby (for now) 25:25 – Running on Heroku, but moving to AWS with Docker and Kubernetes with EKS 28:32 – Terraform is used to manage their infrastructure as code with Circle CI 32:09 – Load balancers, AWS API Gateway and using Lambda for backups 34:09 – Sentry for error reporting and an ELK stack for viewing logs 34:55 – Deploying code from development to running live on the Kubernetes cluster 40:36 – Backups are handled by Amazon’s managed services 42:59 – Best tips? Don’t be afraid to jump into OTP and leverage process state 45:31 – Sascha is on Twitter, he has a personal site and Better Doc is hiring Links 📄 References https://github.com/phoenixframework/phoenix_live_view https://aws.amazon.com/elasticsearch-service/the-elk-stack ⚙️ Tech Stack

In this episode of Running in Production, Jerod Santo and Gerhard Lazu go over their news and podcast platform which is using Phoenix and Elixir. It was rewritten in Phoenix back in 2015 and it runs on Linode’s Kubernetes Engine across a 3 node cluster. Jerod and Gerhard talk about building a context-less Phoenix app, using Turbolinks, caching, transitioning to using Kubernetes, automating as much as possible while focusing on patterns that work for small teams and so much more. Topics Include 6:06 – Motivation for choosing Phoenix and Elixir 10:34 – The MVP didn’t require too many changes to make it deployable 13:43 – It’s an open source monolithic app in 1 git repo and it doesn’t use contexts 17:35 – Gerhard was happy to see it was a monolithic app when it came to deploying it 21:09 – A few useful libraries that help power the podcast CMS 24:09 – Using embedded schemas for handling email subscription settings 25:19 – Using Docker but Jerod hasn’t gone all-in with it in development 33:47 – It’s mostly server side templates with sprinkles of JavaScript and Turbolinks 37:45 – Choosing not to use Live View because Turbolinks is working at the moment 41:51 – nginx, Linode’s load balancer, Fastly as a CDN and handling metrics 47:42 – Motivation for choosing Linode and using their Kubernetes Engine 54:42 – Serving about 200 requests per minute and how caching is done with ETS 59:54 – Creating very thin wrapper modules for certain libraries when it makes sense 1:02:36 – The deployment process from development to production 1:08:29 – What was it like to transition to using Kubernetes? 1:11:45 – Polling for new versions of the app instead of giving CI the keys to the kingdom 1:13:38 – Dealing with inserting secrets into the app through Kubernetes 1:17:48 – The embodiment of dev and ops working very well together 1:19:01 – Database backups, making small changes and health checks 1:27:43 – Best tips (Jerod)? Team up with people who fill in your knowledge gaps 1:30:16 – Best tips (Gerhard)? Be honest about what you like and enjoy doing 1:31:57 – Gerhard has a personal site, Jerod is on Twitter & check out https://changelog.com/ Links 📄 References https://twitter.com/adamstac https://github.com/edeliver/edeliver https://www.youtube.com/watch?v=0hd0FPd47II (3h+ podcast

In this episode of Running in Production, Joel Griffith goes over building a service to run headless browsers using Express, Node and Docker. It’s been up and running since late 2017 and runs on just under 1,000 DigitalOcean servers. Joel talks about handling millions of browser sessions, breaking the app up into a few pieces, using Redis as a primary database, using Stripe, creating a custom Node CLI for helping automate deployment tasks and so much more. Topics Include 3:28 – Being profitable early on and spinning up a VPS for each dedicated account 6:22 – 4-5 million browser sessions per day, with peaks up to 30+ million 8:03 – Motivation for choosing Express and Node 14:31 – The vm2 library helps a ton to eliminate things like remote code execution 19:39 – There’s 4 main parts to the app (containers, load balancer, API server and Redis) 25:12 – For the pay as you go accounts, you top your account off with credits 27:12 – Redis is their primary database 31:29 – Using official SDKs are nice, especially if they are written in TypeScript 32:43 – Preact is being used on the front-end as an alternative to React 37:22 – Docker is being used in development and production 42:01 – Testing Stripe locally in development along with nginx (it’s not Dockerized) 52:36 – Using DigitalOcean to host everything (droplets, cloud firewall and floating IPs) 57:40 – Files like PDFs are dynamically served, they are not static files 59:34 – All of the servers run Ubuntu 20.04 LTS 1:01:41 – There’s close to 1,000 servers running at any given time 1:05:37 – The deployment process from development to production (includes Dokku) 1:11:51 – A custom Node CLI helps automate quite a bit of deployment tasks 1:14:00 – The core tool is open source on GitHub 1:18:38 – Email notifications get sent out if any of the servers get overloaded 1:23:01 – An endless sea of text messages from 500s while on vacation in Bora Bora 1:25:44 – Best tips? Know when to pivot and try to talk with folks who cancel your service 1:29:58 – Joel is on GitHub, checkout https://www.browserless.io/ along with @browserless Links 📄 References https://developers.google.com/web/tools/puppeteer https://github.com/tj?tab=repositories https://openresty.org/en/ <a href="https://www.nginx.com/product

In this episode of Running in Production, Parker Selbert talks about building the licensing site for his Oban project using Phoenix and Elixir. It’s been up and running since March 2019 on Heroku. Parker talks about using Live View for the demo page, how access control is handled with Hex’s private packages, a bunch of libraries he uses, hand rolling his Stripe implementation, building a custom admin with custom analytics and more. Topics Include 4:48 – Motivation for using Phoenix and Elixir to build the licensing server 7:36 – Processing millions of background jobs a day with PostgreSQL as a queue back-end 11:55 – Live View powers the demo page and using prefixed jobs to keep things separate 16:43 – It’s almost always worth using a dedicated background tool even in Elixir 18:14 – Keeping Live View and Phoenix updated to their latest stable versions 22:26 – How licensing for the pro version works at the Hex package level 27:03 – Docker is being used in production but not development 31:38 – Challenges of getting Phoenix to reload files outside of the main code base 34:04 – A bunch of useful libraries that were used to help build the pro site 43:00 – Building out a custom admin to help manage customers, plus custom analytics 46:41 – About $100 a month to host everything on Heroku (2 Dynos + Postgres) 50:47 – The deployment process from development to production 56:21 – Backing up the database is handled very nicely by Heroku 1:00:03 – Best tips? Try to keep as much as you can open and contributable by others 1:00:34 – Check out Parker on Twitter and GitHub, his wife is on GitHub too Links 📄 References https://hex.pm/docs/private https://asdf-vm.com/#/ https://hexdocs.pm/plug/Plug.BasicAuth.html ⚙️ Tech Stack

In this episode of Running in Production, Ben Dominguez goes over building a video course platform to learn Python. It’s written in Flask and the MVP was finished in 3 weeks. He gets about 4,000 visitors a month. It’s hosted Heroku’s hobby tier. Ben is the solo developer on the project. He talks about using techs you’re familiar with, keeping deployments simple with Heroku, using PaymentIntents with Stripe and much more. Topics Include 7:28 – Motivation for moving from WordPress to using Flask and Python 10:55 – It’s a video course platform to learn Python through Fantasy Football 13:08 – A couple of useful Python / Flask libraries that helped move things along quickly 16:07 – The React front-end and Flask API back-end are in the same repo 21:58 – PostgreSQL (prod) / SQLite (dev) are being used and the app is hosted on Heroku 27:56 – Tech stack recap and how Stripe is implemented with PaymentIntents 32:38 – Sending emails out with Mailchimp and gmail’s SMTP servers 35:34 – Everything is running on the hobby tier of Heroku for $7 / month 38:27 – The deployment process from development to production 41:04 – Database backups happen every 24 hours and manually before DB migrations 45:36 – Maybe implementing a custom forum into the platform later 48:44 – Best tips? Use your bread and butter skills to get to market as soon as possible 50:16 – Check out the platform or find FFDataPros on Twitter and GitHub Links 📄 References https://learnpythonthehardway.org/ https://jupyter.org/ ⚙️ Tech Stack

In this episode of Running in Production, Kshitij Sinha walks us through his job finding platform called Creative Hire. It was written in Python and Django and is hosted on AWS’ free tier. It’s been up and running since early 2020 after the MVP was built in about 6 weeks. Kshitij is the solo developer on the project. It’s a monolith split across a few Django apps with 2 git repos (one for the back-end and one for the front-end). It does a quite a bit of NLP to help categorize submissions and more. Topics Include 3:58 – Motivation to rewrite the app in Django and Python 6:29 – It’s 1 monolithic Django app that’s split up with Django apps 11:12 – Django REST Framework is used as an API back-end with React on the front-end 15:48 – It’s about 200k lines of Python and 78k lines of JS split across 2 git repos 18:13 – Picking the right NLP Python library (he went with spaCy in the end) 23:44 – Redis is the back-end for Celery and it’s used for caching too 27:41 – Using Ant Design Pro to help manage front-end assets (which uses Webpack) 30:36 – Docker is being used for everything except nginx runs directly on the host 33:47 – It’s hosted on a single EC2 instance (free tier) on AWS running Ubuntu 18.04 LTS 41:34 – The server was set up manually by mostly following tutorials 42:40 – Using Mailgun and Sentry, also right now Creative Hire is free during early access 45:58 – The process to develop and deploy code to production 53:13 – Database backups happen about once a week and how alerts happen 56:12 – Best tips? Have a test environment and make small changes when pushing new code 58:28 – Besides their site, Creative Hire also has a Facebook page Links 📄 References https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html ⚙️ Tech Stack