Alex's Slip-box

These are my org-mode notes in sort of Zettelkasten style

docker-compose Rails PG Redis Sidekiq Travis-CI

Created on September 18, 2020

Updated on September 16, 2023

Tags: docker, rails

:ID: 19CD4732-5A20-43B9-BAC4-FEB736D190ED

# Instructions

This is a generator for Rails project setup with docker, Postgres, Redis, Sidekiq; with container orchestration using docker-compose. The goal of this is to do everything in Docker including the initial project setup in order to eliminate the need to install anything locally.

  1. Set the :HEADER-ARGS: property for each file with the :tangle path. This is where the generated file will be written. See also Using Header Arguments.
  2. Edit the config Association List below. Select the desired versions and app name.
  3. call org-babel-tangle to generate the files in your chosen tangle directory.
  4. Run the setup script as an entry point to a temp container based on the selected image. The command is generated in a Usage comment in setup.sh.
  5. Add a .env file
  6. Add a .dockerignore. See this gist for ideas of what to ignore. TODO: just generate this, dummy.
  7. Update Procfile.dev to bind the server to 0.0.0.0 web: unset PORT && bin/rails server -p 3000 -b '0.0.0.0'
  8. Build container image docker-compose build
  9. Start the app with docker-compose up

# Skip all the confirm prompts

This will run the config elisp block below several times. Set the org-confirm-babel-evaluate variable to nil to skip prompting.

# Config

(setq config
      '(("ruby-version" . "3.2.2")
        ("node-version" . "20")
        ("pg-version" . "latest")
        ("redis-version" . "latest")
        ("app-name" . "tmp")))
(cdr (assoc key config))

# Dockerfile

This is a multistage docker build. The advantage is having yarn and bundle dependencies being a step that is cached independently. 

ARG RUBY_VERSION=<<config("ruby-version")>>
ARG NODE_VERSION=<<config("node-version")>>

FROM node:$NODE_VERSION-slim as node

WORKDIR /app

COPY package.json yarn.lock ./
RUN yarn install

FROM ruby:$RUBY_VERSION as base

RUN apt-get update -qq \
    && apt-get install --no-install-recommends -y build-essential locales libvips libpq-dev \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists /var/cache/apt/archives

WORKDIR /app

FROM base as gems

COPY Gemfile Gemfile.lock ./
RUN gem install bundler && bundle install --jobs 20 --retry 5

FROM base as build

RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

COPY . ./
COPY --from=node /usr/lib /usr/lib
COPY --from=node /usr/local/share /usr/local/share
COPY --from=node /usr/local/lib /usr/local/lib
COPY --from=node /usr/local/include /usr/local/include
COPY --from=node /usr/local/bin /usr/local/bin
COPY --from=node /opt /opt
COPY --from=node /app/node_modules /app/node_modules
COPY --from=gems /usr/local/bundle /usr/local/bundle

EXPOSE 3000

CMD ["bin/start"]

# docker-compose

This orchestrates the following containers:

  • web
  • database
  • redis
  • sidekiq The sidekiq command is basic in this example. To specify queues for example use bundle exec sidekiq -q default -q other. Do sidekiq --help to see all the options.

Environment variables are provided via a .env file in the project root. This file is automatically generated by the setup script below. 

version: "3.6"

services:
  web:
    tty: true
    stdin_open: true
    build: .
    command: bash -c "rm -f tmp/pids/server.pid && /app/bin/dev"
    volumes:
      - ".:/app"
      - "/app/node_modules" # don't mount node_modules dir
    ports:
      - "3000:3000"
    links:
      - "database"
      - "redis"
    env_file:
      - ".env"
  database:
    image: postgres:<<config("pg-version")>>
    volumes:
      - 'postgres:/var/lib/postgresql/data'
    ports:
      - "5432"
    env_file:
      - ".env"
  redis:
    image: redis:<<config("redis-version")>>
    volumes:
      - "redis:/data"
    ports:
      - "6379"
  sidekiq:
    depends_on:
      - "database"
      - "redis"
    build: .
    volumes:
      - ".:/app"
      - "/app/tmp" # don't mount tmp dir
    command: "bundle exec sidekiq"
    env_file:
      - ".env"

volumes:
  redis:
  postgres:

# Setup the project

After generating the files we still don’t have a Gemfile, Gemfile.lock or any of the Rails’ framework files, etc. But we can still do all the project setup within a docker container and mounting the project directory as a volume and running a few commands.

See Usage comment in the script for how to run it

# setup.sh

This is a one time setup script. It can be deleted after it is run. 

# Usage:
#   docker run --rm -it -v "$PWD":/app -w /app ruby:<<config("ruby-version")>> sh setup.sh

# Install dependencies
gem install rails -v "~> 7"

# Install node from nodesource
https://github.com/nodesource/distributions/blob/master/README.md#installation-instructions
curl -fsSL https://deb.nodesource.com/setup_<<config("node-version")>>.x | bash - && apt-get install -y nodejs

# Install yarn
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
apt update && apt install -y yarn

# Setup rails app
rails new <<config("app-name")>> --database=postgresql --javascript=esbuild --css=bootstrap

cd <<config("app-name")>>
mv * .*  ../
cd ..
rmdir <<config("app-name")>>

# COMMENT THE FOLLOWING IN TO INSTALL webpacker
# bin/rails webpacker:install

bundle add sidekiq --skip-install
bundle add rubocop --group development --require false --skip-install
bundle add rubocop-performance --group development --require false --skip-install
bundle add rubocop-rails --group development --require false --skip-install
bundle add rubocop-rspec --group development --require false --skip-install
bundle add rspec-rails --group "development, test" --skip-install
bundle add factory_bot_rails --group "development, test"

# Configure database
DBCONFIG=$(cat <<EOF
default: &default
  adapter: postgresql
  encoding: unicode
  host: <%= ENV['POSTGRES_HOST'] %>
  user: <%= ENV['POSTGRES_USER'] %>
  password: <%= ENV['POSTGRES_PASSWORD'] %>
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
development:
  <<: *default
  database: <<config("app-name")>>_development
test:
  <<: *default
  database: <<config("app-name")>>_test
production:
  <<: *default
  database: <<config("app-name")>>_production
  username: <<config("app-name")>>
  password: <%= ENV['SUPER_SECRET_DATABASE_PASSWORD'] %>
EOF
)
echo "$DBCONFIG" > config/database.yml

# Add env vars for development
touch .env
echo "POSTGRES_DB=<<config("app-name")>>_development" >> .env
echo "POSTGRES_USER=postgres" >> .env
echo "POSTGRES_PASSWORD=postgres" >> .env
echo "POSTGRES_HOST=database" >> .env
echo "PGUSER=postgres" >> .env
echo "REDIS_URL=redis://redis:6379/1" >> .env

# Run it

After completing the project setup, run:

docker-compose build to build the image docker-compose up to run the containers

Tear it down with:

docker-compose down

See https://github.com/apmiller108/astronomania-api README for example of more commands like debugged with pry and running tests.

# Travis-CI

This is the travis.yml. Most of this is boiler plate from Travis’ docs. The env stuff I had to figure out becuase I am using a .env file to pass config vars into containers with docker-compose. First I had to add the env vars to Travis’ project settings. Then echo them into a .env file for docker-compose to read from.

NOTE: the COMPOSE_VERSION is NOT the docker-compose file version, but the docker-compose release version: Releases · docker/compose · GitHub

NOTE: this file doesn’t get generated. Use it if you want. 

language: bash

sudo: required

services:
  - docker

env:
  COMPOSE_VERSION: 1.26.2

before_install:
 - sudo rm /usr/local/bin/docker-compose
 - curl -L https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose
 - chmod +x docker-compose
 - sudo mv docker-compose /usr/local/bin
 - docker --version
 - docker-compose --version

script:
  - touch .env
  - echo "POSTGRES_DB=${POSTGRES_DB}" >> .env
  - echo "POSTGRES_USER=${POSTGRES_USER}" >> .env
  - echo "POSTGRES_PASSWORD=${POSTGRES_PASSWORD}" >> .env
  - echo "POSTGRES_HOST=${POSTGRES_HOST}" >> .env
  - echo "NASA_API_KEY=${NASA_API_KEY}" >> .env
  - docker-compose up --detach --build
  - docker ps -a
  - docker-compose exec web bin/rails db:schema:load RAILS_ENV=test
  - docker-compose exec web bundle exec rspec

after_script:
  - docker-compose down
  - rm .env

notifications:
  email: false

# Resources

Search Results