Micro-components architecture with python / Django / Drf

I have several applications in my Django project:

- ticker
- payments
- crypto
- referrals
- core

I am using Docker wrapped by Wercker (quite limiting I would say, but saves time).

  • How to downgrade docker on CoreOS?
  • Installing openssh-server, g++, gdb, and gdbserver in a docker container | Visual C++ for Linux Development + Docker
  • docker mysql on different port
  • Docker LAMP stack - where is the location to keep PHP projects?
  • How to install app from git repository via docker?
  • Node.js tcp sockets + linked Docker containers
  • My question is, how to deploy each application as a standalone container as a start, and a standalone VPS Node in the private network later.

    Also, I would really like to deploy the postgres instance into a separate instance in the private network and not just a container.

    My logic tells me that a separate Django project will be required for this purpose.

    We have a little bit of a weird workflow there… writing a docker-compose file during a wercker step and then triggering it in the ENTRYPOINT.

    I am attaching my docker-compose and wercker.yml file.

    docker-compose:

    version: '2'
    services:
        postgis:
            image: mdillon/postgis
            environment:
               POSTGRES_USER: ${POSTGIS_ENV_POSTGRES_USER}
               POSTGRES_PASSWORD: ${POSTGIS_ENV_POSTGRES_PASSWORD}
               POSTGRES_DB: ${POSTGIS_ENV_POSTGRES_DB}
            volumes:
                - /nexchange/database:/var/lib/postgresql/data
            restart: always
        app:
            image: onitsoft/nexchange:${DOCKER_IMAGE_TAG}
            volumes:
                - /nexchange/mediafiles:/usr/share/nginx/html/media
                - /nexchange/staticfiles:/usr/share/nginx/html/static
            links:
                - postgis
            restart: always
        web:
            image: onitsoft/nginx
            volumes:
                - /nexchange/etc/letsencrypt:/etc/letsencrypt
                - /nexchange/etc/nginx/ssl:/etc/nginx/ssl
                - /nexchange/etc/nginx/nginx.conf:/etc/nginx/nginx.conf
                - /nexchange/mediafiles:/usr/share/nginx/html/media
                - /nexchange/staticfiles:/usr/share/nginx/html/static
            ports:
                - "80:80"
                - "443:443"            
            links:
                - app
            restart: always
    

    wercker.yml:

    box: pitervergara/geodjango:nexchange
    ports:
      - "8000"
    
    dev:
      services:
       - id: mdillon/postgis
         env:
           POSTGRES_USER: nexchange
           POSTGRES_PASSWORD: nexchange
           POSTGRES_DB: nexchange
      steps:
        - script:
            name: export django settings module
            code: |
              export DJANGO_SETTINGS_MODULE=nexchange.settings_dev
        - script:
            name: create static and media root
            code: |
              mkdir -p /usr/share/nginx/html/static
              mkdir -p /usr/share/nginx/html/media
        - npm-install
        - maxon/npm-run:
            script: bower
        - maxon/npm-run:
            script: build-js
        - script:
            name: pip install requirements (with cache)
            code: |
              pip_download_cache="$WERCKER_CACHE_DIR/wercker/_pipcache"
              mkdir -p ${pip_download_cache}
              pip install --cache-dir ${pip_download_cache} -r requirements.txt
              has_krakenex=$(python -c "import importlib; k = importlib.find_loader('krakenex'); print(str(k is not None))")
              if [ "${has_krakenex}" == "False" ]; then
                git clone https://github.com/onitsoft/python3-krakenex.git /usr/src/krakenex
                cd /usr/src/krakenex
                python3 setup.py install
                cd -
              fi
              python -c "import krakenex; print('import krakenex sucessfully executed')"
        - script:
            name: Django make migrations
            code: |
              cat <(echo "yes") - | python manage.py makemigrations
        - script:
            name: wait...
            code: |
              sleep 5
        - script:
            name: Django apply migrations
            code: |
              python manage.py migrate
        - script:
            name: Django compile messages
            code: |
              # python manage.py compilemessages
        - script:
          name: Django create superuser
          code: |
              echo "from django.contrib.auth.models import User; User.objects.create_superuser('onit', 'weare@init.ws','weare0nit')" | python manage.py shell
        - script:
            name: Django import Currency fixture
            code: |
              python manage.py loaddata core/fixtures/currency.json
        - script:
            name: Django import Payment Methods fixture
            code: |
              python manage.py loaddata core/fixtures/payment_method.json
        - script:
            name: Django import Payment Preference fixture
            code: |
              python manage.py loaddata core/fixtures/payment_preference.json
        - script:
            name: Django import Withdraw address fixture
            code: |
              python manage.py loaddata core/fixtures/withdraw_address.json
        - script:
            name: Django import affiliate program fixture
            code: |
              python manage.py loaddata core/fixtures/affiliate_program.json
        - script:
            name: Django run ticker update prices command
            code: |
              python manage.py ticker
        - script:
            name: Django collect static
            code: |
              python manage.py collectstatic --noinput
        - script:
            name: Link pre-commit hook script
            code: |     
              chmod +x .pre-commit.sh 
              cd .git/hooks/
              ln -fs ../../.pre-commit.sh pre-commit
              cd -
        - create-file:
            name: Create cron invoked script to run ticker
            filename: /ticker.sh
            overwrite: true
            content: |-
              #!/bin/bash
              #
              export DJANGO_SETTINGS_MODULE=${DJANGO_SETTINGS_MODULE}
              export POSTGIS_ENV_POSTGRES_USER=${POSTGIS_ENV_POSTGRES_USER}
              export POSTGIS_ENV_POSTGRES_PASSWORD=${POSTGIS_ENV_POSTGRES_PASSWORD}
              export POSTGIS_ENV_POSTGRES_DB=${POSTGIS_ENV_POSTGRES_DB}
              export POSTGIS_PORT_5432_TCP_ADDR=${POSTGIS_PORT_5432_TCP_ADDR}
              export POSTGIS_PORT_5432_TCP_PORT=${POSTGIS_PORT_5432_TCP_PORT}
              #
              cd ${WERCKER_SOURCE_DIR}
              /usr/local/bin/python manage.py ticker >> /var/log/cron.log 2>&1
        - script:
            name: Add cron job
            code: |
              chmod +x /ticker.sh
              echo "* * * * * root /ticker.sh" > /etc/crontab
              cron
        - internal/watch:
            code: |
              echo 'Dev server running'
               npm run watch-js & newrelic-admin run-program  python manage.py runserver 0.0.0.0:8000
            reload: false
    
    build:
      services:
       - id: mdillon/postgis
         env:
           POSTGRES_USER: ${POSTGIS_ENV_POSTGRES_USER}
           POSTGRES_PASSWORD: ${POSTGIS_ENV_POSTGRES_PASSWORD}
           POSTGRES_DB: ${POSTGIS_ENV_POSTGRES_DB}
      steps:
        - install-packages:
          packages: netcat
        - script:
            name: create static and media root
            code: |
              mkdir -p /usr/share/nginx/html/static
              mkdir -p /usr/share/nginx/html/media
        - script:
          name: Install node requirements
          code: |
            # https://github.com/wercker/support/issues/227 :(
            rm -fr node_modules && npm install --production
        - maxon/npm-run:
            script: bower-prd
        - script:
            name: Install python requirements
            code: |
              pip_download_cache="$WERCKER_CACHE_DIR/wercker/_pipcache"
              mkdir -p ${pip_download_cache}
              pip install --cache-dir ${pip_download_cache} -r requirements.txt
              has_krakenex=$(python -c "import importlib; k = importlib.find_loader('krakenex'); print(str(k is not None))")
              if [ "${has_krakenex}" == "False" ]; then
                git clone https://github.com/onitsoft/python3-krakenex.git /usr/src/krakenex
                cd /usr/src/krakenex
                python3 setup.py install
                cd -
              fi
              python -c "import krakenex; print('import krakenex sucessfully executed')"
        - script:
            name: Django migrations
            code: |
              python manage.py makemigrations
        - script:
            name: django collect and compile translations
            code: |
              # python manage.py compilemessages
        - script:
            name: Django collect static
            code: |
              python manage.py collectstatic --noinput
        - script:
          name: copy files
          code: |
            mkdir -p /usr/src/app
            cp -r [a-z]* /usr/src/app
            cp -r /usr/share/nginx/html/static $WERCKER_OUTPUT_DIR/staticfiles
            cp -r /usr/share/nginx/html/media $WERCKER_OUTPUT_DIR/mediafiles
        - script:
            name: place docker-compose and nginx.conf files
            code: |
              mv "nginx.conf" "$WERCKER_OUTPUT_DIR/nginx.conf"
              mv "docker-compose.yml" "$WERCKER_OUTPUT_DIR/docker-compose.yml"
        - create-file:
            #
            # PEM_FILE_CONTENT - the key to SSH into server (create key par via wercker web interface. remeber to install public key on server)
            # SSH_USER - the user to SSH into server
            # DEST_HOST_ADDR - server where to deploy  
            #
            # DOCKER_HUB_USER - dockerhub username
            # DOCKER_HUB_PASSWORD - dockerhub password (defined as a protectd var)
            # DOCKER_HUB_REPO - the dockerhub repo where to push (repo must already exists and should be private)
            name: Create production entrypoint
            filename: /entrypoint.sh
            overwrite: true
            content: |-
              #!/bin/bash
              # ###
              # This script is generate in deploy step and:
              #   Exports variables
              #   Apply migrations
              #   Starts gunicorn
              # ###
              export DJANGO_SETTINGS_MODULE=${DJANGO_SETTINGS_MODULE}
              export GUNICORN_PORT=${GUNICORN_PORT}
              export POSTGIS_ENV_POSTGRES_USER=${POSTGIS_ENV_POSTGRES_USER}
              export POSTGIS_ENV_POSTGRES_PASSWORD=${POSTGIS_ENV_POSTGRES_PASSWORD}
              export POSTGIS_ENV_POSTGRES_DB=${POSTGIS_ENV_POSTGRES_DB}
              export POSTGIS_PORT_5432_TCP_ADDR=${POSTGIS_PORT_5432_TCP_ADDR}
              export POSTGIS_PORT_5432_TCP_PORT=${POSTGIS_PORT_5432_TCP_PORT}
              export NEW_RELIC_CONFIG_FILE=${NEW_RELIC_CONFIG_FILE}
              export NEW_RELIC_ENVIRONMENT=${NEW_RELIC_ENVIRONMENT}
              export NEW_RELIC_LICENSE_KEY=${NEW_RELIC_LICENSE_KEY}
              #
              while ! nc -z ${POSTGIS_PORT_5432_TCP_ADDR} ${POSTGIS_PORT_5432_TCP_PORT}
              do
                >&2 echo "PostgreSQL '(${POSTGIS_PORT_5432_TCP_ADDR}:${POSTGIS_PORT_5432_TCP_PORT})' not ready - waiting..."
                sleep 1;
              done
              echo "PostgreSQL '(${POSTGIS_PORT_5432_TCP_ADDR}:${POSTGIS_PORT_5432_TCP_PORT})' is ready - moving on..."
              #
              # Apply migrations
              python /usr/src/app/manage.py migrate
              # Import fixtures
              python /usr/src/app/manage.py loaddata core/fixtures/currency.json
              python /usr/src/app/manage.py loaddata core/fixtures/payment_method.json
              python /usr/src/app/manage.py loaddata core/fixtures/affiliate_program.json
              echo "To load the 'withdraw_address' fixture, uncomment the next line"
              # python manage.py loaddata core/fixtures/withdraw_address.json
              #
              # Prepare log files and start outputting logs to stdout
              # adapted from 
              # http://michal.karzynski.pl/blog/2015/04/19/packaging-django-applications-as-docker-container-images/
              touch /var/log/gunicorn_error.log
              touch /var/log/gunicorn_access.log
              tail -n 0 -f /var/log/*.log &
              # Copy static data to nginx volume
              cp -ra $WERCKER_OUTPUT_DIR/staticfiles/* /usr/share/nginx/html/static
              cp -ra $WERCKER_OUTPUT_DIR/mediafiles/* /usr/share/nginx/html/media
              #
              # Create superuser
              echo "from django.contrib.auth.models import User; User.objects.create_superuser('onit', 'weare@onit.ws','weare0nit')" | python manage.py shell
              #
              # Start Cron Process
              cron
              echo "Gunicorn start"
              exec newrelic-admin run-program gunicorn --chdir /usr/src/app --name nexchange --bind 0.0.0.0:${GUNICORN_PORT} --workers 3 --log-level=info --log-file=/var/log/gunicorn_error.log --access-logfile=/var/log/gunicorn_access.log nexchange.wsgi:application "$@"
        - script:
            name: set entrypoint as executable
            code: |
              chmod +x /entrypoint.sh
        - create-file:
            name: Create cron invoked script to run ticker
            filename: /ticker.sh
            overwrite: true
            content: |-
              #!/bin/bash
              #
              export DJANGO_SETTINGS_MODULE=${DJANGO_SETTINGS_MODULE}
              export POSTGIS_ENV_POSTGRES_USER=${POSTGIS_ENV_POSTGRES_USER}
              export POSTGIS_ENV_POSTGRES_PASSWORD=${POSTGIS_ENV_POSTGRES_PASSWORD}
              export POSTGIS_ENV_POSTGRES_DB=${POSTGIS_ENV_POSTGRES_DB}
              export POSTGIS_PORT_5432_TCP_ADDR=${POSTGIS_PORT_5432_TCP_ADDR}
              export POSTGIS_PORT_5432_TCP_PORT=${POSTGIS_PORT_5432_TCP_PORT}
              #
              cd ${WERCKER_SOURCE_DIR}
              /usr/local/bin/python manage.py ticker >> /var/log/cron.log 2>&1
        - script:
            name: Add cron job
            code: |
              chmod +x /ticker.sh
              echo "* * * * * root /ticker.sh" > /etc/crontab
        - script:
            name: echo python information
            code: |
              echo "python version $(python --version) running"
              echo "pip version $(pip --version) running"
              echo "installed python packages:"
              echo "$(pip freeze | sort)"
        - internal/docker-push:
            username: $DOCKER_HUB_USER
            password: $DOCKER_HUB_PASSWORD
            tag: ${DOCKER_IMAGE_TAG}
            repository: $DOCKER_HUB_REPO
            registry: https://registry.hub.docker.com
            entrypoint: /entrypoint.sh
            ports: ${GUNICORN_PORT}
            working-dir: /usr/src/app
    
    deploy:
      box: pcfseceng/ci-ssh-client
      steps:
        - mktemp:
            envvar: PRIVATEKEY_PATH
        - create-file:
            name: write key
            filename: $PRIVATEKEY_PATH
            content: $PEM_FILE_CONTENT_PRIVATE  
            overwrite: true
        - script:
          name: Do deploy
          code: |
            SSH_OPTIONS="-o StrictHostKeyChecking=no -i $PRIVATEKEY_PATH"
            SSH_DEST="$SSH_USER@$DEST_HOST_ADDR"
            SUBSTR=${WERCKER_GIT_COMMIT:0:9}
            scp ${SSH_OPTIONS} nginx.conf ${SSH_DEST}:/nexchange/etc/nginx/nginx.conf
            scp ${SSH_OPTIONS} docker-compose.yml ${SSH_DEST}:/nexchange/docker-compose.yml
            ssh ${SSH_OPTIONS} ${SSH_DEST} << EOF
              # export the variables that docker-compose will inject into DB container
              export POSTGIS_ENV_POSTGRES_USER=${POSTGIS_ENV_POSTGRES_USER}
              export POSTGIS_ENV_POSTGRES_PASSWORD=${POSTGIS_ENV_POSTGRES_PASSWORD}
              export POSTGIS_ENV_POSTGRES_DB=${POSTGIS_ENV_POSTGRES_DB}
              export DOCKER_IMAGE_TAG=${DOCKER_IMAGE_TAG}
              echo "Printing current env:"
              /usr/bin/env
              echo "env of env"
              # Login to docker hub (for private images)
              sudo docker login \
                -u $DOCKER_HUB_USER \
                -p $DOCKER_HUB_PASSWORD
              # Start new instance
              sudo -E docker-compose -f /nexchange/docker-compose.yml pull
              sudo -E docker-compose -f /nexchange/docker-compose.yml up -d
            EOF
    
    tests:
      steps:
        - script:
            name: export django settings
            code: |
              export DJANGO_SETTINGS_MODULE=nexchange.settings_test
        - script:
            name: Install python requirements
            code: |
              pip_download_cache="$WERCKER_CACHE_DIR/wercker/_pipcache"
              mkdir -p ${pip_download_cache}
              pip install --cache-dir ${pip_download_cache} -r requirements.txt
              has_krakenex=$(python -c "import importlib; k = importlib.find_loader('krakenex'); print(str(k is not None))")
              if [ "${has_krakenex}" == "False" ]; then
                git clone https://github.com/onitsoft/python3-krakenex.git /usr/src/krakenex
                cd /usr/src/krakenex
                python3 setup.py install
                cd -
              fi
              python -c "import krakenex; print('import krakenex sucessfully executed')"
        - script:
          name: Install node requirements
          code: |
            # rm -fr node_modules && npm install
            npm install
        - maxon/npm-run:
            script: bower
        - script:
            name: Backend tests
            code: |
              coverage erase
              coverage run --source="." manage.py test -v 3
              coverage report
              coverage html -d cover
        - script:
            name: Frontend tests
            code: |
              # export PHANTOMJS_BIN=${WERCKER_SOURCE_DIR}/node_modules/.bin/phantomjs
              # npm run-script test
        - script:
            name: Validate New Relic config file
            code: |
              newrelic-admin validate-config ${NEW_RELIC_CONFIG_FILE}
        - script:
            name: Install Phantomjs tests frontend
            code: |
              #cd /usr/local/share
              #sudo wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-x86_64.tar.bz2
              #sudo tar xfj phantomjs-1.9.8-linux-x86_64.tar.bz2
              #sudo ln -s /usr/local/share/phantomjs-1.9.8-linux-x86_64/bin/phantomjs /usr/local/share/phantomjs
              #sudo ln -s /usr/local/share/phantomjs-1.9.8-linux-x86_64/bin/phantomjs /usr/local/bin/phantomjs
              #sudo ln -s /usr/local/share/phantomjs-1.9.8-linux-x86_64/bin/phantomjs /usr/bin/phantomjs
    
        - script:
            name: Test_selenium
            code: |
              cd /pipeline/source
    #          python manage.py test_selenium
    
    static-validation:
      steps:
        - script:
            name: Run static-validation.sh
            code: |
              bash static-validation.sh
    

  • LXC/docker.io & kernel updates
  • Node and docker - how to handle babel or typescript build?
  • Connecting to running docker containers from localhost
  • copy the file from host to docker container prior to run webserver in container
  • Docker: multi kafka brokers failed
  • running django worker and daphne in docker container
  • One Solution collect form web for “Micro-components architecture with python / Django / Drf”

    You can update the INSTALLED_APPS list in the settings.py file using the startup script. This startup script will be called using the Entrypoint.

    Then you can pass parameter to container start command based on which the startup script will decide which app needs to added to the settings.py file.

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.