Setting up Django with Nginx, Gunicorn, PostgreSQL

  1. Home
  2. Blog
04 Dec
Edit on GitHub
Admin / 2502 views

Contents:

 

Intro

Hello. Today we will be learning how to set up usual Django project.

We will be using such tehnology stack Ubuntu 16.04 x64, Django, PostgreSQL, Gunicorn, Supervisor, Virtualenv and NginX.

For hosting we will be using DigitalOcean

Spinning Ubuntu VPS

Visit DigitalOcean and register account , you can use my invite link (you'll get $10 in credit)

Than add your droplet for example you can use the cheapear one (5$ droplet)

After choosing region where your droplet will be located (choose where your customers will be located) set up name and you've done - you've created your first droplet.

Now check your email in a few minutes there will be a mailed with login info.

Then we will connect to Ubuntu 

ssh root@ourServerIpAdress

 

Fixing locales

Let's start with ensuring that our system is up to date. PS `-y` means automatic yes to prompts

$ sudo apt-get update
$ sudo apt-get upgrade -y

There is common problem with locales to fix it use this

$ nano /etc/environment

and add this line add the end

LC_ALL="en_US.UTF-8"

Now reboot to finish fixing locales

$ sudo reboot

 

Installing common programs like top, mc, tree and python-dev etc

Now wait at least 30 seconds and login again to our server and run below command to install all `bone structure` packages

$ sudo apt-get install htop mc tree python3-dev libpq-dev python-dev libxslt1-dev libxml2 -y

 

Setting PostgreSQL

Installing postgresql

$ sudo apt-get install postgresql postgresql-contrib -y

Now we will create PostgreSQL user and set up database for him.

You need to change `USERNAME`, `USERPASSWORD` and `DATABASE_NAME` to your own values. Note that `USERNAME` and `DATABASE_NAME` should be lovercase

$ sudo -u postgres psql --command="create user USERNAME"
$ sudo -u postgres psql --command="alter role USERNAME with password 'USERPASSWORD'"
$ sudo -u postgres psql --command="create database DATABASE_NAME with owner USERNAME template template0 encoding 'UTF8' "
$ sudo -u postgres psql --command="grant all privileges on database DATABASE_NAME to USERNAME"

If you have problems with postgresql try to start cluster, because in some system cluster won't start automatically (remember to change `9.5` to your postgresql version)

$ pg_createcluster 9.5 main --start

Setting up users and permissions

Creating project directories

$ sudo mkdir -p /var/webapps/testproject/

Here we will create a user for our website, named `testuser` and assigned to a system group called `webapps`.

$ sudo groupadd -r webapps
$ sudo useradd -mr -g webapps -s /bin/bash -d /var/webapps/testproject/home testuser

Adding permissions to folders

$ sudo chown -R testuser:users /var/webapps/testproject/
$ sudo chmod -R g+w  /var/webapps/testproject/

 

Virtualenv

Installing package

$ sudo apt-get install python-virtualenv -y

Now login to our `testuser`, note that `-` will move you to user's home directory

$ su - testuser

go one directory up

testuser@server:~$ cd ..

and create virtualenv with python 3 (for python 2 use '=python2')

testuser@server:/var/webapps/testproject$ virtualenv --python=python3 env
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /var/webapps/testproject/env/bin/python3
Also creating executable in /var/webapps/testproject/env/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

than activate it to use all packages from it

testuser@server:/var/webapps/testproject$ source env/bin/activate

and finally create directory where we will be storing source code, in this directory you can either clone your code from git or follow next step

(env) testuser@server:/var/webapps/testproject$ mkdir code

 

Starting sample Django project

Assuming that you have done all previous steps than we will create sample Django project

Installing Django is as easy as

(env) testuser@server:/var/webapps/testproject$ pip install django
Collecting django
(...)
Installing collected packages: django
Successfully installed django

Starting testproject, note that dot at the end means that we will re use current directory

(env) testuser@server:/var/webapps/testproject$ cd code
(env) testuser@server:/var/webapps/testproject/code$ django-admin startproject testproject .

And test it by running 

(env) testuser@server:/var/webapps/testproject/code$ python manage.py runserver 0.0.0.0:8000
Performing system checks...

System check identified no issues (0 silenced).

You have 13 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.

March 26, 2017 - 12:57:32
Django version 1.10.6, using settings 'testproject.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.

And you well be able to access it from http://ourServerIpAdress:8000 

 

Configuring PostgreSQL to work with Django

Installing PostgreSQL database adapter 

(env) testuser@server:/var/webapps/testproject/code$ pip install psycopg2

You can now configure the databases in your `settings.py`:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'DATABASE_NAME',
        'USER': 'USERNAME',
        'PASSWORD': 'USERPASSWORD',
        'HOST': 'localhost',
        'PORT': '',                      # Set to empty string for default.
    }
}

Than we need to create tables for our newly created database

(env) testuser@server:/var/webapps/testproject/code$ python manage.py makemigrations
(env) testuser@server:/var/webapps/testproject/code$ python manage.py migrate

 

Setting up folders

Go one directory up

(env) testuser@server:/var/webapps/testproject$

For now if you were following all steps your project directory should look like this

├── code
│   ├── manage.py
│   └── testproject
├── env
└── home

But for real life project you will need folders for logs, configs and etc

For example in `media` and `static` we will be storing media files and files from `collectstatic` 

(env) testuser@server:/var/webapps/testproject$ mkdir logs conf
(env) testuser@server:/var/webapps/testproject$ mkdir -p www/media
(env) testuser@server:/var/webapps/testproject$ mkdir -p www/static

And for now it should look like

├── code
│   ├── manage.py
│   └── testproject
├── conf
├── env
├── logs
└── www
    ├── media
    └── static

 

Gunicorn

In production we won’t be using Django’s single-threaded development server, but a special application server called gunicorn.

Installing it via

(env) testuser@server:/var/webapps/testproject$ pip install gunicorn
Collecting gunicorn
(...)
Installing collected packages: gunicorn
Successfully installed gunicorn

And test it by running

(env) testuser@server:/var/webapps/testproject$ cd code
(env) testuser@server:/var/webapps/testproject/code$ gunicorn testproject.wsgi:application --bind ourServerIpAdress:8001
[2017-03-26 13:01:34 +0000] [11414] [INFO] Starting gunicorn
[2017-03-26 13:01:34 +0000] [11414] [INFO] Listening at: http://ourServerIpAdress:8001 (11414)
[2017-03-26 13:01:34 +0000] [11414] [INFO] Using worker: sync
[2017-03-26 13:01:34 +0000] [11417] [INFO] Booting worker with pid: 11417

Gunicorn is set up and ready to serve your website. Now we will create config file and save it in `conf/gunicorn.conf.py`.

Note: don't change bind , we will later use it for NginX

proc_name = 'testproject'
bind = '127.0.0.1:8000'
workers = 3
user = 'testuser'
group = 'webapps'
loglevel = 'debug'
errorlog = '/var/webapps/testproject/logs/gunicorn.error.log'
accesslog = '/var/webapps/testproject/logs/gunicorn.access.log'

As a rule-of-thumb set the workers according to the following formula: 2 * CPUs + 1. The idea being, that at any given time half of your workers will be busy doing I/O. For a single CPU machine it would give you 3.

Be careful not to forget to change paths and filenames to match your project in `gunicorn.conf.py`

 

Supervisor

We need to ensure sure that gunicorn is starting automatically with the system and that it can automatically restarts if for some reason it exits unexpectedly.

These is a job for a service called supervisord. 

$ sudo apt-get install supervisor -y

Specifically on Ubuntu 16.04 we need to execute this in order to work with supervisor correctly

$ sudo systemctl enable supervisor
$ sudo systemctl start supervisor

In order to force Supervisor to run gunicorn  we will need to create `/etc/supervisor/conf.d/testproject.conf` in `/etc/supervisor/conf.d` directory with this content

[program:testproject]
environment=LANG="en_US.utf8", LC_ALL="en_US.UTF-8", LC_LANG="en_US.UTF-8"
directory = /var/webapps/testproject/code
command = /var/webapps/testproject/env/bin/gunicorn testproject.wsgi:application -c /var/webapps/testproject/conf/gunicorn.conf.py
user = testuser
loglevel = debug
stdout_logfile = /var/webapps/testproject/logs/supervisor.log
stderr_logfile = /var/webapps/testproject/logs/supervisor.error.log
autostart = true
autorestart = true
redirect_stderr = true

And create log files

$ touch /var/webapps/testproject/logs/supervisor.log
$ touch /var/webapps/testproject/logs/supervisor.error.log

In order to recognise file and run it you should execute this

$ sudo supervisorctl reread
testproject: available
$ sudo supervisorctl update
testproject: added process group

You can now check status of our site

$ sudo supervisorctl status testproject
testproject  RUNNING   pid 11987, uptime 0:00:35

Your website should now be able to automatically start and restart in the case of failure or system reboot.

 

NginX

Here we will be setting up Nginx to work with our static files and media

To install

$ sudo apt-get install nginx -y
$ sudo service nginx start

Check status of setup by visiting page http://ourServerIpAdress . Nginx should greet you with words "Welcome to nginx!"

Here we will create Nginx virtual server configuration for our Django website. Every Nginx virtual server should be configured by a file in the `/etc/nginx/sites-available` directory and enabling them by making symbolic links in the `/etc/nginx/sites-enabled` directory.

Create file `/etc/nginx/sites-available/testproject` with following content (remember to adapt it to your project variables and paths)

server {

    listen 80;
    server_name ourServerIpAdress;

    client_max_body_size 15m;

    access_log /var/webapps/testproject/logs/nginx-access.log;
    error_log /var/webapps/testproject/logs/nginx-error.log;

    # Enable Gzip compression
    gzip on;

    # Compression level (1-9)
    gzip_comp_level 5;

    # Don't compress anything under 256 bytes
    gzip_min_length 256;

    # Compress output of these MIME-types
    gzip_types
        application/atom+xml
        application/javascript
        application/json
        application/rss+xml
        application/vnd.ms-fontobject
        application/x-font-ttf
        application/x-javascript
        application/x-web-app-manifest+json
        application/xhtml+xml
        application/xml
        font/opentype
        image/svg+xml
        image/x-icon
        text/css
        text/plain
        text/x-component;

    location /static/ {
        alias /var/webapps/testproject/www/static/;
        expires 30d;
        add_header Pragma public;
        add_header Cache-Control "public";
    }

    location /media/ {
        alias /var/webapps/testproject/www/media/;
        expires 30d;
        add_header Pragma public;
        add_header Cache-Control "public";
    }


    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header X-Forwarded-Host $server_name;
        proxy_set_header Host $server_name;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header REMOTE_ADDR $remote_addr;
        add_header P3P 'CP="ALL DSP COR PSAa PSDa OUR NOR ONL UNI COM NAV"';
        # uncomment below to enable custom 404 (and other error pages)
        # proxy_intercept_errors on;
    }

    # uncomment this and create file 404.html in /var/webapps/testproject/www/static
    # directory to create custom 404 error page
    # error_page 404 /404.html;
    # location = /404.html {
    #    root /var/webapps/testproject/www/static;
    #    internal;
    # }

}

You can also check syntax for NginX via

$ nginx -t

And create symbolik link to enable it in NginX

$ sudo ln -s /etc/nginx/sites-available/testproject /etc/nginx/sites-enabled/testproject

And restart NginX

$ sudo service nginx restart

As for now you should achieve up and running website on http://ourServerIpAdress 

Final directory structure in `/var/webapps/testproject` will be

├── code
│   ├── manage.py
│   └── testproject
├── conf
│   └── gunicorn.conf.py
├── env
├── home
├── logs
│   ├── gunicorn.access.log
│   ├── gunicorn.error.log
│   ├── nginx-access.log
│   ├── nginx-error.log
│   ├── supervisor.error.log
│   └── supervisor.log
└── www
    ├── media
    └── static

 

Useful commands

To refresh changes in project use this, it will restart gunicorn and supervisor and as a result your changes to source files will be visible on site.

$ sudo service supervisor restart

How to backup database  (unsure about this, help wanted)

This command wil create db.json file with your database backup

python manage.py dumpdata --exclude auth.permission --exclude contenttypes > db.json

When you want to load your backup you need to drop database and load in freshly created database

So in Django you need to migrate before loading backup and then

python manage.py loaddata db.json

 

SIdenotes: this can't be used for large databases, backup and restore databases should be identical in structure.

 

 

If you know about Ansible and Vagrant you can check my Django Seed Project which is deployable via ansible.

Most Popular
Categories
Tags

© All Rights Reserved Redux Blog