Foreword

In the previous article, we already managed to dockerize the project using pdm, so the next step is to deploy the complete DjangoStarter v3 version with Docker.

It's not as straightforward as before this time because the project uses tools like npm and gulp to manage front-end dependencies, and pdm for Python dependencies. That's why I've employed a multi-stage build for this setup.

Additionally, I've replaced uwsgi this time. Although uwsgi is a bit old, its performance is still decent. However, we're now in the ASGI era, and WSGI has its limitations. Plus, the project I'm deploying uses channels, so it made sense to switch to the recommended daphne server, which seems to be working well. I'm still exploring its full capabilities and will document my findings in future articles.

After overcoming numerous challenges, I finally got this entire setup working.

PS: Working on this was really exhausting... I feel like an emotionless internet mover, constantly searching through search engines and official docs (and now GPT), trying different combinations until a functional solution emerges 😂

This article documents my process, and the new Docker deployment solution will soon be merged into the master branch of the DjangoStarter project.

Some Concepts

Deep understanding of how Docker works can help avoid many issues.

After encountering many pitfalls and getting headaches, I always deeply feel that I'm still not skilled enough.

Multi-stage Build

In a Dockerfile, the functionality of using multiple FROM instructions and assigning a name to each FROM via the AS keyword is known as a "multi-stage build".

Multi-stage build allows you to use multiple base images within a single Dockerfile and selectively copy build artifacts from one stage to another during the build process. The advantage of this is that you can use a larger image in the earlier stages for build work (e.g., compiling code), and then in the final stage, retain only the necessary files, placing them into a smaller base image to reduce the size of the final image.

Multi-stage build greatly simplifies the process of building complex images and also helps keep the final image size small.

Here are some important points to note:

• Dependencies between stages do not need to be explicitly declared; during the build, stages can be built concurrently. The build will only wait for a dependent stage to complete upon encountering a COPY --from=<stage_name> or COPY --from=<stage_index> statement.
• Each FROM instruction initiates a new build stage. These stages are independent; environment variables, file system states, etc., from one stage are not automatically passed to the next stage. Each stage can choose to copy build artifacts from any previous stage.
• In a multi-stage build, instructions in the Dockerfile are executed sequentially. The earlier stages do not automatically pass files or states to later stages unless the COPY --from=<stage> instruction is explicitly used.

Docker Volumes Mechanism

Docker volumes are a data volume management mechanism provided by Docker, designed for persistent storage and data sharing between containers and between containers and the host system.

• Data stored in volumes persists even when the container is deleted.
• The same volume can be mounted to multiple containers to enable data sharing.
• Volume data is stored in a specific location on the Docker host, and containers access this data only through mount points.
• Priority (very important): A volume mounted at runtime will override the contents of the corresponding path inside the container. (It was due to this override issue that the files in static-dist couldn't be updated for a long time.)

Therefore, in scenarios involving shared static files, maintaining data consistency becomes crucial.

dockerfile

Let me show you the final dockerfile I came up with.

ARG PYTHON_BASE=3.11
ARG NODE_BASE=18

Python Build

FROM python:$PYTHON_BASE AS python_builder

Setting Python Environment Variables

ENV PYTHONUNBUFFERED=1

Disable Update Check

ENV PDM_CHECK_UPDATE=false

Setting Up Domestic Source

RUN pip config set global.index-url https://mirrors.cloud.tencent.com/pypi/simple/ &&
# Install pdm pip install -U pdm &&
# Configure mirror pdm config pypi.url "https://mirrors.cloud.tencent.com/pypi/simple/"

Copy Files

COPY pyproject.toml pdm.lock README.md /project/

Install Dependencies and Project to Local Package Directory

WORKDIR /project RUN pdm install --check --prod --no-editable

Node Build

FROM node:$NODE_BASE as node_builder

Configure mirror && Install pnpm

RUN npm config set registry https://registry.npmmirror.com &&
npm install -g pnpm

Copy dependency files

COPY package.json pnpm-lock.yaml /project/

Install Dependencies

WORKDIR /project RUN pnpm i

gulp build

FROM node:$NODE_BASE as gulp_builder

Configure Mirror && Install pnpm

RUN npm --registry https://registry.npmmirror.com install -g gulp-cli

Copy Dependency Files

COPY gulpfile.js /project/

Get Packages from Build Stage

COPY --from=node_builder /project/node_modules/ /project/node_modules

Copy Dependency Files

WORKDIR /project RUN gulp move

Django Build

FROM python:$PYTHON_BASE as django_builder

COPY . /project/

Copying Packages from Build Stages

COPY --from=python_builder /project/.venv/ /project/.venv COPY --from=gulp_builder /project/static/ /project/static

WORKDIR /project ENV PATH="/project/.venv/bin:$PATH"

Handling Static Assets

RUN python ./src/manage.py collectstatic

Running Stage

FROM python:$PYTHON_BASE-slim-bookworm as final

Retrieve packages from build stage

COPY --from=django_builder /project/.venv/ /project/.venv
COPY --from=django_builder /project/static-dist/ /project/static-dist
ENV PATH="/project/.venv/bin:$PATH"
ENV DJANGO_SETTINGS_MODULE=config.settings
ENV PYTHONPATH=/project/src
ENV PYTHONUNBUFFERED=1
COPY src /project/src
WORKDIR /project

Multi-stage Build

This Dockerfile contains the following build stages, which can be roughly understood by their names:

• python_builder: Installs the pdm package manager and Python dependencies.
• node_builder: Installs frontend dependencies.
• gulp_builder: Uses the gulp tool to process and integrate frontend assets.
• django_builder: Extracts Python dependencies and frontend assets from the build outputs of the previous containers, then performs tasks like collectstatic (currently only this, but may include others in the future).
• final: The final stage used for running and generating the image.

Key Points

In the process of debugging this Dockerfile, there are several key points to keep in mind:

• Do not use slim images during the build stage to avoid encountering strange issues due to an overly minimal environment.
• Starting from the django_builder stage, which involves running Python, you must add the Python path from the virtual environment (venv) to the system's PATH environment variable.
• Because DjangoStarter v3 uses a new project structure with all source code located in the src directory under the root, you must add this directory to the PYTHONPATH environment variable in the final stage. Otherwise, you may face obscure package import issues (I encountered this while using uwsgi).
• Also in the final stage, I set environment variables like DJANGO_SETTINGS_MODULE and PYTHONUNBUFFERED. Whether they are necessary or not, I set them to match the development environment to preempt potential problems.

nginx configuration

Almost forgot about this nginx configuration (in fact, I posted a WeChat article and forgot to include this part).

It's actually very simple. Just make a few simple modifications based on the examples provided by channels.

upstream channels-backend {
    server app:8000;
}

server {
    listen 8001;
    server_name localhost;

    charset utf-8;
    # Limit user upload file size
    client_max_body_size 100M;

    location / {
        try_files $uri @proxy_to_app;
    }

    location @proxy_to_app {
        proxy_pass http://channels-backend;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $server_name;
    }

    # Static resource path
    location /static {
        alias /www/static-dist;
    }

    # Media resources, user upload file path
    location /media {
        alias /www/media;
    }
}

access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log warn;

server_tokens off;

docker-compose

I used some environment variables in the compose configuration.

This avoids having to modify the project name and other details for each project.

First, here's the configuration, followed by an introduction.

services:
  redis:
    image: redis
    restart: unless-stopped
    container_name: $APP_NAME-redis
    expose:
      - 6379
    networks:
      - default
  nginx:
    image: nginx:stable-alpine
    container_name: $APP_NAME-nginx
    restart: unless-stopped
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf
      - ./media:/www/media:ro
      - static_volume:/www/static-dist:ro
    depends_on:
      - redis
      - app
    networks:
      - default
      - swag
  app:
    image: ${APP_IMAGE_NAME}:${APP_IMAGE_TAG}
    container_name: $APP_NAME-app
    build: .
    restart: always
    environment:
      - ENVIRONMENT=docker
      - URL_PREFIX=
      - DEBUG=true
    #    command: python src/manage.py runserver 0.0.0.0:8000
    command: >
      sh -c "
      echo 'Starting the application...' &&
      cp -r /project/static-dist/* /project/static-volume/ &&
      exec daphne -b 0.0.0.0 -p 8000 -v 3 --proxy-headers config.asgi:application
      "
    volumes:
      - ./media:/project/media
      - ./src:/project/src
      - ./db.sqlite3:/project/db.sqlite3
      - static_volume:/project/static-volume
    depends_on:
      - redis
    networks:
      - default

volumes:
  static_volume:

networks:
  default:
    name: $APP_NAME
  swag:
    external: true

A Few Key Points

nginx

I added an nginx container here to provide web services because I had always been using uwsgi before. However, uwsgi's socket mode does not support static files. To have uwsgi serve static files, you would have to use HTTP mode, but that would lose uwsgi's advantages. Therefore, I've always used nginx to handle static file access.

That said, I no longer install nginx directly on the server. Instead, I now use the swag container for a one-stop solution. So, a web server must be added in the compose file. Given that, I'll stick with nginx since I'm more familiar with it. This issue was previously discussed in this article (Project Completion Summary - Django-React-Docker-Swag Deployment Configuration).

image name

This time, the built image is finally named properly...

Paving the way for the next step: pushing to Docker Hub.

Data Sharing

Previously, I always executed collectstatic locally and then uploaded to the server, so static files weren't an issue. However, this approach was far from elegant and not friendly for CI/CD either.

With DjangoStarter v3, this pain point is also resolved. Front-end dependencies and resource management are now integrated into Docker's build stage. Therefore, Docker volumes are needed to share these static resources between the app and nginx containers.

As mentioned at the beginning, volumes have higher priority, which means even if some static resources are modified later, a rebuild and restart will still use the older version already mounted in the volume. To address this, I mount the app container's /project/static-volume to the shared static_volume, and then copy the files from /project/static-dist into it. I don't directly mount to /project/static-dist, as that would cause the old data in static_volume to overwrite the files generated by collectstatic.

There are other ways to handle this, such as cleaning out the files in static_volume before each startup and then mounting. However, I tried it and it was a bit of a hassle, so I gave up.

Choosing an Application Server

Django (or rather, backend frameworks in the Python ecosystem) doesn't have built-in servers like Kestrel or Tomcat, unlike .NET Core or Spring Boot. Therefore, deploying to a production environment requires the help of an application server.

Common Django application servers include uWSGI, Gunicorn, Daphne, Hypercorn, Uvicorn, and others.

Previously, I always used uWSGI, which is a powerful and highly configurable WSGI server supporting multi-threading, multi-process, and asynchronous work modes, along with extensive plugin support. Its high performance and flexibility make it a top choice for many large projects. However, uWSGI's configuration is relatively complex and may not be beginner-friendly. The high complexity of uWSGI can sometimes lead to configuration errors or difficult debugging in certain scenarios.

In this article's examples, using uwsgi for deployment kept causing issues. Since I had used channels in a previous project, this time I opted for Daphne, which is a core part of the Django Channels project. It is an ASGI server that supports HTTP and WebSocket. It also supports new features like HTTP/2, which is quite good.

If you need to handle WebSocket connections or use Django Channels, Daphne is an ideal choice. Daphne can be combined with reverse proxy servers like Nginx to handle static files and SSL termination.

Next, I also plan to try Gunicorn and the ASGI-based Uvicorn. Uvicorn seems to be used more frequently in FastAPI projects. Since Django has supported asynchronous functionality (i.e., ASGI) since version 3.0, maybe it's time to abandon traditional WSGI servers?

(Actually, Daphne works quite well, and I'm a bit reluctant to try other servers.)

I'll copy some introductions for the others.

Gunicorn (Green Unicorn) is a lightweight WSGI server designed for simplicity and ease of use. Gunicorn's configuration is straightforward, and it provides good performance by default, making it widely used in both development and production environments. Compared to uWSGI, Gunicorn has a lower learning curve and is suitable for most Django projects.

Hypercorn is a modern ASGI server that supports multiple protocols such as HTTP/2, WebSocket, and HTTP/1.1. It can run under various concurrency modes, including asyncio and trio. Hypercorn is ideal for projects that require Django Channels, WebSocket support, or plan to support HTTP/2 in the future.

Uvicorn is a lightweight, high-performance ASGI server based on asyncio, designed for speed and simplicity. It supports HTTP/1.1 and WebSocket and is also an early adopter of HTTP/2. Uvicorn is often used with FastAPI, but it is also suitable for Django, especially when using Django's asynchronous features. Uvicorn's configuration is relatively simple, and its startup speed is very fast, making it suitable for development environments and production environments that require asynchronous processing.

ASGI is the future trend, so it's time to move on from WSGI...

Summary

I spent a lot of time tinkering around earlier, but looking back, the summary is actually not that complicated—just a few key points. However, due to a lack of deep understanding of Docker, WSGI, ASGI, and Python runtime mechanisms, I ended up encountering many pitfalls. In the end, I pieced together this Docker solution through trial and error… 😂

Anyway, that's about it. Moving forward, I’ll continue refining DjangoStarter v3, and there are also some other Django development experiences worth recording recently.

References

• How to Create Efficient Python Docker Images (如何创建高效的 Python Docker 镜像) - https://www.linuxmi.com/python-docker-images.html