A Docker MySQL Failure in Production

The Problem

Yesterday afternoon I was about to go for afternoon tea when a frontend colleague suddenly said the API had gone down. I logged into Sentry and saw an error

(2005, "Unknown MySQL server host 'mysql' (-3)")

What does that mean? Can't connect to the database. The host is mysql because this is in a Docker environment

I SSH'd into the server to check. Nothing seemed wrong—the MySQL container was running normally

Strange

Then I restarted Docker, and found that the MySQL container hadn't restarted properly?

I manually ran docker stop, then docker compose up again

And it was a disaster—after the service came back up, all the MySQL data was gone...

Finding the Cause

I was a bit panicked, after all it was production data, but I also knew data couldn't just vanish into thin air. MySQL is a mature commercial database—it wouldn't make such a basic mistake. So the problem was almost certainly on our side...

Sure enough, when we previously used docker-compose to orchestrate the containers, we'd been lazy and hadn't added a volume mapping for the MySQL container

When we force-stopped it without committing first, the container data was "lost"

I used quotation marks because the data wasn't actually lost—it had become an orphan volume (a volume with no references?)

The Fix

So the situation was clear. Now I just needed to find the volume that the previous MySQL container had used, extract the data files from it, and remap them

The volume file path is /var/lib/docker/volumes/

There were a bunch of volume folders inside, so I had to go through them one by one

You can also run the du command and estimate based on size

Eventually I found a 2.9G directory, opened it, and saw that the data inside was MySQL's

That made things easy

I cd'd into this volume's directory and copied the data files out

cp -r _data /var/lib/mysql

Then I modified the previous docker-compose.yml config file

version: "3"
services:
  mysql:
    image: daocloud.io/mysql
    volumes:
      - /var/lib/mysql:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=1234
    expose:
      - 3306

Just add the volumes config and you're good~

Restart the container

docker-compose up -V

At first I didn't know I needed the -V flag. After mapping the volume, MySQL still had no data

Later I read the docs and learned that without -V it would keep using the original volume

Recap

Although the data was recovered, the problem wasn't fully solved. There's still a question left: how did the original failure happen? Why did the MySQL container suddenly become unreachable?

This isn't clear yet. I'll look at the binlog later to analyze it.

Finally, this incident taught a few lessons

• Don't be lazy—always map volumes in advance
• My Docker knowledge is lacking and half-baked; I need to study it systematically
• docker-compose management is still pretty primitive. Should we look for a chance to switch to a more modern container management approach?

References

• A guide to recovering lost data from a Docker MySQL container: http://i.lckiss.com/?p=4222