The Mystery of Disappearing Disk Space: A Complete Guide to Linux Server Storage Troubleshooting and Optimization

Preface

Recently, our online services have been experiencing some peculiar issues, such as static resources on the webpage failing to load, unexplained errors when requesting the backend, or Redis throwing errors...

When I SSH into the server, things seem even worse—just typing a command feels sluggish...

If I lacked experience before, I might have thought it was some complex technical problem. But as a long-time Linux user, I’ve already realized that behind all these anomalies lies the culprit: "Storage space full"!

So the question then becomes: "Where did all the disk space go?"

Troubleshooting

First, Get a General Idea Using df

First, let's use the most common command.

df -h

Let's take a look at the situation of each disk and mount point.

After running this command on my server, I discovered that the available space in the root directory was only a few hundred KB left…

Use du for Specifics

The du command lets you check the space usage of each subdirectory, which can then be combined with the sort command for ordering.

sudo du -h --max-depth=1 / | sort -hr

Parameter Explanation:

• Since we are directly scanning the root directory, root privileges (root权限) are required.
• --max-depth=1: Only display the usage of the current directory and its immediate subdirectories.
• sort -hr: Sort the results by size in descending order.

To view the total space usage of a directory:

du -sh /

A Better Tool

The du command mentioned earlier still isn't that convenient to use, mainly because it lists too much data.

This time I used the ncdu tool

ncdu is a more user-friendly disk usage analysis tool with an interactive interface.

However, many distributions do not include it by default, so you need to install it first:

sudo apt install ncdu

Analyzing Root Directory Usage with ncdu

sudo ncdu -x /

Using the -x parameter restricts the scan to the current filesystem and prevents crossing mount points. Since there are many mount points for other hard drives under the root directory, and based on the earlier analysis using the df command showing that the main hard drive is full, I only need to examine the main drive.

After ncdu starts, it scans the storage space of each file and then enters an interactive interface where you can intuitively see the space usage of each directory, sorted from largest to smallest.

The Culprit

You can easily see in ncdu that the /var/lib/docker directory occupies over 70% of the storage space. It's an absolute disaster!

Analyzing with du

sudo du -h --max-depth=1 /var/lib/docker | sort -hr

The approximate space usage after that would look something like this (just an example, not actual data):

10G     /var/lib/docker/overlay2
5G      /var/lib/docker/volumes
1G      /var/lib/docker/containers
500M    /var/lib/docker/images

Basically, things like Docker images, container logs, and volumes are what's eating up the disk space.

Cleaning Up Unused Docker Resources

Once you've identified the problem, the solution is straightforward.

First, stop the unused Docker containers.

Then, execute some of the cleanup commands provided by Docker.

Cleaning Up Unused Resources (Images, Containers, Volumes, and Networks)

Docker provides the docker system prune command to clean up unused resources.

docker system prune -a

• -a: Removes all unused images (including those not associated with any container).
• Note: This command will not remove resources that are still in use by running containers, so use it with caution.

Cleaning Unused Volumes Only

If /var/lib/docker/volumes is consuming a significant amount of space:

docker volume prune

Clean Up Unused Networks Only

If /var/lib/docker/network takes up more space:

docker network prune

Removing Unnecessary Containers, Images, and Volumes

Remove Containers That Are Not Running

docker container prune

Deleting Unused Images

docker image prune -a

Delete unused volumes

docker volume prune

Cleaning Up Old Images and Unused Tags

If you are using many images, many old versions may no longer be useful.

List images sorted by size

docker images --format "{{.Repository}}:{{.Tag}} {{.Size}}" | sort -k2 -h

Delete a specific image

docker rmi <image-id>

What Is This Huge Thing?

While analyzing the storage space usage, I also found a file that is ridiculously large. The following file is over 500GB...

/var/lib/docker/containers/e4b5a99b429a612885417460214ea40a6a49a3360c29180af800ff7aef4c03df/e4b5a99b429a612885417460214ea40a6a49a3360c29180af800ff7aef4c03df-json.log

Find the Container That Left the Mess

Let’s see which container is responsible for the mess.

The path to the log file contains the container ID: e4b5a99b429a612885417460214ea40a6a49a3360c29180af800ff7aef4c03df

Let’s find out which container that is:

docker ps | grep e4b5a99b429a

If the container is not found in the list, it may have already been stopped or removed. In such cases, you can use docker inspect to check the specific details:

docker inspect e4b5a99b429a

Analyzing Log Content

You can view the log content using tail or less to check for any abnormal output:

sudo tail -n 50 /var/lib/docker/containers/e4b5a99b429a612885417460214ea40a6a49a3360c29180af800ff7aef4c03df/e4b5a99b429a612885417460214ea40a6a49a3360c29180af800ff7aef4c03df-json.log

I won't paste the specific log content here, but it looks like there shouldn't be any issues—it's just the result of prolonged runtime, accumulated over time...

The method for handling this issue is discussed in the next section.

Cleaning up Docker log files

If /var/lib/docker/containers is taking up a large amount of space, it may be due to oversized container log files.

View Log Files

Each container's logs are stored in /var/lib/docker/containers/<container-id>/<container-id>-json.log.

Use the following command to find the largest log files:

sudo find /var/lib/docker/containers/ -type f -name "*.log" -exec du -h {} + | sort -hr | head -n 10

Manually Clearing Logs

Clear the log file for a specific container:

sudo truncate -s 0 /var/lib/docker/containers/<container-id>/<container-id>-json.log

Set Log File Size Limit

Limit log size in Docker's configuration file (recommended):

1. Edit the Docker configuration file (usually /etc/docker/daemon.json):

   sudo nano /etc/docker/daemon.json
   

2. Add or modify the following configuration:

   {
     "log-driver": "json-file",
     "log-opts": {
       "max-size": "10m",
       "max-file": "3"
     }
   }
   

   • max-size: Maximum 10 MB for a single log file.
   • max-file: Keep 3 log files.

3. Reload Docker:

   sudo systemctl restart docker
   

Migrate /var/lib/docker to a New Disk

You can mount /var/lib/docker to another disk to alleviate storage pressure on the current disk. This is a common practice, especially in scenarios with multiple disks.

Method 1: Directly Migrating /var/lib/docker to a New Disk

Stop the Docker Service

Before migrating the data, you need to stop the Docker service:

sudo systemctl stop docker

Move /var/lib/docker to a New Location

Assuming the new disk is mounted at /mnt/new-disk, execute the following command:

sudo mv /var/lib/docker /mnt/new-disk/docker

Create a Symbolic Link

Link the new path back to /var/lib/docker so that Docker continues to operate with its default path:

sudo ln -s /mnt/new-disk/docker /var/lib/docker

Start the Docker Service

sudo systemctl start docker

Verification

Run the following command to ensure Docker is functioning correctly:

docker info

Method 2: Modifying the Docker Configuration File to Specify a New Storage Location

Stop the Docker Service

sudo systemctl stop docker

Move /var/lib/docker to a New Disk

Migrate the existing data to the new disk mount point. For example, if the new disk is mounted at /mnt/new-disk:

sudo mv /var/lib/docker /mnt/new-disk/docker

Modify Docker Configuration

Edit the Docker configuration file (typically /etc/docker/daemon.json) to specify the new storage path:

sudo nano /etc/docker/daemon.json

Add or modify the following content:

{
  "data-root": "/mnt/new-disk/docker"
}

Start the Docker Service

sudo systemctl start docker

Verification

Check again if Docker is running correctly:

docker info | grep "Docker Root Dir"

You should see the new path (e.g., /mnt/new-disk/docker).

Method 3: Directly Mounting a New Disk to /var/lib/docker

If you want to use a new disk directly as the mount point for /var/lib/docker, you can follow these steps.

Format the New Disk

Assuming the new disk is /dev/sdb1, first format it and create a file system (e.g., ext4):

sudo mkfs.ext4 /dev/sdb1

Mount the New Disk

Mount the new disk to /var/lib/docker:

sudo mount /dev/sdb1 /var/lib/docker

Migrate Existing Data

If there is already data in the /var/lib/docker directory, you need to copy it to the new disk first:

sudo rsync -a /var/lib/docker/ /mnt/new-disk/

Then, mount the new disk back to /var/lib/docker.

Edit /etc/fstab for Automatic Mounting on Boot

Edit the /etc/fstab file and add a line for the mount configuration:

/dev/sdb1  /var/lib/docker  ext4  defaults  0  2

---

Notes

1. Data Migration Risk: When migrating or rebuilding /var/lib/docker, be sure to back up important data (such as persistent volume data (持久卷数据)).

2. Permission Issues: Ensure the permissions of the new directory are consistent with the original directory:

   sudo chown -R root:root /mnt/new-disk/docker
   

3. Check Mount Point: Ensure the new disk is mounted successfully and set up automatic mounting to avoid path loss after a system restart.

Using the above methods, you can successfully mount /var/lib/docker to another disk, alleviating storage pressure and optimizing the storage layout.

Rebuild /var/lib/docker

If space is still insufficient after cleanup, you can rebuild Docker's storage directory (this will delete all containers, images, and data)

Stop the Docker service:

sudo systemctl stop docker

Back up existing Docker data (optional):

sudo mv /var/lib/docker /var/lib/docker.bak

Create a new empty directory:

sudo mkdir /var/lib/docker

Start the Docker service:

sudo systemctl start docker

Summary

During the investigation of the Linux server disk space disappearing issue, I went through a complete practical exercise in storage analysis and optimization.

Key Steps Overview:

1. Preliminary Storage Usage Check
   • Utilized tools like du and ncdu to quickly identify directories with significant space usage.
   • Discovered that the /var/lib/docker directory was consuming a large amount of storage.
2. In-Depth Problem Identification
   • Located the specific container log file path and confirmed which container was generating excessive logs using the container ID.
   • Analyzed the log content further using docker inspect and docker logs.
3. Problem Resolution
   • Cleared the oversized container log file and immediately freed up space using the truncate command.
   • Modified the Docker configuration file (daemon.json) to limit log file size, preventing similar issues from recurring.
4. Verification and Optimization
   • After restarting the Docker service, verified that the service was operating normally.
   • Used docker system prune to clean up unused resources and planned a log management strategy.

Personal Insights and Reflections:

Resolving this issue provided me with profound insights into the following aspects:

• Importance of System Monitoring: Timely monitoring of storage usage can prevent issues from escalating.
• Best Practices in Log Management: Excessively growing log files are a common cause of storage consumption; it is essential to set reasonable log size limits.
• Efficient Use of Tools: Tools like du, ncdu, and Docker commands significantly enhance efficiency during troubleshooting.
• Routine Maintenance Habits: Regularly cleaning up unused container resources (such as stopped containers and unused images) helps maintain system health.

This practice not only resolved the disk space issue but also deepened my understanding of Linux system administration and Docker operations. In future operations, I will place greater emphasis on system monitoring and optimization to proactively prevent similar problems.