Experience Share: Fast Server Backup and Reinstall After Finding a Mining Trojan (Tencent Cloud Platform)

Preface

My cloud server had been compromised with a mining trojan before. At the time I cleaned up the trojan and left it at that.

Recently I found it again—several servers had even been infected with mining trojans. This time I had no choice but to do a thorough cleanup and reinstall all of them 😂

PS: In the AI era, everyone is relying more and more on AI for programming (including us). Network security deserves even more attention. I'll write a separate post later sharing our practical experience.

This article covers:

• Connecting to COS on the server
• Creating backups & a full disk image
• Some general server configs I usually use
• New system setup & server security hardening

Connecting to COS

Tencent Cloud COS has a free quota, which works well for temporarily storing server backups.

You can manage COS files on the server with the coscli tool. Official docs: https://cloud.tencent.com/document/product/436/63144

Configuring coscli requires these settings:

• A sub-account ID with COS management permissions, corresponding to coscli's APPID
• SecretId and SecretKey created via the sub-account
• bucket name
• bucket endpoint (e.g. cos.ap-guangzhou.myqcloud.com)

Here are the steps:

Create a bucket

Create a bucket in the same region as the server

Create a sub-account

Attach this policy QcloudCOSFullAccess

Record the sub-account ID

Then create an API key—that's the SecretId and SecretKey you'll need next

Download and configure

Once the prep work is done, download coscli. This tool is written in Go, so you can just download and run it.

wget https://cosbrowser.cloud.tencent.com/software/coscli/coscli-linux-amd64
mv coscli-linux-amd64 coscli
chmod +x coscli

Run the configuration

./coscli config init

You'll enter an interactive interface. Follow the prompts and only enter the settings prepared above; leave the rest as defaults.

For explanations of the specific settings, see the official docs: https://cloud.tencent.com/document/product/436/63144

Common commands

This coscli tool's commands are basically the same as Linux file management, for example:

# 列出文件
./coscli ls cos://<bucket-name>/[path]

In this article's backup scenario, we mainly use the cp command to upload local files to COS

./coscli cp ./example.txt cos://<bucket-name>/

Swap the two paths to download

./coscli cp cos://<bucket-name>/example.txt ./

Creating backups

These days the servers all deploy various apps with docker, with volumes mounted to local directories.

The backup mainly packages these configs and data, compresses them, and uploads them to COS

At first I used a one-liner to handle this

# 压缩并显示进度
# tar -cf -: 创建压缩流并输出到标准输出
# pv: 计算大小并显示进度条
# gzip: 进行压缩
tar -cf - $SOURCE_DIRS | pv -s $(du -sb $SOURCE_DIRS | awk '{sum+=$1} END {print sum}') | gzip > $BACKUP_NAME

But I found that some docker volume files were created by root, so this command didn't have enough permissions and couldn't pack them

So I later made some changes and ended up with the one-click backup-and-upload script below.

One-click script

In the first part, define the variables: SOURCE_DIRS is the directories to pack, TARGET_COS is, COSCLI_PATH is the path to the coscli binary

#!/bin/bash

# 1. 定义变量
BACKUP_NAME="backup_$(date +%Y%m%d_%H%M%S).tar.gz"
SOURCE_DIRS="apps projects"
TARGET_COS="cos://<bucket_name>"
COSCLI_PATH="$HOME/temp/coscli"

echo "🚀 开始备份目录: $SOURCE_DIRS ..."

# 2. 压缩并显示进度
# tar -cf -: 创建压缩流并输出到标准输出
# pv: 计算大小并显示进度条
# gzip: 进行压缩
# tar -cf - $SOURCE_DIRS | pv -s $(du -sb $SOURCE_DIRS | awk '{sum+=$1} END {print sum}') | gzip > $BACKUP_NAME

# (1) 计算总大小（用于 pv 进度条显示）
SIZE=$(sudo du -sb $SOURCE_DIRS | awk '{sum+=$1} END {print sum}')

# (2) 压缩并显示进度
# 使用 sudo tar 确保能读取所有文件
sudo tar -cf - $SOURCE_DIRS | pv -s $SIZE | gzip > $BACKUP_NAME

# (3) 修改压缩包权限（可选）
# 因为是 sudo 生成的，文件所有者会变成 root，这里可以改回 ubuntu 方便后续管理
sudo chown $USER:$USER $BACKUP_NAME

echo "✅ 压缩完成: $BACKUP_NAME"

# 3. 上传到腾讯云 COS
echo "📤 正在上传到腾讯云 COS..."
$COSCLI_PATH cp $BACKUP_NAME $TARGET_COS

# 4. (可选) 清理本地压缩包
rm $BACKUP_NAME
# echo "🧹 已清理本地临时文件。"

echo "🎉 备份任务已完成！"

gzip's compression ratio is decent; the size dropped by more than 50%

Backing up the database

Although the database volumes were backed up, there can be some pitfalls on startup due to version issues. To be on the safe side, export a separate database backup.

I run PostgreSQL in docker

Export the data with the following command:

docker exec -t pgsql pg_dumpall -U postgres > pgsql_full_backup.sql

For convenience, I also wrote a pgsql-dump function; details below.

Creating a disk image

My idea is to quickly restore production services, then create a disk image of the server for analysis and investigation in a VM.

Images on Tencent Cloud can't be downloaded, so I chose to create an image with dd, which is more flexible.

Some tricks:

Before starting, stop all docker containers, then clean up storage space. Most of the usage should be docker

docker system prune -a

There may be other cases of space usage; you can refer to my previous article for analysis: The Mystery of Vanishing Disk Space: A Full Walkthrough of Linux Server Storage Troubleshooting and Optimization

Before making a full disk image, fill with zeros first so the compressed image will be smaller

dd if=/dev/zero of=/zero.dat bs=1M status=progress

The command above will keep going until remaining space is full; delete the file after it finishes

The generated image cannot be placed on the current disk (e.g. /dev/vda), or you'll end up nesting it.

In the end I set up an SSH pipe and transferred it directly to local

ssh user@server "sudo dd if=/dev/vda bs=4M status=none | gzip -c" > server_evidence.img.gz

Windows can't display progress, so I used Windows Terminal split panes (shortcut: by default Alt + Shift + Plus (+) is vertical split, Alt + Shift + Minus (-) is horizontal split)

# 每隔 5 秒刷新一次文件大小
while($true) { 
    Get-Item .\server_evidence.img.gz | Select-Object Name, @{Name="Size(MB)";Expression={$_.Length / 1MB}}; 
    Start-Sleep -Seconds 5 
}

After it actually finished, Gzip's compression ratio was pretty good—a 20G server disk compressed down to a 6G image file

PS: I'll write an article introducing tmux and Windows Terminal some other day

alias config

To simplify command operations, you can put these in ~/.bashrc

# some more ls aliases
alias ll='ls -alFh'
alias la='ls -A'
alias l='ls -CF'

# docker
alias compose='docker compose'
alias dcrestart='docker compose down && docker compose up -d --build && docker compose logs -f'
alias headscale='docker exec -it headscale headscale'

For a more advanced setup, if you have a lot of them, you can create a separate file to manage them, which is easier to maintain.

It's recommended to create a .bash_aliases file in the home directory, then add this to ~/.bashrc:

if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi

Then dump all the aliases into .bash_aliases. That way the main config file stays very clean.

Functions

Some convenient functions, also put in ~/.bashrc.

After using them for a while I found there were more and more functions, so I needed to think about a more suitable centralized management solution. Currently I think two approaches are fairly viable:

• python (typer, cli tools)
• Justfile/Taskfile

But that's a story for later; I'll organize it another time

One-click PostgreSQL backup export

pgsql-dump() {
    # 参数 1: 用户名 (默认为 postgres)
    # 参数 2: 文件名 (默认为带时间戳的文件名)
    local user="${1:-postgres}"
    local timestamp=$(date +%Y%m%d_%H%M%S)
    local file="${2:-pgsql_backup_${timestamp}.sql}"

    # 检查容器是否正在运行
    if ! docker ps --format '{{.Names}}' | grep -q "^pgsql$"; then
        echo "Error: Docker container 'pgsql' is not running."
        return 1
    fi

    echo "Exporting database with user [${user}]..."
    
    # 执行导出
    if docker exec -t pgsql pg_dumpall -U "$user" > "$file"; then
        echo "Successfully exported database to: ${file}"
    else
        echo "Error: Database export failed."
        return 1
    fi
}

Smart extract & compress

Forgot whether it's tar -zxvf or -jxvf? This function automatically picks the extraction tool based on the file extension.

extract() {
    if [ -f "$1" ] ; then
        case "$1" in
            *.tar.bz2)   tar xvfj "$1"     ;;
            *.tar.gz)    tar xvzf "$1"     ;;
            *.bz2)       bunzip2 "$1"      ;;
            *.rar)       unrar x "$1"      ;;
            *.gz)        gunzip "$1"       ;;
            *.tar)       tar xvf "$1"      ;;
            *.tbz2)      tar xvfj "$1"     ;;
            *.tgz)       tar xvzf "$1"     ;;
            *.zip)       unzip "$1"        ;;
            *.Z)         uncompress "$1"   ;;
            *.7z)        7z x "$1"         ;;
            *)           echo "'$1' cannot be extracted via extract()" ;;
        esac
    else
        echo "'$1' is not a valid file"
    fi
}

Compression command parameters (like tar) are even harder to remember than extraction (e.g. cvzf, cjvf, and so on). This function can automatically call the corresponding compression tool based on the desired file extension.

No more memorizing those tedious parameters—just tell it what format you want:

• Compress to Gzip: pack project.tar.gz ./my_folder
• Compress to Zip: pack images.zip pic1.jpg pic2.jpg
• Compress to 7z: pack backup.7z /var/www/html

pack() {
    # 用法: pack <文件名.扩展名> <目标文件或目录>
    if [ $# -lt 2 ]; then
        echo "Usage: pack <archive_name.ext> <target_files_or_dirs>"
        echo "Supported extensions: .tar.gz, .tar.bz2, .zip, .tar, .7z"
        return 1
    fi

    local archive="$1"
    shift
    local targets="$@"

    case "$archive" in
        *.tar.gz|*.tgz)  tar -cvzf "$archive" $targets ;;
        *.tar.bz2|*.tbz2) tar -cvjf "$archive" $targets ;;
        *.zip)           zip -r "$archive" $targets     ;;
        *.tar)           tar -cvf "$archive" $targets   ;;
        *.7z)            7z a "$archive" $targets       ;;
        *)               echo "Error: Unknown extension for '$archive'" ;;
    esac
}

On Linux, different compression formats suit different scenarios. Understanding their differences helps you choose the right tool:

Quick Backup

Before changing a config, you often need to quickly make a backup of some file.

Usage: backup nginx.conf -> directly generates nginx.conf.20260303_1327.bak

backup() {
    # 快速备份文件：filename -> filename.20240520.bak
    local file="$1"
    if [ -f "$file" ] || [ -d "$file" ]; then
        cp -r "$file" "${file}.$(date +%Y%m%d_%H%M%S).bak"
        echo "Backup created: ${file}.$(date +%Y%m%d_%H%M%S).bak"
    else
        echo "Error: '$file' not found."
    fi
}

Quickly Enter a Docker Container

No more typing the long docker exec -it ... /bin/bash. This function supports a custom shell (defaults to bash; if the container has no bash it will try sh).

dex() {
    local container="${1}"
    local shell="${2:-bash}"
    if [ -z "$container" ]; then
        echo "Usage: dex <container_name> [shell]"
        return 1
    fi
    docker exec -it "$container" "$shell" || docker exec -it "$container" sh
}

Find Files and Search Their Contents

Search for a specified string in all files under the current directory.

Usage: ftext "API_KEY" ./src

ftext() {
    # 参数 1: 搜索词，参数 2: 目录 (可选，默认当前)
    grep -rnw "${2:-.}" -e "$1"
}

docker configuration

/etc/docker/daemon.json

Configure an image accelerator and limit log size

{
  "registry-mirrors": [
    "https://mirror.ccs.tencentyun.com"
  ],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Reinstall

At this point the prep work is basically done. Next, pick a suitable image and reinstall (I recommend Debian).

I previously saw a colleague pull a wild trick: after connecting to the server, they used an official image to dd over the entire disk. The reinstalled system is a pure official system, not Tencent Cloud's modified version. I haven't tested this approach yet though.

PS: In Linux circles this is usually called “DD reinstall”. It's an advanced technique to bypass the cloud vendor's preset images and achieve “freedom of system installation”.

Advantages

• Completely clean: Images provided by cloud vendors usually come with monitoring scripts (such as tat_agent, barad_agent) and security components. After a DD reinstall the system is entirely yours, with no vendor leftovers.
• Bypass restrictions: Some cloud vendors don't offer a particular OS version (for example if you insist on some old Debian or a specific Alpine Linux). With DD you can ignore the console options and install whatever system you want.
• Remove “modifications”: Some vendors modify the kernel or software sources. A DD operation lets you get a 100% official, original experience.

There are now many excellent open-source “one-click DD scripts” in the tech community (such as scripts by Qiushui Yibing or Vicer). They automatically handle NIC drivers and network configuration for you, which is much safer than running the dd command by hand.

New System Configuration

I originally wanted to split this into two articles because of length, but that's too much hassle, so I'll briefly record it here as well.

I'll expand on it later when I have time.

Security Hardening

When configuring a new system, my habit is that the first step is security hardening.

The security part could actually be covered in much more detail. I'm still learning while practicing, so this article can only briefly record it.

Create a Regular User

Debian defaults to the root user, so first create a regular user, set a password, and add them to the sudo group.

adduser user
usermod -aG sudo user

There's a nasty pitfall here: Tencent Cloud seems to have modified the sudo config and stripped the sudo group's privileges 😂, so even after adding the user to the sudo group they still have no permissions.

So you have to run visudo and add the line %sudo ALL=(ALL:ALL) ALL. Now I really get why people want to "DD reinstall".

SSH Restrictions

Edit /etc/ssh/sshd_config, mainly to do a few things: disable password login, disable root login, and change the default port.

# 随便改个端口
Port 12837
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes

Restart the service to take effect sudo systemctl restart ssh

Update Packages

With the above done, you're finally not running naked. Now start updating software.

sudo apt update
sudo apt upgrade -y

Advanced

Here are some advanced security solutions/tools. Due to length I'll just list them briefly; I'll write separate articles on the specific configs later.

Firewall ufw

Much more convenient than iptables

# 默认拒绝
sudo ufw default deny incoming
sudo ufw default allow outgoing
# 允许端口
sudo ufw allow 2222/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
# 启动
sudo ufw enable
# 查看
sudo ufw status

Oh right, Tencent Cloud's management console has another layer of firewall. Remember to block ICMP packets there so others can't ping the server, which can reduce a lot of scanning. The SSH port can be allowed only for your own computer's IP address, which is more secure.

fail2ban

An essential security tool for servers, to prevent brute-force attacks.

The default config is usually enough; it automatically protects the SSH port.

sudo apt install fail2ban
# 查看状态
sudo fail2ban-client status

Rootkit Detection

sudo apt install rkhunter
sudo rkhunter --update
sudo rkhunter --check

sudo apt install chkrootkit
sudo chkrootkit

Periodic Automatic Security Updates

This is optional. If you have a good memory and maintain the server regularly, you can update manually.

sudo apt install unattended-upgrades
sudo dpkg-reconfigure unattended-upgrades

Common Tools

Monitor CPU, network connections, process network usage, etc.

sudo apt install htop iftop nethogs

I won't list the rest; go with whatever you're used to.

Install Docker

All app deployment depends on this, but installing it in China is a bit of a pain now. For Tencent Cloud servers you can refer to the official documentation

I'll just copy-paste a version here; run the following commands in order.

sudo apt-get update
sudo apt-get install ca-certificates curl -y
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://mirrors.cloud.tencent.com/docker-ce/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
echo   "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://mirrors.cloud.tencent.com/docker-ce/linux/debian/ \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" |   sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl start docker

As for the accelerator and such, you can refer to the docker configuration section in this article.

Finally after installing, add the current user to the docker group so you don't have to sudo docker every time.

https://docs.docker.com/engine/install/linux-postinstall/

sudo usermod -aG docker $USER

Miscellaneous

The rest is restoring the alias, 函数, 工具 and other backed-up configs from earlier in this article into the various config files.

Oh right, Debian has another pitfall: when a regular user uses sudo, /usr/sbin is not in the environment variables (the directory for system admin commands like usermod, ip, fdisk, etc.). By default it's only included in the root user's PATH.

So you need to manually add it to ~/.profile

export PATH=$PATH:/usr/sbin:/sbin

Wrap-up

That's it. These past few days of wrestling with the server have been exhausting.

The configs themselves are simple, but actually doing them is tedious and time-consuming. I never had a systematic record before, so every time I ran into configuring/reinstalling a new server I'd keep asking AI or reading docs, which was inefficient.

This article fully records the process so I can refer to it better in similar situations later. Next I also need to gradually build a more convenient, usable workflow and keep improving work efficiency.