Preface

Hello! Long time no see~

I've been thinking about a career transition recently and haven't updated my articles for a while.

But seeing that the video editing tool Clipify I developed earlier has gained many stars reminded me that the promises I made before have only been partially realized. So, I used my spare time over the weekend to refactor the Clipify project's interface and develop some new features.

During this process, I realized that the current tech stack of WinForms + Blazor is like strapping a rocket to a carriage—giving off a steampunk retro vibe. Although the interface developed with front-end technologies looks great, WinForms' inability to be cross-platform isn't a long-term solution.

So, I started trying to migrate to MAUI. Although it still doesn't support Linux, at least it works on Mac, which counts as being cross-platform, I guess…

However, before I could even get the project running, I accidentally committed a bunch of static resources:

Clipify.Maui/wwwroot/lib/
Clipify.Maui/wwwroot/css/*.min.css
Clipify.Maui/wwwroot/temp/

This made git push super slow and would consume a lot of storage space.

Fortunately, with the help of the AI assistant, I managed to resolve this issue.

But I thought I can't keep bothering the AI assistant all the time. Since the problem was solved, I should document it so that I can handle similar situations more easily in the future.

Using Git Native Commands

This is actually a well-known issue

Many articles you can find online will tell you that you can solve it using the following commands

git rm -r --cached Clipify.Maui/wwwroot/lib
git rm --cached Clipify.Maui/wwwroot/css/*.min.css
git rm -r --cached Clipify.Maui/wwwroot/temp

These commands will remove the committed files, but will not delete local files

But even after I did this, it still didn't work, and git push remained very slow

Analysis of the Cause

So the question is, why doesn't git rm --cached work?

The reason is that git rm --cached only removes the Git tracking of the specified file from the current commit (working directory); it does not modify the past commit history.

It solves the problem of "stop tracking from now on," but for files it has tracked historically, Git will still keep them in the object database (.git/objects).

In the previous few commits, I already committed folders like Clipify.Maui/wwwroot/lib.

So, for this scenario, this command is useless. 😂

In simple terms, Git is a version control tool; it does not forget your historical commits. Even if you remove the lib/ folder now with git rm --cached, Git still remembers that it was included in the Nth commit before. Therefore, the .git directory still holds those large blob files, and they will still need to be transferred during a push.

Solution

So how do we solve this?

The method I saw before was writing a script to loop through and delete committed files from each commit.

Now, it's no longer that troublesome, thanks to the git-filter-repo tool, a Python tool used to rewrite Git history.

https://github.com/newren/git-filter-repo

Thanks to scoop, which allows me to get a similar software installation experience on Windows as on Linux/MacOS.

# Install Git Filter Repo
brew install git-filter-repo  # macOS
scoop install git-filter-repo # Windows
# Or install with Python
pip install git-filter-repo

Usage

git filter-repo --path Clipify.Maui/wwwroot/lib/ --invert-paths --force

This command will completely delete all files and commit records related to this path from history

After rewriting the entire Git commit history, the repository size will be significantly reduced, git push will be faster, and the specified files in historical commits will be completely removed. You may need to use --force to force push the code.

After going through this process, using git push to submit was noticeably much faster for me.

Extension: Batch Deletion

The git filter-repo --path parameter specifies a single path and does not support wildcards.

git filter-repo actually has a hidden feature: supports --path-glob (glob wildcard pattern), which is more flexible than --path.

For example, to delete all .png files under public/image-grid:

git filter-repo --path-glob 'public/image-grid/*.png' --invert-paths --force

You can also delete recursively in subdirectories:

git filter-repo --path-glob 'public/image-grid/**/*.png' --invert-paths --force

Extension: How to analyze large files in git commit history

Using [git rev-list + git verify-pack] can find large objects (large files) in Git commit history.

However, the native git commands are quite complex.

I still choose to rely on the power of tools, this time the git-sizer tool.

git-sizer is an official tool from GitHub, used to analyze issues like Git repository size, commit volume, etc.

brew install git-sizer

or

scoop install git-sizer

Install via cargo

cargo install git-sizer

PS：Thanks again to scoop!

Execute git-sizer directly in the project root directory.

This tool will output the following information:

• Largest commits
• Largest blobs (file content)
• Largest directories
• Branches containing large files
• Whether there are hidden bloated objects in history (hidden bloats)

The result I obtained using this tool looks like this:

Processing blobs: 2448                        
Processing trees: 218                        
Processing commits: 42                        
Matching commits to trees: 42                        
Processing annotated tags: 0                        
Processing references: 3                        
| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Biggest objects              |           |                                |
| * Trees                      |           |                                |
|   * Maximum entries      [1] |  1.40 k   | *                              |

[1]  77b7ff2aa0456193bee83eebe9dfb76ebd4f35ac (2d9cc135f1aff0863ce2ef8ceff3cbf6984ee499:Clipify.Maui/wwwroot/lib/font-awesome/svgs/solid)

This way, you can easily locate the largest files and slim down your git repository 🎉

Extension: Finding the Large File List

At first, I was thinking of using git commands combined with pipes.

But that was too much hassle.

In the end, git-filter-repo is amazing!!

One command solves it all.

git filter-repo --analyze

After running, a .git/filter-repo/ folder will be generated in the repository directory.

Inside, you will find:

• analysis/blob-shas-and-paths.txt - A file list with size statistics.
• analysis/directories-all-sizes.txt - A directory list with size statistics.

There are other files too; if you're interested, you can explore them yourself. I mainly focused on these two files.