Foreword

I recently took over a computer vision project.

The code is a complete mess (屎山), but that's not my concern since I'm not looking at it anyway.

My main task was to build the Docker image and test deployment compatibility.

This shouldn't have been too difficult.

However, the network environment on domestic servers is terrible, requiring the configuration of various image sources (Docker images, apt images, pip images), each with its own issues. The result was a combination that consistently failed to build successfully.

Then a thought occurred to me: why not use GitHub Actions to build the image? With GitHub's server network, all the problems would be solved, wouldn't they?

After trying it, it indeed worked, and the image was built successfully.

Along the way, I did run into some pitfalls, though, such as issues with large files.

In previous projects, I never encountered large files because I never committed PyTorch model files to Git along with the code!

Anyone who does that is making a choice that's a bit… questionable. 😅

GitHub's Restrictions on Large Files

GitHub File Size Limits:

• Files larger than 50MB will trigger a warning.
• Files larger than 100MB will be directly rejected during push.

The result is:

Writing objects: 100% (304/304), 463.41 MiB | 3.23 MiB/s, done. Total 304 (delta 32), reused 0 (delta 0), pack-reused 0 (from 0) remote: Resolving deltas: 100% (32/32), done. remote: warning: File fire_detection/yolov5l.pt is 88.50 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB remote: error: Trace: 52267c96d371181694b02dad5eebead817ce6d985e2f8d356c2b458adfbb2a09 remote: error: See https://gh.io/lfs for more information. remote: error: File runs/detect/fire_smoke_detection/weights/best.pt is 166.91 MB; this exceeds GitHub's file size limit of 100.00 MB remote: error: File runs/detect/fire_smoke_detection/weights/best.pt is 250.32 MB; this exceeds GitHub's file size limit of 100.00 MB remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com. To github.com:cppstudio/fire-detection.git ! [remote rejected] master -> master (pre-receive hook declined) error: failed to push some refs to 'github.com:studio/fire-detection.git'

The solution is to use Git LFS (Large File Storage), an official GitHub feature specifically designed for storing large files (model weights, binaries, audio/video).

However, this is also quite stingy—the free tier for GitHub LFS only provides 1GB of storage + 1GB of bandwidth. You'll need to purchase additional quotas if you exceed this limit (or use alternatives like HuggingFace, Baidu Cloud, or OSS for hosting model files).

If you're only storing .pt models, consider hosting the weights on HuggingFace and including the download link in your README.

Installing the git-lfs Tool

It can be easily installed on any system.

# Linux (Ubuntu/Debian)
```bash
sudo apt install git-lfs

macOS (brew)

brew install git-lfs

Windows (scoop)

scoop install git-lfs

Then initialize

```bash
git lfs install

Tracking Large File Types

For example, in the context of this document, .pt model files are large and can be specified for management by LFS.

git lfs track "*.pt"

This will generate a .gitattributes file in the repository, with content similar to:

*.pt filter=lfs diff=lfs merge=lfs -text

Remember to commit this .gitattributes file to git as well.

git add .gitattributes
git commit -m "track large files with Git LFS"

Resubmitting Large Files

If previous commits included large files

You need to resubmit these large files to Git LFS (Large File Storage).

Fortunately, git-lfs provides convenient commands to clean up history; otherwise, you'd have to use the method described in my previously posted article to clean up large files. See: How to Clean Up Accidentally Committed Large Files in Git?

# Automatically replace large files in history with `git lfs migrate`
```bash
git lfs migrate import --include="*.pt"

Force Push

git push origin master --force

After that, just proceed with the normal push operation.

```bash
git push origin master

GitHub will then store large files in Git LFS, instead of in the regular Git repository.

Pulling Large Files

After using Git LFS, large files stored in the repository (such as .pt) will only show a pointer file (a few dozen bytes that record the real file's location in LFS storage) when performing a regular git clone.

To download the actual large file, Git LFS must be installed.

Specific Situation

• If Git LFS is not installed on the machine: After cloning, large files will be "pointer files" and cannot be opened or used.
• If Git LFS is installed on the machine: During cloning, large files will be automatically replaced with the actual files.
• If you have already cloned but large files were not pulled: You can manually run: git lfs pull

Therefore, usually, the project README will include a note:

## Note

This project uses [Git LFS](https://git-lfs.github.com/) to manage large files (such as model weights).

Please run the following commands first:

```bash
git lfs install
git lfs pull
```
## Summary

This experience of dealing with GitHub's large file issue has actually given me a few insights:

1. **GitHub's Limitations**: Files over 100MB are absolutely not allowed; don't even think about forcefully committing them.
2. **The Value of Git LFS**: The official solution is simple and easy to use, but the free tier is quite stingy; it's suitable for small-scale file management.
3. **Alternative Solutions**: If model files are often hundreds of MB, it's recommended to store them on dedicated platforms like HuggingFace, OSS, Baidu Netdisk, etc., and then provide download scripts or links in your code.
4. **Best Practices**: Code is code, models are models—don't mix them in commits; keeping your repository clean makes collaboration and deployment much easier later on.

To sum it up in one sentence: **Learn to use Git LFS for emergencies; but don't treat it as a free cloud storage.**

Additionally, one more comment: This project looks like a typical "college student assignment repository"—code is a mess, models are directly thrown into git, and it's probably some "research result" that a professor offloaded to a graduate student on a whim. Honestly, the fact that it runs at all is already a miracle 😂.