DEV Community

Cover image for Large SDK and GitHub
Frankie Yuen
Frankie Yuen

Posted on

Large SDK and GitHub

#devops #iot #productivity #github

When working with large SDKs, version control can quickly become a challenge. I have been working with rv1126b_linux6.1_sdk_v1.1.0 - totals over 13 GB, exceeding GitHub's 2 GiB per push limit, requiring staged commits.

To profile disk usage:

du -sh rv1126b_linux6.1_sdk_v1.1.0/* | sort -hr | head -20
Enter fullscreen mode Exit fullscreen mode

Top consumers:

4.2G    external
3.7G    prebuilts
2.0G    rtos
1.6G    kernel-6.1
1.1G    docs
954M    ubuntu
634M    tools
354M    debian
340M    device
242M    buildroot
160M    app
127M    u-boot
112M    yocto
80M     rkbin
51M     hal
Enter fullscreen mode Exit fullscreen mode

external and prebuilts dominate due to binaries and toolchains. GitHub rv1126 topic

Git LFS Setup

For files >100MB, enable Git LFS early:

git lfs install
git lfs track "*.bin" "*.img" "prebuilts/**/*" "external/**/*"
echo "prebuilts/**/* filter=lfs diff=lfs merge=lfs -text" >> .gitattributes
git add .gitattributes
Enter fullscreen mode Exit fullscreen mode

Track patterns for SDK binaries (e.g., firmware images, toolchains) to avoid bloat. Commit .gitattributes first. Prune unused LFS objects regularly: git lfs prune.

Git LFS + Sparse Checkout |
Git LFS Patterns |
Git LFS Security

Staged Pushing Workflow

Split initial commit via interactive rebase or directory batches:

# Repack for delta compression (helps with similar binaries)
git repack -a -d --depth=50 --window=250

# Push directories sequentially
git add external/ && git commit -m "Add external" && git push -u origin main
git add prebuilts/ && git commit -m "Add prebuilts" && git push
# Repeat for others <2GiB
Enter fullscreen mode Exit fullscreen mode

Monitoring & Optimization

Post-push verification:

git count-objects -vH -q  # Quick stats
git lfs ls-files          # List tracked LFS files
Enter fullscreen mode Exit fullscreen mode

Clean history if needed: git filter-repo --strip-blobs-bigger-than 100M (rewrites history; force-push). Enable sparse-checkout for clones: git sparse-checkout set prebuilts kernel.

Git LFS Guide |
Large Git Repos |
GitHub Discussion

Advanced: Repo Size Reduction

  • Split giant initial commit: git rebase -i --root, mark as edit, then git reset HEAD~ && git add <subset> && git commit. Split Commits Gist | StackOverflow Rebase
  • Multi-pack-index for large packs: git multi-pack-index repack --batch-size=1g. Git Repack
  • Exclude docs/build artifacts via .gitignore: external/docs/, build/. Sparse Checkout

These techniques keep operations efficient for 10GB+ repos. GitHub Large Repos

Top comments (0)