How to separate a small library out of a big Git tree

Extracting a Small Library from a Large Git Repository

Learn how to cleanly separate a small, reusable library from a larger, existing Git repository into its own independent project, preserving its history.

Working with large, monolithic Git repositories can sometimes become unwieldy, especially when a small, self-contained piece of code evolves into a reusable library. Extracting such a library into its own repository offers numerous benefits, including clearer separation of concerns, independent versioning, easier dependency management, and improved reusability across multiple projects. This article will guide you through the process of extracting a subdirectory from a larger Git repository into a new, standalone repository, while preserving its commit history.

Understanding the Challenge

The primary challenge in extracting a subdirectory is to retain only the commit history relevant to that specific subdirectory, discarding all other changes from the parent repository. Simply copying the files and initializing a new Git repository would lose all historical context, making it difficult to track changes, revert to previous versions, or understand the evolution of the library. Git's powerful filter-repo (or older filter-branch) command is designed precisely for this kind of history rewriting.

flowchart TD
    A[Start with Monorepo] --> B{Identify Subdirectory}
    B --> C[Create New Empty Repo]
    C --> D[Clone Monorepo Locally]
    D --> E["Use `git filter-repo`"]
    E --> F{Push Filtered History}
    F --> G[New Independent Library Repo]
    G --> H[Update Monorepo to use Library]
    H --> I[End]

Workflow for extracting a subdirectory into a new Git repository.

Prerequisites and Setup

Before you begin, ensure you have git-filter-repo installed. This tool is a modern, faster, and safer replacement for git filter-branch. If you don't have it, you can usually install it via pip or your system's package manager. You'll also need a local clone of your large repository and an empty remote repository where your new library will reside.

pip install git-filter-repo
# Or on Debian/Ubuntu:
sudo apt-get install git-filter-repo

Installing git-filter-repo.

The Extraction Process

The core of this process involves cloning your existing repository, applying git-filter-repo to isolate the desired subdirectory's history, and then pushing that filtered history to a new remote repository. Finally, you'll update your original repository to remove the extracted code and potentially add the new library as a submodule or dependency.

1. 1. Create a New Empty Repository

Go to your Git hosting service (GitHub, GitLab, Bitbucket, etc.) and create a new, empty repository for your library. Do not initialize it with a README or license file.

2. 2. Clone the Original Repository

Clone the large, original repository to your local machine. This clone will be used for the filtering process.

3. 3. Navigate into the Cloned Repository

Change your current directory to the root of the newly cloned repository.

4. 4. Filter the Repository History

Run git filter-repo with the --path option, specifying the path to the subdirectory you want to extract. This command will rewrite the repository's history to only include commits that touched files within that subdirectory, effectively making the subdirectory the new root of the repository.

5. 5. Remove Remote Origin and Add New Remote

After filtering, the local repository still points to the original remote. Remove this origin and add the URL of your new, empty library repository as the new origin.

6. 6. Push to the New Remote

Push the filtered history to the main (or master) branch of your new library repository. Use the -u flag to set the upstream branch.

7. 7. Clean Up (Optional but Recommended)

Delete the local clone of the original repository that you used for filtering to avoid confusion. You can always re-clone the original if needed.

# Step 2 & 3: Clone and navigate
git clone git@example.com:your-org/monorepo.git
cd monorepo

# Step 4: Filter the history (replace 'path/to/your/library' with actual path)
git filter-repo --path path/to/your/library

# Step 5: Remove old origin and add new
git remote remove origin
git remote add origin git@example.com:your-org/new-library.git

# Step 6: Push to new remote
git push -u origin main

# Step 7: Clean up
cd ..
rm -rf monorepo

Commands for extracting a subdirectory using git-filter-repo.

Updating the Original Repository

Once the library is extracted, your original monorepo still contains the library's code. You should now remove it and optionally integrate the new library as a dependency. Common approaches include Git submodules, package manager dependencies (e.g., npm, pip, Maven), or simply updating your build process to fetch the library.

# In your original monorepo clone:

# Option 1: Remove the directory and add as submodule
git rm -r path/to/your/library
git commit -m "Removed extracted library"
git submodule add git@example.com:your-org/new-library.git path/to/your/library
git commit -m "Added new-library as submodule"

# Option 2: Remove the directory and update build system
git rm -r path/to/your/library
git commit -m "Removed extracted library, now using new-library as dependency"

Updating the original repository after extraction.