Create a folder and push multiple files under a single commit through GitHub API

Intro

I could not understand how to push multiple files under a single commit, using GitHub API. Some googling landed me on Git Database API Docs. Wasn't very clear how it worked so I tried getting my hands dirty! Documenting it here for anyone who's looking for it.

Disclosure:

1. This API needs atleast 1 commit to work. Empty repositories don't work.

Git Database API

The Git Database API enables you to read and write raw Git objects to your Git database on GitHub and to list and update Git references (branch heads and tags).

The whole process is as follows: create blobs corresponding to files, create tree, create a commit for the changes and finally updating refs to reflect the commit.

Creating blobs

A Git blob (binary large object) is the object type used to store the contents of each file in a repository. The file's SHA-1 hash is computed and stored in the blob object.

To create a blob we need to hit the following endpoint:

POST https://api.github.com/repos/{user}/{repo}/git/blobs

Let's create a blob for a file called main.py with 1 line of python code:

{
    "content":"print(\"hello world !\")",
    "encoding": "utf-8"
}

We can create blobs for each file we want to upload. In the response, we get SHA of the blob which we need to send while creating the tree. Sample response:

{
    "sha": "638eff25696b982124deeb1f3dfcceabfdc81a93",
    "url": "https://api.github.com/repos/BRO3886/git-db-example/git/blobs/638eff25696b982124deeb1f3dfcceabfdc81a93"
}

Creating a tree

Now, we need to create a tree with the files we added. For that, first we need to get the SHA of the base tree. Then we need to create a tree for the files we need to add:

Get SHA for base_tree

The base tree can be thought of as the folder where you'd like to create your commit. If you want to create it at the root folder, you can make request to:

GET https://api.github.com/repos/{user}/{repo}/git/trees/{branch}

It returns a response like:

{
    "sha": "0d43a3b20104b4baa402c09a6c9c6c3298390e4a",
    "url": "{url of tree}",
    "tree": [
        {
            "path": "App",
            "mode": "040000",
            "type": "tree",
            "sha": "{sha}",
            "url": "{url of folder/tree}"
        },
        {
            "path": "README.md",
            "mode": "100644",
            "type": "blob",
            "sha": "{some SHA}",
            "size": 885,
            "url": "{some URL}"
        },
        ...
    ],
    "truncated": false
}

Here, we can extract the SHA from response.sha or SHA of the folder under response.tree[i].sha.

Create tree

Then, we need to hit the create tree api:

POST https://api.github.com/repos/{user}/{repo}/git/trees

With the body as:

{
    "tree":[
        {
            "path":"helloworld/main.py",
            "mode":"100644",
            "type":"blob",
            "sha":"638eff25696b982124deeb1f3dfcceabfdc81a93"
        },
        {
            "path":"helloworld/main2.py",
            "mode":"100644",
            "type":"blob",
            "sha":"638eff25696b982124deeb1f3dfcceabfdc81a93"
        }
        ...
    ],
    "base_tree":"3c408bafa55eda6b1c51de5df0fc36304f37414c"
}

Here request.tree would have an array of blobs that you want to push. For each blob we need the SHA that we got as a response from create blob API.

For the mode:
The file mode; one of 100644 for file (blob), 100755 for executable (blob), 040000 for subdirectory (tree), 160000 for submodule (commit), or 120000 for a blob that specifies the path of a symlink.

GitHub doesn't store folders, so the only way to make a folder is to set the path for the blob as {folderName}/{fileName}

Here, we get another SHA in the response, which we need when we create a commit corresponding to the changes:

{
    "sha": "a69117177bb067933189072b2b8799c63f388f32",
    "url": "https://api.github.com/repos/BRO3886/git-db-example/git/trees/a69117177bb067933189072b2b8799c63f388f32",
    "tree": [
        {
            "path": "README.md",
            "mode": "100644",
            "type": "blob",
            "sha": "bc7b1321063b4075c97bf16e6f8130b6f9fa6537",
            "size": 54,
            "url": "https://api.github.com/repos/BRO3886/git-db-example/git/blobs/bc7b1321063b4075c97bf16e6f8130b6f9fa6537"
        },
        {
            "path": "helloworld",
            "mode": "040000",
            "type": "tree",
            "sha": "82a82f6788b44fe93774597ff2e76ac66ae1e657",
            "url": "https://api.github.com/repos/BRO3886/git-db-example/git/trees/82a82f6788b44fe93774597ff2e76ac66ae1e657"
        }
    ],
    "truncated": false
}

Here we need response.sha for the next step.

Add a commit

This step is pretty simple. We just need the response.sha from the previous step. We make a request to the following endpoint:

POST https://api.github.com/repos/{user}/{repo}/git/commits

Along with the body:

{
    "tree":"a69117177bb067933189072b2b8799c63f388f32",
    "message":"some commit msg",
    "parents": ["3c408bafa55eda6b1c51de5df0fc36304f37414c"]
}

parents: The SHAs of the commits that were the parents of this commit. If omitted or empty, the commit will be written as a root commit. For a single parent, an array of one SHA should be provided; for a merge commit, an array of more than one should be provided.

In our case, for the parent, since we want to add another commit on a particular branch, we need to get the SHA of it.

For that, we need to make a request to the following endpoint:

GET https://api.github.com/repos/BRO3886/git-db-example/git/refs/heads/{branch}

It returns a JSON with details about the branch:

{
    "ref": "refs/heads/main",
    "node_id": "REF_kwDOG87gc69yZWZzL2hlYWRzL21haW4",
    "url": "https://api.github.com/repos/BRO3886/git-db-example/git/refs/heads/main",
    "object": {
        "sha": "3c408bafa55eda6b1c51de5df0fc36304f37414c",
        "type": "commit",
        "url": "https://api.github.com/repos/BRO3886/git-db-example/git/commits/3c408bafa55eda6b1c51de5df0fc36304f37414c"
    }
}

For the parent SHA, we need response.object.sha.

In the response of create commit API, we'll get another SHA:

{
    "sha": "544aa83c4d4a784c4c8490d6548c248b0e57d0ac",
    "node_id": "C_kwDOG87gc9oAKDU0NGFhODNjNGQ0YTc4NGM0Yzg0OTBkNjU0OGMyNDhiMGU1N2QwYWM",
    "url": "https://api.github.com/repos/BRO3886/git-db-example/git/commits/544aa83c4d4a784c4c8490d6548c248b0e57d0ac",
    "html_url": "https://github.com/BRO3886/git-db-example/commit/544aa83c4d4a784c4c8490d6548c248b0e57d0ac",
...
}

We'll need this SHA value for the last step.

Updating ref

This step is for updating the ref from where we pulled the SHA under parents (create commit step). For my case, since the ref was main, I'll update it. Make a request to the following endpoint:

PATCH https://api.github.com/repos/{user}/{repo}/git/refs/heads/{branch}

Along with the body:

{
    "sha":"544aa83c4d4a784c4c8490d6548c248b0e57d0ac"
}

If in the response the object.sha is same as the one sent as a part of the request, along with status code 200, that means your changes will be reflected on GitHub.

---

I hope you got some idea. Here is the GitHub link of the sample repo you can refer: github.com/BRO3886/git-db-example

Comments

[sPITf1re-dev]

Hi and thanks a lot for such a cool guide. It saved me tons of time. I have a question.
This part:
"Get SHA for base_tree"
you received: "sha": "0d43a3b20104b4baa402c09a6c9c6c3298390e4a",

Then you create a tree and use this:
"base_tree":"3c408bafa55eda6b1c51de5df0fc36304f37414c"

which is not what you got in the previous step. But this is the SHA you're obtaining in the next step as the latest commit SHA of a needed branch.

Which one I should actually use as a base tree SHA?
I guess that it's a typo (because you don't use base_tree SHA after obtaining) but not sure.

Please advice, thanks in advance.

[sPITf1re-dev]

UPD: Actually, I did it assuming that there was a typo in your guide and it worked well to me. So I think it is a typo

[Siddhartha Varma]

Hey, yes seems like a typo, thanks for pointing out. Or, I might have wanted to show you can use the sha for any of the folders present in the tree. Basically:

tree
|
|-some_folder
|-{newfile}

For above case use sha of base tree

tree
|
|-some_folder
|   |-{newfile}
|-file.py

For above case, use the sha of some_folder. And this is recursive so you can traverse the folder to get the sha of subfolder.

Hope this answers your query. Sorry I'm a tad bit late in answering...

[Rohit]

Hi Siddhartha,
Thanks for such a good explanation, I need some input in one issue I am facing.
I created one tree under my branch with one file inside tree(folder/file).
structure is like -- branch/folder/file.
Now I want add another tree(folder1/file1) in above tree, but when I commit these thing I am not getting new tree added in above tree, I used all configurations correctly. result should be like (branch/folder{file}/folder1/file1)
resulting structure is like branch{file}/folder1/file1 (also it has file just below branch) one folder got removed i.e. : "folder"
Can you advice.

[S. Amir Mohammad Najafi]

thanks! i want to use them to push multi file into large repository in github action

[thilllon]

the last part, updating ref, saved my life. thx!