How to push files programatically to a repository using Octokit with Typescript

#github #api #typescript #octokit

I've always liked this word programatically, and from time to time I find myself using it when searching Google for a really specific problem. Often, doing something "programatically" leads to an elegant solution for some real-world problem or saves you some time automating a task.

This last week I was set up to develop a product that had dealing with GitHub repos as one of its challenges. I had already worked with GitHub's API before, but mostly for Github Actions and some CI stuff, and this was a bit different, since, as its core, was Git, and Git is not simple.

Basically, I had to create a repository if it didn't exist and push n files into it. At first, I started testing my guesses of how to do it using the GitHub API directly and a personal access token as an auth method. After that, I started coding the solution using GitHub's Octokit/REST, a cool wrapper for its API that supports Typescript and it will help you a lot.

One thing that I have to mention is that this was one of the first times that using Typescript really sped up my work. After Step 2 I didn't stop for testing and, after finishing it, I was pretty sure that I would've missed something but the solution worked seamlessly at first try.

So, one of my first concerns was what do I have to do to mimic a git add . && git commit -m "My commit" && git push using the API? After a bit of Googling, I wrote on my board:

Note: Most of the referencing here uses SHA, a hash of the object within Git. That "code" that every commit has and you see on git log is the commit's SHA.

Get the ref of the branch you wanna push files to. You'll want the SHA of the last commit doc
With the commit's SHA, you will fetch for that commit's tree, which is the data structure that actually holds the files. You'll want its SHA too doc
After this, you will need the list of files you want to upload, this includes the files' paths and its contents. With those you will create a blob for each file. In my case, I was dealing with .mds and .yamls, so I've used utf8 encode and sent each file content's as a plain text that I got from fs.readFile(path, 'utf8'), but there're other options for working with binaries, submodules, empty directories and symlinks. Oh, also: don't worry about files inside directories, it's just about sending the path on the next step for each file that GitHub will organize them accordingly (that's was one of the things I was afraid that I would've to deal with it manually) doc
With all the SHA's from the blobs created on Step 3, you'll create the new tree for the repo with those files. It's here where you'll link the blobs to the paths, and all of them together to the tree. There are some modes that you can read about it on the docs, but I've only used 100644 for regular files. You'll also have to set the SHA retrieved on Step 2 as a parent tree for this one you're creating doc
With the tree (and its SHA) containing all the files, you'll need to create a new commit pointing to such tree. This will be the commit holding all of your changes for the repo. It will also have to have the commit SHA got from Step 1 as its parent commit. Notice this is a common pattern on Git. It's here where you set the commit message too doc
And, finally, you set the branch ref to point to the commit created on the last step, using the commit's returned SHA doc

Done!

One of the problems that I've also had was that I had to create the repository in some cases, and there was no "initial" commit for me to work. After, I found that you can send a auto_init param when creating a repository that GitHub will automatically create a simple README.md initializing the repo. Dodged a bullet there.

So, let's go to the code.

import Octokit from '@octokit/rest'
import glob from 'globby' 
import path from 'path'
import { readFile } from 'fs-extra'

const main = async () => {
  // There are other ways to authenticate, check https://developer.github.com/v3/#authentication
  const octo = new Octokit({
    auth: process.env.PERSONAL_ACESSS_TOKEN,
  })
  // For this, I was working on a organization repos, but it works for common repos also (replace org for owner)
  const ORGANIZATION = `my-organization`
  const REPO = `my-repo`
  const repos = await octo.repos.listForOrg({
    org: ORGANIZATION,
  })
  if (!repos.data.map((repo: Octokit.ReposListForOrgResponseItem) => repo.name).includes(REPO)) {
    await createRepo(octo, ORGANIZATION, REPO)
  }
  /**
   * my-local-folder has files on its root, and subdirectories with files
   */
  await uploadToRepo(octo, `./my-local-folder`, ORGANIZATION, REPO)
}

main()

const createRepo = async (octo: Octokit, org: string, name: string) => {
  await octo.repos.createInOrg({ org, name, auto_init: true })
}

const uploadToRepo = async (
  octo: Octokit,
  coursePath: string,
  org: string,
  repo: string,
  branch: string = `master`
) => {
  // gets commit's AND its tree's SHA
  const currentCommit = await getCurrentCommit(octo, org, repo, branch)
  const filesPaths = await glob(coursePath)
  const filesBlobs = await Promise.all(filesPaths.map(createBlobForFile(octo, org, repo)))
  const pathsForBlobs = filesPaths.map(fullPath => path.relative(coursePath, fullPath))
  const newTree = await createNewTree(
    octo,
    org,
    repo,
    filesBlobs,
    pathsForBlobs,
    currentCommit.treeSha
  )
  const commitMessage = `My commit message`
  const newCommit = await createNewCommit(
    octo,
    org,
    repo,
    commitMessage,
    newTree.sha,
    currentCommit.commitSha
  )
  await setBranchToCommit(octo, org, repo, branch, newCommit.sha)
}


const getCurrentCommit = async (
  octo: Octokit,
  org: string,
  repo: string,
  branch: string = 'master'
) => {
  const { data: refData } = await octo.git.getRef({
    owner: org,
    repo,
    ref: `heads/${branch}`,
  })
  const commitSha = refData.object.sha
  const { data: commitData } = await octo.git.getCommit({
    owner: org,
    repo,
    commit_sha: commitSha,
  })
  return {
    commitSha,
    treeSha: commitData.tree.sha,
  }
}

// Notice that readFile's utf8 is typed differently from Github's utf-8
const getFileAsUTF8 = (filePath: string) => readFile(filePath, 'utf8')

const createBlobForFile = (octo: Octokit, org: string, repo: string) => async (
  filePath: string
) => {
  const content = await getFileAsUTF8(filePath)
  const blobData = await octo.git.createBlob({
    owner: org,
    repo,
    content,
    encoding: 'utf-8',
  })
  return blobData.data
}

const createNewTree = async (
  octo: Octokit,
  owner: string,
  repo: string,
  blobs: Octokit.GitCreateBlobResponse[],
  paths: string[],
  parentTreeSha: string
) => {
  // My custom config. Could be taken as parameters
  const tree = blobs.map(({ sha }, index) => ({
    path: paths[index],
    mode: `100644`,
    type: `blob`,
    sha,
  })) as Octokit.GitCreateTreeParamsTree[]
  const { data } = await octo.git.createTree({
    owner,
    repo,
    tree,
    base_tree: parentTreeSha,
  })
  return data
}

const createNewCommit = async (
  octo: Octokit,
  org: string,
  repo: string,
  message: string,
  currentTreeSha: string,
  currentCommitSha: string
) =>
  (await octo.git.createCommit({
    owner: org,
    repo,
    message,
    tree: currentTreeSha,
    parents: [currentCommitSha],
  })).data

const setBranchToCommit = (
  octo: Octokit,
  org: string,
  repo: string,
  branch: string = `master`,
  commitSha: string
) =>
  octo.git.updateRef({
    owner: org,
    repo,
    ref: `heads/${branch}`,
    sha: commitSha,
  })