Solved: Sync Linear Issues to GitHub Issues for Open Source Transparency

#devops #programming #tutorial #cloud

🚀 Executive Summary

TL;DR: Organizations using Linear for internal planning and GitHub for open-source transparency face challenges in manual issue synchronization. This guide provides a custom, cost-effective Python solution to build a one-way sync, pushing new and updated Linear issues to GitHub using their respective APIs, ensuring public visibility without manual data entry.

🎯 Key Takeaways

A custom one-way synchronization from Linear (GraphQL API) to GitHub (REST API) can be implemented using Python, embedding Linear issue IDs in GitHub issue descriptions to prevent duplicates.
Securely manage API credentials by storing them in a secrets.txt file, loaded by a Python utility, and ensure GitHub PATs have the necessary repo scope.
Automate the synchronization process using cron jobs for periodic execution, and implement robust error handling for API rate limits, authentication failures, and GitHub label mismatches.

Sync Linear Issues to GitHub Issues for Open Source Transparency

Introduction: Bridging the Gap Between Internal Planning and Open Source Transparency

In today’s fast-paced development landscape, many organizations leverage specialized tools to optimize their workflows. Linear, with its sleek interface and powerful project management features, is a favorite among product and engineering teams for internal sprint planning and issue tracking. However, for open-source projects, GitHub Issues remains the gold standard, offering public visibility, community interaction, and seamless integration with code repositories. The challenge arises when you need to keep your public-facing GitHub issues in sync with your internal Linear issues, ensuring transparency without burdening your team with manual data entry.

Manually porting issues from Linear to GitHub is not only tedious and error-prone but also a significant drain on developer time—time better spent building features. Relying on expensive, third-party SaaS solutions for a straightforward synchronization can quickly inflate operational costs. The good news? You can build a robust, custom synchronization mechanism using Python and the respective APIs of Linear and GitHub. This tutorial will guide you through setting up a one-way sync, pushing new and updated Linear issues to your GitHub repository, fostering open-source transparency while maintaining your internal Linear workflow.

Prerequisites for Seamless Integration

Before we dive into the implementation, ensure you have the following tools and credentials ready:

Python 3.x: Our synchronization script will be written in Python.
requests library: This Python library is essential for making HTTP requests to both Linear and GitHub APIs. Install it using pip:

  pip install requests

Linear API Key: You will need an API key from your Linear workspace to authenticate requests. You can generate one in your Linear settings under “API” or “Developer.” Make sure it has read access to issues.
GitHub Personal Access Token (PAT): A GitHub PAT with repo scope (full control of private repositories) or at least public_repo scope (for public repositories) is required to create issues. Generate this in your GitHub developer settings.
Target GitHub Repository: The owner and name of the GitHub repository where issues will be created.
A secrets.txt file: To securely store your API keys and tokens locally.

Step-by-Step Guide: Syncing Linear to GitHub Issues

We will break down the synchronization process into four clear steps, starting from credential setup to the core logic of issue creation.

Step 1: Securely Configure Your API Credentials

For security and ease of management, we’ll store our sensitive API keys and tokens in a secrets.txt file. This file should be placed outside your version control system (e.g., not committed to Git).

Create a file named secrets.txt in your project directory and add your credentials:

LINEAR_API_KEY=YOUR_LINEAR_API_KEY_HERE
GITHUB_PAT=YOUR_GITHUB_PERSONAL_ACCESS_TOKEN_HERE
GITHUB_REPO_OWNER=your_github_username_or_org
GITHUB_REPO_NAME=your-repository-name

Now, let’s create a Python utility function to load these credentials:

import os

def load_secrets(file_path='secrets.txt'):
    secrets = {}
    if not os.path.exists(file_path):
        raise FileNotFoundError(f"Secrets file not found at {file_path}")
    with open(file_path, 'r') as f:
        for line in f:
            line = line.strip()
            if line and not line.startswith('#'):
                key, value = line.split('=', 1)
                secrets[key.strip()] = value.strip()
    return secrets

# Example usage (don't run in production script, just for testing)
# if __name__ == "__main__":
#     try:
#         config = load_secrets()
#         print(f"Loaded Linear API Key: {'*' * len(config.get('LINEAR_API_KEY', ''))}")
#         print(f"Loaded GitHub PAT: {'*' * len(config.get('GITHUB_PAT', ''))}")
#     except FileNotFoundError as e:
#         print(e)

This script parses the secrets.txt file and loads the key-value pairs into a dictionary. This dictionary can then be accessed by our main synchronization script.

Step 2: Fetching Issues from Linear’s GraphQL API

Linear’s API is GraphQL-based, allowing for precise data fetching. We’ll construct a query to retrieve issues that we want to sync. For simplicity, we’ll fetch all issues that are not “Done” or “Canceled”.

import requests
import json
from datetime import datetime, timedelta

# Assume load_secrets() function from Step 1 is available
# config = load_secrets() # Load secrets before calling this function

LINEAR_API_URL = "https://api.linear.app/graphql"

def get_linear_issues(api_key):
    headers = {
        "Authorization": api_key,
        "Content-Type": "application/json"
    }

    # Fetch issues created/updated in the last 7 days to limit data, or adjust as needed
    # For a full sync, remove the 'updatedAt' filter.
    seven_days_ago = (datetime.utcnow() - timedelta(days=7)).isoformat(timespec='milliseconds') + 'Z'

    query = """
    query GetIssues($afterDate: DateTime!) {
      issues(
        filter: {
          or: [
            { updatedAt: { gte: $afterDate } },
            { createdAt: { gte: $afterDate } }
          ],
          state: {
            # Assuming you want to sync issues that are not yet "Done" or "Canceled"
            # Adjust these labels based on your Linear workflow
            name: { nin: ["Done", "Canceled"] }
          }
        },
        first: 50 # Fetch up to 50 issues, paginate if more are expected
      ) {
        nodes {
          id
          title
          description
          url
          state {
            name
            type
          }
          project {
            name
          }
          team {
            name
          }
          creator {
            name
          }
          createdAt
          updatedAt
        }
      }
    }
    """

    variables = {
        "afterDate": seven_days_ago
    }

    payload = {
        "query": query,
        "variables": variables
    }

    try:
        response = requests.post(LINEAR_API_URL, headers=headers, data=json.dumps(payload))
        response.raise_for_status() # Raise an exception for HTTP errors
        data = response.json()
        if "errors" in data:
            print(f"Linear API errors: {data['errors']}")
            return []
        return data["data"]["issues"]["nodes"]
    except requests.exceptions.RequestException as e:
        print(f"Error fetching Linear issues: {e}")
        return []

# Example of how you would call this:
# config = load_secrets()
# linear_issues = get_linear_issues(config['LINEAR_API_KEY'])
# for issue in linear_issues:
#     print(f"Linear Issue: {issue['title']} (ID: {issue['id']})")

This Python code defines a function get_linear_issues that sends a GraphQL query to the Linear API. It retrieves key details for issues that have been recently updated or created and are not in a final state. The state: { name: { nin: ["Done", "Canceled"] } } filter is crucial for controlling which issues are considered for synchronization. Remember to adjust the first: 50 parameter and implement pagination if your Linear workspace has many issues.

Step 3: Creating and Linking Issues in GitHub

Now we’ll take the fetched Linear issues and create corresponding issues in GitHub. To maintain a link between the two, we’ll embed the Linear issue ID and URL directly into the GitHub issue’s description. This also helps in preventing duplicate GitHub issues if the script runs multiple times.

import requests
import json

# Assume load_secrets() function from Step 1 is available
# config = load_secrets() # Load secrets before calling this function

GITHUB_API_URL = "https://api.github.com"

def create_github_issue(pat, owner, repo, linear_issue):
    headers = {
        "Authorization": f"token {pat}",
        "Accept": "application/vnd.github.v3+json",
        "Content-Type": "application/json"
    }

    # Check if this Linear issue already exists in GitHub to avoid duplicates
    # We do this by searching for the Linear ID in existing GitHub issues.
    search_query = f"in:body \"[Linear ID: {linear_issue['id']}]\" repo:{owner}/{repo}"
    search_url = f"{GITHUB_API_URL}/search/issues?q={requests.utils.quote(search_query)}"

    try:
        search_response = requests.get(search_url, headers=headers)
        search_response.raise_for_status()
        search_data = search_response.json()

        if search_data['total_count'] > 0:
            print(f"Linear issue {linear_issue['id']} '{linear_issue['title']}' already exists in GitHub. Skipping.")
            return None # Issue already exists
    except requests.exceptions.RequestException as e:
        print(f"Error searching GitHub issues for {linear_issue['id']}: {e}")
        # Proceed with creation attempt if search fails, with caution
        pass

    # Construct the GitHub issue body
    gh_issue_body = f"## Original Linear Issue\n\n" \
                    f"**Title:** {linear_issue['title']}\n" \
                    f"**Status:** {linear_issue['state']['name']} ({linear_issue['state']['type']})\n" \
                    f"**Project:** {linear_issue.get('project', {}).get('name', 'N/A')}\n" \
                    f"**Team:** {linear_issue.get('team', {}).get('name', 'N/A')}\n" \
                    f"**Created By:** {linear_issue.get('creator', {}).get('name', 'N/A')}\n" \
                    f"**Linear URL:** {linear_issue['url']}\n\n" \
                    f"{linear_issue['description'] or 'No description provided.'}\n\n" \
                    f"--- \n" \
                    f"_[Linear ID: {linear_issue['id']}]_" # Unique identifier for our sync

    # Map Linear status to a GitHub label (optional, but good practice)
    # You'll need to create these labels in GitHub manually first or via API.
    labels = [linear_issue['state']['name']] # Using Linear state name as a label

    payload = {
        "title": f"Linear Sync: {linear_issue['title']}",
        "body": gh_issue_body,
        "labels": labels
    }

    issues_url = f"{GITHUB_API_URL}/repos/{owner}/{repo}/issues"

    try:
        response = requests.post(issues_url, headers=headers, data=json.dumps(payload))
        response.raise_for_status()
        gh_issue_data = response.json()
        print(f"Successfully created GitHub issue: {gh_issue_data['html_url']}")
        return gh_issue_data
    except requests.exceptions.RequestException as e:
        print(f"Error creating GitHub issue for Linear ID {linear_issue['id']} ('{linear_issue['title']}'): {e}")
        if response.status_code == 422: # Unprocessable Entity, often due to invalid labels
            print(f"GitHub API Error Details: {response.json()}")
        return None

# Main sync logic (combining steps 1, 2, 3)
# if __name__ == "__main__":
#     try:
#         config = load_secrets()
#         linear_issues = get_linear_issues(config['LINEAR_API_KEY'])
#         
#         for issue in linear_issues:
#             create_github_issue(
#                 config['GITHUB_PAT'],
#                 config['GITHUB_REPO_OWNER'],
#                 config['GITHUB_REPO_NAME'],
#                 issue
#             )
#         print("Synchronization process completed.")
#     except Exception as e:
#         print(f"An error occurred during synchronization: {e}")

The create_github_issue function first attempts to search GitHub for an existing issue linked to the specific Linear ID. If found, it skips creation. Otherwise, it constructs a descriptive issue body for GitHub, including the Linear issue’s details and a unique identifier ([Linear ID: ...]), then creates the issue. We’re using the Linear issue’s state name as a GitHub label for easy categorization. You will need to ensure these labels exist in your GitHub repository.

Step 4: Automating the Synchronization Process

To ensure continuous transparency, you’ll want to automate this script to run periodically. A common and robust way to do this on Linux systems is by using cron.

First, save your Python script (combining the functions from Steps 1-3) into a single file, for example, sync_script.py. Make sure it’s executable:

chmod +x sync_script.py

Next, open your crontab for editing:

crontab -e

Add a line to run your script every hour (or at your desired frequency). Ensure you specify the full path to your Python executable and script:

0 * * * * /usr/bin/env python3 /home/user/your_project/sync_script.py >> /tmp/logs/linear_github_sync.log 2>&1

This cron job will execute your sync_script.py every hour, redirecting all output (including errors) to /tmp/logs/linear_github_sync.log for auditing. Remember to replace /home/user/your_project/sync_script.py with the actual path to your script.

Common Pitfalls and Troubleshooting

API Rate Limits: Both Linear and GitHub APIs have rate limits. If you’re syncing a large number of issues or running the script very frequently, you might hit these limits.
- Solution: Implement exponential backoff for API calls. Review GitHub’s rate limit headers (X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset) and Linear’s corresponding limits to adjust your script’s frequency or batch requests.
Authentication Errors: Incorrect API keys or Personal Access Tokens (PATs) are a common source of errors.
- Solution: Double-check your secrets.txt file for typos. Ensure your Linear API key has sufficient read permissions and your GitHub PAT has the necessary repo scope to create issues. GitHub PATs can expire or be revoked, so verify their validity.
GitHub Label Mismatch: If your script tries to assign a label to a GitHub issue that doesn’t exist in the repository, the GitHub API will return a 422 Unprocessable Entity error.
- Solution: Manually create the required labels in your GitHub repository (e.g., “Todo”, “In Progress”) that correspond to your Linear issue states. Alternatively, extend the script to check for and create labels dynamically.
Data Mapping Inconsistencies: Linear’s rich fields might not directly map to GitHub’s simpler issue structure.
- Solution: Carefully decide which Linear fields are critical for GitHub. Concatenate or reformat data into the GitHub issue body to preserve information, as shown in our example.

Conclusion

Automating the synchronization of Linear issues to GitHub issues is a powerful way to enhance transparency for your open-source projects without disrupting internal workflows. By leveraging Python and the respective APIs, you’ve created a custom, cost-effective solution that keeps your community informed and engaged. This one-way sync frees your team from manual data transfer, allowing them to focus on what matters most: building great software.

As a next step, consider expanding this script to support two-way synchronization, including updates to existing GitHub issues when their Linear counterparts change, or even closing GitHub issues based on Linear status. Exploring webhooks for real-time updates rather than scheduled cron jobs could also provide a more immediate synchronization experience. Continuous integration platforms like GitHub Actions or GitLab CI/CD can also be used to host and run such synchronization scripts, offering more robust execution environments and advanced scheduling.