Efficient Offboarding of Users in GitLab

Mor — Sun, 08 Sep 2024 18:15:13 +0000

Introduction

When people leave an organization, it can be challenging both operationally and personally. Their absence is felt, and navigating these transitions requires both sensitivity and a solid strategy to keep things on track.

Offboarding users from GitLab is a crucial step to ensure security and continuity within an organization. It's important to handle this process carefully to protect sensitive information while keeping operations running smoothly. This article explores the challenges of offboarding and outlines an effective approach to manage it.

Key Considerations in GitLab User Offboarding

1. User Tokens

Personal Access Tokens (PATs) are revoked when a user is blocked
Potential issues if PATs are used in critical systems or workflows
Need to identify and address dependencies on personal tokens before offboarding

2. Scheduled Pipelines

Scheduled Pipelines (SP) are bound to individual users
Blocking a user disables their associated Scheduled Pipelines
Requires ownership transfer to an active user to maintain pipeline functionality

Automated Offboarding Process

To address these challenges, we've implemented an automated offboarding process that includes safety checks and escalation procedures. Here's an overview of the process:

Initiation: IT department triggers the offboarding process via an API call to a Lambda function.
Initial Validation: The Lambda function performs basic checks, such as verifying the user's email domain.
Jenkins Job Activation: A Jenkins job is triggered to perform more detailed checks: a. Check for active tokens used within the last 14 days b. Identify any Scheduled Pipelines associated with the user
Decision Making:
- If any checks return true (active tokens or SPs exist), a Jira support ticket is created for the DevOps team.
- The DevOps team collaborates with the user's manager to address any potential issues.
- If no issues are found, the process proceeds to block the user.
User Blocking: An API call is made to GitLab to block the user if all checks pass.

Visual Representation of the Offboarding Process

The following flowchart illustrates the step-by-step process of our automated GitLab user offboarding system:

This diagram outlines the key decision points and actions taken during the offboarding process, from initial triggering by IT to the final blocking of the user in GitLab. It highlights the automated checks, escalation procedures, and the role of the DevOps team in resolving any issues that arise.

Jenkins Process Flowchart

To better visualize the workflow implemented in our Jenkins file, we've created the following flowchart. This diagram illustrates the main steps of the GitLab user offboarding process, including user verification, checks for scheduled pipelines and tokens, and the decision-making process for either creating a Jira ticket or blocking the user.

Benefits of This Approach

Risk Mitigation: Prevents accidental disruption of critical systems relying on personal tokens or scheduled pipelines.
Automated Checks: Reduces manual effort and the likelihood of human error in the offboarding process.
Escalation Procedure: Ensures that potential issues are reviewed by appropriate stakeholders before taking action.
Compliance: Helps maintain security by ensuring timely offboarding while considering operational dependencies.

Best Practices for Preventing Token and Pipeline Issues

To minimize complications during the offboarding process, consider implementing these best practices:

Regular Token Audits: Conduct periodic reviews of active tokens and their usage.
Token Expiration Policies: Enforce short-lived tokens with automatic expiration.
Shared Service Accounts: Use dedicated service accounts for critical systems instead of personal accounts.
Pipeline Ownership Documentation: Maintain a centralized registry of Scheduled Pipeline ownership.
Automated Pipeline Transfers: Implement a process to automatically transfer pipeline ownership during offboarding.

Case Study: Improved Offboarding Efficiency

After implementing this automated offboarding process, our organization observed the following improvements:

80% reduction in time spent on manual offboarding tasks
95% decrease in incidents related to unintended service disruptions during offboarding
100% compliance with security policies for timely access revocation

Future Improvements

To further enhance the offboarding process, consider these potential improvements:

Self-Service Portal: Develop a user interface for managers to initiate and track the offboarding process.
Integration with HR Systems: Automate the trigger of the offboarding process based on HR events.
Collect the data from GitLab DB: In our setup, accessing the GitLab DB would be a slightly complicated task and an unwanted overhead for the application. We compromised using the above Python code with a runtime of ~30 min that is acceptable at this time. In different architectures of GitLab (not omnibus), accessing the DB and querying the user info will be much easier and more accessible.

Conclusion

Efficient offboarding of users in GitLab requires a balanced approach that addresses security concerns while maintaining operational continuity. By implementing an automated process with built-in safety checks and escalation procedures, organizations can streamline their offboarding workflow, reduce risks, and ensure a smooth transition when users leave the system. Regular reviews and continuous improvement of the offboarding process will help organizations stay ahead of potential issues and maintain a secure and efficient GitLab environment.

Backup Grafana Dashboards using Azure DevOps

Mor — Sun, 27 Jun 2021 06:40:02 +0000

DevOps is a set of practices that combines software development (Dev) and IT operations (Ops). It aims to shorten the systems development life cycle and provide continuous delivery with high software quality. DevOps is complementary with Agile software development; several DevOps aspects came from the Agile methodology.

Helping to support such a vast operation we as DevOps teams use multiple 3rd party application that need constant monitoring.
One of those tools is Grafana. A great application for better observability across multi-data centers.
Using Grafana we are able to keep track on data coming from MSSQL ElasticSearch , Prometheus , Coralogix , API and more.
With such an extensive operation we need a way to back up all the dashboards we have per environment.
This prove to be a bit tricky since :

[X] How do we keep the same folder structures ?
[X] How easy will it be to restore one or many dashboards ?
[X] Will i have multiple backups per environment ?

What Do We Need To Backup

A dashboard in Grafana is represented by a JSON object, which stores metadata of its dashboard. Dashboard metadata includes dashboard properties, metadata from panels, template variables, panel queries, etc.
See an example dashboard for RabbitMQ:
RabbitMQ-Overview

As you can see the dashboard only points to the data source and to the query type and content.
So we now know that we need to backup just the JSON file of each dashboard stored in Grafana DB:

Backup Options
We explored a few option for backing up Grafana:

Server backup
In case it's needed- we will need to create an new server and dump the image on it. This method is in effect by the SRE-IT department but the disadvantages oversees the good ones
DB backup
In case it's needed- We will need to spin up a new DB ( MySQL or PostgresSQL ) and do a restore. Usually will require some assistance from DBA/DataOps
Dashboard backup
This method , with it's challenges , proved to be a task that will require the DevOps team with no other dependency.

DevOps Building Blocks

Azure Repos

cloud-hosted private Git repos for your project
Microsoft Azure provide free private Git repositories that we use to store our code as well as other projects.

Azure Pipelines

Continuously build, test, and deploy to any platform and cloud
Cloud-hosted pipelines for Linux, macOS, and Windows. Build web, desktop and mobile applications. Deploy to any cloud or on‑premises using installed agents per environment.

Let's start backing up

First we will lay the architecture more clearly :

Azure agents will have the option to access each environment and query the dedicated Grafana API and return the result back to Azure Pipelines.
Those step will repeat each environment we would like to backup i.e QA/Sandbox/Staging/Production

let's see a more detailed view of the flow in one of the above env:

The flow will backup each environment to it's own GIT branch resulting in the below:

Code

The full code can be found here:
https://github.com/kfirlevin/grafana-dashboards-backup

Conclusion

With very easy setup using Azure Devops Repos and Pipelines we were able to backup all of our environments on a daily basis using a simple Powershell code that iterate over our environment .

DEV Community: Mor

Efficient Offboarding of Users in GitLab

Introduction

Key Considerations in GitLab User Offboarding

1. User Tokens

2. Scheduled Pipelines

Automated Offboarding Process

Visual Representation of the Offboarding Process

Jenkins Process Flowchart

Benefits of This Approach

Best Practices for Preventing Token and Pipeline Issues

Case Study: Improved Offboarding Efficiency

Future Improvements

Conclusion

Backup Grafana Dashboards using Azure DevOps

What Do We Need To Backup

DevOps Building Blocks

Azure Repos

Azure Pipelines

Let's start backing up

Code

Conclusion