As a junior data engineer, I have come to realise that building data pipelines involves more than just moving and processing data. Understanding how data should be managed within an organization is also important for ensuring accountability, security and appropriate use.
This realisation led me to explore the principles of data governance, which I will introduce in this article.
What is data governance?
Data governance is a discipline within data management that establishes the responsibilities and decision-making processes required to manage data assets throughout their lifecycle. The objective is to promote data quality, security, availability, and appropriate use. From a data engineer’s perspective, governance requirements help guide engineers to build secure data pipelines that process accurate data and deliver the data to authorized end-users.
What is a data governance framework?
Each organization defines a structured model called a data governance framework. This framework describes the approach to decision-making across different environments within an organization. Without a strong framework, business units will have different views on how to manage data, leading to confusion, inefficiency, and inconsistency.
This framework commonly defines multiple core components:
- Program goals;
- Roles and responsibilities;
- Measures and performance indicators;
- Policies, data standards and procedures;
- Audit management;
- Data governance tools and technologies;
Program goals
Program goals specify the objectives the organization is looking to achieve. For instance, an objective could be enabling data-driven decision-making or enhancing data quality.
Roles and responsibilities
The governance framework defines the roles and responsibilities of all stakeholders within an organization to ensure clear accountability.
- The governance council: a group often composed of senior leaders and relevant stakeholders that oversees the data governance strategy and the direction of the data governance framework.
- The data owners: professionals who are accountable for decisions within their assigned data domains. They help define data-quality expectations and can partake in data governance solutions and policies.
- The data stewards: they handle data management operations on a day-to-day basis. Data stewards often have a task-oriented or operational orientation, whereas data owners that have result-driven or strategic orientation. However, responsibilities can vary between organizations.
- The data consumers: professionals across different departments of an organization who consume enterprise data, such as data analysts, business analysts, and marketing teams. These actors actively use the data and report any anomalies to data stewards.
Measures and performance indicators
The data governance framework should include a set of metrics to monitor the execution of the framework and verify if the program is working. This set of metrics can include cost reduction, data quality scores, policy compliance rates, etc.
Policies, data standards and procedures
First, policies define the rules and expectations on how data should be managed from creation through retention or deletion. Standards are the implementation of the policies by defining naming conventions, metadata, data models, data formats, and more. Finally, procedures describe how each professional responsible within the organization for specific data assets should perform the work.
Audit management
A governance framework should document and monitor internal controls, assess risks, and maintain audit evidence. This helps organizations determine whether their data practices comply with regulatory requirements.
Data governance tools and technologies
Each organization chooses a technology stack to maximize the impact of its data governance program. These technologies range from comprehensive platforms to specialized software solutions. Common core categories of data governance technologies include:
- Data catalogs and metadata management: These technologies collect information about data assets, such as what data exists, where it is stored, and who owns it.
- Access control: These tools enforce role-based access control, offer data masking and identity management to hide sensitive information from the unauthorized users.
- Master Data Management: These platforms help maintain consistent and trusted records for core business entities across systems
- Data lineage tools: These tools help track data origins, monitor data movement, and document data transformations across systems and pipelines.
Top comments (1)
tr.ee/dev-to