Sign in
Technology & Software
Quality 97/100

Automated Data Lineage Graph Mapping

Traces the flow of data from source systems through transformations to final BI/ML sinks.

Creates a technical map of dependencies to facilitate impact analysis during schema changes or pipeline migrations.

Template

You are a Data Architect building a metadata management strategy.

Context

Our ecosystem has become complex, moving data from {{source_databases}} via {{transformation_tools}} into {{downstream_sinks}}. We lack visibility into how a change in the source affects the sinks. We need a conceptual and technical lineage map.

Task

  1. Identify the 'Nodes' (Tables, Topics, Models) and 'Edges' (Pipelines, SQL joins).
  2. Document the transformation logic happening within {{transformation_tools}} (e.g., aggregation, filtering).
  3. Map specific columns from {{source_databases}} to their final state in {{downstream_sinks}}.
  4. Categorize lineage by 'Dataset level' and 'Column level'.
  5. Define a strategy for capturing 'Operational Lineage' (execution time, success/failure status).
  6. Propose a tool-agnostic metadata schema (e.g., OpenLineage) to represent this graph.

Constraints

  • MUST cover both batch and real-time paths if present.
  • MUST identify 'Hidden Dependencies' like external API lookups.
  • MUST NOT exceed 3 levels of depth for the initial mapping.

Output format

  • Lineage Topology Overview

  • Table/Entity Mapping Table

  • Transformation Logic Summary

  • Impact Analysis Protocol

Quality bar

  • Does the map identify clear upstream sources for every downstream sink?
  • Is the granularity sufficient for a developer to perform a 'git grep' impact search?
data-lineage
metadata
governance
pipeline-design
advanced