Sign in
Technology & Software
Quality 97/100

Data Contract Specification for Producer-Consumer Decoupling

Generates a formal data contract using YAML/JSON Schema to prevent breaking changes in downstream pipelines.

Establishes a rigorous schema and SLA definition between data producers and consumers to ensure long-term pipeline stability and automated validation.

Template

You are a Senior Data Architect specializing in distributed systems and data governance.

Context

We are implementing a formal contract between the upstream system {{source_system}} and several downstream ML features. The current data movement for {{data_entities}} is brittle and lacks validation. We require a contract that defines schema constraints, semantic meaning, and a {{freshness_sla}} guarantee.

Task

  1. Define the physical schema using JSON Schema syntax for the specified {{data_entities}}.
  2. Specify data types, nullability, and allowed ranges for each field.
  3. Define semantic metadata including PII markers and business descriptions.
  4. Incorporate the {{freshness_sla}} into a service_level_agreement block.
  5. Design a 'change_management' protocol for handling breaking changes (e.g., versioning strategy).
  6. Outline 3 automated tests (Expectations) that must pass before the data is ingested by downstream consumers.

Constraints

  • MUST use YAML format for the contract definition.
  • MUST NOT allow 'additionalProperties' in the schema definition (strict mode).
  • MUST explicitly flag fields containing sensitive data (PII).

Output format

  • Data Contract Specification

  • Metadata (Owner, Version, Source)

  • Schema Definition (YAML block)

  • Service Level Agreement (Latency, Volume, Quality metrics)

  • Quality Tests (Great Expectations or Soda format)

Quality bar

  • Is the schema strictly typed?
  • Are the SLA metrics measurable and specific?
  • Does the change management policy include a sunsetting strategy?
data-governance
schema-design
data-contracts
mlops
advanced