Low-Latency Generative Ad Copy Engine Architecture Specification

Role: Staff Backend Engineer specializing in High-Throughput Content Delivery and LLM Orchestration Infrastructure.

Context

• Latency Budget (P99): {{p99_latency_budget}}
• LLM Upstream Providers & Fallbacks: {{llm_provider_fallbacks}}
• Tiered Cache TTL & Invalidation Policy: {{cache_ttl_policy}}
• Dynamic Prompt Assembly Engine: {{prompt_assembly_rules}}
• Content Safety & Brand Guardrails: {{content_safety_filters}}
• Tenant Rate Limits: {{rate_limit_policy}}

Task

Draft an advanced backend engineering specification for an automated ad-copy generation API that satisfies strict enterprise latency budgets, implements semantic and exact-match caching, and guarantees graceful degradation under upstream LLM provider outages.

Method

1. Translate {{p99_latency_budget}} into discrete latency allocations across network transit, auth, cache lookup, LLM inference, and validation.
2. Design the dual-layer caching subsystem (L1 in-memory exact match, L2 vector-based semantic similarity) governed by {{cache_ttl_policy}}.
3. Specify the runtime prompt construction pipeline that merges customer variables into {{prompt_assembly_rules}} without introducing injection vulnerabilities.
4. Map out the upstream orchestration layer detailing connection pooling, active health checks, and fallback mechanisms across {{llm_provider_fallbacks}}.
5. Design the streaming response middleware incorporating inline {{content_safety_filters}} before tokens reach the calling marketing frontend.
6. Detail tenant throttling, token quota enforcement, and tier prioritization according to {{rate_limit_policy}}.
7. Formulate a degradation matrix defining fallback behavior (e.g., pre-generated static templates) when upstream providers breach error budgets.

Constraints

• MUST NOT allow unvalidated or unsanitized prompt interpolations to reach upstream model gateways.
• MUST implement an automated circuit breaker that routes traffic to secondary providers within 200ms of primary failure.
• P99 cache hits MUST be served in under 15ms.
• Include exact OpenAPI 3.1 path definitions and error payload schemas.

Output format

• 1. System Topology & Latency Budget Allocation (Table format, 200 words)
• 2. Dual-Layer Caching & Eviction Specification (300 words)
• 3. Dynamic Prompt Assembly & Guardrail Middleware Flow (250 words)
• 4. Provider Failover, Circuit Breaking & Fallback Matrix (300 words)
• 5. OpenAPI 3.1 Schema & Error Handling Contracts (YAML or JSON snippet)

Self-review

• Is every stage of the request pipeline accounted for within the specified P99 latency budget?
• Does the failover logic handle context window differences between primary and secondary LLM providers?
• Are safety filter latencies decoupled or parallelized with stream generation where appropriate?