Multi-Source Research Synthesis Agent Quality Assessment Matrix

Role: Senior Research Methodologist and Meta-Synthesis Evaluation Director.

Context

• Synthesis agent pipeline: {{synthesis_pipeline_name}}
• Corpus domain and scope: {{research_corpus_domain}}
• Source document batch: {{source_document_batch}}
• Benchmark ground truth synthesis: {{expert_ground_truth_synthesis}}
• Minimum citation precision requirement: {{citation_precision_threshold}}
• Primary conflict resolution policy: {{conflict_resolution_policy}}

Task

Produce an exhaustive multi-dimensional evaluation matrix that assesses how accurately {{synthesis_pipeline_name}} extracts, aggregates, and resolves competing claims from {{source_document_batch}} within {{research_corpus_domain}} against {{expert_ground_truth_synthesis}}.

Method

1. Index and categorize all empirical claims, effect sizes, and qualitative assertions inside {{source_document_batch}}.
2. Cross-reference agent-generated citations against exact source paragraphs to verify reference validity.
3. Measure claim extraction precision against {{citation_precision_threshold}} to flag fabricated claims.
4. Identify contradictory assertions across disparate sources within {{research_corpus_domain}}.
5. Audit agent synthesis behavior against {{conflict_resolution_policy}} to verify balanced reporting of conflicting data.
6. Evaluate thematic coverage and structural completeness against {{expert_ground_truth_synthesis}}.
7. Score synthesis outputs across claim grounding, nuance retention, source weight balancing, and omission rate.
8. Construct a systematic scoring matrix summarizing performance across all synthesis dimensions.

Constraints

• MUST explicitly distinguish between hallucinated citations and misplaced in-line references.
• MUST NOT score a synthesis as acceptable if contradictory source findings are silently omitted.
• Every synthesis claim must be traced to verifiable document identifiers.
• Score variance across evaluation dimensions must be statistically substantiated.

Output format

• Synthesis Audit Matrix (Markdown table with columns: Thematic Dimension, Extraction Accuracy %, Citation Grounding %, Conflict Reconciliation Score, Omission Penalty, Overall Rating)
• Claim Attribution Breakdown (Table detailing Source ID, Extracted Claim, Ground Truth Status, Attribution Verdict)
• Discrepancy Analysis (Max 300 words analyzing systemic synthesis errors)
• Quality Scorecard Summary (4-bullet metric summary)

Self-review

• Check that every cited claim in the matrix links to an item in {{source_document_batch}}.
• Validate that attribution verdicts align strictly with {{citation_precision_threshold}}.
• Confirm conflict reconciliation scoring accurately reflects {{conflict_resolution_policy}}.