Vision-Language Spatial Hallucination Remediation Brief

Role: Principal Multimodal QA Architect specializing in vision-language alignment and spatial grounding.

Context

• Multimodal Model Endpoint: {{vlm_model_endpoint}}
• Input Image Modality: {{failing_image_modality}}
• Ground Truth Spatial Coordinates: {{ground_truth_coordinates}}
• Current System Prompt Chain: {{system_prompt_chain}}
• Visual Misalignment Symptoms: {{misalignment_symptoms}}
• Downstream JSON Schema: {{downstream_action_schema}}

Task

Author a high-precision debugging brief diagnosing visual grounding drift, spatial hallucinations, and coordinate scaling errors in {{system_prompt_chain}} when processing {{failing_image_modality}} via {{vlm_model_endpoint}}, delivering a hardened system prompt and coordinate normalization methodology that strictly validates against {{downstream_action_schema}}.

Method

1. Analyze the token-to-patch mapping behavior of {{vlm_model_endpoint}} for high-frequency spatial features in {{failing_image_modality}}.
2. Cross-reference the model's reported bounding coordinates against {{ground_truth_coordinates}} to quantify IoU (Intersection over Union) error margins and directional skew.
3. Dissect {{system_prompt_chain}} for ambiguous spatial referents, anchor-point ambiguities, and ungrounded spatial assumptions.
4. Diagnose the root trigger of {{misalignment_symptoms}} (e.g., dynamic image resolution downsampling, aspect-ratio padding distortion, or coordinate system mismatch).
5. Develop a normalized spatial framing protocol (e.g., 0-1000 integer grid vs normalized float coordinates) optimized for the vision encoder's native patch grid.
6. Restructure the system prompt with chain-of-thought grounding steps (e.g., identify visual anchors before emitting coordinate tuples).
7. Validate that the remediated output adheres strictly to {{downstream_action_schema}} without parsing exceptions.

Constraints

• MUST provide an IoU degradation assessment based on {{ground_truth_coordinates}} versus predicted outputs.
• MUST NOT recommend switching model endpoints; fixes must be constrained to prompt engineering, input pre-processing, and schema constraints.
• All coordinate normalization instructions MUST specify origin point [0,0] orientation (top-left vs bottom-left).
• The revised prompt MUST integrate explicit verification steps for bounding boxes.

Output format

• Spatial Alignment Diagnostic: 100-140 words.
• Coordinate Error & Patch Mapping Analysis: Structured breakdown of scaling, offset, and aspect-ratio distortions.
• Hardened System Prompt: Production-ready system prompt including pre-grounding reasoning and output formatting.
• Input Pre-Processing & Tile Partitioning Guide: Concrete image preparation rules (resolution limits, aspect ratio padding, DPI guidelines).

Self-review

• Does the hardened prompt guarantee compliance with {{downstream_action_schema}}?
• Are coordinate origin conventions explicitly defined and unambiguous?
• Did I directly address the visual artifacts specific to {{failing_image_modality}}?