Vision Language Model Visual Grounding Failure Isolation Harness

Role: Senior Multimodal AI Reliability Architect specializing in vision-language alignment and spatial grounding verification.

Context

• Multimodal Target API: {{vlm_model_endpoint}}
• Evaluation Image-Prompt Pair Set: {{multimodal_input_pairs}}
• Grounding Failure Classification: {{grounding_failure_mode}}
• Spatial Error Tolerance (pixels/normalized): {{spatial_tolerance_threshold}}
• Logging and Telemetry Mode: {{diagnostic_logging_level}}
• Infrastructure Runtime: {{execution_environment}}

Task

Develop an automated multimodal debugging script for {{execution_environment}} that runs comprehensive visual grounding assertion tests against {{vlm_model_endpoint}}, isolating whether failures stem from image resolution downsampling, visual token drop, or text prompt ambiguity.

Method

1. Ingest test pairs from {{multimodal_input_pairs}} and validate image pre-processing pipelines against native resolution scaling.
2. Construct API requests to {{vlm_model_endpoint}} enforcing structured JSON output for bounding box coordinates and reasoning chains.
3. Implement an intersection-over-union (IoU) and generalized IoU (gIoU) evaluation engine comparing predictions against {{spatial_tolerance_threshold}}.
4. Profile hallucination signatures specific to {{grounding_failure_mode}} by running counterfactual text prompt ablations.
5. Execute image patch occlusion sweeps (sliding window mask) to calculate visual feature saliency on failed grounding attempts.
6. Format output telemetry according to {{diagnostic_logging_level}}, isolating whether the root cause is vision encoder degradation or prompt syntax.
7. Output automated assertions that exit with non-zero failure codes when grounding drift exceeds configured boundaries.

Constraints

• The script MUST implement robust asynchronous request batching and retry logic with exponential backoff.
• The script MUST NOT require proprietary third-party VLM evaluation platforms.
• Grounding metrics MUST compute both precision and recall across overlapping bounding boxes.
• The codebase must be modularized into ingestion, inference, analysis, and reporting stages.

Output format

1. Installation & Environment Configuration (bash setup block)
2. End-to-End Grounding Debugger Script (complete typed Python/Node script)
3. Diagnostic Error Taxonomy Table (markdown listing error codes, root causes, and trigger criteria)
4. Prompt & Image Preprocessing Remediation Checklist (ordered action steps)

Self-review

• Ensure the code parses coordinates normalized to [0, 1000] or [0, 1] based on target API conventions.
• Confirm {{grounding_failure_mode}} has a dedicated test branch in the assertion logic.
• Verify all script variables and imports are syntactically valid and free of unresolved references.