Anti-Money Laundering Stream Processing Deadlock and Lag Diagnostic

Role: Principal Distributed Systems Debugging Engineer specializing in real-time financial compliance and streaming architectures.

Context

• Compliance Pipeline Service: {{aml_pipeline_service}}
• Kafka Consumer Group: {{kafka_consumer_group}}
• JVM JMX Endpoint: {{jvm_jmx_endpoint}}
• Suspect Batch Identifier: {{suspect_transaction_batch_id}}
• State Store Backend: {{state_store_backend}}
• Incident Reference: {{incident_ticket_id}}

Task

Create a modular Bash and Python debugging script that analyzes JVM thread states, RocksDB lock contention, and partition lag on {{kafka_consumer_group}} to pinpoint root causes for transaction ingestion stalls in {{aml_pipeline_service}}.

Method

1. Sample partition offsets and consumer lag metrics across all assigned partitions in {{kafka_consumer_group}}.
2. Poll thread states and lock monitors via {{jvm_jmx_endpoint}} to capture JVM thread dumps during pipeline stalls.
3. Parse thread dumps to detect blocked threads, cyclic monitor dependencies, and lock contention on {{state_store_backend}}.
4. Trace execution bottlenecks triggered by {{suspect_transaction_batch_id}} during graph analysis or complex ruleset matching.
5. Inspect Garbage Collection duration and memory pool utilization to identify stop-the-world pauses causing consumer heartbeat timeouts.
6. Check disk I/O metrics and write-stall flags on the embedded {{state_store_backend}} instance.
7. Correlate partition rebalance timestamps with uncommitted transaction offsets to identify cyclic poison-pill records.
8. Generate a consolidated diagnostic JSON payload tied to {{incident_ticket_id}} with step-by-step unblocking actions.

Constraints

• The diagnostic script MUST NOT restart running JVM services or clear state directories automatically.
• JMX connections MUST use non-blocking connection timeouts to avoid hanging the diagnostic execution.
• Memory allocation for heap analysis within the script MUST NOT exceed 512MB on the target host.
• Output telemetry MUST mask all personal identifiable financial data (PII) extracted from transaction payloads.

Output format

• Section 1: Shell Orchestration Wrapper (JMX fetch, thread dump sampling, Kafka lag collector, 30-50 lines).
• Section 2: Python Thread and State Dump Parsing Engine (lock cycle detector and lag analyzer, 80-130 lines).
• Section 3: Incident Diagnostic JSON Output Schema referencing {{incident_ticket_id}}.
• Section 4: Triage & Unblocking Action Plan for Production Operations.

Self-review

• Verify that {{aml_pipeline_service}}, {{kafka_consumer_group}}, {{jvm_jmx_endpoint}}, {{suspect_transaction_batch_id}}, {{state_store_backend}}, and {{incident_ticket_id}} are present.
• Ensure the thread analysis algorithm handles recursive lock detection correctly.
• Confirm all script sections are syntactically complete without placeholder comments.