On-Device Multimodal Inference Architecture Brief

Role: Principal Android ML Systems Engineer specializing in edge generative AI and LiteRT hardware acceleration.

Context

• Target Application: {{app_name}}
• Supported Hardware Profiles: {{target_soc_tiers}}
• Quantization Strategy: {{quantization_scheme}}
• Strict Peak Memory Cap: {{memory_footprint_ceiling_mb}}
• Input Streams: {{multimodal_input_modalities}}
• Target Processing Time: {{inference_latency_sla_ms}}

Task

Produce an exhaustive technical architecture brief outlining the on-device execution strategy for multimodal models in {{app_name}}, guaranteeing adherence to runtime latency, thermal limits, and hardware-accelerated memory boundaries.

Method

1. Analyze the interaction between {{multimodal_input_modalities}} and map incoming buffers to zero-copy memory abstractions.
2. Evaluate delegate routing across CPU, GPU, and NPU for {{target_soc_tiers}} to eliminate kernel fallback penalties.
3. Model runtime memory allocations under {{quantization_scheme}} against {{memory_footprint_ceiling_mb}} including KV cache and staging buffers.
4. Define tensor preprocessing pipelines in native C++ using NDK to conform to {{inference_latency_sla_ms}}.
5. Design graceful fallback mechanisms for non-compliant hardware running outside {{target_soc_tiers}}.
6. Establish thread affinity and background priority classes to prevent main-thread UI jank during generation bursts.
7. Detail thermal management triggers that downscale model precision when SoC skin temperature climbs.
8. Construct a telemetry plan for capturing frame drop rates, delegate initialization failures, and out-of-memory crashes.

Constraints

• MUST maintain total memory allocation below {{memory_footprint_ceiling_mb}} across all operating scenarios.
• MUST NOT perform image normalization or tensor transformation on the JVM/Kotlin UI thread.
• All delegate bindings must fail fast to a predetermined fallback pipeline without crashing the host process.
• Quantization artifacts must be profiled specifically for {{quantization_scheme}} across heterogeneous SoCs.
• Target cold-start delegate initialization must be quantified within the latency model.

Output format

• Executive Summary (1 paragraph)
• Hardware Acceleration & Delegate Mapping Matrix (table)
• Memory Allocation Budget Breakdown (tabular breakdown with peak totals)
• Preprocessing & Buffer Flow Architecture (step-by-step pipeline)
• Thermal Throttling & Degraded State Policy (bulleted criteria)
• Implementation Milestones & Risk Mitigations (3-5 items)

Self-review

• Verify every memory calculation directly reflects {{memory_footprint_ceiling_mb}}.
• Confirm that delegate fallbacks are provided for every listed tier in {{target_soc_tiers}}.
• Ensure pipeline latency stays within {{inference_latency_sla_ms}} without UI thread blocking.