Indexing Pipeline Specification: Load, Split, Embed, Store
Specify the offline indexing half of a RAG system with concrete loader, splitter, embedding and vector-store choices plus refresh policy.
An indexing spec with per-stage configuration, metadata schema including source URLs, expected chunk counts and a persistence/refresh plan.
Role
You are specifying the offline half of a retrieval system.
Task
Write the indexing specification for a corpus drawn from {{corpus_sources}}. Stage 1 Load: apply {{loader_strategy}}, storing each document's text plus a metadata record with {{metadata_fields}} including a resolvable source URL, and define timeout and skip-on-error behaviour. Stage 2 Split: use chunk size {{chunk_size}} and overlap {{chunk_overlap}}, and justify both against the retrieval unit you want. Stage 3 Embed: use {{embedding_model}} and state its vector dimensionality assumption. Stage 4 Store: use {{vector_store_choice}}, describe persistence and how the index is refreshed on {{refresh_schedule}}. Report the expected document and chunk counts as instrumentation to log.
Context
Retrieval quality is mostly decided offline: chunk boundaries, metadata completeness and refresh cadence set the ceiling that no prompt tuning can raise.
Inputs
- {{corpus_sources}}
- {{loader_strategy}}
- {{chunk_size}}
- {{chunk_overlap}}
- {{embedding_model}}
- {{vector_store_choice}}
- {{metadata_fields}}
- {{refresh_schedule}}
Constraints
- Every chunk must retain a citable source reference
- Do not re-fetch the corpus at query time
- State failure handling for unreachable sources
- Persist the index rather than rebuilding per process start
Output Format
Markdown with one section per stage, a metadata schema table, and a logging/instrumentation list.
Quality Criteria
- Chunk parameters justified, not defaulted silently
- Metadata sufficient for citation
- Refresh policy concrete and scheduled
- Instrumentation enables drift detection