# pgvector in Rust: Operational Practice and Concurrent Batch Embedding

URL: https://mechcrate.dev/docs/corpus/database/pgvector-rust-batch-embedding/
Source: docs/development/pgvector-rust-batch-embedding.md

---

# pgvector in Rust: Operations + Batch Embedding Pipelines

Operational truths for running pgvector from Rust and embedding at scale, with vendor-benchmark evidence. Inline `[n]` cites key to `sources`.

## 1. Rust access + the pooler traps

The `pgvector` crate (0.4.x for sqlx 0.8/0.9) exposes `Vector`/`HalfVector`/`SparseVector`/`Bit` behind cargo features (`halfvec` is NOT default) — binding is plain `bind()/try_get()` [2].

**Trap 1 — statement cache vs transaction-mode poolers.** sqlx caches prepared statements per connection (default capacity 100). Through PgBouncer transaction mode — including **Neon's `-pooler` endpoints** — cached statement names collide across physical backends: intermittent `prepared statement "sqlx_s_N" already exists` under load. Fix: `statement_cache_capacity(0)` (or `?statement-cache-capacity=0`) when connecting through a pooler; keep the default on direct connections [3]. PgBouncer ≥1.22 supports protocol-level prepared statements, but verify your pooler's version rather than assume.

**Trap 2 — session GUCs leak through pools.** `hnsw.ef_search` (default 40) is session-level: a bare `SET` on a pooled connection contaminates unrelated queries that later land on that backend. Always scope it: `BEGIN; SET LOCAL hnsw.ef_search = N; SELECT …; COMMIT;` — in sqlx, a transaction with the `SET LOCAL` issued on `&mut *tx` [1].

**Bulk writes**: `INSERT … SELECT FROM UNNEST($1::uuid[], $2::vector[]) ON CONFLICT … DO UPDATE` beats multi-VALUES (~4× at 100k rows); binary COPY only wins above ~10k rows/batch.

## 2. HNSW: tuning, size, maintenance

- Defaults: `m=16`, `ef_construction=64`, `ef_search=40` [1]. Production band for 1536-dim embeddings across real benchmarks: `m=16–32`, `ef_construction=128–256`; set `ef_search ≥ LIMIT` [4][7].
- **`maintenance_work_mem` is the build lever**: builds are dramatically faster when the graph fits (pgvector NOTICEs when it doesn't); raise `max_parallel_maintenance_workers` (default 2). Supabase: 1M×1536 builds in ~5–9.5 min with 15 workers + 30GB, vs 39–87 min prior [7]. Neon: cap `maintenance_work_mem` at 50–60% of compute RAM; scale compute up to build, down to serve [4][5].
- Memory: index ≈ 1.5–2× raw vectors (1536-dim ≈ 6KB/row). ANN is only fast while the graph is cache-resident — autoscaling down can push it out and cliff your first queries after scale-up.
- **Index eligibility**: the query must have `ORDER BY <distance-op> ASC` + `LIMIT`, with the ORDER BY directly on the operator — `ORDER BY 1 - (embedding <=> $1) DESC` silently skips the index [1].
- **When NOT to index**: exact search is fine below ~10k–100k rows (parallel seq scan, 100% recall — `SET max_parallel_workers_per_gather = 4`). Start without an index; add HNSW when queries actually slow [4].
- **Churn degrades recall, not just space**: MVCC re-embeds leave dead tuples in the graph; documented failure path to near-0% recall until vacuum. pgvector's own advice: `REINDEX INDEX CONCURRENTLY` first, then `VACUUM` (HNSW vacuum is slow) [1]. Always `CREATE INDEX CONCURRENTLY` on live tables.
- **Versions**: **0.8.6 is current** (2026-07-29): fixes an IVFFlat build buffer overflow on 32-bit systems, an array→`sparsevec` cast not limiting non-zero elements, and IVFFlat scan memory usage under nested-loop joins [21]. **CVE-2026-3172** (buffer overflow in parallel HNSW builds, can leak other relations' data) affects 0.8.0/0.8.1 — check `SELECT extversion FROM pg_extension WHERE extname='vector'` and be ≥0.8.2 [10]. **CVE-2026-18022** (integer wraparound → out-of-bounds write during IVFFlat index build; ≤0.8.5, fixed 0.8.6) is rated CVSS High but **only 32-bit builds are affected** — 64-bit deployments (Neon, RDS, standard x86-64/arm64) are not practically vulnerable; upgrade as hygiene, not urgency [22]. Watch item: PostgreSQL 19 GA lands Sept 2026 — verify pgvector compatibility before moving.

## 3. Quantization: what's proven

- **halfvec (fp16) is the near-free win at 1536 dims**: ~50% storage, ~23% faster builds, 50% faster prewarm, equivalent recall/latency — confirmed independently by Neon and a pgvector maintainer [5][6]. Try it via expression index first (`USING hnsw ((embedding::halfvec(1536)) halfvec_cosine_ops)`) — queries must use the identical cast to hit the index [1].
- **Binary quantization is dataset-dependent and contested**: Katz measured 91.6% recall at 1536-dim with 4× over-fetch rescoring; Neon measured ~50% on the same dimensionality and recommends against. Measure on your data or skip [5][6]. Reinforcing signal: the TurboQuant proposal ("10× smaller vector indexes", PR #989) was **closed unmerged** in 2026-07 — upstream still ships nothing beyond `binary_quantize` [23].
- **Matryoshka dimensions**: `text-embedding-3-small` was trained at {512, 1024, 1536} — the API `dimensions` param gives a 3× storage cut *before* halfvec and cheaper distance math; requires re-embedding, so decide while the corpus is small [20].

## 4. Filtered vector search: the recall trap

With HNSW, a plain `WHERE` filter is applied **after** the index scan: at `ef_search=40`, a filter matching 10% of rows returns ~4 results *regardless of LIMIT* [8][9]. pgvector 0.8.0's iterative scans fix this (`hnsw.iterative_scan = strict_order | relaxed_order`; knobs `max_scan_tuples`=20k, `scan_mem_multiplier`); AWS measured filtered recall going **10% → 100%** with 1.5–1.8× latency improvement [8].

Decision ladder: highly selective filter → B-tree on the filter column and let the planner skip HNSW; moderately selective → `SET LOCAL hnsw.iterative_scan = strict_order`; latency-critical → `relaxed_order`, **but then you MUST re-sort via a `MATERIALIZED` CTE** or ordering is silently wrong [9]; few fixed filter values → partial HNSW indexes; many values at scale → partitioning [1].

## 5. The batch-embedding pipeline (Rust/tokio)

**API hard limits** (OpenAI-compatible `/embeddings`) [15]: 8,192 tokens per input · **2048 inputs per request** · **300,000 tokens summed per request** · empty strings rejected. The token cap binds first in real corpora (~600 inputs at 500-token chunks) — **batch by tokens, not count** (tiktoken `cl100k_base`, count-only path), with headroom (~250k) for tokenizer skew.

**The shape** (each element evidence-backed):

```
chunks → token-count + oversize guard (>8192 → split/dead-letter)
      → token-aware batcher (n==MAX || tokens>250k)
      → .map(|b| async { rpm.until_ready(); tpm.until_n_ready(b.tokens); retry(embed(b)) })
      → .buffer_unordered(N)                      // N ≈ target_RPS × mean_latency; start 16
      → bounded mpsc::channel(64) → writer task   // UNNEST upsert, ON CONFLICT DO UPDATE
```

- `buffer_unordered` starts futures in order, yields in completion order, and IS the backpressure mechanism — the source stops being polled when N futures are in flight [11]. IO-bound fan-out doesn't need JoinSet/multi-core.
- **The footgun**: `FuturesUnordered` futures only progress while polled — a slow `.await` (like a DB insert) in the consuming loop **stalls every in-flight HTTP request** [12]. Hence the bounded-channel hand-off to a dedicated writer; bounded (never `unbounded_channel`) so a lagging DB propagates backpressure instead of buffering the corpus in RAM [19].
- **Rate limiting**: OpenAI enforces RPM and TPM *independently* (org-wide — a backfill starves production traffic). Two `governor` limiters: requests (1 cell/call) and tokens (`until_n_ready(batch_tokens)`); set to ~80% of your dashboard's actual limits, not blog tables [13][16]. Honor `retry-after-ms` on 429s — it's calibrated to your request.
- **Retries**: `backon` (the `backoff` crate is unmaintained) — exponential + jitter around the HTTP call only; retry 429/5xx/timeout, never 400/401; 400s bisect-and-retry to isolate poison items. Embedding calls are effect-idempotent — make the *write* idempotent and aggressive retry is safe [14].
- **Cheap wins**: share one `reqwest::Client` with `pool_max_idle_per_host ≥ N`; `encoding_format: "base64"` (a 256×1536 float JSON response is ~7.5MB to parse).
- **Backfills**: consider the **Batch API** — 50% cheaper, 50k embedding inputs per batch, typically 1–6h, separate quota from your synchronous RPM/TPM [17]. Right architecture: Batch API for bulk reindex, the streaming pipeline for incremental.
- Track per-chunk state (`pending/done/failed` + dead-letter) so runs are resumable and retries never re-bill [16].

## Synthesis (inferred)

Applied to the mx techniques corpus (Rust/tokio/sqlx/pgvector on Neon pooled endpoint, sequential ≤256-input batches, no retries/rate-limiting, HNSW defaults, ~2k chunks):

1. **Pooler safety now**: `statement-cache-capacity=0` on the Neon `-pooler` URL (§1 trap 1) — the failure is intermittent-under-load, the worst kind. Verify pgvector ≥0.8.2 (CVE).
2. **Filtered-search audit**: our `category`/`language` filters ride on HNSW with default `ef_search` — the §4 trap class. At 2k chunks the honest fix is simpler: exact search (drop/skip the index) gives 100% recall in milliseconds; revisit HNSW + iterative scans past ~50k chunks.
3. **Ingest hardening, in order**: token-aware batching (latent 400 as chunks grow), `backon` retries + `ON CONFLICT` idempotent writes, then `buffer_unordered(16)` + bounded-channel writer + two-axis governor when corpus growth makes sequential ingest slow.
4. **At next re-embed**: evaluate `dimensions: 512` (Matryoshka) and halfvec via expression index — both near-free at this corpus size; skip binary quantization.
5. Re-embedding backfills should end with `REINDEX INDEX CONCURRENTLY` (§2 churn/recall) — or nothing, if we've dropped the index at this scale.