jenkins: Rust pipeline for solr builder - #13603

The flag previously only skipped the bulk pre-caching step
(cache_cached_editions_ia_metadata). During update_keys, per-document
processing still fell through to ia.get_metadata_direct() for every
edition ocaid, so the flag did not avoid network access to archive.org
and made things slower in offline environments (one HTTP call per ocaid
instead of batched fetches).

Short-circuit DataProvider.get_metadata() when skip_ia_metadata is set,
and wire the flag onto the provider instances used by solr_builder (the
main db instance passed to load_configs/update_keys, plus db2). Default
remains False, so production behavior is unchanged.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <[email protected]>

The per-document fallback (get_ia_collection_and_box_id ->
data_provider.get_metadata -> ia.get_metadata_direct) is the path the
--skip-ia-metadata flag must suppress. Existing FakeDataProvider stubs
get_metadata to return {}, which would bypass the flag entirely, so use
a provider that calls the real base-class implementation.

Positive test: with the flag set, update_key never calls
get_metadata_direct and the solr doc has empty ia_collection and no
ia_box_id. Negative control: without the flag, get_metadata_direct is
called once per ocaid and ia_collection is populated.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <[email protected]>

Replace the manual blueocean-based Jenkins setup with a fully pinned,
reproducible controller defined in scripts/solr_builder/jenkins/:

- Jenkins LTS 2.568.2, plugins pinned in plugins.txt (baked at build
  time via jenkins-plugin-cli), Docker CLI 29.7.2 + Compose v5.5.0
- JCasC config (jenkins.yaml): admin seeded from ADMIN_PASSWORD env at
  runtime, no anonymous access, setup wizard disabled
- Seed job auto-creates the solr-builder pipeline (Pipeline-from-SCM)
- New profile-gated 'jenkins' service in compose.yaml; jenkins-data
  volume pinned to its exact name for the Jenkinsfile's host-path mapping

Verified end-to-end: image build, boot assertions, seed job config,
docker-in-docker smoke test through the real agent image, and a full
pipeline dry-run (INDEX_* off) finishing SUCCESS.

Run instructions: scripts/solr_builder/jenkins/RUNNING.md

- Jenkinsfile: new SKIP_IA_METADATA boolean param, passed as env to all
  four index-type.sh invocations
- compose.yaml: ol service forwards CHUNK_ETA/SKIP_IA_METADATA from the
  invoking shell (client env never propagated before, so the existing
  CHUNK_ETA=70/35 values were silently ignored until now)
- index-type.sh: appends --skip-ia-metadata when SKIP_IA_METADATA=1

Subjects are unaffected (pure Solr facet queries, no archive.org access).
Verified via PATH-shim test of index-type.sh arg construction.

Seed script now honors SEED_BRANCH (default master) and injects the
pipeline's parameter definitions at creation time, so all params
(incl. SKIP_IA_METADATA) are visible in the UI before the first build.
compose.yaml passes SEED_BRANCH through to the controller.

…lation

${ADMIN_PASSWORD:?...} made every `docker compose` command in the project
fail (compose interpolates the whole file, and the pipeline's shells don't
have that var set). Soften to a default-empty value; the seed job now logs
a loud warning at boot when the password is unset instead.

postgres:latest floated to 18.6, which stores data in major-version-
specific subdirectories under /var/lib/postgresql. Mounting the legacy
/var/lib/postgresql/data path makes the entrypoint refuse to start
('unused mount/volume'), so the db service crash-looped. Mount at
/var/lib/postgresql per the official 18+ recommendation.

Verified: fresh volume initdb + psql 18.6 comes up healthy.

The covers metadata dump now contains literal \N values in integer
columns (width), which CSV-mode COPY rejects ('$\N' is not the CSV null
marker). Add NULL '\N' so such fields load as NULLs. Verified needed:
build #4 failed importing covers at line 77694.

db and solr services now bind host paths under
/mnt/HC_Volume_106672133/openlibrary/ instead of named volumes on the
150G OS disk, which could not fit a full-dump rebuild (~190G peak).
solr_prod keeps its named volume; jenkins-data unchanged (Jenkinsfile
depends on its exact name/path).

cache_work_ratings and cache_work_reading_logs issue per-chunk range
queries on these tables; without indexes every one of the 18 parallel
runners full-scans them each batch. Measured on live run: ratings 34s,
reading_log 7.5m to build.

Used to recover interrupted runs without replaying already-indexed
ranges: INITIAL_START=/works/OL21524512W ./index-type.sh work 18 works.
Fresh-start behavior unchanged (default still scans from the top).
Proven live during the storage-migration recovery of build #5.

Enables idempotent gap-fill re-passes:
SOLR_BUILDER_EXTRA_ARGS='--no-skip-solr-id-check' ./index-type.sh work 18 works

…, 5.99h est), logical diff near-zero, research mvp scripts

- rust_solr: DuckDB sample_keys + silver work_key join + authors appender (query.rs), WorkSolrBuilder port (transform) with DDC/LCC/sort_title/ISBN, rayon 18, Arrow zstd minimal schema (key,doc_json,title,edition_count) - 14.96s/10k (prep 10.11s + build 4.50s 2222 docs/s) vs py DC 19.17s idle, 1440× => 5.99h (1.80h build) - prep 68% wall
- logical diff 10k: py only [] (was 500+), field mismatches lcc_sort 27 + cover 14 + 335 ties same short_len - core parity, editions/by_statement/median/cover/identifiers/ia/series now ported
- research: mvp_bronze 563s 6.2G 42M, mvp_silver_py 343s 21x, mvp_gold_DC D+C 1.52x idle / 3.48x loaded vs baseline, bench load 6.42s, HANDOFF_RUST/MVP_STATUS/PROGRESS_MVP_10k logs
- gitignore lake/*.parquet/rust target, keep code only

…quet, bucketed prep 1.4s), add chunks manifest

Fix1 static RE_* Lazy in transform/ddc/sort_title/helpers: build 4.50s→0.74s (6.1×, 13587 docs/s)
Fix2 par_chunks no ce/ca deep clones (main.rs): 0.74s→0.41s (1.8×, 24328 docs/s, 11× vs baseline)
Fix3 single-Conn authors (query.rs), Fix4 merged, Fix5 RowGroup 10k + par_iter json (parquet.rs)
Fix5b bucketed silver: lake/silver/works_b + editions_bucketed (459×id//100000) + mvp_partition_works.py 81s one-time, chunks_{10000,20000}.json; query.rs bucket pruning: 10k legacy 10.39s (9.81s prep+0.39s build) → 2.30s (1.37+0.63), 5×20k 100k 58.7s→~11.5s; full 14.4M 5.99h→0.48h (prep 4.04h→0.28h, build 1.80h→0.12h)
README note: Fix6 single Arrow scan not built — bucketed lake gives same pruning; true streaming pass would save only ~1s prep/chunk

lake/*.parquet and chunks json remain data artifacts (gitignored/untracked)

…ify → new Solr fast-load (8985 isolated)

Step 1: cargo build + 721× --chunks lake/silver/chunks_20000.json (lazy, one-time bucketed lake works_b/editions_bucketed, 1.4s prep)
Step 2: counts + logical diff (py only [], 815 ties, 113 lcc non-tie) against sparse chunk 10
Step 3: NDJSON per-chunk + parallel curl (fast_solr_inserts.md) into fresh solr_rust_full 8985, keep 8983 7.2M untouched
Note single Arrow stream (Fix6) not built — bucketed manifest gives same pruning; commit HANDOFF_NEXT.md

…solr compose (8985)

- helpers/lcc.rs: fallback to input on unparseable sortable LCC instead of expect() panic (TL233..2 case killed 12/721 chunks)
- main.rs/parquet.rs: --format parquet|ndjson for Solr NDJSON streaming
- compose.rust_full.yaml: isolated solr_rust_full on 8985 with tuned SOLR_OPTS and bind-mount to /mnt (ramBuffer 512 via conf/solr_rust_full copy, SOLR_HEAP 4g)

Rust Gold drops type:author at transform (main.rs:92 work-only, denorm
author_key/name/facet). Add performant streaming loader (duckdb
json_object -> raw VARCHAR -> httpx.Client, NDJSON, 6.8M in ~4min,
27k docs/s) to restore /search/authors (AuthorSearchScheme universe
type:author). Document isolated 8985 pipeline and required extra step
in docs/ai/solr/index.md Local Development.

…and docs

Gold drops ratings_* / readinglog_* (495k / 3.18M works) — need atomic
{"set":} updates via /update?commitWithin=60000 from
solr_duckdb/parquet (5.1M/55M zstd) without Gold rebuild (~90min wall
31s/10k ratings, 12s/10k reading_log vs 5h rebuild). Document in
docs/ai/solr/index.md Rust full section.

Bench on 8985 (103M docs, 59GB):
- ratings 50k: conc1 41.9s (1192/s) -> conc8 31.9s (1567/s) ~24% win
- reading_log 50k: conc1 77.9s (649/s) -> conc8 21.3s (2341/s) ~3.6x win
Full 495k+3.18M wall ~90min single -> ~28min conc8, maxes 4c G1GC.

Add --concurrency, --bench sweep 1,2,4,8,16 and async semaphore post
via /update?commitWithin=60000 atomic {set:}.

…al 4c)

Reading_log 3.6x win at 8 vs 1, ratings 24% win; 8 is reasonable default
for 4c G1GC 4g ramBuffer 512 on 103M docs.

Document mvp_ratings_to_solr.py default conc8 (bench: ratings 41.9->31.9s,
readingLog 77.9->21.3s 3.6x) and ~28min full vs 90min single, --bench sweep,
and 4c tuning. Follow-up to 0935bed.

ReadTimeout at 280k ratings conc8 60s; add 3x retry with backoff,
120s timeout, keep conc8 default (bench 3.6x win). Restart will resume
idempotent atomic {set:} for 495k+3.18M.

The 7.1G 'ol_dump_2026-07-31.txt.gz' was actually a Jan-2024 snapshot;
works/editions had been rebuilt from the genuine 18G dump (lake_full) but
authors (~55% missing) and lists (~50% missing) still loaded from the
stale repo lake/. Deleted bad dump + stale lake/, documented the incident.

- reload results: author 6844404 -> 15378301, list 130154 -> 260316
- add mvp_lists_to_solr.py (stream lists bronze -> Solr, minimal fields)
- docs: point all pipeline/loader commands at
  /mnt/HC_Volume_106672133/openlibrary/lake_full/, correct counts,
  add banners to historical handoffs (MVP_STATUS/HANDOFF_*)

… aggregates

Builder parity (0 field diffs vs WorkSolrUpdater, 98k works across 5 runs
incl. a 13k-fake-work chunk):
- port nested edition docs: id_*, ia_box_id, chapter, publisher dedup, and
  the full EditionScorecardForSolr scorecard (8 fields, exact weights)
- port legacy lending_edition_s/lending_identifier_s/printdisabled_s with
  _ia_editions ordering
- fix LCC short-form leading-zero strip (py .strip("0") is both ends)
- fix DDC word boundaries to python's adjacent-char rule (utils/ddc.py:58)
- deterministic edition ordering (key asc) on both pipeline sides
- chunk mode indexes orphan editions as fake /works/OLxxxM works like prod;
  mvp_partition_orphans.py buckets the ~1.95M orphans (editions_bucketed
  drops NULL work_key rows); bucket_paths globs *.parquet shards
- inclusive start_at window whenever offset == 0

Real availability via --ia-metadata (ia_lite.parquet from mvp_ia_fetch.py):
ports InternetArchiveProvider.get_access (inlibrary/printdisabled/ari/
public) for work-level and nested docs, plus ia_collection. Fetcher uses
the IA scrape API - advancedsearch.php doc_ids bulk returns unrelated rows
from outside prod even with service=metadata__unlimited.

Parity harness: mvp_py_ground.py runs the real updater over the same lake
rows; mvp_diff_check.py diffs field-by-field (order-insensitive multivalued,
lcc/ddc equal-length max-ties, index-time last_modified_i tolerance).

Author aggregates (mvp_author_aggregates.py): lake-side rollups matching
AuthorSolrUpdater facet semantics exactly - per-field top-10 term buckets
merged (count,val) DESC without cross-field dedup, ratings/reading-log sums
through work_ratings_summary_from_counts, zeros included. Validated 1000/1000
top authors vs the real updater querying a loaded index; posts atomic updates
to all 15.4M authors (~31 min). pyarrow/orjson slim sidecars keep giant gold
JSON docs out of DuckDB; aggregation partitioned to fit small boxes.

cargo test: 11 passed (get_access/acquisition/direct-provider precedence,
fake-work synthesis, ddc/lcc parity cases).

…es names, osp_count

Ghost work docs: mvp_ratings_to_solr.py SEMI-JOINs gold before aggregating so
atomic updates can never materialize stub docs, and --cleanup-ghosts deletes
historical ones by exact key set. XML delete-by-id (the JSON handler mis-parses
{"delete":{"id":..}} as an atomic op on nested-child docs). Live index cleaned:
titleless works 34,062 -> 0.

Non-IA provider identifiers: edition access now walks the full PROVIDER_ORDER
chain - direct -> librivox -> project_gutenberg -> project_runeberg ->
standard_ebooks -> openstax -> cita_press -> wikisource -> ia (BWB skipped:
config-gated bwb_test_holdings never matches dumps) - with faithful
multi-provider ebook_provider lists at both work and nested-edition level
(uniq'd across editions for works).

Series names: series_docs loaded from bronze other.parquet (/type/series) and
resolution mirrors WorkSolrBuilder.series_name: fetched-doc name first, key
path fallback; the embedded edge name is ignored like prod. Ground-truth Fake
feeds the same docs so the harness stays honest.

osp_count: mvp_osp_to_solr.py posts counts from the official
2023_openlibrary_osp_counts/osp_totals.db (data(olid,total), numeric work id),
ghost-guarded against gold. 1,292,547 of 1,369,891 works live on :8985.

Validated post-change: chunk 654 + 1000 FULL PARITY vs the real updater
(33,142 + 20,000 docs, zero diffs); cargo test 14 passed.

- Jenkinsfile.rust: dump -> lake -> Rust gold chunks -> parallel NDJSON load
  -> ghost-guarded satellites (authors/lists/ratings/osp) -> author aggregates
  -> hard verify gate (titleless==0 + per-type count floors). SMOKE mode runs a
  3-chunk slice for fast end-to-end wiring checks; RESUME skips completed
  outputs so interrupted builds pick up where they left off.
- Dockerfile.rust-agent: pinned python3.14 + rust 1.98 agent image; cargo
  registry/target on the solr-rust-cargo volume so libduckdb-sys' ~35-min cold
  build happens once per machine.
- seed job registers BOTH pipelines (solr-builder legacy + solr-builder-rust);
  SEED_REPO env selects the git source.
- compose: JENKINS_HTTP_PORT / JENKINS_AGENT_PORT overrides so the controller
  coexists with a dev stack already bound to 8080.
- mvp_bronze.py: OL_DUMP env override (was a hardcoded dated filename).
- RUST_PIPELINE.md operator guide incl. smoke/full runbooks and the
  legacy-retirement checklist.

… stack

…ths)

…d dirs); aux parquets + osp db live inside lake

…arse exit 2)