This package is an implementation of the Unipept API that's being used by the Unipept Web Application and Desktop application to succesfully perform analysis of metaproteomics samples.
The public API is documented at https://unipept.ugent.be/apidocs. The routes under
/private_api, /mpa and /datasets serve the Unipept web application and have no public
documentation. api/src/routes.rs is the full list of routes. Each route declared
through define_routes! also answers with a .json suffix.
/api/v1 is deprecated. It stays available because many tools still call it, but it now serves
the same routes and the same content as /api/v2. Note that v1 taxonomy differed slightly from
v2 in the past; that difference is gone, so a v1 caller gets v2 semantics today. New clients
should use /api/v2.
/healthanswers 200 whenever the process is serving, for a load balancer to poll./health/databaseanswers 200 when OpenSearch answers and 503 when it does not, so an OpenSearch outage can be told apart from the API itself being down.
The index has two implementations of every structure — one holding owned memory, one borrowing a
memory mapping — and which one a binary uses is decided when it is compiled, not by a flag.
The features are forwarded to sa-server, which names one concrete type per structure; nothing
in the search path branches on them.
| build | what it does |
|---|---|
| (no features) | everything preloaded into owned memory. This is the default. |
--features mmap |
everything memory-mapped |
--features mmap,preloaded-text |
mapped, except the protein text |
--features mmap,preloaded-proteins |
mapped, except the protein metadata |
--features mmap,preloaded-mapping |
mapped, except the suffix-to-protein mapping |
The three preloaded-* features combine freely, giving nine configurations in all. Each is a
no-op without mmap, where everything is preloaded already. There is no preloaded-sa: the
suffix array follows mmap and is roughly 72% of the index, so it dominates residency either way.
A release publishes three of these builds: preloaded (no features), mmap, and hybrid (mmap
with all three preloaded-* features). Each host selects one with VARIANT in its environment
file.
cargo build --release --features mmap
The running server reports what it was built with at startup, since there is no other way to tell:
Index storage backend: sa=mmap text=preloaded proteins=mmap mapping=mmap
Figures below are from sa-index/BENCHMARKS.md in
the index repository, measured on one machine against the 223 GB UniProt index. Read the shape,
not the absolute numbers.
If the whole index is guaranteed resident in RAM, preloading pays: against plain mmap,
preloaded-proteins is +22.1%, preloaded-text,preloaded-proteins is +46.1%, and the fully
preloaded build is +57.6%.
If it is not, use plain mmap. Preloaded memory is anonymous and cannot be reclaimed under
pressure. From the first memory ceiling that binds, no preloading arm beats mmap by more than
the noise floor, preloaded-proteins is behind it at every ceiling, and the fully preloaded
build is OOM-killed at every ceiling tested. Below roughly a third of the index the preloaded
arms do not degrade gracefully — they collapse, at tens of times mmap's fault rate.
Since the choice is compiled in, it is a deployment decision rather than something a restart can
correct. When the memory ceiling might move, mmap is the safe build.
Independent of the backend, and the largest single effect in the index's benchmark record. Under a memory ceiling, raising the thread count well above the core count buys back +65.8% to +94.9%, because a major page fault blocks its thread and each faulting thread otherwise idles a core. With the index fully resident the same setting costs about 10%. A deployment knob, not a default.
CONTRIBUTING.md lists the toolchains and the checks a PR must pass.
The steps below start a development instance. A production host is deployed from
.deploy/, described on the wiki page
Deploying the Unipept API.
You can use the included devcontainer in order to start working on this API. The devcontainer will automatically download the most recent version of the Unipept Index built from SwissProt. Follow these steps in order to easily work on the Unipept API in the devcontainer:
- You first have to build the binaries by running
cargo build --release. - Make the directory where we can store the logfiles for a running instance of the Unipept API:
mkdir -p /var/log/unipept-api. - Finally, the Unipept API can be started with this command:
./target/release/unipept-api -i "/unipept-index-data" -d "http://localhost:9200" -p 80 > /var/log/unipept-api/api.log 2> /var/log/unipept-api/api_error.log.