Add interactive mode - #61

Introduction `-sysf FNAME` / `--system-file FNAME`
-e` escapes both prompt and the system

Add UTF-8 Encoding in read_text.

feat: AMD Instinct MI300X + MI355X (gfx942/gfx950) ROCm support

* Initial work on interactive mode.

* Improve interactive mode. Make rev. prompt optional.

* Update README to explain interactive mode.

* Fix OS X build

* POC: per row scale

This is a POC how to work around opinionated ggml to
have scales per row rather than per block.
Only implemened for Zen4 and only for iq2_tn.

* POC per row scale: iq2_tn on NEON

* POC per row scale: iq2_tn on Metal

* Per row scale Metal templates

* iq1_tn: shrink to 1.625 bpw (NEON and Metal)

* POC per row scale: CUDA

* POC per row scale: add CUDA TODOs

There are two places in ggml-cuda.cu left where it is assumed
that type_size * n_per_row / block_size is the way to compute
and handle row sizes. This does not affect simple usage,
but will lead to issues when tensors are split between GPUs.

* Per row scales - CUDA

The only place left where there are unnecessary assumptions being made
is in the Flash Attention code. As we are not using any quants that
use per row scales for quantized KV cache, it should be OK for now.

* Update IQ1_TN and IQ2_TN bpw shown to user

---------

Co-authored-by: Iwan Kawrakow <[email protected]>

… path) (ggml-org#61)

* ggml: rows-indexed state read for the fused GDN op (ring decode path)

On the ring-enabled decode path every GDN layer paid two extra dispatches
per token just to feed the fused op its input state: a get_rows gather of
the per-seq live states into a contiguous scratch, then a cpy of that
gather into slot 0 of the (D, K, n_seqs) state input. Both are pure reads
of the recurrent cache -- ~786k floats each way per layer on the 27B
target -- serialized into a launch-bound decode graph, 96 dispatches and
~300 MB of scratch traffic per token across 48 layers.

Add ggml_gated_delta_net_rows: the op takes the 2D cache view plus the
per-seq row indices (inp->s_copy_main) as src[6] and reads each
sequence's live state directly at cache row rows[seq]. K moves to
op_params so both variants share one backend code path. The graph side
gains build_rs_cache_view (rs_zero clear + extra-states relocation, no
main gather) and qwen35 wires it on the ring path, with
GGML_GDN_STATE_GATHER=1 restoring the legacy gathered path for A/B.

Implemented on CPU and Metal (function-constant-gated read base, no
kargs change). All other backends that support GATED_DELTA_NET reject
src[6] in supports_op so rows-mode ops fall back instead of silently
reading src[5] as a scratch.

test-backend-ops gains rows-mode cases (single/multi-token, multi-seq,
snapshot overflow, KDA): 38/38 OK on MTL0, CPU leg green. Real-eval
gate: accept counts bit-identical to the gathered path at n_max 1..4
(alpaca x24). Measured on M5 Pro (cont6k Q1_0 x bin6l1 q4_0, ring 4):
harness AR 32.0 -> 35.3 tok/s (+10.5%), spec@n3 33.5 -> 35.3 (+5.5%);
ring-free llama-bench unchanged (~42), as expected.

* ggml: fold recurrent GDN snapshot writes on Metal

* metal gdn: always populate snapshot tail on write-fold, handle K==1 rows

Two correctness fixes to the folded rows-mode GDN epilogue:

- The write-fold followed the SET_ROWS view chain to prove the scatter
  consumes the GDN result, but not that it is the snapshot tail's sole
  consumer. The kernel now always writes the op's own documented output
  tail AND additionally scatters into the cache row, so a second consumer
  or an output/eval callback never observes an uninitialized region.

- WRITE_ROWS scatter existed only in the K>1 branch; a rows-mode graph with
  K==1 suppressed the SET_ROWS but wrote only the output tail, losing the
  cache update. The K==1 final-state branch now scatters to the cache row
  as well.

Gate: test-backend-ops GATED_DELTA_NET 39/39 on MTL0; e2e accept invariant
76/116 tau 2.3103 unchanged (default / fold-disabled / gathered).

* qwen35: gate GDN rows mode to Metal-only GPU device sets

rows mode uses the src[6] GDN variant, implemented on CPU and Metal only;
other GPU backends reject it in supports_op, which would move the recurrent
op (and its state traffic) to CPU. Select rows mode only when every GPU
device in the model is Metal (ACCEL/BLAS devices are skipped).

* ggml: disable OpenMP for Emscripten/WASM builds

The WASM CI build enables OpenMP (-DGGML_USE_OPENMP -fopenmp=libomp), but
Emscripten cannot emit the common symbols libomp's reduction helpers need
(.gomp_critical_user_.reduction.var), so ggml-quants.c fails to compile.
WASM has no host threads to benefit from OpenMP -- force it off for the
Emscripten target instead of failing the build.

…verflow, write-fold guards, K==1 tests) (ggml-org#62)

* address review: CPU workspace sizing, write-fold guards, K==1 rows tests

- ggml-cpu: size the GDN scratch from the op-param K (snapshot slots), not
  src[5]->ne[1] -- in rows mode that dim is the cache row count, so a 1-row
  cache with K>1 (batch-1 block decode) undersized the scratch and overflowed
  the work buffer.
- metal write-fold: honor ctx->use_fusion (GGML_METAL_FUSION_DISABLE), and
  verify the SET_ROWS target is exactly the snapshot tail (per-row state width,
  index count, dest row width) before suppressing it -- descent from the GDN
  output alone let a mis-sized view be fused, reading row indices out of bounds.
- rows-mode state view: document + assert the main/extra row-range disjointness
  invariant that makes the deferred (read-after-relocate) main read safe.
- tests: add rows-mode K==1 cases to exercise the K==1 final-state branch.

* review round 2: byte-offset write-fold check, honest rows-mode ordering note, 1-row-cache K>1 test

- write-fold: also verify the SET_ROWS view begins at the snapshot-tail byte
  offset (attn_size + (K-min(T,K))*state_size_per_snap), not just matching
  size/counts -- a same-sized view at another offset no longer folds.
- rows-mode state view: drop the incorrect disjointness assert (s_copy returns
  idx*size+src0, an arbitrary slot, so it did not establish disjointness).
  Document the real read-before-relocation hazard (multi-seq; not reachable on
  the single-seq decode path) as tracked follow-up.
- tests: add a rows-mode 1-row-cache K>1 case that reproduces the CPU
  workspace under-size the planner fix prevents.

* write-fold: require compact snapshot-row stride before folding

ggml_set_rows only requires contiguous rows (nb[0]); it permits an arbitrary
row stride nb[1] that its kernel honors, but the fused GDN epilogue scatters
the contiguous snapshot tail. Require the compact [D, n_write] layout
(ne[0]==D, nb[0]==type_size, nb[1]==D*type_size) so a strided view falls
through to the real SET_ROWS instead of being mis-scattered.

The 1-at-the-colon/8-elsewhere split was measured against one
hand-written schema, a flat two-property object. Every closer in such an
object sits at the top level, which is precisely where the two indent
positions coincide -- so the schema could not show that they differ, and
the bound it produced is wrong in both directions at once.

ECMA-262 governs five joints by three mechanisms. SerializeJSONObject
emits `": "` as a fixed literal, so nothing precedes a colon and one
space follows it, and its separator is `",\n" + indent`, so the newline
follows the comma and nothing precedes that either. Indentation is a
separate gap, clamped to 10 by 25.5.2.1, and it lands at two positions
that do NOT take the same amount: a member sits at gap*level, a closer
sits back one level at gap*(level-1).

  class              conforming maximum
  key -> `:`         0
  value -> `,`       0
  `:` -> value       1
  `{ [ , ` -> item   1 + gap * level
  value -> `} ]`     1 + gap * (level-1)

Not read off the spec and hoped for: Protean's
TestConformingWhitespace_MatchesClosedForm enumerates every conforming
layout of all six schemas it constrains -- eleven each, the gap being
clamped -- and asserts each joint holds exactly this, cross-checked
against both Go's MarshalIndent and the real JSON.stringify (whose gap 11
and 12 output is byte-identical to gap 10).

So 8 was:

  - too LARGE at a closer. At the top level a closer sits at gap*0, one
    newline, at every gap. Live, that joint measured 1 x20, 2 x2, 8 x1 --
    the last two rows non-conforming, and the 8 sitting exactly on the
    bound, i.e. a degenerate run capped rather than ended. Given 8
    characters of room it survived in ~12% of generations. It now has one.
  - too SMALL at a member, which must clear 1 + 10*level. That is 11 for a
    flat schema and 31 for Protean's WorkflowDesign (object -> array ->
    object) -- the schema the original 19/100 runaway was measured on. At 8
    it was unpretty-printable at any gap above 2, so conforming output was
    rejected at three separate joints. That is the condition that displaces
    a character into a free string and returns something schema-valid and
    wrong, which is the failure bounding exists to avoid.

The emitter already tracks the nesting for its recursion guard, so every
bound is now derived from it and nothing is picked.

Tests: the flat case gains the reject rows for its closer (2, 8 and 40
characters) and for whitespace before a comma, plus an accept row at
gap 10 that the old bound refused. A second case covers a NESTED schema,
because depth is the variable the flat one cannot show: it accepts gap 2,
4 and 10 -- gap 4 was rejected outright before -- and rejects gap 11 as
past the clamp, 25 characters at an inner closer (between that closer's 21
and a member's 31, so it separates the two bounds rather than merely
exceeding both), and whitespace before the top-level `}`.

Verified through the real llguidance sampler over the real Gemma 4 E2B
vocabulary; ctest -R "chat|grammar" is 7/7.

Still to measure live, and NOT claimed here: whether tightening the
top-level closer to one character ends the ggml-org#61 stall without
reintroducing the string corruption. Both directions need Protean's
TestLive_ValidatorRunaway and TestLive_CanonicalizerStrings against a
router built from this commit.

* feat(server): add backend protocol foundation — cpp-httplib v0.46.0 (WS), SessionManager, protocol serde, fix token2wav namespace build

* feat(server): WS /backend handler + HTTP close endpoint — full_duplex input.append loop, fail-fast state checks, base64→WAV/JPEG temp files

* feat(server): turn_based messages parsing, non-streaming mode, T2W audio callback → WS audio_delta

* feat(server): support C++ (llama.cpp-omni) backend via the backend protocol (ggml-org#54)

* feat(server): support C++ (llama.cpp-omni) backend via the backend protocol

Signed-off-by: dongfengwuyishi <[email protected]>

* fix(server): derive MiniCPM-o sub-model paths from the LLM model path

Signed-off-by: dongfengwuyishi <[email protected]>

* fix(server): address backend protocol review issues for session close, parsing, metrics, and cleanup

Signed-off-by: dongfengwuyishi <[email protected]>

---------

Signed-off-by: dongfengwuyishi <[email protected]>

* rebase: migrate WS backend to server-omni.cpp, fix httplib header-only upgrade

- Resolve rebase conflicts (common.h: merge tts_bin_dir+vpm_batch_encode,
  token2wav: remove stray namespace close/reopen, CMakeLists: accept master)
- Move WS /backend + /sessions/:session_id/close from old server.cpp to
  llama-omni-server (server-omni.cpp)
- Add SessionManager to omni_server_state
- Replace split httplib.h/httplib.cpp with header-only cpp-httplib v0.46.0
  (httplib_stub.cpp with CPPHTTPLIB_IMPLEMENTATION)
- Add session/protocol/ws_handler to llama-omni-server build target

* fix(server): align omni backend protocol completion and model path resolution (ggml-org#61)

Signed-off-by: dongfengwuyishi <[email protected]>

---------

Signed-off-by: dongfengwuyishi <[email protected]>
Co-authored-by: dongfengwuyishi <[email protected]>