fix(dedup): map exact-substring ranges to original text - #516

Summary

• map duplicate token ranges through tokenizer offsets instead of decoded-string replacement
• merge overlapping ranges and remove only the detected spans from the original text
• include trailing combining marks omitted from tokenizer offset alignment
• retain the decode-based sync fast path for documents without duplicate ranges
• add regressions for Unicode normalization, repeated occurrences, multibyte characters, and overlapping ranges

Problem

ESRangeRemover currently decodes duplicate token IDs and removes the decoded string with
str.replace. This assumes encoding and decoding round-trip to the same Unicode code points. With
tokenizer normalization, decomposed text such as u\u0303 can encode to the same token as NFC ũ
but decode to different underlying text, causing the Stage 3 sync check or duplicate removal to fail.
str.replace can also remove identical occurrences that were not part of the detected token range.
Some normalizers also omit a trailing combining mark from the aligned token offset, so the mapped
text span must include unaligned marks attached to its final character.

This follows the retokenization-and-offset direction suggested in #121 and leaves all three stage
file formats unchanged.

Validation

• pytest -q tests/pipeline/dedup — 27 passed, 3 existing lighteval/NumPy 2.0 skips
• make quality
• git diff --check

Synthetic Stage 3 benchmark: 10,000 documents, 50 tokens per document, five-run median. Stage 1 and
Stage 2 are excluded from timing.

The common no-range path keeps the existing decode check. Re-tokenization is limited to documents
with duplicate ranges, or to the non-round-trippable fallback needed for token-ID synchronization.

This is related to #121 but does not change the Stage 1 uint16 token format; tokenizers with IDs
above 65,535 remain outside this PR's scope.