Replace rust-timsort with an in-tree powersort - #8421
Conversation
Add crates/vm/src/sorting.rs implementing Tim Peters' timsort with powersort's merge-ordering policy (CPython 3.11+): run detection, binary insertion for short runs, galloping merge (merge_lo/merge_hi), and power-based merge ordering (powerloop). Comparison is passed in as a fallible `is_lt` closure, so the algorithm stays generic over the element type and free of interpreter details. Not yet wired into list.sort(); replaces rust-timsort in a follow-up.
Wire list sorting through crate::sorting::timsort and remove the rust-timsort dependency. Fixes the O(N^2) behavior on random input (1M random floats: ~16min -> ~0.8s), now within ~4x of CPython.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughThe VM replaces the external ChangesVM sorting implementation
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related issues
Possibly related PRs
Sequence Diagram(s)sequenceDiagram
participant ListSort
participant Timsort
participant IsLtComparator
ListSort->>Timsort: Sort list values and keys
Timsort->>IsLtComparator: Compare elements with is_lt
IsLtComparator-->>Timsort: Return ordering or error
Timsort-->>ListSort: Return sorted values or error
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
fanninpm
left a comment
There was a problem hiding this comment.
We have activated the redundant_test_prefix Clippy lint, so Clippy is complaining about test functions whose names begin with test. Here are suggestions of ways you can rename these functions:
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (4)
crates/vm/src/lib.rs (1)
95-95: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winNarrow the module visibility to
pub(crate).
crates/vm/src/sorting.rsexports onlypub(crate) fn timsort, and the sole consumer iscrates/vm/src/builtins/list.rs. Declaring the modulepubadds an empty public namespace to the crate API and makes an accidental futurepubitem externally visible.♻️ Proposed change
-pub mod sorting; +pub(crate) mod sorting;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/vm/src/lib.rs` at line 95, Change the sorting module declaration in the crate root from public to crate-visible by using `pub(crate) mod sorting`, while leaving the existing `timsort` implementation and its internal consumer unchanged.crates/vm/src/sorting.rs (3)
652-697: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy liftConsider moving elements instead of cloning them.
The
T: Clonebound forcesmerge_loandmerge_hito clone on every element movement. ForPyObjectRefeach move costs one reference-count increment plus one decrement on the overwritten slot, while CPython moves raw pointers. This is a plausible contributor to the remaining gap against CPython (0.78s vs 0.21s in the PR description).A bitwise-move implementation (
ptr::copy_nonoverlappinginto aMaybeUninitbuffer, with the buffer treated as logically uninitialized after each element is moved out) removes both theClonebound and the refcount traffic. It also removes the duplicate-element hazard on the comparator error path, because moved-out slots are never live twice.Treat this as a follow-up, not a merge blocker. The unsafe code needs careful drop and panic-safety handling.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/vm/src/sorting.rs` around lines 652 - 697, The sorting implementation currently requires T: Clone and clones elements during merge operations. As a follow-up, update timsort and its merge helpers such as merge_lo and merge_hi to move elements through a carefully managed MaybeUninit buffer using non-overlapping pointer copies, removing the Clone bound while preserving correct ownership, drop, comparator-error, and panic-safety behavior.
699-741: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd tests for stability, galloping merges, and comparator errors.
Current coverage misses the highest-risk paths:
- No stability test. Sort
(key, index)pairs onkeyonly and assert the indices stay ascending within each key group.pseudorandom_collectionproduces a permutation, because 7919 is prime and coprime with 500. The input has no duplicate keys, so runs of equal elements never occur. Add an input with many repeated keys and at least 64 elements to drivemerge_loandmerge_higalloping.one_thousand_elementsis one fully descending run, so it returns before any merge. Add a multi-run input (for example alternating ascending and descending blocks).- No test uses a non-
()error type. Add a test whereis_ltreturnsErrafter N calls, and assert both that the error propagates and that the sorted slice is still a permutation of the input. That test covers the failure-recovery gap flagged inmerge_loandmerge_hi.Do you want me to generate these tests?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/vm/src/sorting.rs` around lines 699 - 741, Expand the sorting tests around the `sort` helper with a stability test for `(key, index)` pairs using key-only comparison, repeated-key inputs of at least 64 elements that exercise galloping merges, and a multi-run input such as alternating ascending and descending blocks. Add a comparator-error test using a non-`() ` error after a fixed number of calls, asserting the error propagates and the resulting slice remains a permutation of the original input.
1-3: 📐 Maintainability & Code Quality | 🔵 TrivialLeftover porting TODO references a symbol that does not exist here.
MERGESTATE_TEMP_SIZEis a CPython C-level constant and has no counterpart in this file. The note describes an intentional design decision (dynamicVecbuffer), not pending work. Consider converting it to a short explanatory//comment or removing it. A second TODO at Line 618 tracks CPython 3.12+ incremental minrun.Do you want me to open a tracking issue for the incremental minrun improvement?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/vm/src/sorting.rs` around lines 1 - 3, Remove the obsolete TODO above MIN_GALLOP because MERGESTATE_TEMP_SIZE is not defined in this Rust implementation. If the dynamic Vec buffer choice should remain documented, replace the TODO with a brief explanatory comment; leave the separate incremental-minrun TODO near the later sorting logic unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/vm/src/builtins/list.rs`:
- Around line 644-652: Update the comments immediately above the is_lt closure
to remove the stale try_sort_by_gt and is_gt references, and describe the
current __lt__-based comparison behavior, including how reverse changes the
operand order.
In `@crates/vm/src/sorting.rs`:
- Around line 164-178: Update both merge routines in crates/vm/src/sorting.rs at
lines 164-178 and 344-359 to recover buffered elements before propagating
comparator errors: in the first routine restore remaining len_a elements from
self.buf[cursor_a..] into values[dest..], and in the second restore remaining
len_b elements from self.buf[..len_b] into values[dest + 1 - len_b..=dest]. Use
an inner helper and wrapper per merge function if needed so every error path
performs the appropriate recovery before returning.
---
Nitpick comments:
In `@crates/vm/src/lib.rs`:
- Line 95: Change the sorting module declaration in the crate root from public
to crate-visible by using `pub(crate) mod sorting`, while leaving the existing
`timsort` implementation and its internal consumer unchanged.
In `@crates/vm/src/sorting.rs`:
- Around line 652-697: The sorting implementation currently requires T: Clone
and clones elements during merge operations. As a follow-up, update timsort and
its merge helpers such as merge_lo and merge_hi to move elements through a
carefully managed MaybeUninit buffer using non-overlapping pointer copies,
removing the Clone bound while preserving correct ownership, drop,
comparator-error, and panic-safety behavior.
- Around line 699-741: Expand the sorting tests around the `sort` helper with a
stability test for `(key, index)` pairs using key-only comparison, repeated-key
inputs of at least 64 elements that exercise galloping merges, and a multi-run
input such as alternating ascending and descending blocks. Add a
comparator-error test using a non-`() ` error after a fixed number of calls,
asserting the error propagates and the resulting slice remains a permutation of
the original input.
- Around line 1-3: Remove the obsolete TODO above MIN_GALLOP because
MERGESTATE_TEMP_SIZE is not defined in this Rust implementation. If the dynamic
Vec buffer choice should remain documented, replace the TODO with a brief
explanatory comment; leave the separate incremental-minrun TODO near the later
sorting logic unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yml
Review profile: CHILL
Plan: Pro Plus
Run ID: f6b47947-f422-4a9e-bed8-db1c9f5a6f9b
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (5)
Cargo.tomlcrates/vm/Cargo.tomlcrates/vm/src/builtins/list.rscrates/vm/src/lib.rscrates/vm/src/sorting.rs
💤 Files with no reviewable changes (2)
- crates/vm/Cargo.toml
- Cargo.toml
youknowone
left a comment
There was a problem hiding this comment.
awesome, TIL what's powersort. Thank you!
Summary
Replaces the
rust-timsortcrate with an in-tree powersort implementation.rust-timsortdegrades to O(N²) on random input. Sorting 1M random floats took 16 minutes (#6093). This implements CPython 3.11+'s powersort (Tim Peters' timsort with powersort's merge-ordering policy).What changed
crates/vm/src/sorting.rs: run detection, binary insertion for short runs, galloping merge (merge_lo/merge_hi), and power-based merge ordering (powerloop). Generic over the element type via a fallibleis_ltclosure, so the algorithm stays free of interpreter details and is unit-tested withi32(no VM needed).list.sort()now goes through it, and therust-timsortdependency is removed.Results
Measured against CPython 3.14 (release build, 1M elements):
Correctness verified against CPython (reverse, key, stability, and a debug-mode stress test across many sizes/patterns to exercise the
merge_hiboundaries).Follow-ups
The remaining ~4x vs CPython is constant factors, not algorithm. I plan to follow up with refcount-free moves and type-specialized comparators. Opening as a draft, so happy to hear whether those should land here or as separate PRs.
Notes
Offsets in the galloping and merge routines use
isizeto mirror CPython's signedPy_ssize_t, casting back tousizefor indexing where values are non-negative.Summary by CodeRabbit