Skip to content

Replace rust-timsort with an in-tree powersort - #8421

Merged
youknowone merged 7 commits into
RustPython:mainfrom
kangdora:powersort
Aug 2, 2026
Merged

Replace rust-timsort with an in-tree powersort#8421
youknowone merged 7 commits into
RustPython:mainfrom
kangdora:powersort

Conversation

@kangdora

@kangdora kangdora commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Replaces the rust-timsort crate with an in-tree powersort implementation.

rust-timsort degrades to O(N²) on random input. Sorting 1M random floats took 16 minutes (#6093). This implements CPython 3.11+'s powersort (Tim Peters' timsort with powersort's merge-ordering policy).

What changed

  • crates/vm/src/sorting.rs: run detection, binary insertion for short runs, galloping merge (merge_lo/merge_hi), and power-based merge ordering (powerloop). Generic over the element type via a fallible is_lt closure, so the algorithm stays free of interpreter details and is unit-tested with i32 (no VM needed).
  • list.sort() now goes through it, and the rust-timsort dependency is removed.

Results

Measured against CPython 3.14 (release build, 1M elements):

workload before (rust-timsort) after CPython 3.14
1M random floats ~16 min 0.78s 0.21s
1M sorted ints n/a 0.05s 0.01s
1M small-range ints n/a 0.39s 0.08s

Correctness verified against CPython (reverse, key, stability, and a debug-mode stress test across many sizes/patterns to exercise the merge_hi boundaries).

Follow-ups

The remaining ~4x vs CPython is constant factors, not algorithm. I plan to follow up with refcount-free moves and type-specialized comparators. Opening as a draft, so happy to hear whether those should land here or as separate PRs.

Notes

Offsets in the galloping and merge routines use isize to mirror CPython's signed Py_ssize_t, casting back to usize for indexing where values are non-negative.

Summary by CodeRabbit

  • Bug Fixes
    • Improved list sorting reliability and consistency in ascending and descending order.
    • Sorting now preserves stable ordering for equal items and handles already sorted, reversed, duplicate, large, and randomly ordered lists effectively.
    • Sorting errors are reported correctly instead of being silently lost.
    • Adaptive sorting improves efficiency across different input patterns while maintaining predictable results.

Add crates/vm/src/sorting.rs implementing Tim Peters' timsort with powersort's merge-ordering policy (CPython 3.11+): run detection, binary insertion for short runs, galloping merge (merge_lo/merge_hi), and power-based merge ordering (powerloop). Comparison is passed in as a fallible `is_lt` closure, so the algorithm stays generic over the element type and free of interpreter details.
Not yet wired into list.sort(); replaces rust-timsort in a follow-up.
Wire list sorting through crate::sorting::timsort and remove the
rust-timsort dependency. Fixes the O(N^2) behavior on random input
(1M random floats: ~16min -> ~0.8s), now within ~4x of CPython.
@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: af4e7003-c7a2-4eb1-a413-fa6ed2667735

📥 Commits

Reviewing files that changed from the base of the PR and between 526820b and b4fda72.

📒 Files selected for processing (2)
  • crates/vm/src/builtins/list.rs
  • crates/vm/src/sorting.rs
🚧 Files skipped from review as they are similar to previous changes (2)
  • crates/vm/src/builtins/list.rs
  • crates/vm/src/sorting.rs

📝 Walkthrough

Walkthrough

The VM replaces the external timsort dependency with an in-tree Powersort-based implementation. List sorting now uses the new comparator interface. The implementation includes adaptive runs, galloping merges, error propagation, and unit tests.

Changes

VM sorting implementation

Layer / File(s) Summary
Timsort engine and validation
crates/vm/src/sorting.rs
Adds run detection, binary insertion sorting, galloping merges, Powersort scheduling, comparator error propagation, and tests for ordered, reversed, duplicate, large, and pseudorandom inputs.
List sorting integration and dependency removal
Cargo.toml, crates/vm/Cargo.toml, crates/vm/src/lib.rs, crates/vm/src/builtins/list.rs
Removes external timsort declarations, exposes the sorting module, and updates keyed and unkeyed list sorting to call the in-tree implementation with ascending or descending is_lt comparisons.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related issues

  • RustPython issue 6093: Replaces the external rust-timsort dependency with an in-tree Powersort-based timsort implementation.

Possibly related PRs

Sequence Diagram(s)

sequenceDiagram
  participant ListSort
  participant Timsort
  participant IsLtComparator
  ListSort->>Timsort: Sort list values and keys
  Timsort->>IsLtComparator: Compare elements with is_lt
  IsLtComparator-->>Timsort: Return ordering or error
  Timsort-->>ListSort: Return sorted values or error
Loading

Suggested reviewers: moreal, shaharnaveh

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes replacing the rust-timsort dependency with an in-tree powersort implementation.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kangdora kangdora changed the title Powersort Replace rust-timsort with an in-tree powersort Jul 31, 2026

@fanninpm fanninpm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have activated the redundant_test_prefix Clippy lint, so Clippy is complaining about test functions whose names begin with test. Here are suggestions of ways you can rename these functions:

Comment thread crates/vm/src/sorting.rs Outdated
Comment thread crates/vm/src/sorting.rs Outdated
Comment thread crates/vm/src/sorting.rs Outdated
Comment thread crates/vm/src/sorting.rs Outdated
Comment thread crates/vm/src/sorting.rs Outdated

@ShaharNaveh ShaharNaveh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow, tysm!

@youknowone youknowone added the z-ca-2026 Tag to track Contribution Academy 2026 label Aug 1, 2026
@kangdora
kangdora marked this pull request as ready for review August 1, 2026 04:57

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (4)
crates/vm/src/lib.rs (1)

95-95: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Narrow the module visibility to pub(crate).

crates/vm/src/sorting.rs exports only pub(crate) fn timsort, and the sole consumer is crates/vm/src/builtins/list.rs. Declaring the module pub adds an empty public namespace to the crate API and makes an accidental future pub item externally visible.

♻️ Proposed change
-pub mod sorting;
+pub(crate) mod sorting;
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/vm/src/lib.rs` at line 95, Change the sorting module declaration in
the crate root from public to crate-visible by using `pub(crate) mod sorting`,
while leaving the existing `timsort` implementation and its internal consumer
unchanged.
crates/vm/src/sorting.rs (3)

652-697: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift

Consider moving elements instead of cloning them.

The T: Clone bound forces merge_lo and merge_hi to clone on every element movement. For PyObjectRef each move costs one reference-count increment plus one decrement on the overwritten slot, while CPython moves raw pointers. This is a plausible contributor to the remaining gap against CPython (0.78s vs 0.21s in the PR description).

A bitwise-move implementation (ptr::copy_nonoverlapping into a MaybeUninit buffer, with the buffer treated as logically uninitialized after each element is moved out) removes both the Clone bound and the refcount traffic. It also removes the duplicate-element hazard on the comparator error path, because moved-out slots are never live twice.

Treat this as a follow-up, not a merge blocker. The unsafe code needs careful drop and panic-safety handling.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/vm/src/sorting.rs` around lines 652 - 697, The sorting implementation
currently requires T: Clone and clones elements during merge operations. As a
follow-up, update timsort and its merge helpers such as merge_lo and merge_hi to
move elements through a carefully managed MaybeUninit buffer using
non-overlapping pointer copies, removing the Clone bound while preserving
correct ownership, drop, comparator-error, and panic-safety behavior.

699-741: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add tests for stability, galloping merges, and comparator errors.

Current coverage misses the highest-risk paths:

  • No stability test. Sort (key, index) pairs on key only and assert the indices stay ascending within each key group.
  • pseudorandom_collection produces a permutation, because 7919 is prime and coprime with 500. The input has no duplicate keys, so runs of equal elements never occur. Add an input with many repeated keys and at least 64 elements to drive merge_lo and merge_hi galloping.
  • one_thousand_elements is one fully descending run, so it returns before any merge. Add a multi-run input (for example alternating ascending and descending blocks).
  • No test uses a non-() error type. Add a test where is_lt returns Err after N calls, and assert both that the error propagates and that the sorted slice is still a permutation of the input. That test covers the failure-recovery gap flagged in merge_lo and merge_hi.

Do you want me to generate these tests?

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/vm/src/sorting.rs` around lines 699 - 741, Expand the sorting tests
around the `sort` helper with a stability test for `(key, index)` pairs using
key-only comparison, repeated-key inputs of at least 64 elements that exercise
galloping merges, and a multi-run input such as alternating ascending and
descending blocks. Add a comparator-error test using a non-`() ` error after a
fixed number of calls, asserting the error propagates and the resulting slice
remains a permutation of the original input.

1-3: 📐 Maintainability & Code Quality | 🔵 Trivial

Leftover porting TODO references a symbol that does not exist here.

MERGESTATE_TEMP_SIZE is a CPython C-level constant and has no counterpart in this file. The note describes an intentional design decision (dynamic Vec buffer), not pending work. Consider converting it to a short explanatory // comment or removing it. A second TODO at Line 618 tracks CPython 3.12+ incremental minrun.

Do you want me to open a tracking issue for the incremental minrun improvement?

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/vm/src/sorting.rs` around lines 1 - 3, Remove the obsolete TODO above
MIN_GALLOP because MERGESTATE_TEMP_SIZE is not defined in this Rust
implementation. If the dynamic Vec buffer choice should remain documented,
replace the TODO with a brief explanatory comment; leave the separate
incremental-minrun TODO near the later sorting logic unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/vm/src/builtins/list.rs`:
- Around line 644-652: Update the comments immediately above the is_lt closure
to remove the stale try_sort_by_gt and is_gt references, and describe the
current __lt__-based comparison behavior, including how reverse changes the
operand order.

In `@crates/vm/src/sorting.rs`:
- Around line 164-178: Update both merge routines in crates/vm/src/sorting.rs at
lines 164-178 and 344-359 to recover buffered elements before propagating
comparator errors: in the first routine restore remaining len_a elements from
self.buf[cursor_a..] into values[dest..], and in the second restore remaining
len_b elements from self.buf[..len_b] into values[dest + 1 - len_b..=dest]. Use
an inner helper and wrapper per merge function if needed so every error path
performs the appropriate recovery before returning.

---

Nitpick comments:
In `@crates/vm/src/lib.rs`:
- Line 95: Change the sorting module declaration in the crate root from public
to crate-visible by using `pub(crate) mod sorting`, while leaving the existing
`timsort` implementation and its internal consumer unchanged.

In `@crates/vm/src/sorting.rs`:
- Around line 652-697: The sorting implementation currently requires T: Clone
and clones elements during merge operations. As a follow-up, update timsort and
its merge helpers such as merge_lo and merge_hi to move elements through a
carefully managed MaybeUninit buffer using non-overlapping pointer copies,
removing the Clone bound while preserving correct ownership, drop,
comparator-error, and panic-safety behavior.
- Around line 699-741: Expand the sorting tests around the `sort` helper with a
stability test for `(key, index)` pairs using key-only comparison, repeated-key
inputs of at least 64 elements that exercise galloping merges, and a multi-run
input such as alternating ascending and descending blocks. Add a
comparator-error test using a non-`() ` error after a fixed number of calls,
asserting the error propagates and the resulting slice remains a permutation of
the original input.
- Around line 1-3: Remove the obsolete TODO above MIN_GALLOP because
MERGESTATE_TEMP_SIZE is not defined in this Rust implementation. If the dynamic
Vec buffer choice should remain documented, replace the TODO with a brief
explanatory comment; leave the separate incremental-minrun TODO near the later
sorting logic unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: f6b47947-f422-4a9e-bed8-db1c9f5a6f9b

📥 Commits

Reviewing files that changed from the base of the PR and between f08933b and 526820b.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (5)
  • Cargo.toml
  • crates/vm/Cargo.toml
  • crates/vm/src/builtins/list.rs
  • crates/vm/src/lib.rs
  • crates/vm/src/sorting.rs
💤 Files with no reviewable changes (2)
  • crates/vm/Cargo.toml
  • Cargo.toml

Comment thread crates/vm/src/builtins/list.rs
Comment thread crates/vm/src/sorting.rs
@kangdora
kangdora marked this pull request as draft August 1, 2026 05:54
@kangdora
kangdora marked this pull request as ready for review August 1, 2026 07:17

@youknowone youknowone left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome, TIL what's powersort. Thank you!

@youknowone
youknowone merged commit 6131363 into RustPython:main Aug 2, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

z-ca-2026 Tag to track Contribution Academy 2026

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants