csv: port CPython reader state machine - #8381
Conversation
📝 WalkthroughWalkthroughCSV dialects now preserve arbitrary ChangesCSV terminator and parser update
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
📦 Library DependenciesThe following Lib/ modules were modified. Here are their dependencies: [x] lib: cpython/Lib/csv.py dependencies:
dependent tests: (4 tests)
Legend:
|
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (3)
crates/stdlib/src/csv.rs (3)
1516-1525: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winAvoid a hard
assert_eq!panic on a runtime buffer invariant.If csv-core ever emits the record without the sentinel as the final byte, this panics and takes down the interpreter instead of surfacing a Python-level error.
buffer_offsetis also used unguarded for the- 1index. A checked strip keeps the fast path identical and degrades safely.🛡️ Proposed fix
- assert_eq!(buffer[buffer_offset - 1], CSV_CORE_TERMINATOR_SENTINEL); - let mut output = buffer[..buffer_offset - 1].to_vec(); + let emitted = &buffer[..buffer_offset]; + let body = emitted + .strip_suffix(&[CSV_CORE_TERMINATOR_SENTINEL]) + .ok_or_else(|| new_csv_error(vm, "internal error: missing record terminator"))?; + let mut output = body.to_vec(); output.extend_from_slice(self.dialect.lineterminator.as_bytes());🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/stdlib/src/csv.rs` around lines 1516 - 1525, Replace the unchecked sentinel assertion and buffer_offset - 1 access in the CSV serialization path with a checked strip operation that verifies buffer_offset is nonzero and the final byte is CSV_CORE_TERMINATOR_SENTINEL. Preserve the existing output construction when the sentinel is present, and return the established Python-level error through the surrounding error-handling path when the invariant fails.
841-841: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valuecsv-core quoting decisions now ignore the real terminator bytes.
Since the builder is told the terminator is the sentinel
\n, csv-core's "necessary" quoting logic will never quote a field because it contains a byte of a custom terminator (e.g.!@#). This is harmless forQUOTE_ALL, and mostly harmless forQUOTE_NONNUMERIC(strings are always quoted), but an unquoted stringified non-string field containing a terminator byte would round-trip incorrectly. Worth a follow-up test or an explicit note here.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/stdlib/src/csv.rs` at line 841, Update the writer setup around writer.terminator in the CSV serialization path so csv-core’s necessary-quoting logic recognizes the actual custom terminator bytes rather than only the sentinel newline. Preserve existing behavior for standard terminators and add a regression test covering an unquoted stringified non-string field containing a custom terminator byte.
237-250: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueUse a single exception type for invalid
lineterminatorhandling.
prase_lineterminator_from_objrejects"lineterminator": ""with_csv.Error, whileFormatOptions::from_argsrejects the same condition withTypeError. This makes Python-levelexceptbehavior depend on whether the value comes from a dialect object or keyword arguments. Also, CPython allowslineterminator=""for writers, so if full CSV parity is desired, remove the empty-check from both paths.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/stdlib/src/csv.rs` around lines 237 - 250, Make invalid lineterminator handling consistent across prase_lineterminator_from_obj and FormatOptions::from_args by using the same exception type for empty values. If matching CPython writer behavior, remove the empty-string rejection from both paths so lineterminator="" is accepted; otherwise apply the shared exception consistently.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@extra_tests/snippets/stdlib_csv.py`:
- Around line 252-255: Update the csv.reader assertion in the CRLF behavior test
to include an actual "\r\n" record boundary while retaining the custom "!@#"
terminator in the input, and assert that records split only at CRLF while "!@#"
remains field data. Keep the existing validation for the trailing custom
terminator as appropriate.
---
Nitpick comments:
In `@crates/stdlib/src/csv.rs`:
- Around line 1516-1525: Replace the unchecked sentinel assertion and
buffer_offset - 1 access in the CSV serialization path with a checked strip
operation that verifies buffer_offset is nonzero and the final byte is
CSV_CORE_TERMINATOR_SENTINEL. Preserve the existing output construction when the
sentinel is present, and return the established Python-level error through the
surrounding error-handling path when the invariant fails.
- Line 841: Update the writer setup around writer.terminator in the CSV
serialization path so csv-core’s necessary-quoting logic recognizes the actual
custom terminator bytes rather than only the sentinel newline. Preserve existing
behavior for standard terminators and add a regression test covering an unquoted
stringified non-string field containing a custom terminator byte.
- Around line 237-250: Make invalid lineterminator handling consistent across
prase_lineterminator_from_obj and FormatOptions::from_args by using the same
exception type for empty values. If matching CPython writer behavior, remove the
empty-string rejection from both paths so lineterminator="" is accepted;
otherwise apply the shared exception consistently.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yml
Review profile: CHILL
Plan: Pro Plus
Run ID: 155e72fb-eec0-4847-b10b-6aae2448c20c
⛔ Files ignored due to path filters (1)
Lib/test/test_csv.pyis excluded by!Lib/**
📒 Files selected for processing (3)
.cspell.jsoncrates/stdlib/src/csv.rsextra_tests/snippets/stdlib_csv.py
fb97015 to
d94c6d9
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@extra_tests/snippets/stdlib_csv.py`:
- Around line 277-285: Rename the local variable input in the CSV reader test to
source or data, and update both csv.reader calls to use the renamed variable
while preserving the existing assertions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yml
Review profile: CHILL
Plan: Pro Plus
Run ID: 2d476e0d-8c0b-4487-ae20-388503fd3a43
📒 Files selected for processing (2)
crates/stdlib/src/csv.rsextra_tests/snippets/stdlib_csv.py
🚧 Files skipped from review as they are similar to previous changes (1)
- crates/stdlib/src/csv.rs
Replace the csv-core reader and per-item quote scanner with one Rust implementation of CPython's nine-state reader parser. Keep parser state across iterator items and distinguish virtual item boundaries from true iterator exhaustion. Centralize field completion so quote provenance, empty-field None conversion, float conversion, strict parsing, field limits, blank rows, and escaped or quoted newlines share one path. Apply the existing reentrant-iterator generation check to every iterator item consumed by a record. Also accept empty ASCII line terminators like CPython, report a missing csv-core writer sentinel as _csv.Error instead of panicking, and name the FSM entry point process_parser_input. Remove the expected-failure markers from the eleven reader tests that now pass. Assisted-by: Codex:gpt-5.6-sol
f8d9250 to
d19db71
Compare
|
@coderabbitai resume |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
🧹 Nitpick comments (1)
crates/stdlib/src/csv.rs (1)
1326-1470: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueOptional: factor out the shared row-writing scaffolding.
writerow_quoted_strings,writerow_quote_none, andwriterow_minimaldiffer only in the per-field quoting decision; theArgIterablecoercion,match_class!stringification, delimiter insertion, terminator append, and UTF-8 conversion are copied three times. A single helper taking a per-field closure (or a small enum) would remove ~60 duplicated lines and keep future terminator/quoting fixes in one place.As per coding guidelines, "When branches differ only in a value but share common logic, extract the differing value first, then call the common logic once to avoid duplicate code".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/stdlib/src/csv.rs` around lines 1326 - 1470, Optionally extract the shared row-writing flow from writerow_quoted_strings, writerow_quote_none, and writerow_minimal into one helper, parameterizing only each mode’s per-field quoting decision. Preserve the existing iterable conversion, field stringification, delimiter handling, line terminator, UTF-8 conversion, and error behavior while removing the duplicated scaffolding.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@crates/stdlib/src/csv.rs`:
- Around line 1326-1470: Optionally extract the shared row-writing flow from
writerow_quoted_strings, writerow_quote_none, and writerow_minimal into one
helper, parameterizing only each mode’s per-field quoting decision. Preserve the
existing iterable conversion, field stringification, delimiter handling, line
terminator, UTF-8 conversion, and error behavior while removing the duplicated
scaffolding.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yml
Review profile: CHILL
Plan: Pro Plus
Run ID: 0b0f6fcf-e294-4a7b-ad01-88547d7921f4
⛔ Files ignored due to path filters (1)
Lib/test/test_csv.pyis excluded by!Lib/**
📒 Files selected for processing (3)
.cspell.jsoncrates/stdlib/src/csv.rsextra_tests/snippets/stdlib_csv.py
🚧 Files skipped from review as they are similar to previous changes (1)
- .cspell.json
✅ Action performedReviews resumed. |
youknowone
left a comment
There was a problem hiding this comment.
Thank you for working on this painful task!
Summary
RustPython's CSV reader combined
csv-corewith custom parsing andpreprocessing paths. The backends did not share the same parser state, and an
iterator-item boundary was treated like parser EOF. As a result, behavior such
as multiline records, unquoted escapes, strict parsing, blank records, and
quoted-field provenance was split across partially overlapping paths.
Replace those reader paths with one byte-based Rust implementation of
CPython's nine-state CSV reader state machine:
exhaustion;
QUOTE_NOTNULL,QUOTE_STRINGS, and numeric conversion;line_num; andconsumed by a record.
For example, a quoted field can now span iterator items without inserting a
newline at the item boundary:
The existing CPython tests now cover malformed quotes, iterator-item EOL
handling, true EOF, NUL and escape handling, quoting conversions, space
delimiters, quoted and escaped newlines, and blank
DictReaderinput. Removethe eleven
expectedFailuremarkers whose unchanged test bodies now pass.Review follow-ups also accept empty ASCII
lineterminatorvalues like CPython,report a missing csv-core writer sentinel as
_csv.Errorinstead of panicking,and name the FSM entry point
process_parser_input.Scope
The reader state machine remains byte-based in this change. Full Unicode
delimiter, quote, and escape character support is deferred; the parser shape,
iterator lifecycle, and field-completion path are designed so that character
representation can be changed separately.
Writer behavior is not redesigned here. The implementation builds on #8328,
which is now part of
main, and preserves the quote-awareskipinitialspacesemantics from #8304. The empty-terminator and checked-sentinel changes are
narrow compatibility and safety follow-ups; the remaining csv-core
necessary-quoting limitation for custom terminator bytes is outside this
reader-focused change.
Testing
uv tool run prek run --all-filescargo fmt --checkcargo run --release Lib/test/test_csv.pycargo run -- extra_tests/snippets/stdlib_csv.pycargo clippy -p rustpython-stdlib --all-targetsWorkspace test command
rustpython-capiis tested separately in CI; its test binary previously couldnot load
libpython3.14.so.1.0in this local environment.rustpython-compiler-sourceis deprecated and is excluded from CI workspacebuilds as well.
cargo test --workspace \ --exclude rustpython-capi \ --exclude rustpython_wasm \ --exclude rustpython-compiler-source \ --exclude rustpython-venvlauncher \ --features threading \ -- --quietAI assistance: Codex:gpt-5.6-sol
Summary by CodeRabbit
Summary by CodeRabbit