Skip to content

csv: port CPython reader state machine - #8381

Merged
youknowone merged 2 commits into
RustPython:mainfrom
widehyo1:implement-cpython-csv-reader-fsm
Aug 1, 2026
Merged

csv: port CPython reader state machine#8381
youknowone merged 2 commits into
RustPython:mainfrom
widehyo1:implement-cpython-csv-reader-fsm

Conversation

@widehyo1

@widehyo1 widehyo1 commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Summary

RustPython's CSV reader combined csv-core with custom parsing and
preprocessing paths. The backends did not share the same parser state, and an
iterator-item boundary was treated like parser EOF. As a result, behavior such
as multiline records, unquoted escapes, strict parsing, blank records, and
quoted-field provenance was split across partially overlapping paths.

Replace those reader paths with one byte-based Rust implementation of
CPython's nine-state CSV reader state machine:

  • preserve parser state across Python iterator items;
  • distinguish the virtual end of an iterator item from true iterator
    exhaustion;
  • recognize escapes in quoted and unquoted fields;
  • implement strict closing-quote and true-EOF errors;
  • retain quoted versus unquoted field provenance for QUOTE_NOTNULL,
    QUOTE_STRINGS, and numeric conversion;
  • count every successfully obtained string item in line_num; and
  • apply the existing reentrant-iterator generation check to every item
    consumed by a record.

For example, a quoted field can now span iterator items without inserting a
newline at the item boundary:

import csv

reader = csv.reader(['a,"b', 'c",d'])
assert next(reader) == ["a", "bc", "d"]

The existing CPython tests now cover malformed quotes, iterator-item EOL
handling, true EOF, NUL and escape handling, quoting conversions, space
delimiters, quoted and escaped newlines, and blank DictReader input. Remove
the eleven expectedFailure markers whose unchanged test bodies now pass.

Review follow-ups also accept empty ASCII lineterminator values like CPython,
report a missing csv-core writer sentinel as _csv.Error instead of panicking,
and name the FSM entry point process_parser_input.

Scope

The reader state machine remains byte-based in this change. Full Unicode
delimiter, quote, and escape character support is deferred; the parser shape,
iterator lifecycle, and field-completion path are designed so that character
representation can be changed separately.

Writer behavior is not redesigned here. The implementation builds on #8328,
which is now part of main, and preserves the quote-aware skipinitialspace
semantics from #8304. The empty-terminator and checked-sentinel changes are
narrow compatibility and safety follow-ups; the remaining csv-core
necessary-quoting limitation for custom terminator bytes is outside this
reader-focused change.

Testing

  • uv tool run prek run --all-files
  • cargo fmt --check
  • cargo run --release Lib/test/test_csv.py
    • 128 tests run
    • 6 skipped
    • 6 expected failures
  • cargo run -- extra_tests/snippets/stdlib_csv.py
  • cargo clippy -p rustpython-stdlib --all-targets
  • the configured workspace test suite
Workspace test command

rustpython-capi is tested separately in CI; its test binary previously could
not load libpython3.14.so.1.0 in this local environment.
rustpython-compiler-source is deprecated and is excluded from CI workspace
builds as well.

cargo test --workspace \
    --exclude rustpython-capi \
    --exclude rustpython_wasm \
    --exclude rustpython-compiler-source \
    --exclude rustpython-venvlauncher \
    --features threading \
    -- --quiet

AI assistance: Codex:gpt-5.6-sol

Summary by CodeRabbit

Summary by CodeRabbit

  • New Features
    • Added support for multi-character, empty, and non-ASCII CSV line terminators.
    • Improved CSV reader and writer handling for quoting, escaping, delimiters, CRLF line breaks, and chunked input.
  • Bug Fixes
    • Fixed empty line terminators to concatenate output records correctly.
    • Improved strict end-of-file and field-size handling.
  • Tests
    • Added coverage for custom terminators, dialect behavior, quoting, escaping, and embedded line breaks.
  • Chores
    • Updated the spelling allowlist.

@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

CSV dialects now preserve arbitrary lineterminator strings, including multi-character and empty values. Writers emit configured terminators, while readers use a custom parser for CSV record handling. Tests cover quoting, escaping, dialect registration, and CRLF behavior.

Changes

CSV terminator and parser update

Layer / File(s) Summary
Dialect and format option contracts
crates/stdlib/src/csv.rs
lineterminator is validated as ASCII, stored as an owned string, and propagated through dialect resolution.
Custom CSV reader state machine
crates/stdlib/src/csv.rs
The previous csv-core reader path is replaced with a parser handling fields, quoting, escaping, record boundaries, strictness, limits, and EOF.
Writer quoting and terminator emission
crates/stdlib/src/csv.rs
Writer helpers inspect string terminators and replace the csv-core sentinel with the configured output terminator.
Terminator compatibility validation
extra_tests/snippets/stdlib_csv.py, .cspell.json
Tests cover multi-character and empty terminators, dialect registration, CRLF reader behavior, and the Crnl spelling allowlist.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related issues

Possibly related PRs

Suggested reviewers: shaharnaveh, youknowone

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The parser meets most stateful CPython-style reader requirements, but Unicode delimiter, quote, and escape support required by #8310 is explicitly deferred. Add Unicode delimiter, quote, and escape support, or establish and validate an architecture that clearly preserves a path to those requirements.
Docstring Coverage ⚠️ Warning Docstring coverage is 10.53% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Out of Scope Changes check ✅ Passed The changes remain focused on the CSV reader and directly related compatibility tests and spell-check updates.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the primary change: porting the CPython CSV reader state machine.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

[x] lib: cpython/Lib/csv.py
[x] test: cpython/Lib/test/test_csv.py (TODO: 6)

dependencies:

  • csv

dependent tests: (4 tests)

  • csv: test_csv test_genericalias
    • importlib.metadata: test_importlib test_zoneinfo

Legend:

  • [+] path exists in CPython
  • [x] up-to-date, [ ] outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (3)
crates/stdlib/src/csv.rs (3)

1516-1525: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Avoid a hard assert_eq! panic on a runtime buffer invariant.

If csv-core ever emits the record without the sentinel as the final byte, this panics and takes down the interpreter instead of surfacing a Python-level error. buffer_offset is also used unguarded for the - 1 index. A checked strip keeps the fast path identical and degrades safely.

🛡️ Proposed fix
-        assert_eq!(buffer[buffer_offset - 1], CSV_CORE_TERMINATOR_SENTINEL);
-        let mut output = buffer[..buffer_offset - 1].to_vec();
+        let emitted = &buffer[..buffer_offset];
+        let body = emitted
+            .strip_suffix(&[CSV_CORE_TERMINATOR_SENTINEL])
+            .ok_or_else(|| new_csv_error(vm, "internal error: missing record terminator"))?;
+        let mut output = body.to_vec();
         output.extend_from_slice(self.dialect.lineterminator.as_bytes());
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/stdlib/src/csv.rs` around lines 1516 - 1525, Replace the unchecked
sentinel assertion and buffer_offset - 1 access in the CSV serialization path
with a checked strip operation that verifies buffer_offset is nonzero and the
final byte is CSV_CORE_TERMINATOR_SENTINEL. Preserve the existing output
construction when the sentinel is present, and return the established
Python-level error through the surrounding error-handling path when the
invariant fails.

841-841: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

csv-core quoting decisions now ignore the real terminator bytes.

Since the builder is told the terminator is the sentinel \n, csv-core's "necessary" quoting logic will never quote a field because it contains a byte of a custom terminator (e.g. !@#). This is harmless for QUOTE_ALL, and mostly harmless for QUOTE_NONNUMERIC (strings are always quoted), but an unquoted stringified non-string field containing a terminator byte would round-trip incorrectly. Worth a follow-up test or an explicit note here.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/stdlib/src/csv.rs` at line 841, Update the writer setup around
writer.terminator in the CSV serialization path so csv-core’s necessary-quoting
logic recognizes the actual custom terminator bytes rather than only the
sentinel newline. Preserve existing behavior for standard terminators and add a
regression test covering an unquoted stringified non-string field containing a
custom terminator byte.

237-250: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

Use a single exception type for invalid lineterminator handling.

prase_lineterminator_from_obj rejects "lineterminator": "" with _csv.Error, while FormatOptions::from_args rejects the same condition with TypeError. This makes Python-level except behavior depend on whether the value comes from a dialect object or keyword arguments. Also, CPython allows lineterminator="" for writers, so if full CSV parity is desired, remove the empty-check from both paths.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/stdlib/src/csv.rs` around lines 237 - 250, Make invalid lineterminator
handling consistent across prase_lineterminator_from_obj and
FormatOptions::from_args by using the same exception type for empty values. If
matching CPython writer behavior, remove the empty-string rejection from both
paths so lineterminator="" is accepted; otherwise apply the shared exception
consistently.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@extra_tests/snippets/stdlib_csv.py`:
- Around line 252-255: Update the csv.reader assertion in the CRLF behavior test
to include an actual "\r\n" record boundary while retaining the custom "!@#"
terminator in the input, and assert that records split only at CRLF while "!@#"
remains field data. Keep the existing validation for the trailing custom
terminator as appropriate.

---

Nitpick comments:
In `@crates/stdlib/src/csv.rs`:
- Around line 1516-1525: Replace the unchecked sentinel assertion and
buffer_offset - 1 access in the CSV serialization path with a checked strip
operation that verifies buffer_offset is nonzero and the final byte is
CSV_CORE_TERMINATOR_SENTINEL. Preserve the existing output construction when the
sentinel is present, and return the established Python-level error through the
surrounding error-handling path when the invariant fails.
- Line 841: Update the writer setup around writer.terminator in the CSV
serialization path so csv-core’s necessary-quoting logic recognizes the actual
custom terminator bytes rather than only the sentinel newline. Preserve existing
behavior for standard terminators and add a regression test covering an unquoted
stringified non-string field containing a custom terminator byte.
- Around line 237-250: Make invalid lineterminator handling consistent across
prase_lineterminator_from_obj and FormatOptions::from_args by using the same
exception type for empty values. If matching CPython writer behavior, remove the
empty-string rejection from both paths so lineterminator="" is accepted;
otherwise apply the shared exception consistently.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: 155e72fb-eec0-4847-b10b-6aae2448c20c

📥 Commits

Reviewing files that changed from the base of the PR and between 5db61a0 and fb97015.

⛔ Files ignored due to path filters (1)
  • Lib/test/test_csv.py is excluded by !Lib/**
📒 Files selected for processing (3)
  • .cspell.json
  • crates/stdlib/src/csv.rs
  • extra_tests/snippets/stdlib_csv.py

Comment thread extra_tests/snippets/stdlib_csv.py
@widehyo1
widehyo1 force-pushed the implement-cpython-csv-reader-fsm branch from fb97015 to d94c6d9 Compare July 26, 2026 09:29

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@extra_tests/snippets/stdlib_csv.py`:
- Around line 277-285: Rename the local variable input in the CSV reader test to
source or data, and update both csv.reader calls to use the renamed variable
while preserving the existing assertions.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: 2d476e0d-8c0b-4487-ae20-388503fd3a43

📥 Commits

Reviewing files that changed from the base of the PR and between d94c6d9 and b391726.

📒 Files selected for processing (2)
  • crates/stdlib/src/csv.rs
  • extra_tests/snippets/stdlib_csv.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/stdlib/src/csv.rs

Comment thread extra_tests/snippets/stdlib_csv.py Outdated
@youknowone youknowone added the z-ca-2026 Tag to track Contribution Academy 2026 label Jul 28, 2026
widehyo1 added 2 commits July 30, 2026 11:16
Replace the csv-core reader and per-item quote scanner with one Rust
implementation of CPython's nine-state reader parser. Keep parser state
across iterator items and distinguish virtual item boundaries from true
iterator exhaustion.

Centralize field completion so quote provenance, empty-field None conversion,
float conversion, strict parsing, field limits, blank rows, and escaped or
quoted newlines share one path. Apply the existing reentrant-iterator
generation check to every iterator item consumed by a record.

Also accept empty ASCII line terminators like CPython, report a missing
csv-core writer sentinel as _csv.Error instead of panicking, and name the
FSM entry point process_parser_input.

Remove the expected-failure markers from the eleven reader tests that now
pass.

Assisted-by: Codex:gpt-5.6-sol
@widehyo1
widehyo1 force-pushed the implement-cpython-csv-reader-fsm branch from f8d9250 to d19db71 Compare July 30, 2026 03:08
@widehyo1

widehyo1 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai resume

@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
crates/stdlib/src/csv.rs (1)

1326-1470: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Optional: factor out the shared row-writing scaffolding.

writerow_quoted_strings, writerow_quote_none, and writerow_minimal differ only in the per-field quoting decision; the ArgIterable coercion, match_class! stringification, delimiter insertion, terminator append, and UTF-8 conversion are copied three times. A single helper taking a per-field closure (or a small enum) would remove ~60 duplicated lines and keep future terminator/quoting fixes in one place.

As per coding guidelines, "When branches differ only in a value but share common logic, extract the differing value first, then call the common logic once to avoid duplicate code".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/stdlib/src/csv.rs` around lines 1326 - 1470, Optionally extract the
shared row-writing flow from writerow_quoted_strings, writerow_quote_none, and
writerow_minimal into one helper, parameterizing only each mode’s per-field
quoting decision. Preserve the existing iterable conversion, field
stringification, delimiter handling, line terminator, UTF-8 conversion, and
error behavior while removing the duplicated scaffolding.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/stdlib/src/csv.rs`:
- Around line 1326-1470: Optionally extract the shared row-writing flow from
writerow_quoted_strings, writerow_quote_none, and writerow_minimal into one
helper, parameterizing only each mode’s per-field quoting decision. Preserve the
existing iterable conversion, field stringification, delimiter handling, line
terminator, UTF-8 conversion, and error behavior while removing the duplicated
scaffolding.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: 0b0f6fcf-e294-4a7b-ad01-88547d7921f4

📥 Commits

Reviewing files that changed from the base of the PR and between f8d9250 and d19db71.

⛔ Files ignored due to path filters (1)
  • Lib/test/test_csv.py is excluded by !Lib/**
📒 Files selected for processing (3)
  • .cspell.json
  • crates/stdlib/src/csv.rs
  • extra_tests/snippets/stdlib_csv.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • .cspell.json

@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Reviews resumed.

@youknowone youknowone left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for working on this painful task!

@youknowone
youknowone merged commit 16bb018 into RustPython:main Aug 1, 2026
27 checks passed
@widehyo1
widehyo1 deleted the implement-cpython-csv-reader-fsm branch August 1, 2026 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

z-ca-2026 Tag to track Contribution Academy 2026

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RFC] csv: discuss the reader parser architecture for CPython compatibility

2 participants