Skip to content

fix(stats): handle empty documents - #512

Open
Excelius-Wang wants to merge 1 commit into
huggingface:mainfrom
Excelius-Wang:fix/empty-document-stats
Open

Excelius-Wang wants to merge 1 commit into
huggingface:mainfrom
Excelius-Wang:fix/empty-document-stats

Conversation

@Excelius-Wang

@Excelius-Wang Excelius-Wang commented Aug 20, 2026

Copy link
Copy Markdown

Summary

  • return 0.0 for ratios and averages whose denominator is empty
  • apply the same behavior across document, line, paragraph, sentence, word, and word-contamination stats
  • add regression coverage for empty documents across all affected stats blocks

Empty or fully filtered documents currently raise ZeroDivisionError in several
stats blocks. This can stop a data-audit pipeline before downstream cleaning has
a chance to handle the sample. The helper preserves existing results for
non-empty inputs and keeps count metrics such as n_lines unchanged.

Tests

  • focused stats regression tests — 7 passed, 6 subtests passed
  • Ruff lint and format checks
  • git diff --check

Note

Cursor Bugbot is generating a summary for commit f7213bb. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant