Skip to content

Add Hypothesis regression tests for tomllib - #158565

Draft
pganssle wants to merge 2 commits into
python:mainfrom
pganssle:tomllib-properties
Draft

pganssle wants to merge 2 commits into
python:mainfrom
pganssle:tomllib-properties

Conversation

@pganssle

@pganssle pganssle commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

⚠️DO NOT READ THIS YET⚠️: I asked an AI to draft me this and I have put it here in a draft PR so that I could look at it and look at the diff. This has not been verified by a human. You are welcome to read this but be warned that it is not reflective of the quality of PR description that I would normally send to another person.

This adds Hypothesis tests for tomllib, covering numeric representations, escaped strings and keys, table definitions, and datetimes. Each property has explicit examples so it also runs through the existing fallback when Hypothesis is not installed.

Unlike several other times I've added Hypothesis tests, this did not turn up a bunch of new bugs. The only new finding was a minor error-reporting issue: an unfinished escape can leave TOMLDecodeError.pos beyond the end of its own doc. There is a focused expected-failure test for that, and I am preparing a separate report.

I think the stronger case for these tests is that they could have caught some of the parser's earlier bugs. To check that, I ran four of the properties unchanged against Tomli releases from before and after the relevant fixes, with explicit examples and the Hypothesis database disabled. Hypothesis found a counterexample in each old release, and the same property passed on the fixed release:

Property Bug it caught Failing / passing release
Duplicate-key rejection Inline-table keys silently overwrote previous values 0.2.0 / 0.2.1
Independent children of array-of-table entries A subtable name could not be reused in a later array entry (Tomli #22) 0.2.8 / 0.2.9
Unicode scalar validation Surrogate escapes were accepted (Tomli #27) 0.2.9 / 0.2.10
Datetime representations Lowercase t was rejected (Tomli #149) 1.2.2 / 1.2.3

These fixes predate CPython's adoption of Tomli, and the current fixtures already cover those bugs. The point is that the properties found them without being given the historical examples, which gives me somewhat more confidence in their value as regression tests than just observing that they pass today. The duplicate-key fix is recorded in the 0.2.1 release notes.

The numeric tests also caught two deliberately introduced errors that the existing suite missed: parsing hexadecimal integers as zero, and losing the sign of negative zero. The remaining properties broaden the combinations of keys, nesting, values and formatting that are checked against known results; I have less direct evidence for those, so they are in a separate commit from the numeric tests.

After rebasing, test_tomllib passes with Hypothesis 6.155.3 and with the fallback, with the one expected failure described above. The full TOML suite took about 0.9 seconds with Hypothesis and 0.07 seconds without it on my local build. The historical comparisons used up to 200 generated examples per property, with generation and shrinking enabled.

Exercise integer bases and digit separators, signed zero, subnormal floats, and the parse_float hook. Fault injection showed that the existing suite missed hexadecimal values parsed as zero and loss of the sign of negative zero.

Include explicit boundary examples so the tests also run without Hypothesis.
Generate Unicode keys, equivalent table forms, invalid redefinitions, arrays of tables, datetime boundaries, excess fractional precision, and formatting variations. These mostly broaden behaviors already represented by fixed fixtures.

Explain the Unicode boundaries and other explicit examples, which also run through the Hypothesis fallback shim.
@StanFromIreland

Copy link
Copy Markdown
Member

These seem like something for Tomli, IIRC they have more extensive test, including the official TOML test suite?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests Tests in the Lib/test dir

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants