Conversation
Exercise integer bases and digit separators, signed zero, subnormal floats, and the parse_float hook. Fault injection showed that the existing suite missed hexadecimal values parsed as zero and loss of the sign of negative zero. Include explicit boundary examples so the tests also run without Hypothesis.
Generate Unicode keys, equivalent table forms, invalid redefinitions, arrays of tables, datetime boundaries, excess fractional precision, and formatting variations. These mostly broaden behaviors already represented by fixed fixtures. Explain the Unicode boundaries and other explicit examples, which also run through the Hypothesis fallback shim.
Member
|
These seem like something for Tomli, IIRC they have more extensive test, including the official TOML test suite? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds Hypothesis tests for
tomllib, covering numeric representations, escaped strings and keys, table definitions, and datetimes. Each property has explicit examples so it also runs through the existing fallback when Hypothesis is not installed.Unlike several other times I've added Hypothesis tests, this did not turn up a bunch of new bugs. The only new finding was a minor error-reporting issue: an unfinished escape can leave
TOMLDecodeError.posbeyond the end of its owndoc. There is a focused expected-failure test for that, and I am preparing a separate report.I think the stronger case for these tests is that they could have caught some of the parser's earlier bugs. To check that, I ran four of the properties unchanged against Tomli releases from before and after the relevant fixes, with explicit examples and the Hypothesis database disabled. Hypothesis found a counterexample in each old release, and the same property passed on the fixed release:
twas rejected (Tomli #149)These fixes predate CPython's adoption of Tomli, and the current fixtures already cover those bugs. The point is that the properties found them without being given the historical examples, which gives me somewhat more confidence in their value as regression tests than just observing that they pass today. The duplicate-key fix is recorded in the 0.2.1 release notes.
The numeric tests also caught two deliberately introduced errors that the existing suite missed: parsing hexadecimal integers as zero, and losing the sign of negative zero. The remaining properties broaden the combinations of keys, nesting, values and formatting that are checked against known results; I have less direct evidence for those, so they are in a separate commit from the numeric tests.
After rebasing,
test_tomllibpasses with Hypothesis 6.155.3 and with the fallback, with the one expected failure described above. The full TOML suite took about 0.9 seconds with Hypothesis and 0.07 seconds without it on my local build. The historical comparisons used up to 200 generated examples per property, with generation and shrinking enabled.