Repository navigation
Test runner should allow specific subtest filtering, via --test-name-pattern or otherwise #46728
Description
Activity
- changed the title
[-]Test runner should allow specific subtest filtering[/-][+]Test runner should allow specific subtest filtering, via --test-name-pattern or otherwise[/+]on Feb 19, 2023 I think we should be able to support this without any additional overhead other than the case where
--test-name-patternis used. In this block we already compare the test name against all of the user provided patterns. We could also split the user provided patterns on some separator (>in the example above) and using theparentpointer, loop until we know if we have a fully qualified match or not.I'll take a look at this if nobody else has started on it.
Hi guys, how about test on concated names?
var describeNames = ['level1', 'level2']; var itName = 'should pass' var testNamePattern = /level1>level2>should pass/ testNamePattern.test([...describeNames,itName].join('>')) // true
I feel simpler and more reliable than inventing pattern syntax.
All other runners that come to mind do filter both on
describeandtestas part of the same sequence, as in the example in the issue report.I have checked, and these are the behaviors of other test runners:
jest --testNamePatternalways runsdescriberegardless of the pattern.mocha --grep- runs alldescriberegardless of the pattern.playwright --grep- runs alldescriberegardless of the pattern.- ava, tape, node-tap - no
describe
All other runners that come to mind do filter both on describe and test as part of the same sequence, as in the example in the issue report.
this seems to be incorrect @connor4312
I would expect describes to be included.
@cjihrig do you still expect that, given this information?
Reacted by Mark StosbergThat would still be my expectation, but I guess I would be wrong 😄. I'm fine with following the behavior of everyone else here.
Reacted by Caio BorghiJest certainly supports filtering by
describe. It's the main way I work with it.Ah, sorry, I misunderstood.
describes are run since thegreppattern is a pattern, not a prefix, so all tests have to get enumerated by running alldescribeblocks. So yes I think your intention is correct in saying "describe should not be filtered"Ok, so In regards to the comment made by @targos, I have tested this:
describe("yes", function() { it("no", () => {}); it("yes", () => {}); describe("maybe", function() { it("no", () => {}); it("yes", () => {}); }); }); describe("no", function() { it("no", () => {}); it("yes", () => {}); describe("maybe", function() { it("no", () => {}); it("yes", () => {}); }); });
playwright test --grep "yes": all ran besides no -> no, no -> maybe -> nomocha --grep "yes": all ran besides no -> no, no -> maybe -> nojest --testNamePattern=yes: all ran besides no -> no, no -> maybe -> no
meaning
describecan be used as a filter, but only to rule in its descendants - not to rule them outReacted by Colin Ihrig, Michaël Zasso and Connor PeetHello,
I'm implementing a Node.js test runner integration in WebStorm.
I would like to save common developers workflows for the test runner integration.One of these common workflows involves running a special test from a file that contains multiple tests.
This can be particularly useful when debugging a specific use case:debug-selected-test.mov
To implement this workflow, it's essential to have the ability to configure a test runner to execute a specific test.
Unfortunately, I couldn't find a way to do this with the current--test-name-patternimplementation.Code Example
Let's imagine that we have tests for callback and promise-based functionality (inspired by
test-fs-write-file-flush.jsfrom the Node.js repository):describe(`callback version`, () => { test(`valid`, () => new Promise((resolve, reject) => { validatePositiveNumber(1, (error, isValid) => { try { assert.ok(isValid); resolve(); } catch (e) { reject(e); } }); })); // invalid, etc }); describe(`promise based version`, () => { test(`valid`, () => validatePositiveNumber(1) .then(isValid => { assert.ok(isValid); }) ); // invalid, etc });
Let's attempt to filter the
validtest case from thecallback versionsuite,
using Node.js 20.4+ with the merged 'test_runner: make --test-name-pattern recursive #48382' PR.- Filtering with
--test-name-pattern="callback version"will run all tests incallback versionsuit (example). - Filtering with
--test-name-pattern="valid"will run all tests in thecallback versionand thepromise based versionsuites because they containvalidtest cases (example). - Filtering with 2
--test-name-patternparameters--test-name-pattern="callback version"and--test-name-pattern="valid"also runs all tests in this case. (example).
However, for Node.js versions earlier than 20.4, this approach works with the code example (example). - Filtering with
--test-name-patternand combined names as^callback version valid$doesn't run any test. (example).
Can you please suggest a way to filter the
validtest cases from thecallback versionsuite in this code example?The example in other tests runners
I checked the example in other test runners, and here are the results:
- AVA doesn't support suites
- Jasmine works well with
--filter="^callback version valid$"parameter, supports suites bydescribe(). - Jest works well with
--testNamePattern="^callback version valid$"parameter, supports suites bydescribe(). - Mocha works well with
--grep="^callback version valid$"parameter, supports suites bydescribe(). - Playwright works well with
--grep="callback version valid"parameter, supports suites bydescribe(). - NODE TAP works well with
--grep="^callback version$" --grep="^valid$"parameters, supports some kind of grouping for subtests. - Tape doesn't support test filtration and grouping.
- Vitest works well with
--testNamePattern="^ callback version valid$", supports suites bydescribe().
Suggestion to extend
--test-name-patternSo it appears that in general if some test runner supports grouping by
describe(), the runner gives a way to filter a specific test through all levels of the groups. Perhaps we should consider extending the--test-name-patternparameter to support this type of filtration more commonly.After reviewing other test runners, it appears that we can simply extend the
--test-name-patternto support the full path to the test. For the code example provided, it would look like^callback version valid$. This solution allows us to address the issue without requiring any breaking changes and offers a straightforward and widely-adopted approach.Could please say what you think about it?
Reacted by Michał Drobniak, Raymond Augé and Toni Villena- Filtering with
2 remaining items
Hello @mdrobny,
Thank you so much for your contribution! I'm looking forward these fixes in next version of Node.js.
Have a lovely time ✨
Reacted by Toni VillenaUpdate
So because this change is considered a breaking change, it will be released in Node.js 22 in April (if PR gets merged before that 😄 🤞🏼 )
- added a commit that references this issue
on Feb 29, 2024 - added a commit that references this issue
on Mar 1, 2024 Thank you! I will eagerly adopt it as soon as possible 🙂
Hello @mdrobny,
I Implemented support for your fixes for WebStorm and all JetBrains IDE's. With your fixes all works perfect!
Thank you so much.
Have a lovely time.Reacted by Michał DrobniakAdopted for VS Code in connor4312/nodejs-testing@a490ae2, works great 👍
Reacted by Michał Drobniak and Moshe Atlow
Version
19.3.0
Platform
Darwin MacBook-Pro-2.local 22.3.0 Darwin Kernel Version 22.3.0: Thu Jan 5 20:53:49 PST 2023; root:xnu-8792.81.2~2/RELEASE_X86_64 x86_64
Subsystem
test_runner
What steps will reproduce the bug?
Given a set of tests like this
... it is impossible to command line filter (
--test-name-filter) to run only the "works" test inmy suite 2.Being able to be precise about tests to run is important when working on an editor integration. If a user asks to "run this test", then only that test should run. This is especially relevant in the use case of integration tests, where running tests may have side effects.
How often does it reproduce? Is there a required condition?
100%
What is the expected behavior?
I should be able to run
my suite 2 > worksby itself. In every test runner I've worked with (jest, mocha, or vitest for example) names are generated for glob purposes by space-delimiting nested tests. So I expected to be able to pass--test-name-pattern="^my suite 1 my suite 2 works$"to run only that test.(This does mean that identically named tests in the same context cannot be differentiated. But this is a common problem across test runners that also affects result output, and I already have code that emits a diagnostic warning ('yellow squiggle') if the user does this.)
What do you see instead?
I must pass
--test-name-pattern="^my suite 1$" --test-name-pattern="^my suite 2$" --test-name-pattern="^works$", which runs the undesired test.Additional information
Apologies for the late followup on #42984. I didn't get a chance to work on the vscode integration for this for a few months, and only noticed this issue later.