Repository navigation
Concurrent parallel invocations can lose worker-options files #21974
Description
Activity
This indeed looks very similar to #21393. Maybe it is actually the same issue. How often do you see this?
Btw, running mypy with pre-commit requires extra care, see #13916 (in particular parallelizing mypy invocations is usually a bad idea, unless you invoke mypy on completely independent groups of files). Finally, GitHub usually gives very cheap machines for actions, with just few cores, so likely by trying to force more parallelism you may make things slower.
For the issue itself, I think it may be possible to fix it by adding a bit of retry/back-off logic in the workers. I could be that there is a tiny delay before a file is visible to the other process.
- marked Rare race condition on macOS with parallel checking #21393 as a duplicate of this issue
on Sep 13, 2026 - changed the title
[-]Concurrent parallel invocations can lose worker-options files on Linux[/-][+]Concurrent parallel invocations can lose worker-options files[/+]on Sep 13, 2026 Thank you @ilevkivskyi for taking a look at this so quickly.
In 434 runs on one project in the last week, it appears I have hit this four times:
- 9 September: main, Python 3.13 (https://github.com/adamtheturtle/literalizer/actions/runs/34294381100/job/102287688551).
- 9 September: PR Support module aliases in fine grained mode #4861, Python 3.12 (https://github.com/adamtheturtle/literalizer/actions/runs/34387059139/job/102585871582).
- 10 September: PR Sync typeshed #4887, Python 3.13 (https://github.com/adamtheturtle/literalizer/actions/runs/34418401026/job/102688283473).
- 10 September: main, Python 3.12 (https://github.com/adamtheturtle/literalizer/actions/runs/34489393511/job/102911865600).
Thanks also for the thoughts on pre-commit - I'll keep those in mind and maybe reconsider my setup.
Reacted by Ivan LevkivskyiI investigated the worker startup path and have a small fix ready following the retry/backoff direction suggested above.
The coordinator already bounds worker startup using
WORKER_START_TIMEOUT/WORKER_START_INTERVAL, while the worker currently treats an initialFileNotFoundErrorreading--options-dataas immediately fatal. My patch adds a bounded retry only forFileNotFoundErrorat that read boundary; other filesystem and deserialization errors still propagate normally.I also added deterministic tests covering immediate success, transient recovery, timeout exhaustion, unrelated filesystem errors, and options serialization, and the existing parallel checks, mypy self-check, and lint suite are passing.
I'll do one final collision/rebase check and open the PR.
- added a commit that references this issue
on Sep 14, 2026 @adamtheturtle Note I merged couple PRs that should help with this. Can you try the current mypy master and see if the flakiness improved? If you can't use the mypy master, these fixes should be available in mypy v2.4.0 next week.
Wow thank you @ilevkivskyi ! I'll move the high-traffic literalizer project to mypy master and I'll update this issue if there are any failures.
(and thank you @Shoryamishra61 )
Reacted by Shoryamishra61- added 4 commits that reference this issue
on Sep 21, 2026
Crash
On Ubuntu 24.04 with Python 3.12 and Mypy 2.3.1, four concurrent Mypy
invocations using
--num-workers=4intermittently failed to start their buildworkers. Several workers reported:
Full GitHub Actions log:
https://github.com/adamtheturtle/literalizer/actions/runs/34489393511/job/102911865600
The four commands shared the working directory and default
.mypy_cache, buteach checked a different temporary Python file. The failure is intermittent;
an unchanged rerun passed. Serializing the invocations and using one Mypy
worker avoids it.
This resembles #21393, but that report is specifically a macOS test failure;
this occurrence was on Linux with concurrent consumer invocations.