AddressSanitizer: Skip tests directly in Python, not with external config

@vstinner

BPO	46633
Nosy	@vstinner, @pablogsal
PRs	bpo-46633: Skip tests on ASAN and/or MSAN builds #31632 [3.10] bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) #31634 [3.9] bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) (GH-31634) #31644

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-03-02.17:13:40.631>
created_at = <Date 2022-02-04.10:09:11.092>
labels = ['tests', '3.11']
title = 'AddressSanitizer: Skip tests directly in Python, not with external config'
updated_at = <Date 2022-03-02.17:13:40.630>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-03-02.17:13:40.630>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2022-03-02.17:13:40.631>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2022-02-04.10:09:11.092>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 46633
keywords = ['patch']
message_count = 11.0
messages = ['412499', '412501', '412506', '414247', '414248', '414250', '414262', '414266', '414370', '414379', '414380']
nosy_count = 2.0
nosy_names = ['vstinner', 'pablogsal']
pr_nums = ['31632', '31634', '31644']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue46633'
versions = ['Python 3.11']

Linked PRs

It seems like bpo-45200: "Address Sanitizer: libasan dead lock in pthread_create() (test_multiprocessing_fork.test_get() hangs)" is not fixed yet.

In the GitHub Action job, test_multiprocessing_fork is skipped because it's too slow, so the job doesn't hang.

Yesterday, I modified the ASAN buildbot to double its timeout from 20 to 40 minutes:
python/buildmaster-config@5a37411

But it's useful, when it hangs, it hangs forever. Exmaple on the AMD64 Arch Linux Asan Debug 3.9 buildbot (with the new config):

---
(test.test_multiprocessing_fork.WithProcessesTestPicklingConnections) ... ok
Timeout (0:35:00)!
---
https://buildbot.python.org/all/#/builders/588/builds/332

Tests are tuned for ASAN, but the configuration is copied and inconsistent between the GitHub Actions job and the buildbot configuration.

I propose to move this configuration directly into Python.

test_decimal.py checks for "-fsanitize=address" in CFLAGS and skip some tests if it's present.

Notes:

test.support has check_sanitizer() function. Tests using it:
- test_crypt
- test_idle
- test_tix
- test_tk
- test_ttk_guionly
test_decimal also checks for memory sanitizer and skip some tests if it's present:

MEMORY_SANITIZER = (
    '-fsanitize=memory' in _cflags or
    '--with-memory-sanitizer' in _config_args
)

test_faulthandler suppress crash report, similir to support.SuppressCrashReport, directly in the C code used by tests: faulthandler_suppress_crash_report() function.
Objects/obmalloc.c checks "#if __has_feature(address_sanitizer)": is ASAN enabled?
GitHub Action config: .github/workflows/build.yml. Skipped tests:
- test___all__
- test_concurrent_futures
- test_multiprocessing_fork
- test_multiprocessing_forkserver
- test_multiprocessing_spawn
- test_peg_generator
- test_tools

Comment:
---
# Skip test_tools test_peg_generator test_concurrent_futures because
# there are too slow: between 5 and 20 minutes on this CI.

# Skip multiprocessing and concurrent.futures tests which are affected by
# bpo-45200 bug: libasan dead lock in pthread_create().

# test___all__ is skipped because importing some modules directly can trigger
# known problems with ASAN (like tk or crypt).
---

Buildbot configuration: UnixAsanBuild class of master/custom/factories.py. Options:

  ASAN_OPTIONS='detect_leaks=0:allocator_may_return_null=1:handle_segv=0'

With the comment:
---
# See https://bugs.python.org/issue42985 for more context on why
# SIGSEGV is ignored on purpose.
---

Skipped tests:

test_ctypes
test_capi
test_crypt
test_decimal
test_faulthandler
test_interpreters

With the comment:
---
# These tests are currently raising false positives or are interfering with the ASAN mechanism,
# so we need to skip them unfortunately.
---

I have recently added some decorators in test.
support to deactivate tests if running under the sanitizers.

I created a PR to no longer skip tests in buildbots:
python/buildmaster-config#314

Status with the 2 pending PRs:

Tests only skipped on ASAN:

_test_multiprocessing

Skip on ASAN and MSAN:

test___all__
test_concurrent_futures
test_crypt
test_decimal.test_maxcontext_exact_arith()
test_idle
test_peg_generator
test_tix
test_tk
test_tools
test_ttk_guionly

No longer skipped:

test_capi
test_ctypes
test_faulthandler
test_interpreters

I created bpo-46887: ./Programs/_freeze_module fails with MSAN: Uninitialized value was created by an allocation of 'stat.i'.

"""
No longer skipped:

test_capi
test_ctypes
test_faulthandler
test_interpreters
"""

I built Python manually with:

./configure --with-pydebug CC=clang LD=clang --with-address-sanitizer

These tests no longer with ASAN_OPTIONS used on our CI:

ASAN_OPTIONS='detect_leaks=0:allocator_may_return_null=1:handle_segv=0' ./python -m test (...)

New changeset 9204bb7 by Victor Stinner in branch 'main':
bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632)
9204bb7

New changeset 9326445 by Victor Stinner in branch '3.10':
[3.10] bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) (GH-31634)
9326445

New changeset 359bc39 by Victor Stinner in branch '3.9':
[3.10] bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) (GH-31634) (GH-31644)
359bc39

python/buildmaster-config#314

I merged this PR as well. The new buildbot configuration will be deployed soon.

@gpshead

Lib/test/test_asyncio/test_subprocess.py and Lib/test/test_distutils.py are now skipped on ASAN tests by the commit: f6dd14c (cc @gpshead)

FWIW I disabled those two because they triggered ASAN on Ubuntu 20.04's gcc 9.4 and in github's CI (which might use that?). They do not trigger ASAN on more recent Debian bullseye or Ubuntu 22.04 based systems with a more recent gcc.

In the _posixsubprocess.c case it was at a callsite that has ~25 parameters (ugh) and just gained one more. But the asan assertion would only trigger during one consistent but seemingly random test_asyncio.test_subprocess test rather than all the time; so if there is a bug in the ASAN implementation itself it isn't clear what it was.

I enabled test_faulthandler on ASAN: commit 58f9c63.

pythongh-90791: Enable test___all__ on ASAN build

pythongh-90791: test.pythoninfo logs ASAN_OPTIONS env var

gh-90791: Enable test___all__ on ASAN build (#108286)

I enabled more tests in test___all__: commit a541e01

gh-90791: test.pythoninfo logs ASAN_OPTIONS env var (#108289)

pythongh-90791: test.pythoninfo logs ASAN_OPTIONS env var (python#108289

[3.11] gh-108822: Backport libregrtest changes from the main branch (#…

Navigation Menu

AddressSanitizer: Skip tests directly in Python, not with external config #90791

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

AddressSanitizer: Skip tests directly in Python, not with external config #90791

Description

Linked PRs

Activity

vstinner commented on Feb 4, 2022

vstinner commented on Feb 4, 2022

pablogsal commented on Feb 4, 2022

vstinner commented on Mar 1, 2022

vstinner commented on Mar 1, 2022

vstinner commented on Mar 1, 2022

vstinner commented on Mar 1, 2022

vstinner commented on Mar 1, 2022

vstinner commented on Mar 2, 2022

vstinner commented on Mar 2, 2022

vstinner commented on Mar 2, 2022

vstinner commented on May 5, 2022

gpshead commented on May 6, 2022

vstinner commented on Aug 22, 2023

vstinner commented on Aug 22, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions