Repository navigation
mimalloc should fail to allocate 784 271 641 GiB: test_decimal killed by the Linux kernel with OOM on a Free Threading build #114331
Description
Activity
Same failures on all supported Fedora versions.
It can be gcc bug, if we test with inline assembly instead, suspection will be proved.
I got the same failure with clang: build.log
Reacted by Donghee NaReacted by Donghee Namialloc is fine with allocating 842105263157894744 bytes, whereas glibc malloc() fails with NULL. But when Python tries to use this memory (write data), it's killed by the Linux kernel with Out Of Memory (OOM).
842105263157894744 bytes is around 784 271 641 GiB: it's big. I tested on a machine which only has 8 GiB of memory: so clearly, 784 271 641 GiB cannot be allocated.
The difference between a regular build and a --disable-gil build is that --disable-gil implies to use mimalloc allocator.
# ./python -m test.pythoninfo|grep mem pymem.allocator: mimalloc_debug
./python -m test -v test_decimal -m test_maxcontext_exact_arithprogram is KILLED. Running it in gdb doesn't help, since it's killed by the Linux kernel.$ journalctl -k -r Jan 22 09:02:48 s390x-kvm-112.lab.eng.rdu2.redhat.com kernel: Out of memory: Killed process 1549896 (python) total-vm:822368422334244kB, anon-rss:7572064kB, file-rss:8kB, shmem-rss:0kB, UID:0 pgtables:30506kB oom_score_adj:0"total-vm:822368422334244kB" that's a lot of memory.
I put a breakpoint on PyMem_Malloc() if size is larger than 1 GiB:
Breakpoint 1, PyMem_Malloc (size=842105263157894744) at Objects/obmalloc.c:861 861 if (size > (size_t)PY_SSIZE_T_MAX) (gdb) where #0 PyMem_Malloc (size=842105263157894744) at Objects/obmalloc.c:861 #1 0x000003fff00a1fe2 in mpd_alloc (nmemb=nmemb@entry=105263157894736843, size=size@entry=8) at ./Modules/_decimal/libmpdec/mpalloc.c:92 #2 0x000003fff00a21e6 in mpd_switch_to_dyn (result=result@entry=0x3ffffffa360, nwords=nwords@entry=105263157894736843, status=status@entry=0x3ffffffa6b4) at ./Modules/_decimal/libmpdec/mpalloc.c:217 #3 0x000003fff00a68c0 in mpd_qresize (result=result@entry=0x3ffffffa360, nwords=nwords@entry=105263157894736843, status=status@entry=0x3ffffffa6b4) at ./Modules/_decimal/libmpdec/mpdecimal.c:517 #4 0x000003fff00a99f4 in mpd_qshiftl (result=result@entry=0x3ffffffa360, a=a@entry=0x3ffffffa360, n=n@entry=1999999999999999998, status=status@entry=0x3ffffffa6b4) at ./Modules/_decimal/libmpdec/mpdecimal.c:2520 #5 0x000003fff00b516c in _mpd_qsqrt (result=result@entry=0x200012eb8c8, a=a@entry=0x200012eb808, ctx=ctx@entry=0x20001587520, status=status@entry=0x3ffffffa6b4) at ./Modules/_decimal/libmpdec/mpdecimal.c:7951 #6 0x000003fff00b57d0 in mpd_qsqrt (result=result@entry=0x200012eb8c8, a=a@entry=0x200012eb808, ctx=0x20001587520, status=status@entry=0x3ffffffa98c) at ./Modules/_decimal/libmpdec/mpdecimal.c:8043 #7 0x000003fff008f7e6 in dec_mpd_qsqrt (self=0x200012eb7e0, args=<optimized out>, kwds=<optimized out>) at ./Modules/_decimal/_decimal.c:4407 (gdb) py-bt Traceback (most recent call first): File "/root/cpython/Lib/test/test_decimal.py", line 5665, in test_maxcontext_exact_arith self.assertEqual(Decimal(4).sqrt(), 2)The test is decorated with:
@unittest.skipIf(sys.platform.startswith("aix"), "AIX: default ulimit: test is flaky because of extreme over-allocation") @unittest.skipIf(is_emscripten, "Test is unstable on Emscripten") @unittest.skipIf(check_sanitizer(address=True, memory=True), "ASAN/MSAN sanitizer defaults to crashing " "instead of returning NULL for malloc failure.")
On a regular Python build, I see the same memory allocation:
0:00:00 load avg: 0.84 [1/1] test_decimal test_maxcontext_exact_arith (test.test_decimal.CWhitebox.test_maxcontext_exact_arith) ... Breakpoint 1, PyMem_Malloc (size=842105263157894744) at Objects/obmalloc.c:861 861 if (size > (size_t)PY_SSIZE_T_MAX) (gdb) py-bt Traceback (most recent call first): File "/root/cpython/Lib/test/test_decimal.py", line 5665, in test_maxcontext_exact_arith self.assertEqual(Decimal(4).sqrt(), 2)There are 4 large memory allocations, all of them fail:
- size=842105263157894744
- size=842105263157894744
- size=421052631578947376
- size=421052631578947376
- changed the title
[-]3.13.0a3 freethreding build fails to build on s390x: killed during the profile task[/-][+]mimalloc should to allocate 784 271 641 GiB: test_decimal killed by the Linux kernel with OOM on a Free Threading build[/+]on Jan 22, 2024 @DinoV @colesbury: mimalloc needs maybe some tuning here?
--disable-gilon x86-64 seems to behave differently. On my laptop with 32 GiB of RAM:0:00:00 load avg: 1.01 [1/1] test_decimal test_maxcontext_exact_arith (test.test_decimal.CWhitebox.test_maxcontext_exact_arith) ... Breakpoint 1, PyMem_Malloc (size=842105263157894744) at Objects/obmalloc.c:861 861 if (size > (size_t)PY_SSIZE_T_MAX) Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.8-16.fc39.x86_64 xz-libs-5.4.4-1.fc39.x86_64 zlib-1.2.13-4.fc39.x86_64 (gdb) n 867 return _PyMem.malloc(_PyMem.ctx, size); (gdb) n mimalloc: warning: unable to allocate OS memory (error: 12 (0xc), size: 0xbafc24672800000 bytes, align: 0x2000000, commit: 1, allow large: 1) mimalloc: warning: unable to allocate OS memory (error: 12 (0xc), size: 0xbafc24672800000 bytes, align: 0x2000000, commit: 1, allow large: 1) mimalloc: error: unable to allocate memory (842105263157894768 bytes) 868 } (gdb) n mpd_alloc (nmemb=105263157894736843, size=8) at ./Modules/_decimal/libmpdec/mpalloc.c:93 93 } (gdb) n mpd_switch_to_dyn (result=0x7ffffffe22c0, nwords=105263157894736843, status=0x7ffffffe25cc) at ./Modules/_decimal/libmpdec/mpalloc.c:218 218 if (result->data == NULL) { (gdb) p result->data $1 = (mpd_uint_t *) 0x0See "warning: unable to allocate OS memory" error.
Using
strace, I seemmap()syscalls failing with ENOMEM.mmap(NULL, 842105263166062592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Ne peut allouer de la mémoire) write(2, "mimalloc: warning: ", 19) = 19 write(2, "unable to allocate OS memory (er"..., 123) = 123 mmap(NULL, 842105263166062592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Ne peut allouer de la mémoire) write(2, "mimalloc: warning: ", 19) = 19 write(2, "unable to allocate OS memory (er"..., 123) = 123 write(2, "mimalloc: error: ", 17) = 17 write(2, "unable to allocate memory (84210"..., 53) = 53 mmap(NULL, 842105263166062592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Ne peut allouer de la mémoire) write(2, "mimalloc: warning: ", 19) = 19 write(2, "unable to allocate OS memory (er"..., 123) = 123 mmap(NULL, 842105263166062592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Ne peut allouer de la mémoire) write(2, "mimalloc: warning: ", 19) = 19 write(2, "unable to allocate OS memory (er"..., 123) = 123 write(2, "mimalloc: error: ", 17) = 17 write(2, "unable to allocate memory (84210"..., 53) = 53 mmap(NULL, 421052631587225600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Ne peut allouer de la mémoire) write(2, "mimalloc: warning: ", 19) = 19 write(2, "unable to allocate OS memory (er"..., 123) = 123 mmap(NULL, 421052631587225600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Ne peut allouer de la mémoire) write(2, "mimalloc: warning: ", 19) = 19 write(2, "unable to allocate OS memory (er"..., 123) = 123 write(2, "mimalloc: error: ", 17) = 17 write(2, "unable to allocate memory (42105"..., 53) = 53 mmap(NULL, 421052631587225600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Ne peut allouer de la mémoire) write(2, "mimalloc: warning: ", 19) = 19 write(2, "unable to allocate OS memory (er"..., 123) = 123 mmap(NULL, 421052631587225600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Ne peut allouer de la mémoire) write(2, "mimalloc: warning: ", 19) = 19 write(2, "unable to allocate OS memory (er"..., 123) = 123 write(2, "mimalloc: error: ", 17) = 17 write(2, "unable to allocate memory (42105"..., 53) = 53@DinoV @colesbury: Tell me if you need more information about s390x.
Thanks for debugging this and find the rooting cause. My inclination is to address this in
_decimal.c(or possiblytest_decimal.py) rather than trying to add heuristics to mimalloc. This makes the fourth platform/configuration with problems with this test -- the behavior it assumes does not seem reliable.In the meantime, if we want to build/test a free-threaded build on s390x, we can add another
unittest.skipIfto the test.I would be fine with a skip on s390x, since apparently, the underlying memory allocator or Linux kernel behave differently on s390x and x86-64.
7 remaining items
- added a commit that references this issue
on Mar 28, 2024 - added a commit that references this issue
on Mar 28, 2024 Thanks Victor!
Reacted by Victor Stinner and Miro Hrončok- added 2 commits that reference this issue
on Apr 2, 2024
Bug report
Bug description:
Hello. When we were updating Python 3.13 from a2 to a3 in Fedora, I wanted to enable the freethreding on s390x and ppc64le, as it was previously disabled due to #112535
However, the freethreding build fails to build in Fedora Linux 40 on s390x with:
Full log: build.log
I'll try to build this on older Fedora Linux versions as well to eliminate other factors, such as GCC 14:
CPython versions tested on:
3.13
Operating systems tested on:
Linux
Linked PRs