Skip to content

year 2038 problem in compileall.py #79171

Closed
Closed
year 2038 problem in compileall.py#79171
Labels
3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifebuildThe build process and cross-build

Description

@bmwiedemann
bmwiedemann
mannequin
opened on Oct 15, 2018
BPO 34990
Nosy @jwilk, @serhiy-storchaka, @matrixise, @bmwiedemann, @ammaraskar, @miss-islington, @tirkarthi
PRs
  • WIP: bpo-34990: year 2038 problem in compileall.py #9892
  • bpo-34990: Change pyc headers to use 64-bit timestamps #19651
  • bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int #19708
  • [3.10] bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int (GH-19708) #27928
  • [3.9] bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int (GH-19708) #27929
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-08-24.15:13:34.226>
created_at = <Date 2018-10-15.11:22:16.253>
labels = ['build', '3.9', '3.10', '3.11']
title = 'year 2038 problem in compileall.py'
updated_at = <Date 2022-02-21.11:46:17.662>
user = 'https://github.com/bmwiedemann'

    bugs.python.org fields:

    activity = <Date 2022-02-21.11:46:17.662>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-08-24.15:13:34.226>
closer = 'ammar2'
components = ['Build']
creation = <Date 2018-10-15.11:22:16.253>
creator = 'bmwiedemann'
dependencies = []
files = []
hgrepos = []
issue_num = 34990
keywords = ['patch']
message_count = 13.0
messages = ['327743', '327747', '327748', '327749', '327750', '327751', '327753', '327774', '360270', '367679', '400200', '400214', '400215']
nosy_count = 7.0
nosy_names = ['jwilk', 'serhiy.storchaka', 'matrixise', 'bmwiedemann', 'ammar2', 'miss-islington', 'xtreak']
pr_nums = ['9892', '19651', '19708', '27928', '27929']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue34990'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    Activity

    1. bmwiedemann commented on Oct 15, 2018

      @bmwiedemann
      bmwiedemannmannequin
      on Oct 15, 2018
      MannequinAuthor

      To reproduce:
      touch -d 2038-01-20 /usr/lib/python3.6/site-packages/six.py
      python3 /usr/lib64/python3.6/compileall.py

      File "/usr/lib64/python3.6/compileall.py", line 198, in compile_path
      legacy=legacy, optimize=optimize)
      File "/usr/lib64/python3.6/compileall.py", line 90, in compile_dir
      legacy, optimize):
      File "/usr/lib64/python3.6/compileall.py", line 138, in compile_file
      mtime)
      struct.error: 'l' format requires -2147483648 <= number <= 2147483647

      It could use either
      64 bit int (requires new .pyc format with different magic number) or
      unsigned 32 bit int (gives us only another 68 years)

    2. bmwiedemann
      added
      3.7 (EOL)end of life
      buildThe build process and cross-build
      on Oct 15, 2018

    3. matrixise commented on Oct 15, 2018

      @matrixise
      matrixise
      on Oct 15, 2018
      Member

      With 3.8a

      Traceback (most recent call last):
  File "/home/stephane/src/github.com/python/cpython/Lib/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/stephane/src/github.com/python/cpython/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/stephane/src/github.com/python/cpython/Lib/compileall.py", line 326, in <module>
    exit_status = int(not main())
  File "/home/stephane/src/github.com/python/cpython/Lib/compileall.py", line 303, in main
    if not compile_file(dest, args.ddir, args.force, args.rx,
  File "/home/stephane/src/github.com/python/cpython/Lib/compileall.py", line 142, in compile_file
    expect = struct.pack('<4sll', importlib.util.MAGIC_NUMBER,
struct.error: 'l' format requires -2147483648 <= number <= 2147483647

    4. matrixise commented on Oct 15, 2018

      @matrixise
      matrixise
      on Oct 15, 2018
      Member

      But until 2038, maybe there will be a new format for the .pyc file.

      We should keep this issue and try to fix it for 3.8 or 3.9?

    5. bmwiedemann commented on Oct 15, 2018

      @bmwiedemann
      bmwiedemannmannequin
      on Oct 15, 2018
      MannequinAuthor

      It does not need to be fixed tomorrow, but 2037 is too late, because by then there will be a lot of legacy systems around.
      (Un)fortunately many systems live 10+ years now

    6. vstinner commented on Oct 15, 2018

      @vstinner
      vstinner
      on Oct 15, 2018
      Member

      Timestamp with year >= 2038 are accepted: importlib._bootstrap_external._code_to_timestamp_pyc() uses (int(x) & 0xFFFFFFFF). It's not a bug, but by design. compileall should just do the same. Sorry, I don't know if it's specified somewhere, but I know that it's done on purpose.

    7. matrixise commented on Oct 15, 2018

      @matrixise
      matrixise
      on Oct 15, 2018
      Member

      So we need to fix compileall.py.

      maybe we could add the label 'easy' to this issue.

    8. tirkarthi commented on Oct 15, 2018

      @tirkarthi
      tirkarthi
      on Oct 15, 2018
      Member

      A reproducer in Python that can be added to test_compileall if needed :

      def test_compile_all_2038(self):
    with open(self.source_path, 'r') as f:
        os.utime(f.name, (2147558400, 2147558400)) # Jan 20, 2038 as touch
    self.assertTrue(compileall.compile_file(pathlib.Path(self.source_path)))

      ./python.exe -m unittest -v test.test_compileall.CompileallTestsWithSourceEpoch.test_compile_all_2038
      test_compile_all_2038 (test.test_compileall.CompileallTestsWithSourceEpoch) ... ERROR

      ======================================================================
      ERROR: test_compile_all_2038 (test.test_compileall.CompileallTestsWithSourceEpoch)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/test/test_py_compile.py", line 30, in wrapper
    return fxn(*args, **kwargs)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/test/test_compileall.py", line 114, in test_compile_all_2038
    self.assertTrue(compileall.compile_file(pathlib.Path(self.source_path)))
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/compileall.py", line 142, in compile_file
    expect = struct.pack('<4sll', importlib.util.MAGIC_NUMBER,
struct.error: 'l' format requires -2147483648 <= number <= 2147483647

      Thanks

    9. tirkarthi commented on Oct 15, 2018

      @tirkarthi
      tirkarthi
      on Oct 15, 2018
      Member

      Victor seems there was some discussion about 2038 problem in the original PR but I don't know if it's related to this. Reference : #4575 (comment)

      Thanks

    10. bmwiedemann commented on Jan 19, 2020

      @bmwiedemann
      bmwiedemannmannequin
      on Jan 19, 2020
      MannequinAuthor

      ping.
      Another 19th of January passed.

      I'd still like to see progress on this, because this hinders my other y2038 bug discovery work.

    11. bmwiedemann
      added
      3.8 (EOL)end of life
      3.9 (EOL)end of life
      on Jan 19, 2020

    12. vstinner commented on Apr 29, 2020

      @vstinner
      vstinner
      on Apr 29, 2020
      Member

      I would prefer to mimick importlib._bootstrap_external which uses:

      def _pack_uint32(x):
    """Convert a 32-bit integer to little-endian."""
    return (int(x) & 0xFFFFFFFF).to_bytes(4, 'little')

      Using 64-bit timestamp (PR 19651), treat timestamp as unsigned (PR 9892 and PR 19708) have drawback:

      • 64-bit timestamp make .pyc files larger
      • unsigned timestamp no longer support timestamp before 1969 which can cause practical issues

      "& 0xFFFFFFFF" looks dead simple, uses a fixed size of 4 bytes and doesn't have any limitation of year 2038.

      The timestamp doesn't have to be exact. In practice, it sounds very unlikely that two timestamps are equal when compared using (ts1 & 0xFFFFFFFF) == (ts2 & 0xFFFFFFFF). I expect file modification times to be close by a few days, not separated by 2**32 seconds (136 years).

      Use hash based .pyc to avoid any issuse with file modification time: it should make Python more deterministic (more "reproducible").
      https://docs.python.org/dev/reference/import.html#pyc-invalidation

    13. serhiy-storchaka commented on Aug 24, 2021

      @serhiy-storchaka
      serhiy-storchaka
      on Aug 24, 2021
      Member

      New changeset bb21e28 by Ammar Askar in branch 'main':
      bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int (GH-19708)
      bb21e28

    14. ammaraskar commented on Aug 24, 2021

      @ammaraskar
      ammaraskar
      on Aug 24, 2021
      Member

      New changeset 9d3b6b2 by Miss Islington (bot) in branch '3.9':
      [3.9] bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int (GH-19708)
      9d3b6b2

    15. ammaraskar commented on Aug 24, 2021

      @ammaraskar
      ammaraskar
      on Aug 24, 2021
      Member

      New changeset 0af681b by Miss Islington (bot) in branch '3.10':
      [3.10] bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int (GH-19708)
      0af681b

    16. ammaraskar
      added
      3.10 (EOL)end of life
      3.11only security fixes
      and removed
      3.7 (EOL)end of life
      3.8 (EOL)end of life
      on Aug 24, 2021
    17. ammaraskar
      closed this as completedon Aug 24, 2021
    18. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    19. bmwiedemann
      added a commit that references this issue on Sep 3, 2022

      Update python38 to version 3.8.13 / rev 25 via SR 1000772

      a40cb94

    20. suryansh18saxena commented on Jun 22, 2025

      @suryansh18saxena
      suryansh18saxena
      on Jun 22, 2025

      Hi! I'm a first-time contributor and would love to fix this. Could you please assign me this issue? Thanks!

    21. vstinner commented on Jun 23, 2025

      @vstinner
      vstinner
      on Jun 23, 2025
      Member

      Hi! I'm a first-time contributor and would love to fix this. Could you please assign me this issue? Thanks!

      Hi, this issue is closed, there is nothing to do.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifebuildThe build process and cross-build

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions