Skip to content

multiprocessing calls flush on sys.stdout at exit even if it is None (pythonw) #75985

Closed
Closed
multiprocessing calls flush on sys.stdout at exit even if it is None (pythonw)#75985
Labels
3.7 (EOL)end of life3.8 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@poxthegreat
poxthegreat
mannequin
opened on Oct 17, 2017
BPO 31804
Nosy @terryjreedy, @pitrou, @applio, @poxthegreat
PRs
  • bpo-31804: multiprocessing calls flush on sys.stdout at exit even if … #5492
  • bpo-31804: Fix multiprocessing.Process with broken standard streams #6079
  • [3.7] bpo-31804: Fix multiprocessing.Process with broken standard streams (GH-6079) #6080
  • [3.6] bpo-31804: Fix multiprocessing.Process with broken standard streams (GH-6079) #6081
    		•
    Files
  • process.py.patch: patch file with trivial bugfix
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2018-03-11.19:10:52.978>
created_at = <Date 2017-10-17.14:59:34.932>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = 'multiprocessing calls flush on sys.stdout at exit even if it is None (pythonw)'
updated_at = <Date 2018-03-11.19:10:52.977>
user = 'https://github.com/poxthegreat'

    bugs.python.org fields:

    activity = <Date 2018-03-11.19:10:52.977>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2018-03-11.19:10:52.978>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2017-10-17.14:59:34.932>
creator = 'Pox TheGreat'
dependencies = []
files = ['47224']
hgrepos = []
issue_num = 31804
keywords = ['patch']
message_count = 12.0
messages = ['304512', '304678', '309708', '309716', '309721', '309759', '309762', '309853', '313608', '313613', '313616', '313617']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'pitrou', 'davin', 'Pox TheGreat']
pr_nums = ['5492', '6079', '6080', '6081']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31804'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

    Activity

    1. poxthegreat commented on Oct 17, 2017

      @poxthegreat
      poxthegreatmannequin
      on Oct 17, 2017
      MannequinAuthor

      If you start Python by pythonw then sys.stdout and sys.stderr are set to None. If you also use multiprocessing then when the child process finishes BaseProcess._bootstrap calls sys.stdout.flush() and sys.stderr.flush() finally. This causes the process return code to be not zero (it is 1).

    2. poxthegreat
      added
      type-crashA hard crash of the interpreter, possibly with a core dump
      stdlibStandard Library Python modules in the Lib/ directory
      on Oct 17, 2017
    3. terryjreedy
      added
      3.7 (EOL)end of life
      on Oct 20, 2017

    4. pitrou commented on Oct 20, 2017

      @pitrou
      pitrou
      on Oct 20, 2017
      Member

      This looks to be a duplicate of https://bugs.python.org/issue28326

    5. pitrou
      closed this as completedon Oct 20, 2017

    6. poxthegreat commented on Jan 9, 2018

      @poxthegreat
      poxthegreatmannequin
      on Jan 9, 2018
      MannequinAuthor

      Unfortunately this is NOT a duplicate of https://bugs.python.org/issue28326. That issue is about a closed output stream. In that case sys.stdout and sys.stderr are file like objects which have been closed.

      This issue is about sys.stdout and sys.stderr being None! This is because pythonw was used not python.

    7. poxthegreat
      reopened this on Jan 9, 2018

    8. pitrou commented on Jan 9, 2018

      @pitrou
      pitrou
      on Jan 9, 2018
      Member

      @pox, nevertheless, the fix committed in #4073 should also fix this issue. Do you disagree?

    9. terryjreedy commented on Jan 9, 2018

      @terryjreedy
      terryjreedy
      on Jan 9, 2018
      Member

      Pox, please retest with either 3.6.4 or 3.7.0a3 (or .0a4 when released, soon). Both were released after the merge that should fix this.

    10. poxthegreat commented on Jan 10, 2018

      @poxthegreat
      poxthegreatmannequin
      on Jan 10, 2018
      MannequinAuthor

      Retested it with a freshly installed 3.6.4 version. Used the following code to test:

      import sys
import multiprocessing


def foo():
    return 'bar'


if __name__ == '__main__':
    proc = multiprocessing.Process(target=foo)
    proc.start()
    proc.join()
    with open('process_exit_code.txt', 'w') as f:
        f.write(sys.version)
        f.write('\nprocess exit code: ')
        f.write(str(proc.exitcode))

      It is very important to run the script with pythonw, not just with python. This is the content of the resulting process_exit_code.txt file on my machine:
      3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)]
      process exit code: 1

      As it can be seen the problem was not fixed. The process exit code should be 0. By default the new multiprocessing process created uses the same interpreter as the creator process, so it uses pythonw too.

    11. pitrou commented on Jan 10, 2018

      @pitrou
      pitrou
      on Jan 10, 2018
      Member

      I'm not using Windows, so I'm unable to test using pythonw. You'll have to provide a fix, or someone else will have to.

    12. poxthegreat commented on Jan 12, 2018

      @poxthegreat
      poxthegreatmannequin
      on Jan 12, 2018
      MannequinAuthor

      I have already uploaded a patch file but it is not in the required format. Also I realize that most of the confusion was because I forgot to provide the OS version. Perhaps it would be good to have a separate field for that.

      I will upload a patch as it is described in the developer guide.

    13. poxthegreat
      added
      type-bugAn unexpected behavior, bug, or error
      and removed
      type-crashA hard crash of the interpreter, possibly with a core dump
      on Jan 12, 2018

    14. pitrou commented on Mar 11, 2018

      @pitrou
      pitrou
      on Mar 11, 2018
      Member

      New changeset e756f66 by Antoine Pitrou in branch 'master':
      bpo-31804: Fix multiprocessing.Process with broken standard streams (bpo-6079)
      e756f66

    15. pitrou
      added
      3.8 (EOL)end of life
      on Mar 11, 2018

    16. pitrou commented on Mar 11, 2018

      @pitrou
      pitrou
      on Mar 11, 2018
      Member

      New changeset ff5d213 by Antoine Pitrou (Miss Islington (bot)) in branch '3.7':
      bpo-31804: Fix multiprocessing.Process with broken standard streams (GH-6079) (GH-6080)
      ff5d213

    17. pitrou commented on Mar 11, 2018

      @pitrou
      pitrou
      on Mar 11, 2018
      Member

      New changeset 069b8d2 by Antoine Pitrou in branch '3.6':
      [3.6] bpo-31804: Fix multiprocessing.Process with broken standard streams (GH-6079) (GH-6081)
      069b8d2

    18. pitrou commented on Mar 11, 2018

      @pitrou
      pitrou
      on Mar 11, 2018
      Member

      This is all fixed in the Python 3 branches now. I won't bother with Python 2 as it has quite a different code structure and backporting would take too much of my time.

      Pox TheGreat, thanks for reporting and the initial patch!

    19. pitrou
      closed this as completedon Mar 11, 2018
    20. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of life3.8 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions