Skip to content

Node 18.13 HTTP/2 live lock #46234

Closed
Closed
Node 18.13 HTTP/2 live lock#46234
Labels
http2Issues and PRs related to the http2 subsystem.

Description

@ronag
ronag
opened on Jan 17, 2023

This is a tricky one but we have started noticing processes getting stuck on 100% cpu (live lock) since updating to Node 18.3.

Doing a debug breakpoint in the process shows us a lot of time is spent in:

(gdb) bt
#0  0x0000000000b9a6e4 in node::http2::Http2Session::OnStreamClose(nghttp2_session*, int, unsigned int, void*) ()
#1  0x0000000000af2e3c in node::Environment::RunAndClearNativeImmediates(bool) ()
#2  0x0000000000af3446 in node::Environment::CheckImmediate(uv_check_s*) ()
#3  0x000000000165abb9 in uv__run_check (loop=loop@entry=0x5278e80 <default_loop_struct>) at ../deps/uv/src/unix/loop-watcher.c:67
#4  0x00000000016532f0 in uv_run (loop=0x5278e80 <default_loop_struct>, mode=UV_RUN_DEFAULT) at ../deps/uv/src/unix/core.c:420
#5  0x0000000000aafa2d in node::SpinEventLoop(node::Environment*) ()
#6  0x0000000000bb11f4 in node::NodeMainInstance::Run() ()
#7  0x0000000000b26c44 in node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResult const*) ()
#8  0x0000000000b2a83f in node::Start(int, char**) ()
#9  0x00007f87ed3abd0a in __libc_start_main (main=0xaa5910 <main>, argc=2, argv=0x7ffd7edcaee8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd7edcaed8) at ../csu/libc-start.c:308
#10 0x0000000000aad7ee in _start ()

We suspect https://github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_V18.md#18.13.0 but it's difficult to know for sure.

We've rolled back to 18.12.1 and will be observing if it keeps happening.

Activity

  1. udnes99 commented on Jan 17, 2023

    @udnes99
    udnes99
    on Jan 17, 2023

    Also experienced issues after upgrading to 18.13.0. Specifically when running within a container deployed to Google Kubernetes Engine. We actually started to experience a 100% CPU freeze using Node 16. However, we did not figure out which version of Node 16 that caused trouble, and our fix was ironically to upgrade to node 18 (which worked until recently).

  2. ronag commented on Jan 17, 2023

    @ronag
    ronag
    on Jan 17, 2023
    MemberAuthor

    @nodejs/http @nodejs/http2

  3. ronag
    changed the title [-]Node18.13 HTTP2 live lock[/-] [+]Node 18.13 HTTP/2 live lock[/+] on Jan 17, 2023

  4. santigimeno commented on Jan 17, 2023

    @santigimeno
    santigimeno
    on Jan 17, 2023
    Member

    It looks it could be: dee882e94f though it'd be great having a way to reproduce it.

  5. mcollina commented on Jan 17, 2023

    @mcollina
    mcollina
    on Jan 17, 2023
    SponsorMember

    I concur.

  6. udnes99 commented on Jan 18, 2023

    @udnes99
    udnes99
    on Jan 18, 2023

    I went through the changelogs, and discovered the same commit was introduced in Node 16.19.0, which is the time at which we started noticing issues, bur were unable to repro. Great job with debugging!

  7. ronag commented on Jan 18, 2023

    @ronag
    ronag
    on Jan 18, 2023
    MemberAuthor

    @santigimeno Could you make a revert PR?

  8. mcollina commented on Jan 18, 2023

    @mcollina
    mcollina
    on Jan 18, 2023
    SponsorMember

    I suggest we have a regression test for this before backporting the revert across all lines.

  9. santigimeno
    added a commit that references this issue on Jan 18, 2023

    Revert "src: let http2 streams end after session close"

    3619cb3

  10. santigimeno commented on Jan 18, 2023

    @santigimeno
    santigimeno
    on Jan 18, 2023
    Member

    Revert PR is here: #46249 and moved to test to the known_issues folder

  11. VoltrexKeyva
    added
    http2Issues and PRs related to the http2 subsystem.
    on Jan 18, 2023

  12. alexpusch commented on Feb 17, 2023

    @alexpusch
    alexpusch
    on Feb 17, 2023

    We encountered this ugly issue as well - node process hangs sporadically after node 18.3 upgrade. Took me a while to trace it here.
    Our http2 stream use is probably Google Pubsub grcp client

  13. killagu commented on Feb 21, 2023

    @killagu
    killagu
    on Feb 21, 2023
    Contributor
    2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0
2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0
2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0
2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0
2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0
2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0
2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0
2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0
2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0
2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0
2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0
2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0
2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0
2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0
2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0
2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0
2023-02-21 15:48:41 [INFO] Http2Session client (858) stream 7 closed with code: 0

    Node v16.19.1 has the same problem. Session close never done.

  14. nodejs-github-bot
    closed this as completedin d1b29b4on Feb 21, 2023
  15. juanarbol
    added a commit that references this issue on Mar 5, 2023

    Revert "src: let http2 streams end after session close"

    73a8f46
  16. targos
    added a commit that references this issue on Mar 13, 2023

    Revert "src: let http2 streams end after session close"

    518b890
  17. BethGriggs
    added a commit that references this issue on Mar 27, 2023

    Revert "src: let http2 streams end after session close"

    086bb2f
  18. MayaLekova
    added a commit that references this issue on May 22, 2026

    Revert "src: let http2 streams end after session close"

    d1b29b4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    http2Issues and PRs related to the http2 subsystem.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions