Repository navigation
node-daily-v18.x-staging builds fail on rhel8-s390x since 1 Nov #3950
Description
Activity
Note to check whether this only affects test-ibm-rhel8-s390x-4 (all four failing daily v18.x-staging builds all ran on that machine).
other platforms haven't failed
This includes, so far, rhel9-s390x.
Note to check whether this only affects test-ibm-rhel8-s390x-4 (all four failing daily v18.x-staging builds all ran on that machine).
Resumed CI for nodejs/node#55708 (was failing on test-ibm-rhel8-s390x-4 with the same error) -- https://ci.nodejs.org/job/node-test-commit-linuxone/46644/nodes=rhel8-s390x/ is running on test-ibm-rhel8-s390x-1
FWIW I tried cherry-picking all the npm updates to see if that would improve the outcome, but no, it fails with the same error
https://ci.nodejs.org/job/node-test-commit-linuxone/46648/nodes=rhel8-s390x/console
10:12:25 npm error Exit handler never called! 10:12:25 npm error This is an error with npm itself. Please report this error at: 10:12:25 npm error <https://github.com/npm/cli/issues> 10:12:25 10:12:25 10:12:25 npm error A complete log of this run can be found in: /home/iojs/build/workspace/citgm-smoker/npm_cache/_logs/2024-11-05T10_06_54_406Z-debug-0.log 10:12:25 make[1]: *** [Makefile:754: tools/doc/node_modules] Error 1
So from the log it looks like npm failed to reach the registry. The fetch error on the audits endpoint would indicate that it's trying (and failing) IPv6. Node.js 18 doesn't have happy eyeballs, which would probably explain why the other release lines haven't been affected on this machine.
[iojs@test-ibm-rhel8-s390x-4 ~]$ cat /home/iojs/build/workspace/citgm-smoker/npm_cache/_logs/2024-11-05T10_06_54_406Z-debug-0.log ... 43 http fetch GET https://registry.npmjs.org/@types/mdast/-/mdast-3.0.11.tgz attempt 1 failed with ENETUNREACH 44 http fetch GET https://registry.npmjs.org/@types/hast/-/hast-2.3.4.tgz attempt 1 failed with ENETUNREACH 45 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 1 failed with ENETUNREACH 46 silly audit bulk request failed undefined 47 verbose audit error FetchError: request to https://registry.npmjs.org/-/npm/v1/security/audits/quick failed, reason: connect ENETUNREACH 2606:4700::6810:1c22:443 47 verbose audit error at ClientRequest.<anonymous> (/home/iojs/build/workspace/node-test-commit-linuxone/deps/npm/node_modules/minipass-fetch/lib/index.js:130:14) 47 verbose audit error at ClientRequest.emit (node:events:517:28) 47 verbose audit error at _destroy (node:_http_client:882:13) 47 verbose audit error at onSocketNT (node:_http_client:902:5) 47 verbose audit error at process.processTicksAndRejections (node:internal/process/task_queues:83:21) { 47 verbose audit error code: 'ENETUNREACH', 47 verbose audit error errno: 'ENETUNREACH', 47 verbose audit error syscall: 'connect', 47 verbose audit error address: '2606:4700::6810:1c22', 47 verbose audit error port: 443, 47 verbose audit error type: 'system' 47 verbose audit error } 48 silly audit error undefined 49 silly audit report null 50 http fetch GET https://registry.npmjs.org/@types/mdast/-/mdast-3.0.11.tgz attempt 2 failed with ENETUNREACH 51 http fetch GET https://registry.npmjs.org/@types/hast/-/hast-2.3.4.tgz attempt 2 failed with ENETUNREACH 52 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 2 failed with ENETUNREACH 53 http fetch GET https://registry.npmjs.org/unist-util-select/-/unist-util-select-5.0.0.tgz attempt 1 failed with ENETUNREACH 54 http fetch GET https://registry.npmjs.org/rehype-stringify/-/rehype-stringify-9.0.3.tgz attempt 1 failed with ENETUNREACH 55 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 3 failed with ENETUNREACH 56 http fetch GET https://registry.npmjs.org/@types/hast/-/hast-2.3.4.tgz attempt 3 failed with ENETUNREACH 57 http fetch GET https://registry.npmjs.org/unist-util-select/-/unist-util-select-5.0.0.tgz attempt 2 failed with ENETUNREACH 58 http fetch GET https://registry.npmjs.org/rehype-stringify/-/rehype-stringify-9.0.3.tgz attempt 2 failed with ENETUNREACH 59 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 1 failed with ETIMEDOUT 60 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 1 failed with ETIMEDOUT 61 http fetch GET https://registry.npmjs.org/@types/debug/-/debug-4.1.8.tgz attempt 1 failed with ETIMEDOUT 62 http fetch GET https://registry.npmjs.org/@types/ms/-/ms-0.7.31.tgz attempt 1 failed with ETIMEDOUT 63 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-3.0.0.tgz attempt 1 failed with ETIMEDOUT 64 http fetch GET https://registry.npmjs.org/css-selector-parser/-/css-selector-parser-2.3.2.tgz attempt 1 failed with ETIMEDOUT 65 http fetch GET https://registry.npmjs.org/highlight.js/-/highlight.js-11.8.0.tgz attempt 1 failed with ETIMEDOUT 66 http fetch GET https://registry.npmjs.org/property-information/-/property-information-6.2.0.tgz attempt 1 failed with ETIMEDOUT 67 http fetch GET https://registry.npmjs.org/rehype-raw/-/rehype-raw-6.1.1.tgz attempt 1 failed with ETIMEDOUT 68 http fetch GET https://registry.npmjs.org/rehype-stringify/-/rehype-stringify-9.0.3.tgz attempt 3 failed with ENETUNREACH 69 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 2 failed with ENETUNREACH 70 http fetch GET https://registry.npmjs.org/@types/debug/-/debug-4.1.8.tgz attempt 2 failed with ENETUNREACH 71 http fetch GET https://registry.npmjs.org/@types/ms/-/ms-0.7.31.tgz attempt 2 failed with ENETUNREACH 72 http fetch GET https://registry.npmjs.org/property-information/-/property-information-6.2.0.tgz attempt 2 failed with ENETUNREACH 73 http fetch GET https://registry.npmjs.org/unist-util-select/-/unist-util-select-5.0.0.tgz attempt 3 failed with ENETUNREACH 74 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 2 failed with ENETUNREACH 75 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-3.0.0.tgz attempt 2 failed with ENETUNREACH 76 http fetch GET https://registry.npmjs.org/css-selector-parser/-/css-selector-parser-2.3.2.tgz attempt 2 failed with ENETUNREACH 77 http fetch GET https://registry.npmjs.org/highlight.js/-/highlight.js-11.8.0.tgz attempt 2 failed with ENETUNREACH 78 http fetch GET https://registry.npmjs.org/@types/mdast/-/mdast-3.0.11.tgz attempt 3 failed with ETIMEDOUT 79 http fetch GET https://registry.npmjs.org/rehype-raw/-/rehype-raw-6.1.1.tgz attempt 2 failed with ENETUNREACH 80 http fetch GET https://registry.npmjs.org/property-information/-/property-information-6.2.0.tgz attempt 3 failed with ENETUNREACH 81 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-3.0.0.tgz attempt 3 failed with ENETUNREACH 82 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 3 failed with ENETUNREACH 83 http fetch GET https://registry.npmjs.org/css-selector-parser/-/css-selector-parser-2.3.2.tgz attempt 3 failed with ENETUNREACH 84 http fetch GET https://registry.npmjs.org/highlight.js/-/highlight.js-11.8.0.tgz attempt 3 failed with ENETUNREACH 85 http fetch GET https://registry.npmjs.org/@types/ms/-/ms-0.7.31.tgz attempt 3 failed with ETIMEDOUT 86 http fetch GET https://registry.npmjs.org/@types/debug/-/debug-4.1.8.tgz attempt 3 failed with ETIMEDOUT 87 http fetch GET https://registry.npmjs.org/@types/unist/-/unist-2.0.6.tgz attempt 3 failed with ETIMEDOUT 88 http fetch GET https://registry.npmjs.org/rehype-raw/-/rehype-raw-6.1.1.tgz attempt 3 failed with ENETUNREACH 89 info ok 90 error Exit handler never called! 91 error This is an error with npm itself. Please report this error at: 92 error <https://github.com/npm/cli/issues> ...
I've opened #3952, which has been deployed on to three of the four rhel8-s390x test machines (I'll update the last one when the citgm job it is currently running completes).
Note to check whether this only affects test-ibm-rhel8-s390x-4 (all four failing daily v18.x-staging builds all ran on that machine).
other platforms haven't failed
This includes, so far, rhel9-s390x.
Well annoyingly rhel9-s390x is now failing, after I started leave 😞.
- added a commit that references this issue
on Nov 22, 2024
Starting on 1 Nov, node-daily-v18.x-staging builds have been consistently failing on rhel8-s390x.
e.g.
From https://ci.nodejs.org/job/node-test-commit-linuxone/46599/nodes=rhel8-s390x/console
The warning can be ignored (expected for non-releases). I'm slightly concerned that the log references a path in the citgm-smoker workspace (from a node-test-commit-linuxone job!).
Both of the above builds ran on commit nodejs/node@1c4decc so we can rule out any of the subsequent changes landed on
v18.x-staging. Since it doesn't appear at first to be due to a code change (and other platforms haven't failed), I'm opening for investigation here in nodejs/build.