Repository navigation
Future of the Node HTTP Client #38533
Description
Activity
Maybe add something in the docs that point to undici. However, I don’t think this is something we have done in the past.
I found some prior art that may help: our
util.log(string)core function recommends using a third party module, but it's deprecated, which I'm unsure whether would be an appropriate designation in this situation (perhaps more likely legacy).Stability: 0 - Deprecated: Use a third party module instead.
Refs: https://nodejs.org/dist/latest-v16.x/docs/api/util.html#util_util_log_string
Reacted by Robert NagyI share @ronag concerns and I would also add one of my own.
The reason why very few would like to work on the HTTP client is because every change breaks the huge number of modules in the ecosystem that monkeypatch it. Fixing the majority of those bugs requires either breaking some other users or finding some clever hacks that will come hunting us back over the years.
Since this is on the TSC agenda: @nodejs/tsc
I think @jasnell is talking about shipping WHATWG ReadableStream this year (right?), we can also land the polyfill in core in the meantime and just ship fetch (on top of unidici perhaps).
Then we can direct people to the new API and put the old API in maintenance mode and possibly eventually even deprecate it.
Another possible alternative would be to land unidici as part of core (or as a dep) and build the existing HTTP API on top of it (kine of like what browsers did with fetch and XHR or how domains build on async_hooks now).
Another possible alternative would be to land unidici as part of core (or as a dep) and build the existing HTTP API on top of it (kine of like what browsers did with fetch and XHR or how domains build on async_hooks now).
It's not possible. The current API is an extremely leaky abstraction that multiple modules in the ecosystem monkeypatch. Doing so would not reduce the friction for the community.
It's not possible. The current API is an extremely leaky abstraction that multiple modules in the ecosystem monkeypatch. Doing so would not reduce the friction for the community.
I'm sure this is pretty common in servers - but who's patching the clinet?
I'd prefer we keep the existing API with a legacy status and introduce a new one based on
undici.Reacted by SK, NemoStein and Eliaz BobadillaI'd prefer we keep the existing API with a legacy status and introduce a new one based on undici.
Note that if landing
fetchis an eventual goal that would mean we'd have to maintain 3 HTTP APIs. I am fine with this if the fetch we land is based on unidici to the point we only de-facto maintain two APIs and fetch is just a wrapper around thatJust a side note. I actually prefer if undici is not included in core. It is much easier to maintain and develop as npm module.
Reacted by Szymon Marczak, Nick K. and max-hkI'd prefer we keep the existing API with a legacy status and introduce a new one based on undici.
Note that if landing
fetchis an eventual goal that would mean we'd have to maintain 3 HTTP APIs. I am fine with this if the fetch we land is based on unidici to the point we only de-facto maintain two APIs and fetch is just a wrapper around thatThere is https://github.com/Ethan-Arrowood/undici-fetch which we are considering merging into undici once it's a bit more mature.
It would be unfortunate to have to recommend people to use an external module for doing HTTP requests...
Reacted by Benjamin Gruenbaum, Antoine du Hamel, Gary Crye, Chengzhong Wu, Andrei Pechkurov, bl-ue, Dustin Deus, RyccoSN, Carlos Fuentes, Andrii Oriekhov and 13 moreKind of OT on your side note:
Just a side note. I actually prefer if undici is not included in core. It is much easier to maintain and develop as npm module.
I think that's a bigger problem and we need a process for deps to live in core without requiring a full CI run or the same process. The reason it's much nicer to maintain/develop outside of core is (presumably, from stuff I was involved in):
- You don't need to wait 48 hours (or a week) to land PRs.
- You don't need to wait for mostly flakey CI that's mostly irrelevant.
- You can easily set up your own infra without needing to bother more people.
- You don't need to go through the consensus seeking process.
It would be great if we could solve that and make contributing relatively standalone stuff (like unidico or AbortSignal) more convenient and less tedious.
One approach you may want to take is to keep the nodejs/unidici repo and treat it like a dep - that way you'd get the same "freedom" stuff like libuv gets
There is https://github.com/Ethan-Arrowood/undici-fetch which we are considering merging into undici once it's a bit more mature.
That would be cool though I would prefer it if users had
fetchas a convenient API andunidici(presumably with a more "standard" name) as a low-level API (assuming callback-land http becomes legacy).Kind of OT on your side note:
What does OT mean?
I think that's a bigger problem and we need a process for deps to live in core without requiring a full CI run or the same process. The reason it's much nicer to maintain/develop outside of core is (presumably, from stuff I was involved in):
Yea I get that. Just thinking that it would be nice if possible.
There is https://github.com/Ethan-Arrowood/undici-fetch which we are considering merging into undici once it's a bit more mature.
That would be cool though I would prefer it if users had
fetchas a convenient API andunidici(presumably with a more "standard" name) as a low-level API (assuming callback-land http becomes legacy).Yea, that's the idea. In terms of API we have a bit of a layered approach in undici. Whether or not we split undici into minimal packages (e.g. undici-core, undici-pool, undici-agent, undici-api, undici-fetch) or just have everything together (undici) is something to consider.
Reacted by Benjamin Gruenbaum20 remaining items
Is there already a conclusion regarding moving undici into core?
I fully understand that keeping it as a separate NPM module results in an easier workflow. But I assume the same would be valid for all core components.core has a well defined maintenance/release strategy including LTS. Are the same rules applicable to undici once it is the recommended HTTP client?
node has citgm to watch the impact on the ecosystem. Currently it's verified that a new node release doesn't break undici but I assume once undici as the recommended HTTP client something similar should be done for undici releases.
Please note that my intend is not to block anything here. My main point is to understand the long term strategy regarding this.
Is there already a conclusion regarding moving undici into core?
No.
But I assume the same would be valid for all core components.
Indeed. This is an argument that goes back and forth, e.g. stream utils such as pipeline and finished (and now destroy) could also more easily live as npm packages. I'm not sure what the strategy in general is here in regards to what is and isn't in core.
core has a well defined maintenance/release strategy including LTS. Are the same rules applicable to undici once it is the recommended HTTP client?
I have a hard time seeing that it would. Not sure whether or not that would make sense. It would be as any other npm package, if it's not in core. I think LTS strategy makes sense for node core since it's not possible for users to cherry-pick what to or not to upgrade. This does not apply to npm packages where users can choose when and how to upgrade semver major.
node has citgm to watch the impact on the ecosystem. Currently it's verified that a new node release doesn't break undici but I assume once undici as the recommended HTTP client something similar should be done for undici releases.
That would be great. However, we would need help to put that into practice. I think there was some project that was working on making something like citgm for npm packages (something along the lines "will I break you" or something)?
Please note that my intend is not to block anything here. My main point is to understand the long term strategy regarding this
Good feedback. My understanding is that the consensus is that undici is the way forward but there are still questions that remain open.
I'm much in favor of leaving undici outside of core due to ease of development and maintenance. Those are my personal primary concerns as a undici developer and those priorities might not align with the priorities of node core and the ecosystem.
Reacted by Pelle Wessman, Matteo Collina, Andrii Oriekhov and Claus KlingbergIf undici is merged into core, it would follow the same LTS/maintenance cycle. If it remains separate, then it would have it's own cycle, similar to the
readable-streammodule. Personally, I'd like to see it merged into core sooner rather than later -- however, right now it's premature as that would hinder it's ability to evolve as quickly as it needs right now.We didn't have the right people to talk about this at the TSC meeting today, but it seems like it ties into the larger issue of nodejs/TSC#1041.
Reacted by Pelle WessmanWe agreed in the TSC meeting today to remove from the tsc-agenda and possibly add it back on after nodejs/TSC#1041
is resolvedIf I understand correctly undici has all fixes to node http that are long due, and changing http itself is an impossible puzzle since so much code in the wild monkeypatches it.
In that case I recommend to include undici in node alongside http and mark it as legacy.
Just a side note. I actually prefer if undici is not included in core. It is much easier to maintain and develop as npm module.
Why is is so much easier ?
Why is is so much easier ?
Faster release cycles, easier for others to contribute to, independent CI environments, etc.
Reacted by max-hk, Pelle Wessman, Vicary and Andrii OriekhovFaster release cycles, easier for others to contribute to, independent CI environments, etc.
To keep those benefits is it possible to include whatever the last release is from unici into node whenever node is released (similar to npm I think)
Reacted by Steven, Nicky McCurdy, Muthu Kumar and Andrii OriekhovThis issue has been marked as stale due to 210 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.This issue has been automatically closed after 30 days of inactivity following its stale status (no activity for a total of 120 days).
If this is still relevant, feel free to reopen it or leave a comment with additional details so we can continue the discussion.
I see a slight problem with the situation with the node core http client. We are getting issues reported in the issue tracker. However, there is little interest by contributors (that are familiar with the code) to look into these issues and resolve them.
Part of the reason for this is that some of the people that usually engage with http client issues is @mcollina, @dnlup or myself. Whom I believe are mostly focusing on nodejs/undici, which we feel is a better alternative and the future, and have limited interest in putting further effort into the node core http client.
This is a slightly suboptimal situation. I would personally like to discuss whether and how we could encourage users over to NodeJS/undici and eventually deprecating the current node client.
Maybe add something in the docs that point to undici. However, I don’t think this is something we have done in the past.
Thoughts?