Repository navigation
Discussion: New “ESM by default” mode #49432
Description
Activity
- added
on Sep 1, 2023 I don't think this is feasible to change the default if package.json is present due to the massive amount of modules available on npm.
My 2 cents on how to solve this in a much easier way: let's make
npm initadd the necessary field inpackage.json.Reacted by Raz Luvaton, Erick Wendel, Jake Bailey, Alexander Praetorius, Bosco Domingo, Brandon Bennett, Natalia Venditto and Ahmed ElawadReacted by Rasmus Porsager, SeungBin Kim and goosyI don’t think this is feasible to change the default if package.json is present due to the massive amount of modules available on npm.
I’ve considered this and I don’t think it’s a blocker. As discussed in #49295 (comment), there are many potential solutions, the simplest of which is to just crawl through subfolders adding
"type": "commonjs"topackage.jsonfiles that lack it. I don’t think this needs to be a worry or something we need to deal with at this point.Also, the proposal isn’t to go straight to changing Node’s default. That will be months if not years away, once this flag has shipped for a long time and issues have been resolved. The time for considering the impact of changing Node’s default is once this flag has been available for a while and we can see how it works. We may never change Node’s default, and that’s fine; this flag and mode has merit regardless. The point of mentioning that this might lead to changing Node’s default is so that we design with that potential goal in mind, so that whatever we add has a clear way to opt out if/when it becomes the new default.
@GeoffreyBooth how does adding that field work for people using other languages/processes that spawn child processes for existing codebases? I'm mostly curious since it is a much larger breaking change than most we see and affects all sorts of runners of node applications in non-simplistic ways.
Reacted by Alexander Praetoriushow does adding that field work for people using other languages/processes that spawn child processes for existing codebases? I’m mostly curious since it is a much larger breaking change than most we see and affects all sorts of runners of node applications in non-simplistic ways.
First off, adding a flag isn’t a breaking change. The breaking change would only be if we make the flag the new default behavior, which I don’t think we need to worry about just yet. It’s premature to consider larger ecosystem effects of making this new mode the default when we haven’t designed, much less implemented, the flag to enable this new mode.
The flag is no different than achieving these effects via module customization hooks (the changes that the hooks can achieve, anyway). Just as other languages/processes wouldn’t know how you’re customizing the files in your project when using hooks, they wouldn’t know whether your project is meant to run under this new flag or not. This is a problem we already have, and it’s on those other tools to provide a way for the user to signal to them that they should expect Node to be in this new mode, like how
tsconfig.jsonis used. Shipping the flag long before we change Node’s default allows the ecosystem time to adapt.One thing we can do to help ease the transition is to start making the
"type"field mandatory. As in, you don’t need to have apackage.jsonfile, but if you have one, it needs a"type"field. Then at least for applications, which are our dominant use case, there’s still a deterministic way for tools that work with apps to know how all the files within that application should be interpreted. We could make this one of the effects of the flag, so that in the new mode we need explicitpackage.jsontypes.Reacted by Haroen Viaene and Hans BrendeReacted by Shalvah@GeoffreyBooth I'm not opposing the flag in any way, I'm just concerned about the default swap which has been attempted in the past in #32394
Then at least for applications, which are our dominant use case, there’s still a deterministic way for tools that work with apps to know how all the files within that application should be interpreted. We could make this one of the effects of the flag, so that in the new mode we need explicit package.json types.
I'm more skeptical on this due to files being loaded for config/init like ~/.*rc files outside of the application directory.
I think the idea of having multiple releases in general is not great - if we want to do it, builds with pointer compression enabled would’ve been a thing already and then we would have another headache about what combinations of releases we want to maintain….
A runtime flag that makes ESM the default sounds good to me though, why can’t we just have that and allow it in NODE_OPTIONS? That should be enough for the use cases mentioned in the OP already
I’m more skeptical on this due to files being loaded for config/init like ~/.*rc files outside of the application directory.
There’s an infinitely long tail of ecosystem tools and libraries that don’t support whatever latest and greatest features we ship. Several years ago there were presumably tools that relied on the fact that
.jsfiles were CommonJS, and those tools needed to update to check for thetypefield before making that assumption; this is no different. If we limit ourselves by what today’s tools expect out of Node, then we’d never be able to change anything, even additive/non-breaking changes. Users can make the choice of deciding to use this flag or deciding to use a tool that's incompatible with it.I think the idea of having multiple releases in general is not great
What do you mean by “multiple releases”?
What do you mean by “multiple releases”?
multiple binaries, e.g. if you can have node-esm, then you can also also have node-pc (pointer compression), then you can also have a combination of node-esm-pc, then you can have…other combinations
What do you mean by “multiple releases”?
multiple binaries, e.g. if you can have node-esm, then you can also also have node-pc (pointer compression), then you can also have a combination of node-esm-pc, then you can have…other combinations
Yes, that’s perhaps an argument against shipping an alternate binary (see #49407). I think we should worry about the flag first before contemplating binaries; I think the flag stands on its own, and maybe there’s a solution for a binary with that flag enabled that doesn’t involve us needing to ship or maintain it, but we can worry about it later.
A runtime flag that makes ESM the default sounds good to me though, why can’t we just have that and allow it in NODE_OPTIONS? That should be enough for the use cases mentioned in the OP already
Shebangs can’t include flags on many platforms. See #49295 (comment) and linked and following comments. I don’t know if shell scripts have much need for pointer compression, but assuming they don’t, then the shebang case is an argument for a separate binary, as shell scripts being unable to specify flags is a particular constraint that only a separate binary can solve, and hopefully this is the only Node option that such scripts need control over.
Anyway to your larger point, yes, let’s ship the flag first and go from there. Many platforms do support flags in shebangs, and there are other use cases solved by the flag, so we’re still making a lot of progress just with the flag by itself at first.
Reacted by Kevin Gibbons84 remaining items
It shouldn’t be an issue, the ESM loader is able to load CJS, why would we error?
I was thinking how would we handle something like
node --entry-url ./entry.cjs?foo=barbut I guess you’re right, it would get handled however animport()of that would be handled currently, which I assume is along the lines of “convert the URL to a file path and import/require it.” So--entry-urlwould become another of those “if this is used, the user has opted into having the ESM loader handle the entry point” flags like--importand--loader.Anyway getting back to the UX question first though, which do we want:
- 👀
--importis used for this case, likenode --import ./entry.js, and if the user wants the REPL with--importthey need to add-ior--interactive - ❤️
--entry-urlis a new boolean flag that causes the entry point string to be parsed as a URL and loaded by the ESM loader:node --entry-url --enable-source-maps ./entry.js?foo=bar
Reacted by Stephen Belanger, Jordan Harband and Livia MedeirosReacted by Jacob Smith and Guy Bedford- 👀
I know people are still debating the best approach for supporting URL entry points, but since one of the options under consideration is a semver-major change and the others aren’t, I opened a PR for the semver-major one in case that wins consensus: #49946. Let’s hopefully decide whether or not to go with that approach before the 21.0.0 cutoff on Tuesday.
I think it would be better to open spinoff issue so we can have consensus on more details.
For example, do we want to support relative URLs? Do we want to support bare specifiers? Do we want to support shortcut specifiers? Do we want to allow users to override mime-type? Should we support loading CJS with URL? Should we seek for pjsons forfile:URLs?For the UX question, it might be worth mentioning that third option: providing main entry URL as named argument.
As for when it should land, I see it as a feature tied to
import.meta.mainrather than feature tied to--default-type. If we are planning to have.mainsoon, I'd suggest to land this feature directly afterwards (if not in the same PR, assuming thatimport.meta.mainisn'tsemver-major).
The reasons for that are that it's much harder (if possible at all) to test if it actually works as main entry point, and it would be unclear for users that there's difference between main entry point and import until we provide a mechanism to differentiate.I think it would be better to open spinoff issue so we can have consensus on more details.
We can, but since the first PR implemented everything else I think the URL entry stuff is all that’s left to design, so maybe we can just finish it here?
For example, do we want to support relative URLs?
Yes. The input would be relative to the file URL of
process.cwd().Do we want to support bare specifiers?
I don’t think so. That’s what
--importis for, or one could donode --entry-url data:text/javascript,import('bare-specifier').Do we want to support shortcut specifiers?
What is a shortcut specifier?
Do we want to allow users to override mime-type?
From an HTTPS URL, you mean? I think not as part of the CLI, that can be achieved via a module customization hook.
Should we support loading CJS with URL? Should we seek for pjsons for
file:URLs?Yes and yes. It would be loaded by the ESM loader, the same as if you had referenced a relative or absolute file URL via
import().What is a shortcut specifier?
Subpath imports, the ones starting from
#.Do we want to allow users to override mime-type?
From an HTTPS URL, you mean? I think not as part of the CLI, that can be achieved via a module customization hook.
From any non-
file:URL, to be precise.
For example, we could expand--input-typeto enablenode --input-type=commonjs --entry-url https://example.com/legacy.cjsandnode --input-type=commonjs --entry-url 'data:text/javascript,console.log("am CJS");'(I don't think we should, but it's worth considering).Subpath imports, the ones starting from
#.No, I think we shouldn’t implement the ESM resolution algorithm here.
#would just mean a hash, like a URL.From any non-
file:URL, to be precise.We already have
--experimental-network-imports. The waydata:andhttps:imports work currently is that Node uses the MIME types from the header of the resource. That can be overridden by theloadmodule customization hook; I don’t think we want to provide another way to customize this.- added 2 commits that reference this issue
on Feb 15, 2024 This issue has been marked as stale due to 210 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.Not stale
Not stale
Yes it is, the last comment was from almost 3 years ago. Maybe you meant "still relevant"?
I would argue that is no longer relevant, now that we have syntax detection enabled by default, I think the use-case is mostly fulfilled, and I don't think anyone is working on this. I don't see the point of keeping that issue open, closing as not planned.
Reacted by Jordan Harband
Building off of #49295 (comment), we’re considering a new mode where all of the current places where Node defaults to CommonJS would instead default to ESM. I think this should be enabled by flag, and once it ships we can potentially ship a separate binary where it’s enabled by default; and/or we can someday make the new mode Node’s default in a semver-major change.
The use cases solved by such a mode are (and I’ll edit this comment to update this list as people think of more):
I want to write ESM syntax by default without needing to opt into it, such as via
package.jsonfile or file extension or--input-type=module.I want to write what are typically called shell scripts in ESM JavaScript, where I can have a single extensionless file anywhere on my disk that uses ESM syntax and it doesn’t need a nearby
package.jsonfile or a symlink or wrapper file in order to run as ESM.I want consistency in how the CLI handles references to files; currently
--importaccepts URL strings but the main entry point must be a path string (and therefore can’t be adata:URL orhttps:URL, such as to work with--experimental-network-imports).If this mode becomes enabled by default in a future version of Node, when using that version of Node I want to be able to start a new project and write ESM syntax without needing to take any steps to opt into enabling ESM support.
To handle these use cases, the changes that this mode should make are (and likewise I’ll edit this to reflect consensus):
When a
.jsfile is outside of any definedpackage.jsonscope, because there are nopackage.jsonfiles in its folder or any folders above it, it will be treated as ESM JavaScript.When a
.jsfile is in a package scope where its nearest parentpackage.jsonlacks a"type"field, it will be interpreted per the conditions we establish in Discussion: In an ESM-first mode, how should apackage.jsonfile with notypefield be handled? #49494: when the package scope is under a folder namednode_modules, the file will be treated as CommonJS; otherwise it will be treated as ESM.When an extensionless file is outside of any defined
package.jsonscope, it can be run as an entry point or imported per the conditions we establish in Discussion: In an ESM-first mode, how should extensionless entry points be handled? #49431: if the file begins with the Wasm magic bytes, it will be evaluated as Wasm, and likewise for any other future file types with defined headers that don’t conflict with JavaScript; otherwise it will be evaluated as ESM JavaScript. (Still subject to--experimental-wasm-modulesuntil that stabilizes.)Input from STDIN or
--evalwould be treated as ESM unless--input-type=commonjsis passed. Or in other words, the default value of--input-typewould flip fromcommonjstomodule.The CLI will parse the main entry point as an URL string, similar to how the value of
--importis parsed.process.argv[1]will not be parsed by the CommonJS loader.This new flag will be named per the discussion in #49541.
Prior art: #32394, #49295, #49407
@LiviaMedeiros @nodejs/loaders @nodejs/wasi @nodejs/tsc