Repository navigation
feature: Local Modules #146
Description
Activity
@srcspider I agree that
require('../../../../../../../foo/bar')is painful but I think your solution is a bit too complicated. What if we could refer to the project root by using a special symbol instead, e.g.require('#foo/bar')
I would have preferred
@but it's been taken by org namespaces in npmi still agree with @Raynos on this one (trying to cite correctly): write applications in a flat structure (read:
lib/*.jsandlib/*/*.js). if an application can't be written using that structure, it's too big. don't write it or rethink it.I second that, if your application is getting out of hand with nested directories it's a solid sign that you need to split it up even more. You can even do this within an application and avoid the need to publish small, single-use packages by using an approach such as linklocal (I'm hoping that the npm client will add first-class support for features like this, but that might be a pipe dream).
@rvagg there are many scenarios where that approach is not practical, some apps really are big and complicated and separating into smaller modules brings with it its own set of problems. If iojs is to grow it must be possible (and fun!) to write these bigger apps using it. Your linklocal suggestion sounds like a hack around the problem - what is the difference between writing a module like that and simply having another folder under
/lib?Fwiw I've seen people writing their own global
appquire()functions to achieve this in bigger projects, which works but is not nice@phpnode can you find an example of a big app where you'd do
require("../../../../x")legitimately and it would not be beneficial to refactor it into smaller separate modules (preferably on GitHub)?I think that would really help the case here.
I've thought about this a lot (and I'm probably biased as I've written an editor where instead of
requireing modules/packages it finds the packages/modules you reference by name). I think the problem is not just relative requires, but also having very repetitiverequires in every file, e.g., requiringlodash,React(browserified) or any other utility package in every module.So just fixing the relative requires isn't going to solve the bigger issue of having repetitive and/or hard to reason with
requires all over the place.My own conclusion is that we probably shouldn't change the module loader or resolution algorithm, which is working wonderfully well (I mean, come on, it's fascinating that multiple versions of packages in a tree work as well as they do without coordination!). Instead we should create new tools to better manage our namespaces.
Again, here's why I think that way:
- Module names change. Sucks to update references everywhere.
- Module locations change. Again, sucks to update references everywhere.
- Some modules are required everywhere. Not so nice to have repetitive
requires in every module. - Can't use
global.
Instead of fixing this in core I'd rather think of how to fix this in the ecosystem. And determine if core needed to change in some small way to allow more ecosystem experimentation.
I've been struggling with this as well. It doesn't always make sense to break everything up into isolated packages if each piece is only relevant in the context of its sibling packages. Reality of "business" means not every component can be open-sourced or is immediately of open-source quality. Problem domains often messy and it's hard to get the ideal tree structure correct from the outset. Publishing stuff to a registry incurs a lot of overhead and makes architectural refactoring very difficult.
As rod mentioned, I've been trying to solve these problems with linklocal. I don't think this belongs in core at all, though wouldn't mind seeing this in npm one day.
linklocalallows you to structure your app exactly like so:app component1 component2 component2.2 component3 component4Then you can pull dependencies in-between components like any other dependency:
require('component1'). You just specify those dependencies explicitly in each component's package.json using standardfile:dependencies (these exist as of npm@2.0.0):"name": "component1", "dependencies": { "component2":"file: ../component2", "component4":"file: ../component4" }
then you call
linklocaland everything is symlinked into the appropriatenode_moduleslocations, (not just in the top-level):> linklocal -r component1 component2 component2.2 component3 component4 Linked 5 dependenciesWe've also been combining this with scoped packages to prevent future namespace collisions, and only require package.json changes if we eventually want to take advantage of scopes to publish to a private registry.
linklocalis basically a local, recursive,npm link– it doesn't clobber/clutter npm's global namespace, and will recursively symlink any localfile:dependencies for you automatically.linklocalfits a nice middle-ground between the alib/*directory and a private npm registry.lib/*gives you high flexibility but minimal structure or encapsulation. On the other hand, a private npm registry (& git urls/tarballs to a lesser degree) give you encapsulation and structure and localised dependencies, but creates great overheads during development. This is an especially arduous process when cross-project changes are required e.g.cd component1; git add; git commit; npm version; git tag; git push --all; cd component2; npm install --save component1@latest; git add; git commit; etc….linklocal gives you the flexibility of
lib/*with the structure of a registry, but without the and management overheads. It's currently being used in two moderate-size projects and works well, though it currently requires you to jump through a few hoops with npm. It's not for libs, it's for apps. Yes, ideally apps should be no different to libs, but that is very hard to achieve, at least for me. I don't think I could go back to building apps withoutlinklocalor an equivalent.</advertisement>@benjamingr I was typing a long reply here but @timoxley did a better job of explaining the problem, even though I disagree with his solution. I'm not aware of any large projects that do this legitimately on github, I do it myself in oriento but I think I could potentially solve those issues by using the (apparently now discouraged)
peerDependenciesfeature. However, that is an open source module, not a proprietary application.When developing applications it's very common for changes to affect more than one module, much more so than when developing small open source components and keeping these changes in sync is a lot of overhead. It's exactly the same problem faced by people who make use of git submodules in their apps.
It is conceptual purity at the expense of pragmatism and developer productivity. When there is no hope of reusing the various parts of the application between projects what is gained by having distinct modules?
there's
NODE_PATHthere as well, soNODE_PATH=lib/ nodewill let youlib/fooasrequire('foo')from all over your code...however it's being deprecated afaikWe've been using
linklocalin production for a few months now and it's pretty much solved all our localrequire()issues. I'd be very happy to see something similar being built into thenpmclient.@srcspider I agree that require('../../../../../../../foo/bar') is painful but I think your solution is a bit too complicated. What if we could refer to the project root by using a special symbol instead, e.g.
require('#foo/bar')
Sorry, not familiar with how the
#there is meant to work. Is it just searching forpackage.json? if so then that's either requiring an extra file that's hard to stick in a ignore clause on the editor. Note, you can safely tell your editor to not show the.iojsfiles I suggested and terminals will ignore them automatically.There are several other problems.
You said that refers to the root of the project. But that seems less flexible then what I suggested as an existing workaround which is just to name the directory you stick your modules in "node_modules" so on resolution names bubble up and then hit it, think it's the npm one and jump back in. That satisfies the functionality you describe with
require('#foo/bar')but doesn't require your module to be on the route. Doesn't break if you try to move the code around, doesn't require special files. And it all works alreadyMy problem wasn't referring to local modules, the node_modules solution works well enough. My problem is "managing local modules." Essentially having more freedom on the layout of the project (which the
#modulesyntax sounds like it just makes worse if I understood it correctly) and the ability to manage shared local libraries of modules with out much fuss.Note that in my example with
lib/mergeI could have hadlib/mergeinside thesomeAppModulesand moved it later on in any of the two library modules with out any code needed to change.With regard to the custom
requirefunction; I use a lot of browserify and abused aliasing to achieve the same effect but I prefer the "no build configuration" required option when possible.Doesn't actually achive what I described in the example. Note that at the end I describe how the different libraries can easily choose if they want to have their own
lib/mergeit's just a matter of having the module inside their structure. The PATH solution doesn't solve that.I also hate it for another reason: IT only works if you're gonna work on a single project all your life.
I often find projects go on waiting periods and I work on others. Or you're working on an isolated piece of code. Having magical
lib/mergeis just plain annoying, and managing PATH is annoying as well. It's a "becomes everyone's problem" solution.Doesn't solve my problem. It essentially creates the same problem as PATH, it's doesn't dynamically resolve. Creates a lot of configuration. I would prefer to configure locations of many modules, not individual module locations. It's much easier to reason with.
Note that even right now you can do this,
project/ node_modules/ # npm modules src/ node_modules/ # shared private code client/ node_modules/ # project specific codeNo configuration. Dynamically resolves. No extra tooling required. And you should be able to easily transition the code to a npm module if you ever need to, with little to no changes to the code just moving files.
The issue is only that you can't have that same structure there be flat, and you can't rename the
node_modulesin any way to make them actually describe more accurately what they actually are. Obviously it also gets really awkward when you need more then 1 shared library.off-topic, but to answer this
I think the problem is not just relative requires, but also having very repetitive requires in every file, e.g., requiring lodash, React (browserified) or any other utility package in every module.
You can easily solve this by bundling at the code level. For example if you have something like
Chiken, and that chicken has aModel,Repo,Viewand whatever else types associated with it you can just bundle everything under the nameChiken, by just having aChiken/index.jsthat looks like this:module.exports = { Model: require('./Model'), Repo: require('./Repo'), View: require('./View'), Domestic: require('./Domestic') // object with it's own Model, Repo, etc // ... }
You can now just do
require('system/Chiken')and you have access toChiken.Model,Chiken.Repo,Chiken.View, etc. React accepts the.in recent versions so you can use<Chiken.View />no problem.If you feel like being optimal you can always still do
var ChikenModel = require('system/Chiken/Model')This principle applies to utility functions as well,
module.exports = { merge: require('./merge'), href: require('./href'), hsl: require('./hsl'), Model: require('./Model'), // ... };
var lib = require('lib'); var Chiken = function (data) { lib.Model.call(this, data) }; Chiken.prototype = lib.merge(lib.Model.prototype, { // ... }); module.exports = Chiken;
Personally, I like aliasing locally any important functions like Promises, Request management, Logging, etc, so their implementation is in only one place in code and can be easily swapped out for a different one. I also prefer referring and loading directly very specialized libraries like
React,lodash, etc, and most of the time I do use just explicit declaration for utility functions just to keep them short.There are a few techniques that can be applied to large applications to make
require('../../../../../x')go away.Technique 1: Have multiple test folders
One of the common problems I see in large apps is files like
endpoints/user/routes/x.jstest/endpoints/user/routes/x.js- which leads to
require('../../../../endpoints/user/routes/x.js')
Instead I recommend:
endpoints/user/routes/x.jsendpoints/user/test/routes/x.js- which leads to
require('../../routes/x.js')
Technique 2: Dependency injection.
One of the common problems I see in large applications is statefully requiring completely different sub systems in your application.
Let's say you have files like
business/user/repo/save-user.js- `endpoints/user/routes/save.js
- which leads to
require('../../../business/user/repo/save-user.js')
We've found that requiring other sub systems statefully has always been a terrible idea. It's not flexible and hard to test. The coupling also makes refactors a nightmare.
The better practice is dependency injection. In our case we would have something like
// BAD: var saveUser = require('../../../business/user/repo/save-user.js') module.exports = saveUserRoute; function saveUserRoute(req, res, opts) { var saveUser = opts.services.user.repo.saveUser; /* real code */ }There are other techniques as well, but these two around tests and sub systems get rid of most of the nasty require statements.
Note that this seperation of "multiple packages in one git repository" does not have the overhead of multiple git repositories but has all the benefits of small packages. Especially if you keep one package.json per git repository.
We've found that requiring other sub systems statefully has always been a terrible idea. It's not flexible and hard to test. The coupling also makes refactors a nightmare.
Whats the difference between pushing opts into every method call like you did there and just doing:
Module-Localized Testing
project/ node_modules/ # npm modules src/ node_modules/ module1 tests/ node_module/ ...dependency injection... index.js module2 tests/ node_module/ ...dependency injection... index.js module3 tests/ node_module/ ...dependency injection... index.js
Your test file, (example is simplified)
var Tested = require('./temp/Tested'); // ...run test code for Tested...
cd src/node_modules/module1 mkdir tests/temp cp Something.js tests/temp/Tested.js run-tests rm -rf tests/tempHere's a version with the "modules_root"
.iojsfile suggested in the first post:project/ node_modules/ # npm modules src/ module1 tests/ ...dependency injection... index.js .iojs module2 tests/ ...dependency injection... index.js .iojs module3 tests/ ...dependency injection... index.js .iojs .iojs(note: it's pretty easy to make the
.iojsinvisible in most editors)You can create a specialized "mock" module and then just do
module.exports = require('mock/Something');to reuse test related code. It shouldn't result in any more code then the dependency injection method, only your dependencies would be dynamically resolved.Project-Localized Testing
You can save more code, and avoid false positives by having the test code outside the main source tree, so that after the temporary copy your piece of code can't access anything in the normal source tree, just mocks and whatever else you tell it.
project/ node_modules/ # npm modules src/ node_modules/ module1 module2 module3 tests/ node_modules/ Workspace/ ...you copy your the tested file(s) here... lib/ Promise.js # mock test/ module1Tests module2Tests
var Module1 = require('Workspace/Module1'); // ...test Module1...
So long as you wrap any important 3rd-party libraries into a local name space, eg.
require('bluebird') -> require('lib/Promise'), you don't have to worry about
global or npm dependencies getting in the way of your tests (aliasing locally is just good in general, even if you don't intend to write tests).The nice thing about the two method above is they can be applied with out much effort; so long as you haven't created a tangled mess of things. So you can use them with out refactoring too much of your project or including boilerplate.
You would also not have a need for all the wrapping, configuration, deferring and more importantly since the the resolution is based on the directory tree it's very clear how the resolution happens (anything could happen with dependency injection on the other hand; though I trust most people keep it sane).
Just for comparison here's the code for creating the
Chickentype with the two methods:Dynamically Resolved
var merge = require('lib/merge'); var Model = require('lib/Model'); var Chicken = function (data) { Model.call(this, data) }; Chicken.prototype = merge(Model.prototype, { // ... }); module.exports = Chicken;
// file: example.js var Chiken = require('models/Chicken'); var Food = new Chicken();
Dependency Injection
module.exports = function (di) { var merge = di.lib.merge; var Model = di.lib.Model; var Chicken = function (data) { Model.call(this, data) }; Chicken.prototype = merge(Model.prototype, { // ... }); return Chiken; }
// file: di.conf.js var di = {}; // Load Utilities // -------------- di.merge = require('lib/merge')(); // ... // Load Utility Types // ------------------ di.Model = require('lib/Model')(di); // ...
// file: main.js var di = require('di.conf'); var example = require('example'); example(di);
// file: example.js modul.exports = function (di) { var Chiken = di.Chicken; var food = new Chicken(); }
Personally everything in my project including non-javascript stuff is in some build pipeline so isolating code in a testable context is practically just as complicated to run as the dependency injection example; initial costs of writing build scripts not considered.
Its worth mentioning this technique from substack's browserify handbook:
https://github.com/substack/browserify-handbook#avoiding-
I find it very useful at times.I agree that
require('../../../../../../../foo/bar')is painful but I think your solution is a bit too complicated. What if we could refer to the project root by using a special symbol instead, e.g.require('#foo/bar')That would be really good to have.
You realize the model system is frozen right ?
We cannot make any changes to the
requirefunction.No, I did not realize of it. May I ask why is anything "frozen" at this stage and why changes are not allowed in the
requirefunction? Thanks a lot.Apologies for my tone.
The module system is "locked" ( not frozen, wrong terminology ).
See https://github.com/iojs/io.js/blob/v0.12/doc/api/modules.markdown#modules
If we change the semantics of
requireyou can publish a module to npm that will only be usable with iojs version 1.0. This means that you can no longer use modules in npm using node 0.10 or node 0.12This might not seem scary, however because npm supports nested dependencies and ranges you might require a module A that supports node 0.10 and a dependency of A that is three levels deep say module E might break backwards compatibility by using this new require semantics in a patch or minor version change, this means module A no longer works. This can be a very unexpected and frustrating change.
node has made an enormous effort to not break backwards compatibility between 0.8 and 0.10, and between 0.10 and 0.12.
I doubt the folks involved with iojs would want a backwards incompatible change between node 0.12 and io 1.0
Thanks for the comment.
If we change the semantics of require you can publish a module to npm that will only be usable with iojs version 1.0. This means that you can no longer use modules in npm using node 0.10 or node 0.12.
Clear. Being honest, I was just considering this new feature (using "#" to indicate root of the project) for custom projects rather than for NPM libraries (as typicall libraries are small enough not require features like that one).
But yes... there is no way to categorize/distinguish them, so it would be dangerous anyhow.
If we change the semantics of require you can publish a module to npm that will only be usable with iojs version 1.0. This means that you can no longer use modules in npm using node 0.10 or node 0.12
The semantics can be locked specifically to the
node_modulesdirectory andrequire. Since we're talking local modules so long as you correctly create the distinction of "these are universal package conventions" and "these are the conventions of your flavor of node" you can pretty much do anything.Many languages have this distinction: 3rd party modules load one way, your local modules load whichever way you feel is best, and the system work fine.
The issue is mainly really "how would npm block misuse of the conventions" as well as how would stuff like browserify handle such conventions with out much fuss.
Here is, in my opinion the simplest solution to these problems:
- introduce an alternative loading system for local packages only, eg.
localfunction - add transformation to require plugin for use by common build systems and things like browserify
- this would be standardized as how all flavors of node, not just iojs bypass this problem
Example
var Model = local('lib/Model'); var Example = function (data) { Model.call(this, data); }; module.exports = Example;
The
localfunction above is identical torequirewith the exception that it also allows for extra node-flavour rules; such as the.iojsfiles suggested at the start of the topic.npmexplicitly doesn't allow for the use oflocal(it's easy to scan for), thereby ensuring modules can be universally used.Example usage, obviously very little changes:
node ./src/main.js browserify -t iojs ./src/main.js -o ./public/main.js
// in whatever build system you're using browserify({ transform: [ 'iojs' ] })
So in summary,
requirestays locked as-is, npm can keep serving universal modules, browserify stays as-is, different flavors of node can implement whatever local modules management strategy they wish so long as it uses the functionlocal(or whatever is convened as a standard "local modules loading" require-function).Are there any other political issues with adding local-only modules functionality?
- introduce an alternative loading system for local packages only, eg.
I also never had any problems with too long paths. Actually you anyway should just pass a variable (like a
koainstance) around. See here for an example.If my package is named
"foo"in thepacakge.json, it could be as simple asrequire('foo/somethings');? It assumes that thefoodirectory is namedfooand node also search fromdirename($PWD).@srcspider The module system is, as @Raynos mentioned, locked: no changes will be made to it. This precludes changing the API for modules, both explicitly (modifying require directly), and implicitly (changing the expectations of what's available to modules -- e.g., new globals). Much of Node's value is tied up in the node package ecosystem; the easiest way to destroy that value it would be to change the module system and introduce incompatibilities, however subtle or unintentional.
That being said, as I understand the proposal, it should be able to be implemented as a standalone package on npm using require.extensions and perhaps a bit of monkey-patching. An exploration of what code written against such an API would look like in a more fleshed-out fashion would be valuable for future module systems.
Closing this for now. It is very unlikely that we'll make such a drastic change to such a delicate, locked API, but feel free to continue the conversation here. I would be interested in the results of your experiment, if you choose to release a package on npm!
I would like to make a brief plug for using this problem space as an opportunity to model ways of introducing ES6 modules into Node and writing a flexible, potentially metadata-driven
System.loaderimplementation that allows you to use something like your.iojsmanifest approach, @srcspider. The whole point of decoupling ES6 module loading from parsing and linking is to make it possible to write alternative loaders using the ES6 reflection API, so I bet it would be educational to see how far you could take this building on top ofcurrent transpilation and shimming tools likeesnextand6to5.- added a commit that references this issue
on Apr 6, 2017
Currently one of the biggest pain points with node development is the problem of referring to local modules (ie. how to avoid the unreadable
require('../../../../../something')).The more modules you create in your project the biggest the issue becomes.
Node conventions recommends splitting things into modules and uploading to npm however that has some issues of it's own:
There's a simple solution for both, and that's to create "fake" node_modules directories. Something like this:
This works great since any code inside
src/node_modulescan call internal modules and node will resolve them correctly once it's search algorithm goes intosrc/.It also doesn't require any path aliasing (which I believe is frown upon?), so you can just stick everything in
src/into a module, point tosrc/node_modules/main.jsas the entry point and your entire application is now reusable (either by the world or members of your private group).The problem with that solution is that it's very annoyingly:
Solution
Changes to module search algorithm
node_modulesdirectory is not found in directory, node should search for a.iojsfile.iojsfile is found then the configuration is read and rules in it applyExample
.iojsfiles that resolves our problemIn each custom module library you place the following, unless it's literally called "node_modules"
In the root you place
.iojsfile if you wish to expose modules inside them directly to other modules:We can now have something like:
All three directories in
src/would have a copy of the "modules_root".iojsfile. And insrcyou would have a copy of the "node_modules_dirs".iojs.Let's say for example both
sharedLibrary1,sharedLibrary2,sharedPrivate1andsharedPrivate2all have a filelib/merge.js.Because the
.iojsapplies while still inside them callingrequire('lib/merge')would resolve tosharedPrivate1/lib/mergein one andsharedPrivate2/lib/mergein the other; which is exactly what we want.If you call
require('lib/merge')insomeAppModuleswill resolve tosharedLibrary1/lib/mergebecause it's the first one specified in thesrc/.iojsfile, however if you call it insharedLibrary2it will resolve tosharedLibrary2/lib/mergesince it will get resolved bysrc/sharedLibrary2/.iojsbefore it gets resolved bysrc/.iojsBackwards compatible convection of
someAppModulesto a single application NPM moduleOf course this assumes you convert some parts of your code to rely on generic modules as opposed to local ones. Easy peasy since everything has a nice grep-able name.