Managing node_modules/

The whole point of Orogene is to manage node_modules/ directories for consumption by various tooling, such as Node.js, transpilers, bundlers, linters, etc.

The process of managing node_modules/ and making sure all the appropriate dependencies are in place is called "apply" in orogene. Application happens through a number of commands, automatically. For example, oro reapply, oro add, oro remove, and oro run all make sure to apply your dependencies as needed, to make sure your node_modules/ corresponds to your manifest and lockfile as much as possible. You can disable this behavior, as desired.

Application blow-by-blow

By default, the apply operation will do the following things:

  1. Read dependencies declared in package.json.
  2. Generate a dependency tree based on those.
    1. If a lockfile (package-lock.kdl or package-lock.json) already exists, it will be used to supplement the resolution.
    2. If any dependencies in the lockfile are missing or don't correspond to expected dependencies, the tree will ignore that particular lockfile data and resolve the item(s) itself.
  3. Scan through the existing node_modules/, if any, and prune anything "extraneous" (aka, removed/outdated dependencies, stray files, etc).
  4. Extract any missing packages into node_modules, into one of two modes:
    1. isolated (default, preferred)
    2. hoisted (possibly more compatible, discouraged because it exposes phantom dependencies)
  5. Execute any preinstall scripts on the entire tree, including the root package.
  6. Link/shim any bins in the dependencies to their appropriate node_modules/.bin directories.
  7. Execute install and postinstall scripts on the entire tree, including the root package.
  8. Finally, the updated lockfile is written to package-lock.kdl.

Modifying Application

All commands that execute implicit apply accept the same options for modifying its behavior. To see the full list, refer to the apply command options, which are also listed in the -h/--help for any command that performs implicit apply.

Some options of note:

--no-apply

Skips the entire apply operation. This option will have no effect on the oro apply command, which forces an apply regardless of your settings.

This option can be useful if you don't like implicit applies and would like to generally configure Orogene to skip them by default, but let you use --apply when you happen to want to apply something. To do this, add apply false to your oro.kdl options node.

--locked

Instead of using lockfiles as a suggestion/optimization, this option will force an error if the dependencies declared in package.json don't correspond to the ones declared in an existing package-lock.kdl.

This option has no effect in oro add/oro remove, for hopefully obvious reasons.

--lockfile-only

Resolves the dependency tree and writes the lockfile as appropriate, but skips anything having to do with node_modules itself, including pruning and scripts.

--no-lockfile

Skips writing, or updating the lockfile entirely. As of right now, this will still read the lockfile to inform resolution.

Adding or Removing Dependencies

You can modify your current project's dependencies three different ways:

  1. Use the oro add command.
  2. Use the oro remove command.
  3. Edit package.json manually and rerun oro apply.

When any of these methods are used, orogene will update the current dependency tree such that the changed dependencies are reflected. This process means that the dependencies specified in package-lock.kdl won't necessarily be obeyed: In general, Orogene will do its best to preserve a dependency's previous position in the tree, but if a conflict happens, it may be moved or replaced.

In order to guarantee that Orogene only accepts tree modifications on oro add/oro remove, you can use the --locked option.

Specifier syntax

A package specifier in orogene is a string describing which package should be resolved. This string, in some cases (such as in the CLI) can include a name under which to install that dependency. Specifiers are used both through the add CLI commands, or in the value field in your dependency objects, next to the requested package names.

Alias

Syntax: <alias>@<specifier>

Alias specifiers work a lot like NPM specifiers, but are able to force a dependency to use a certain name in the node_modules tree.

When specifying an alias for an NPM registry package, you must use the npm: prefix before the NPM specifier itself. All other specifiers can be provided as-is.

Examples: underscore@npm:lodash, underscore@github:lodash/lodash

NPM

Syntax: <package-name>[@(<semver> | <tag>)]

NPM specifiers refer to "regular" NPM packages from a registry. The <package-name> can be any valid scoped or unscoped package name. If followed by an @, either a semver range or a dist-tag can be specified.

When used in package.json, the <package-name> must be omitted, and it is recommended that only semver ranges be used.

Examples: lodash, lodash@3, @axodotdev/oranda@1.2.3

Path

Syntax: <./relative/path> | <C:\absolute\path>

Path specifiers refer to the directory in the local machine where a package exists. They can either be relative (in which case they must be prefixed by ./ or .\), or absolute (in which case they must start with either / or a drive letter).

Examples: ./path/to/my/proj, C:\src\foo

Hosted Git

Syntax: <host>:<org>/<proj>[#(<rev> | semver:<semver>)]

Hosted git specifiers refer to packages on a handful of well-known hosted git platforms, and can be used as a shorthand. You may optionally provide either a git rev or a semver:-prefixed semver range following a # to resolve to a particular version, instead of the latest HEAD.

Supported platforms: github, gitlab, gist, bitbucket.

Examples: github:orogene/orogene, gist:foo/bar#deadcafe, gitlab:baz/quux#semver:^1.2

Git

Syntax: [git+]<git-url>[#(<rev> | semver:<semver>)]

Arbitrary git URLs can also be provided to Orogene. For git:// URLs, no prefix is necessary, but other URL types must include a git+ prefix. You may optionally provide either a git rev or a semver:-prefixed semver range following a # to resolve to a particular version, instead of the latest HEAD.

Examples: git://github.com/lodash/lodash, git+ssh://codeberg.org/foo/bar.git#semver:^1.2.3

Phantom Dependencies

"Phantom dependencies" refers to a phenomenon where dependencies that weren't directly declared in a package's package.json can be imported from those packages. It arises primarily from "hoisted" node_modules trees where, in order to reduce duplication, shared dependencies are moved as high up the tree as possible, exposing them to everyone else at that level or deeper.

By default, Orogene uses an "isolated" linking strategy to avoid this: All dependencies are first written to <root>/node_modules/.oro-store/<dep>-12345/node_modules/<dep>, then every dependency inside the store, along with the root package, gets their own individual node_modules that contains symlinks back into the store, only for dependencies directly declared in their package.json. This avoids the phantom dependencies problems and ensures that you don't accidentally forget to add a dependency to your package.json just because of the state of your dependency tree at any given point in time.

The catch with this isolated mode is that most NPM packages in the public registry have been written in a world where the NPM CLI defaults to "hoisted" installations, so a number of packages out in the wild depend on this behavior, purely by accident. The best fix for this is simply to patch the offending package and install an updated version.

When this is not possible, you can use --hoisted to force Orogene to apply dependencies in a classic, flattened-as-much-as-possible style. As with other options, this can be added to your oro.kdl as needed.