A faster JavaScript packager for Serverless applications.
The Serverless framework is a fantastic one-stop-shop for taking your code and packing up all the infrastructure around it to deploy it to the cloud. Unfortunately, for many JavaScript applications, some aspects of packaging are slow, hindering deployment speed and developer happiness.
With the serverless-jetpack plugin, many common, slow Serverless packaging scenarios can be dramatically sped up. All with a very easy, seamless integration into your existing Serverless projects.
First, install the plugin:
$ yarn add --dev serverless-jetpack
$ npm add --save-dev serverless-jetpackAdd to serverless.yml
plugins:
- serverless-jetpack... and you're off to faster packaging awesomeness! 🚀
The plugin has the following options:
mode: Eitheryarn(default) ornpm. The installation tool to use which must be already installed on your system. Note: If you are usingnpm,npm@5.7.0+is strongly recommended so that the plugin can usenpm cifor much faster installations.lockfile: Defaults toyarn.lockformode: yarnandpackage-lock.jsonformode: npm.- You can set it to a different relative location like:
lockfile: ../../yarn.lockfor monorepo projects wherein the lockfile exists outside of the package directory. - Setting
lockfile: nullwill skip using lockfiles entirely. This will be slower and more dangerous (since you can wind up with different dependencies than your root project). But it is available if your project does not use lockfiles.
- You can set it to a different relative location like:
stdio: Enable/disable shell output foryarn|npm installcommands. Defaults tofalse.
Which can be integrated into a more complex serverless.yml configuration like:
package:
# Any `include`, `exclude` logic is applied to the whole service, the same
# as built-in serverless packaging.
# include: ...
exclude:
- "*"
- "**/node_modules/aws-sdk/**" # included on Lambda.
- "!package.json"
plugins:
# Add the plugin here.
- serverless-jetpack
custom:
# Optional configuration options go here:
serverless-jetpack:
mode: npm # Default `yarn`
lockfile: ../../package-lock.json # Different location
stdio: true # Default `false`
functions:
base:
# ...
another:
# ...
package:
# These work just like built-in serverless packaging - added to the
# service-level exclude/include fields.
exclude:
- "**"
include:
- "src/**"
- "node_modules/**"
- "package.json"The Serverless framework can sometimes massively slow down when packaging applications with large amounts / disk ussage for devDependencies. Although the framework enables excludeDevDependencies during packaging by default, just ingesting all the files that are later excluded (via that setting and normal include|exclude globs) causes apparently enough disk I/O to make things potentially slow.
Observing that a very common use case for a Serverless framework is:
- A
package.jsonfile defining production and development dependencies. - A
yarn.lockfile if usingyarnor apackage-lock.jsonfile if usingnpmto lock down and speed up installations. - One or more JavaScript source file directories, typically something like
src.
The serverless-jetpack plugin leverages this use case and gains a potentially significant speedup by observing that manually pruning development dependencies (as Serverless does) can be much, much slower in practice than using honed, battle-tested tools like yarn and npm to install just the production dependencies from scratch -- by doing a fresh yarn|npm install in a temporary directory, copying over source files and zipping that all up!
Process-wise, the serverless-jetpack plugin uses the internal logic from Serverless packaging to detect when Serverless would actually do it's own packaging. Then, it inserts its different packaging steps and copies over the analogous zip file to where Serverless would have put it, and sets internal Serverless artifact field that then causes Serverless to skip all its normal packaging steps.
Jetpack does most of what Serverless does globbing-wise with include|exclude at the service or function level. Serverless does the following (more or less):
- Glob files from disk with a root
**(all files) and theincludepattern, following symlinks, and create a list of files. - Apply service + function
exclude, thenincludepatterns in order to decide what is included in the package zip file.
This is potentially slow if node_modules contains a lot of ultimately removed files, yielding a lot of completely wasted disk I/O time. Also, following symlinks is expensive, and for node_modules almost never useful.
Jetpack, by contrast does the following:
- Glob files from disk with a root
**(all files) and theincludepattern, except fornode_modules(never read) and without following symlinks to create a list of files. - Apply service + function
exclude, thenincludepatterns in order. - Separately
npm|yarn installproductionnode_modulesinto a dedicated dependencies build directory. Run the same glob logic andexclude+includematching over just the newnode_modules. - Then zip the files from the two separate matching operations.
This does have some other implications like:
- If your
include|excludelogic intends to glob indevDependencies, this won't work anymore. But, you're not really planning on deploying non-production dependencies are you? 😉
This plugin assumes that the directory from which you run serverless commands is where node_modules is installed and the only one in play. It's fine if you have things like a monorepo with nested packages that each have a serverless.yml and package.json as long as each one is an independent "root" of serverless commands.
Having additional node_modules installs in nested directory from a root is unlikely to work properly with this plugin.
It is a best practice to use lockfiles (yarn.lock or package-lock.json) generally, and specifically important for the approach this plugin takes because it does new yarn|npm installs into a temporary directory. Without lockfiles you may be packaging/deploying something different from what is in the root project. And, production installs with this plugin are much, much faster with a lockfile than without.
To this end, the plugin assumes that a lockfile is provided by default and you must explicitly set the option to lockfile: null to avoid having a lockfile copied over. When a lockfile is present then the strict (and fast!) yarn install --frozen-lockfile --production and npm ci --production commands are used to guarantee the packaged node_modules matches the relevant project modules. And, the installs will fail (by design) if the lockfile is out of date.
Many projects use features like yarn workspaces and/or lerna to have a large root project that then has many separate serverless functions/packages in separate directories. In cases like these, the relevant lock file may not be in something like packages/NAME/yarn.lock, but instead at the project root like yarn.lock.
In cases like these, simply set the lockfile option to relatively point to the appropriate lockfile (e.g., lockfile: ../../yarn.lock).
npm ci was introduced in version 5.7.0. Notwithstanding the general lockfile logic discussed above, if the plugin detects an npm version prior to 5.7.0, the non-locking, slower npm install --production command will be used instead.
The serverless framework only excludes the first match of serverless.{yml,yaml,json,js} in order. By contrast, Jetpack just glob excludes them all. We recommend using a glob include if your deploy logic depends on having something like serverless.js around.
The following is a simple, "on my machine" benchmark generated with yarn benchmark. It should not be taken to imply any real world timings, but more to express relative differences in speed using the serverless-jetpack versus the built-in baseline Serverless framework packaging logic.
When run with a lockfile (producing the fastest yarn|npm install), all of our scenarios have faster packaging with serverless-jetpack. In some cases, this means over a 6x speedup. The results also indicate that if your project is not using a lockfile, then built-in Serverless packaging may be faster.
As a quick guide to the results table:
Scenario: Contrived scenarios for the purpose of generating results.simple: Very small production and development dependencies.individually: Same dependencies assimple, but withindividuallypackaging.huge: Lots and lots of development dependencies.
Mode: Useyarnornpm?Lockfile: Use a lockfile (fastest) or omit?Type:jetpackis this plugin andbaselineis Serverless built-in packaging.Time: Elapsed build time in milliseconds.vs Base: Percentage difference ofserverless-jetpackvs. Serverless built-in. Negative values are faster, positive values are slower.
The rows that are bolded are the preferred configurations for serverless-jetpack, which is to say, configured with a lockfile and npm@5.7.0+ if using npm.
Machine information:
- os:
darwin 18.5.0 x64 - node:
v8.16.0 - yarn:
1.15.2 - npm:
6.4.1
Results:
| Scenario | Mode | Lockfile | Type | Time | vs Base |
|---|---|---|---|---|---|
| simple | yarn | true | jetpack | 5687 | -42.16 % |
| simple | yarn | true | baseline | 9832 | |
| simple | npm | true | jetpack | 6163 | -31.37 % |
| simple | npm | true | baseline | 8980 | |
| simple | yarn | false | jetpack | 7199 | -27.73 % |
| simple | yarn | false | baseline | 9961 | |
| simple | npm | false | jetpack | 17714 | 50.96 % |
| simple | npm | false | baseline | 11734 | |
| individually | yarn | true | jetpack | 8259 | -50.44 % |
| individually | yarn | true | baseline | 16665 | |
| individually | npm | true | jetpack | 9787 | -50.10 % |
| individually | npm | true | baseline | 19613 | |
| individually | yarn | false | jetpack | 9242 | -45.27 % |
| individually | yarn | false | baseline | 16888 | |
| individually | npm | false | jetpack | 17426 | -35.68 % |
| individually | npm | false | baseline | 27094 | |
| huge | yarn | true | jetpack | 13473 | -71.85 % |
| huge | yarn | true | baseline | 47864 | |
| huge | npm | true | jetpack | 9451 | -71.45 % |
| huge | npm | true | baseline | 33105 | |
| huge | yarn | false | jetpack | 15403 | -38.98 % |
| huge | yarn | false | baseline | 25243 | |
| huge | npm | false | jetpack | 25483 | -9.95 % |
| huge | npm | false | baseline | 28300 |