DrewML

An Old-Fashioned Technology Stack

In the world of modern front-end, the architecture of Magento 2's client-side would be considered outdated. This isn't super surprising, though: development work on Magento 2 first started in 2010. Quite a bit has changed on the web in the last 9 years!

To help put that in context, here are some random notes about the web of 2010 to help frame things:

• Steve Jobs declared war on Flash
• The term Responsive Web Design was coined
• RequireJS was still a year away from a 1.0 release
• AngularJS and KnockoutJS were first released
• Browserify (first node module bundler) was still a year away from being announced
• Internet Explorer 8 was the latest version of everyone's favorite browser
• Google Chrome was 2 years old and was only a desktop browser

It's not uncommon for web apps with a long shelf life to begin getting slower as more features are added: Magento is not an exception.

I recently had the opportunity, along with my co-workers Vitalii Zabaznov and Dmytro Poperechnyy, to work on improving the Magento platform's client-side performance by modernizing the way JavaScript assets are chunked and delivered to shoppers. To understand the impact of these improvements, we'll need pause and discuss the problems that exist today.

JavaScript Dominates

If you've spent any time attempting to improve client-side performance over the last few years, you might already know that JavaScript frequently dominates any measurements you look at. If this is news to you, that's totally ok! The surface area of the web is massive, and browsers are complex beasts: it's impossible to be an expert in everything.

Before proceeding, if you'd like to learn more about the negative effects JavaScript can have on web performance, I've found the following articles on the topic to be extremely approachable:

• The Cost of JavaScript in 2018
• The Cost of JavaScript in 2019

If you don't have time to read the linked articles, I'd like to call out one quote in particlar from the always helpful Addy Osmani:

Byte-for-byte, JavaScript is still the most expensive resource we send to mobile phones, because it can delay interactivity in large ways.

Although Magento is a pretty typical server-rendered PHP app (and not a Single Page Application), it loads a pretty massive amount of JavaScript by default. On a Product Details page with the sample data package installed, 1.4 MB (451 KB gzipped) of minified JavaScript is delivered to mobile devices. This is the single biggest bottleneck I've seen in any Magento stores I've traced.

The Waterfall Problem

Most of the JavaScript in Magento is authored in the Asynchronous Module Definition (AMD) format, and loaded via RequireJS. A single module looks roughly like this:

Writing modules in this format ensures that RequireJS can load modules both asynchronously and in the correct order. The benefits here are primarily:

• JavaScript loaded asynchronously does not block rendering
• Opportunities for race conditions when loading > 1 file decrease
• Code can be written in a modular fashion, leading to easier testing and long-term maintenance.

This modularity, though, can be devastating to performance. As an application grows and dependency chains become deeper, this leads to a waterfall effect when loading the files.

To illustrate the problem, consider a small app consisting of 5 AMD modules.

Assuming that Foo is the entry point for the application, this creates the following network waterfall:

Unfortunately, the browser cannot discover a dependency until it has both downloaded and executed the parent. In this example, the browser cannot begin downloading Bar and Bizz until Foo has completed.

To see this problem on a larger scale, here is an extremely zoomed and blurred out example of the waterfall effect when loading a stock Magento 2 homepage:

Luckily, because Magento stores are primarily server-rendered, the asynchronous loading of these JavaScript modules does not block rendering of the critical page content. However, the deferring of JavaScript execution can end up freezing the UI for significant periods of time when it otherwise looks ready to use.

Bundling - The Wrong Way

Magento has a built-in bundling feature to address the waterfall effect when loading JavaScript assets. Unfortunately, this feature trades one problem for several others, and in most cases the trade-off is not worth it.

Dead Code

The built-in bundling feature works by concatenating all JavaScript modules it finds into one or several large files. Because no consideration is given to where different code is used in the storefront, every visitor of the store has to download all the code for all pages.

To help put this in context, let's compare the page weight of 2 identical product pages in Magento 2.3.2, with and without bundling:

Blocking

The built-in bundling feature allows you to configure the number of separate bundles it generates. However, this feature does not make a difference, because all scripts are loaded in a blocking fashion in the head.

Deoptimization

The built-in bundling feature does not directly include JavaScript modules inline within a bundle. Instead, if wraps them in JavaScript strings and runs each module through eval at runtime. Hiding code from the browser until the last minute like this bypasses a significant number of performance optimizations present in modern browsers.

Paper Cuts Add Up

All these problems add up very quickly, even in a store with no modifications. As an example, I set up a store on Magento 2.3.2, and recorded the performance of loading the site on a mid-tier mobile device on an average 3G network, both with and without bundling. I think the results are telling:

Bundling - The Manual Way

The Magento DevDocs has an article that provides instructions to manually bundle a store with per-page bundles, by writing code to scrape the pages of a store in development or staging.

This process can yield good results, but the manual work involved tends to not scale for large stores or teams, and is extremely error prone. It also has the same blocking problem as the built-in bundling feature.

Just use webpack

I've addressed this a bit in the past.

Believe me when I say I'd much prefer to use an existing tool for this. In 2018 I invested a pretty significant amount of time working on a webpack plugin that attempted to shim some necessary functionality.

Progress was made, but at a certain point the cost to build and maintain the webpack plugin began to outweigh the cost of writing something that better fit Magento's architecture.

The only other way to move the Magento platform onto a modern bundler would require some significant breaking changes for both store owners and extension authors. Although this is tempting for the sake of performance, there is already a team working on a modern rebuild of the storefront. I just cannot justify large breaking changes when people are planning to start migrating their stores in the next few years.

For posterity's sake, here's an incomplete list of features that are not compatible with modern bundlers:

mage-init and x-magento-init

Similar to the idea of web components, Magento has a declarative mechanism that can be used to instantiate widgets on the front-end. These declarations can have a dependency on 1 or more JavaScript modules. These can be included on any page, at any time.

Inline require/define

Magento is a server-rendered application, so most markup is typically written in .phtml partials. It's common for developers to open a script tag and interface directly with the module registry via window.require and window.define. These can be included on any page, at any time.

"Mixins"

RequireJS was patched in the core of Magento with functionality that allows a developer to intercept requests for modules, and mutate or replace the result.

"Deps"

A Magento package can include a file named requirejs-config.js, which is merged with all other packages' RequireJS configurations. A deps key can be included, which signals that a file should be included and executed on all pages.

uiComponents

Magento provides (another) abstraction for reusable UI widgets. These have a special template property that points to an HTML file, which is loaded via RequireJS.

Introducing Baler

On July 25th of this year, I made my first commit to a project now known as Baler. The original idea of Baler was to see if I could build a drop-in replacement for the RequireJS Optimizer that knows all the intricasies of how JavaScript works within Magento.

I'm excited to say that these efforts were fruitful! Based on the progress so far, I'm confident we're headed in the right direction.

How Baler Works

Baler has 2 separate strategies for optimization

Core Bundle

For each Magento theme, Baler collects a list of "entry points," which are JavaScript modules that will be included on every page of the storefront.

Baler then follows each entry points' dependencies recursively, building up a directed graph of all dependencies that can be statically analyzed. The graph is then flattened, sorted, and minified into a single bundle.

Finally, Baler creates a new version of requirejs-config.js for a theme, with instructions that tell the RequireJS runtime to not fetch already-bundled modules 1 by 1. This is the trick that allows the core bundle to be loaded asynchronously, with no risks of race conditions for existing stores.

Dynamic Dependencies

Baler parses and analyzes all .phtml templates in a store, and extracts as much information as possible about inline dependencies for JavaScript modules and widgets. It then generates a mapping of template files to dependencies, and writes a JSON manifest to disk.

Not enough information is available during this stage to determine which templates are used together, so bundling of modules here is at high risk of shipping dead or duplicate code.

Instead, Baler takes advantage of HTTP/2 multiplexing and delivers these modules individually. The trick here is that a package (Magento_Baler) can inject preload tags into the head, effectively flattening the graph and eliminating the waterfall effect. This means you ship a few more bytes due to loss in compression (larger blobs of text compress better), but you make up those bytes by not shipping any modules not used on the current page.

The (automated) process works roughly like this:

1. Request comes in to Magento store
2. Magento's layout rules run, yielding a final layout
3. A list is collected of all .phtml files in the layout that will be rendered to the client
4. Using the manifest generated by baler during build, collect a list of all JavaScript dependencies needed for each loaded template
5. Add preload tags to the head of the document, to give the browser an early signal for all modules in the graph. Modules already in the core bundle are excluded

State of Baler

The "Core Bundle" functionality is already fully implemented in Baler. This means that it's already capable of delivering scripts used on all pages of a store asynchronously in the head.

Until the "Dynamic Dependencies" work has been completed, dynamic dependencies will continue to load in with the waterfall effect. However, this shouldn't stop you from giving it a shot today. All testing done so far shows improvements on most metrics compared to built-in bundling or no bundling.

Comparison

The following stats were collecting from a Magento 2.3.2 store, with sample data installed, visiting the "Echo Fit Compression Short" product record:

I've skipped including runtime loading/execution measurements in this post for now, to instead encourage you to start playing with Baler yourself!

If you're ready to get started, the alpha guide is waiting for you.

Summary

• The built-in bundling feature in Magento is bad for performance, and should be disabled
• Waterfall loading of modules isn't great
• Baler is a new tool to tackle both these problems
• Magento is pretty cool for letting me burn up so much time looking for a solution to this, even after multiple failed attempts