molsson's blog

Migrating from MySQL to PostgreSQL

Tue, 09 Jul 2019 00:00:00 +0000

I recently migrated an old node webapp from MySQL to PostgreSQL. The application is a little over 10k lines of code and does not use any ORM / SQL builder tool. To convert the schema and move the data I used a little Common Lisp tool called pgloader which can convert from various other databases like MySQL, sqlite or MSSQL.

The reasons¹ why I did this migration was:

PostgreSQL feels a bit more stable / long term (I don’t like that Oracle bought MySQL). This was the last side-project I had that was still using MySQL, so now I don’t have to deal with MySQL anymore.
- Porting this project to MongoDB would have been a lot more work and the license situation is not ideal there and also I don’t like the “schemaless” aspect of MongoDB.
MySQL’s timestamp datatype has the 2032 problem and a long range of various other similar little technical limitations.
PostgreSQL Ubuntu packages are quite well maintained. There are separate packages for 9.4, 9.5, 9.6, 10, 11, 12 available in parallel on 18.04

Just installing pgloader using “sudo apt install pgloader” didn’t work because it crashed and I found some GitHub issue that was fixed but the apt packaged version did not have the fix yet. Instead I tried to run the docker version, but it was quite painful to get that working. I had to reconfigure both mysql and postgres to accept connections from any interface rather than just localhost, then set a password for the postgres user (it doesn’t allow password logins by default) and add a user in MySQL that was allowed to connect from a remote IP. Finally, I had to use the docker host IP (visible on the docker0 interface if you run ip addr) both as the source and the destination. I ended up running something like:

docker run --rm --name pgloader dimitri/pgloader:latest pgloader \
    mysql://myuser:mysqlpasswo@172.17.0.1/sourceDbName \
    pgsql://postgres:passwordForPostgresUser@172.17.0.1/targetDbName

Later, when I went through a similar process for a staging environment I installed the latest version of pgloader from apt.postgresql.org instead and that was a lot easier.

Migrating the source code

The first step was to convert the Mysql SQL placeholders from ? to PostgreSQL $N placeholders:

-  const rows = await db.query('SELECT * FROM users WHERE user_id = ?', [userId])
+  const rows = await db.query('SELECT * FROM users WHERE user_id = $1', [userId])

Doing this manually, would be very tedious/boring so I wrote a program that edited my source code instead. babel is a great tool for JavaScript source-to-source transformation, but it doesn’t preserve source formatting when it emits code. However, there is a great utility on npm called @codemod/cli that parses and emits code using recast instead of the babels default parser. I implemented my SQL migration utility as a babel plugin and I used the excellent JavaScript AST Explorer to prototype the first version. It was possible to implement the entire plugin in just 45 lines of code:

/*
 Tiny babel plugin that helps migrate node Mysql SQL to Postgres SQL.
 Run via:
 npx -p @codemod/cli codemod --plugin ./scripts/mysql-to-postgres.js src/server/foo.js
 */
const needsMigration = (sql) => {
    const ustr = sql.toUpperCase();
    return (
        ustr.includes("?") &&
        (ustr.includes("SELECT") ||
            ustr.includes("INSERT") ||
            ustr.includes("UPDATE") ||
            ustr.includes("WHERE") ||
            ustr.includes("DELETE"))
    );
};
const migrateSql = (sql) => {
    let i = 0;
    return sql.replace(/\?/g, (_) => `$${++i}`);
};

module.exports = function (babel) {
    const { types: t } = babel;
    return {
        name: "mysql-to-postgres-sql-converter",
        visitor: {
            StringLiteral(path) {
                if (needsMigration(path.node.value)) {
                    path.replaceWith(
                        t.StringLiteral(migrateSql(path.node.value)),
                    );
                }
            },
            TemplateElement(path) {
                if (needsMigration(path.node.value.cooked)) {
                    path.replaceWith(
                        t.templateElement(
                            {
                                raw: migrateSql(path.node.value.raw),
                                cooked: migrateSql(path.node.value.cooked),
                            },
                            path.node.tail,
                        ),
                    );
                }
            },
        },
    };
};

This fixed all my SQL and preserved all source formatting but unfortunately it also added a few semicolons. I was able to remove these automatically using eslint though:

eslint --parser babel-eslint --parser-options=ecmaVersion:6 \
       --parser-options=sourceType:module --no-eslintrc --no-ignore \
       --fix --rule 'quotes: [2, single]' src/server/foo.js

After this I did a few manual fixups like (some fixes can be done using a tool like https://aarrange.com/mysql-to-postgresql):

Changed all my timestamp datatypes to timestamp(3) with time zone. By default postgres has microsecond precision but that causes problems if you use node because the JavaScript Date can only represent milliseconds. If you really need microsecond precision you can use a custom type parser (see note at the bottom of this page).
Adding RETURNING ... on INSERT statements with auto incrementing ids.
Change a few INSERT ... ON DUPLICATE KEY UPDATE into INSERT ... ON CONFLICT ....
Fixed all references to the MySQL specific field .affectedRows.
Fixing references to camel case column names. pgloader renames camel case columns to all lowercase since this is the convention on postgresql (you can enable case sensitive column names in postgres but I thought it was better to stay with postgresql default settings).
Fixed some rand(someSeed) calls to transactions with SET LOCAL SEED = someSeed.
Ported by web session handling from express-mysql-session to connect-pg-simple.
Rewrote the setup / deploy scripts to install postgres and create a new database + schema.
Created a new blank db-migrate seed.
Making sure that all E2E tests were passing again. I have a very good test suite for this project, and it was quite amusing to see all the, sometimes quite unexpected, places that broke when I changed the database.

After the migration was done, I noted that:

PostgreSQL ended up taking just 94MB of RAM memory (RSS) versus the 194MB that MySQL had required for the exact same data / workload (which is great because this project is deployed on a tiny Digital Ocean droplet with 1GB RAM).
I learned that PostgreSQL doesn’t support logical column ordering yet, which kind of sucks.
PostgreSQL has a TODO page that lists things that are not implemented yet.
pg_dump emits SQL to lock down the search path to protect against CVE-2018-1058 and this ended up causing lots of problems when I was setting up db-migrate package.

1: Most people seem to prefer PostgreSQL, even though there are certainly examples of the opposite for certain very specific situations.

Introduction to Elasticsearch

Mon, 08 Apr 2019 00:00:00 +0000

Elasticsearch (ES) is a distributed open-source search and analytics engine built on the text search library Apache Lucene. Given enough machines, an ES cluster can typically execute search and aggregation queries across very large datasets. While ES is great if you want traditional full text search over a set of text documents (similar to Google search), it is also very useful for searching and aggregating over large sets of more structured or semi-structured datasets like system logs, network traffic data and collected system metrics.

The very first version of Elasticsearch (0.4.0) was written by Shay Banon (kimchy) who is the current CEO of Elastic. The very first version of ES was published on Github Feburary 8th 2010. Shay had previously (2006-2010) developed another search related open source project called Compass¹ which was a high level API wrapping Lucene that also added a few features. After 209 days, the first PR by another developer was merged into ES, by that time kimchy had managed to land an incredible 1145 additional commits (5.5 commits per calendar day). Clearly, Shay knew very well both what he wanted to build and how it should be built. On 13th Feb 2012, Shay created the company “Elasticsearch BV” together with Steven Schuurman (who became the initial CEO), Uri Boness and Simon Willnauer.

Steven Schuurman had previous co-founded various other companies. Before Elastic he was the CEO of Orange11, a software company that amongst other things developed enterprise search solutions based on Solr / Lucene. Uri Boness has been working as “Software Architecture / Head of Search” at Orange11 and Simon Willnauer was (and still is) a core committer in the Apache Lucene project.

Two years later, on the 12th Feburary 2014, Elasticsearch 1.0 was released. Shay Banon eventually took over the CEO role in early 2017, even though he was still committing code up to mid-2015. In fact, a quick “git log” in the ES repo reveals that the developer with the most commits is still, to this day, Shaw Banon. Here is a graph showing “how many percent of ES commits to date that was authored by kimchy” :

Meanwhile, Jordan Sissel had been working on Logstash (an open source ruby/java based ingestion tool that could process and stash log data into multiple backends). Logstash predated Elasticsearch but using Logstash with Elasticsearch became very common and Jordan joined Elastic in 2013². Similarly, Kibana was an open source project for visualizing and working with Elasticsearch data. It started outside of Elastic but Rashid Khan, who worked for Elastic, quickly became the lead developer. Because Elastic grew out of various open source projects, it also grew into a very distributed company with employees in over 35 countries and quite a few employees working remote. Elastic NV eventually filed for an IPO in June 2018 and was listed on the New York Stock Exchange in October 2018. The IPO was at $36 per stock but they closed at $70 after the first trading day. The ESTC market cap is 6.11B as of today, and at the time of the IPO, Schuurman owned a 19 percent stake and Banon owned 12 percent.

The Elastic stack

ES is often used together with Logstash and Kibana (forming the ELK-stack). Logstash is a tool for processing data from log files and feeding them into ES (it has plugins for inputs, transforms and outputs so it’s essentially an ETL tool), and Kibana is a web UI that allows you to query ES using search queries in Lucene syntax. Kibana also lets you aggregate data and render it into various charts and visualizations. It is implemented in typescript with a node backend and React for the frontend. The Elastic stack also includes a set of programs, referred to as “Beats”, that feed various other kinds of data into ES (sometimes via Logstash). Examples include FileBeat (feeds file data like logs into ES), MetricBeat (collects system metrics like CPU/network/memory usage and feeds it into ES) and PacketBeat (records network traffic and feeds it into ES).

This chart shows the number of unique git authors seen per 90 days for the major Elastic.co repositories:

The above chart includes some open source “one-off patching”, and if you instead count the number of unique git authors that had 5 or more commits in each 90 day window, you get this chart instead (I suspect this latter metric is better correlated with the actual number of full time developers working on a project).

Example: How the Elastic stack can work together to process logs

When collecting system logs, a common setup is to run FileBeat on each machine and have it send data to Logstash. Logstash will then parse the raw log files into sets of field values to make them more search friendly (this is referred to as “grokking”). For example, if your log lines have the following format:

2019-02-17 00:05:02,061 INFO FoobarComponent  Connecting to service at foo:1234
2019-02-17 00:05:03,038 DEBUG FoobarComponent  Loaded 60 objects.
2019-02-17 00:05:04,012 DEBUG CoreComponent  Finished initialization.

You can use grokking to indicate that the first two “words” are a timestamp, the third word is a log level, the fourth word is a component name and the rest is the “log message”. You can define custom grokking using regexp patterns etc, or you can rely on pre-defined grok patterns that ships with Logstash (it knows how to parse common log file types like Apache access/error logs etc). Once the data is sent to ES the log lines will have been translated to something like this instead:

{
    "timestamp": "2019-02-17 00:05:02,061",
    "severity": "INFO",
    "component": "FoobarComponent",
    "message": "Connecting to service at foo:1234",
}

In the end, this makes it possible to find this data in Kibana (or programmatically via ES) using a query like severity:INFO AND message:connecting.

Before FileBeat existed a lot of people ran Logstash directly on each node but it’s written in Ruby + Java so it typically consumes quite a lot of memory compared to FileBeat (which is implemented using the Go programming language). Also FileBeat is more lightweight because, unlike Logstash, it only forwards data; it doesn’t know how to parse hundreds of different log formats etc. Another nice benefit is that the network protocol used between FileBeat and Logstash has a notion of back-pressure so if the Logstash server gets overloaded it can tell FileBeat to slow down a bit temporarily and spool the data. Searching logs is of course just one use case for ES among many many others.

Why non “full text search” queries are not good enough

Before adding ES to your infrastructure it’s important to understand what problem it solves and why. Suppose you want to implement full text search over a SQL database of “products”, where each product has a .title, .description and set of comments (stored in a different table). It’s not really possible to implement search using a crude solution like:

...
   FROM products p
   LEFT JOIN comments c ON (p.product_id = c.product_id)
   WHERE
        p.title LIKE '%searchstring%' OR
        p.description LIKE '%searchstring%' OR
        c.comment_text LIKE '%searchstring%'

There are many reasons why the above query won’t work well. The database cannot build an index for this so this type of query will be slow unless the table is really small. If the user searches for word1 word2 they will get no results even if a product contains word2 word1 in the description. If the user searches for satisfies, documents containing satisfy will not be returned. Each item will either match the condition or it will not, the database will not rank the search results. For example, if “productA” has the search term both in its title and in its description but “productB” only had the search term in its description, then it would be reasonable to expect that “productA” would be returned as the top search result but the SQL query won’t ensure that.

In relational databases you should normalize the data, splitting it into many tables and connecting them with foreign keys (this helps ensure that there is one source of truth and that the data is consistent). However, when you implement a search feature you typically want to search in several of these fields / columns so you are often forced to JOIN the data which can be expensive. If a row points at another row via a foreign key, those rows needs to live on the same machine / shard for the JOIN to be efficient; so the more such dependencies you have within the data, the more difficult it becomes to do horizontal scaling / efficient automatic sharding.

Wait, what about the built-in full text search in my RDBMS ?

If you dip into vendor specific SQL, most RDBMS have some sort of “full text index” that you can create for a text column to mitigate the matching issues above.

For example, PostgreSQL actually offers two different kinds of full text search indices. GIN based full text indexes are very fast for searching but slower to build and take up more space (these are based on an “inverted index” just like ES). GIST based indexes are slower for searching (but still pretty fast) while providing faster inserts / updates. Creating and searching using a PostgreSQL GIN index is quite straightforward:

ALTER TABLE products ADD "product_vectors" tsvector;
CREATE INDEX idx_fts_vec ON products USING gin(product_vectors);
INSERT INTO products
    (id, title, description, product_vectors)
VALUES (
    42,
    'Nikon D5',
    'The Nikon D5 is a full frame professional DSLR camera',
    (
        to_tsvector('Nikon D5') ||
        to_tsvector('The Nikon D5 is a full frame professional DSLR camera') ||
        to_tsvector('wow super nice camera') /* <-- text from comments */
    )
);
/* The actual search is done using: */
SELECT id FROM products WHERE product_vectors @@ to_tsquery('searchword');

Now, if you already depend on the SQL database, then using the built-in full text search saves you having to install, integrate, upgrade and maintain separate software for handling search. Another benefit of using the built-in full text search is that the database can automatically keep the full text index up-to-date, just like it does with a regular database index. With Elasticsearch, the application has to submit a re-indexing request to ES whenever the SQL data changes (so there will typically be a short period of time where the data is out of sync). On the other hand, rebuilding a GIN-index in PostgreSQL for example, can be quite expensive and it really depends on your use case whether you want this to happen immediately or not.

If your dataset is small enough, or if search is not a primary focus for your users; you might might get away with using built-in full text indices. For example, Stackoverflow used³ this approach for the first few years, before they switched to Elasticsearch.

Built-in full text search works until it doesn’t

One of the main arguments against built-in full text indices is that the RDBMS typically is the least scalable part of your infrastructure, so anything that can move a significant fraction of the load elsewhere is valuable. Elasticsearch also offers much better fine-tuning / customization of the full text indexing process. This gives you more detailed control over which search terms should match which documents (e.g. the matching of “satisfy” vs “satisfies” in the above example).

Another key benefit is that even if the data is normalized in the SQL server, you can index it in denormalized form in Elasticsearch. From the example above, above we can create a single “document” in ES that contains the product title, description and all the comments for that product. In practice, this means that with ES you get horizontal scaling that actually works.

In general, you have to think carefully about:

how much space are you willing to allocate for the index
how fast must the search queries execute
how important is horizontal scaling
how fine-tuned control over indexing / search matching do you need
how quick must the re-indexing be

Competitors to Elasticsearch

Even though Elasticsearch is extremely popular, there is of course other software you can use for search. Just to give a super quick overview; other search engine solutions include the open-source project Apache Solr and the outdated / discontinued Sphinx project as well as the closed source products Splunk (on prem or SaaS) and Algolia (SaaS only). Many cloud providers also have offerings in this space, e.g. Amazon CloudSearch. AWS also operates a hosted Elasticsearch service which runs a forked version of ES (it’s not 100% API compatible but most of the commonly used APIs are available). There are also a few up-and-coming open source projects that look interesting but that cannot yet be compared to Elasticsearch, for example tantivy (search library similar to Lucene, but implemented in rust) and Toshi (search engine implemented in rust, uses tantivy). Two other ones are the search library bleve (implemented in golang, named after this) and blast which is a server using Bleve. Also note that the full-text search in Couchbase is powered by Bleve. Another nice full text solution (in rust) is Sonic. A few people have tried to implement Lucene index-compatible search libraries in non-Java languages but none of them took off. Xapian is a fairly mature search engine library (like Lucene) and Xapiand is a server built around it.

Another (larger) open source project that should be mentioned is Vespa, which was open sourced by Yahoo in September 2017. Vespa has a page in their documentation that compares it to Elasticsearch. Vespa was not used for the main Yahoo search engine but rather it was powering the alltheweb.com search engine that Yahoo acquired in 2003 from Overture which had in turn acquired it when it bought parts of the Norwegian company FAST (other parts of this company was acquired by Microsoft). FAST itself had its origin in the old “FTP Search” website operated by Tor Egge at NTNU. Finally, there is also manticore search, which is an open source project founded by some of the core members from Sphinx (and it started as a fork of Sphinx).

To visualize the size of these projects, here is the number git commits over time (compared to ES). One thing to note is that Lucene and Solr lives in the same git repository (and ES uses Lucene). Also, note that the first commit in the Vespa repository is a 2.5 MLOC behemoth diff that represents the squashed history up to that point (i.e. this codebase is literally decades old).

I also plotted the number of commits pushed to each project per 180 day window (this says nothing about the quality or size of the commits ofc). It shows that Vespa has a higher commit rate than ES right now:

This final plot shows the number of git authors with 5 or more commits per 90 days (I use this as a proxy metric for counting the number of “full time developers” working on a project). For example, the Xapiand project have essentially 3 developers working on it:

How Elasticsearch is licensed

In ES 6.2 and earlier, the Elasticsearch Github repository contained only Elasticsearch and Elasticsearch was and is open source. On top of that, there has always been a set of commercial plugins available as a separate download called X-Pack. Even, critically important functionality like authentication and TLS was part of X-Pack, which also contained functionality for alerting, reporting and some machine learning features. For Elasticsearch 6.3, Elastic decided to add all the X-Pack code into the main Elasticsearch git repository but the new code was still not published under an open source license like the rest of Elasticsearch. Shay Banon described the rationale for that change here.

There are various projects that tries to offer open source replacements for the X-Pack plugins. For example, SearchGuard is an open source authentication plugin for Elasticsearch, even though they eventually launched a paid version of their open source plugin. There is an overview of various X-Pack alternatives available here.

AWS offers a hosted Elasticsearch service which runs a slightly modified version of Elasticsearch (which is API compatible with the open source version of ES for the most common API calls). This service offers authentication for example, but it does not have X-Pack installed. This service competes with the hosted cloud service that Elastic themselves maintain where you get ES instances with X-Pack preinstalled.

In March 2019, AWS launched Open Distro for Elasticsearch which is essentially a set of open source ES plugins that AWS uses to provide security, alerting, SQL support and performance analysis tools.

Example of what Elasticsearch offers: Search

Elasticsearch offers synonymization, handling of typos, match highlighting, auto completion and it can compute a relevance score for each search result indicating how well that particular item matched the search query. In particular:

Synonymization allows you to configure your indexing so that a document that mentions a “courageous knight” is returned whenever a user searches for “brave knight” and so on, based on custom or predefined lists of synonyms.
Typo handling ensures that a document that mentions something “outrageous” is found when a user searches for “outragous” (or vice versa). ES also comes with a “Phrase Suggester” that can be used to implement “Did you mean XYZ” suggestions to the end user.
Match highlighting gives you a machine readable map of which particular words matched which search term (so that you can make them bold in the search results or similar).
Auto complete can be implemented using ES in-memory “completion suggesters”.
Relevance scoring means that documents that match the search query very well will be given a higher score and will be returned up top among the search results.

It’s not a good idea to rank documents based only on the number of search terms they contain, because suppose you had a database of recipes and two of them were titled How to make the best pasta ever and Tasty Fudge. If a user searches for how to make the tastiest fudge ever and you only ranked results based on the number of search terms that matched, then the pasta recipe would get the highest ranking because 5 words/terms are matching, whereas with the fudge recipe there is only one match (or two if you have enabled stemming so that tasty/tastiest count as a match). If the database has many recipies with titles that begin with How to make then matches for the words how, to and make are not going to be very useful for understanding which document(s) the user is really looking for.

The problem is that matches for common words like “the” and “to” should not increase document ranking as much as a match for the less common word “fudge”. This idea is formalized in an algorithm called TF/IDF which stands for Term Frequency / Inverse Document Frequency. This algorithm was the default algorithm for relevance ranking in ES for many years. In practice though, extremely common words like “the” was often not even indexed in the first place (such words are referred to as stopwords) but the general idea holds; that matches for less common terms cause higher relevance scores compared to hits for words that are more common in the overall index. Other than term frequence and inverse document frequency, Elasticsearch also computes something called the “field-length norm” which means that if two documents both have a single match, but one of the documents have the match in a shorter field, then that document gets slightly higher relevance score. You can also customize relevance scores heavily, for example you might want to let a “product rating” field weigh into the ranking.

The current version of Elasticsearch uses another algorithm called Okapi BM25 for computing relevance scores. This algorithm is quite similar to TF/IDF but handles stop words more gracefully.

Example of what Elasticsearch offers: Cardinality Estimation

When a SQL database executes a query like SELECT COUNT(DISTINCT(type)) FROM products it will insert all .type values into a hash set and then count the number of elements in the end. If the dataset had N values and M unique values, then this requires O(M) memory and takes O(N) time (since each hash set insert takes O(1) amortized). For high cardinality sets, M might exhaust available RAM so that the database needs to sort it on disk there which is very slow. Also, if you split the data into two parts and determine the number of unique .type values in each part, that information cannot be combined to deduce the number of unique .type values in the entire dataset. If the set of unique values is so large that it needs to span across a large cluster of machines, then the above algorithm is simply impracticable. This problem is called the count-distinct problem.

There are radically more efficient algorithms available, given that you are willing to settle for approximately correct counts. For example, you could hash all your items into integers and set them in a bitset (this has the same time complexity but it has orders of magnitude better space complexity but of course each hash collision would cause the final count to be slightly off); this is called “linear counting”. An even better solution was published in 1984, namely the Flajolet–Martin algorithm. Eventually that work was refined into “LogLog Counting”, which in turn evolved into HyperLogLog (HLL, the most commonly used algorithm for this problem today) which was first published in 2007 by Philippe Flajolet et al. In general, HLL is a cornerstone of “big data” and variants of this algorithm is used in Redis, Spark, Redshift, BigQuery etc. The HLL commands in Redis are prefixed with “PF” in honor of Philippe Flajolet.

If you count the number of distinct words in Shakespeare’s works using a hash set, linear counting and HLL you get these results:

Counting Algorithm	Bytes Used	Count	Error
HashSet	10447016	67801	0%
Linear Counting	3384	67080	1%
HyperLogLog (HLL)	512	70002	3%

The cardinality aggregation feature in Elasticsearch uses a slightly improved HLL variant called HyperLogLog++, and this makes it possible to count the number of distinct elements in a field on a per shard basis and then use the results of those computations to provide an approximate total. This algorithm is explained more in-depth by Adrien Grand (ES/Lucene committer) in this video.

Nodes, Shards, Indices and Routing

An Elasticsearch cluster consists of a set of nodes (typically a node is a single machine running ES). Elasticsearch stores documents (essentially blobs of JSON) into indices; an ES index is conceptually similar to SQL table. Each document is marked with a built-in field called “_id” which uniquely identifies that document. An ES cluster can have multiple indices, for example one for “products” and one for “customers”. Each index is typically partitioned into primary shards so that each document is stored in exactly one primary shard. For example, you can load a document with ID UMNkYH from the products index using:

$ curl -s 'es-hostname:9200/products/_search?q=_id:UMNkYH'

When ES receives a “find by document id” request like this, it will hash the document ID to determine which primary shard the document is stored in (this process is called routing). Similarly, when a set of new documents are inserted into the index, the uniformity of the hash function ensures that they are added evenly across all shards. The number of shards is set when the index is created and can not be changed, unless you create a new index and reinsert all your data into it (which often ends up being a quite expensive operation). When ES receives a regular search request like “find all products with a price below 100” or “find all documents that contain the words foo and bar”, it does not know which shards contain the requested documents so it will instead broadcast the query to all shards and then merge the results from each shard.

How to execute basic queries

There are two formats that can be used when sending queries to ES. The curl command in the previous section runs a URI search and passes the search criteria via the q= parameter. The other format is request body search where the search criteria is passed as JSON in the request body instead. If you rewrite the “find by id” query from above as a request body query you get:

$ curl -s 'es-hostname:9200/products/_search' -H "application/json" \
       --data-raw '{"query":{"term":{"_id":"UMNkYH"}}}'

ES returns JSON and you can get it pretty-printed by adding pretty as a URL param to the request:

$ curl -s 'es-hostname:9200/products/_search?q=price:42&pretty'
$ curl -s 'es-hostname:9200/products/_search?pretty' -H "application/json" \
       --data-raw '{"query":{"term":{"_id":"UMNkYH"}}}'

Another (better) way to pretty print the JSON is to pipe it to jq or fx.

The syntax used within the q= is very similar to the Lucene syntax you enter in Kibana. Although, you have to keep URL escaping in mind so replace space with + etc. For example, if you want to find all products with category set to “camera” and a price of 1999, use:

$ curl -s 'es-hostname:9200/products/_search?q=price:1999+AND+category:camera' | \
    jq -C . | less -RSFX

When you need to query datetime fields with this syntax ES offers some basic “date math” support that you can use within square brackets. However, note that curl processes square brackets as shell globbing unless you pass the -g parameter to curl. So if you need to find all products postings with category “camera” that was modified in the last 3 hours, use:

$ curl -s -g 'es-hostname:9200/products/_search?q=lastmodified:[now-3h+TO+now]' | \
    jq -r .hits.total

In ES versions before 5.x, a lot of people installed a plugin called kopf. This plugin allowed you to get an overview of nodes, shards, indices and it also had a “rest” tab where you could submit JSON queries. If the kopf plugin is installed, you can open it by navigating to es-hostname:9200/_plugin/kopf. In ES 5.x and later you can use cerebo which is essentially kopf but as a standalone program (written by the same author).

Another really nice way to submit JSON queries is via the “devtools console” in Kibana.

Examples of a few other ES API calls

# Show cluster name, health and node/shard count
$ curl -s 'es-hostname:9200/_cluster/health' | \
    jq -r '{ cluster_name, status, number_of_nodes, active_primary_shards }'

# List available indices
$ curl -s 'es-hostname:9200/_cat/indices'

# List available indices, with column captions
$ curl -s 'es-hostname:9200/_cat/indices?v'

# List available indices, storage sizes in bytes instead of human readable units
$ curl -s 'es-hostname:9200/_cat/indices?v&bytes=b'

# List nodes with current CPU/mem usage
$ curl -s 'es-hostname:9200/_cat/nodes?v'

Replicas and cluster health

Each primary shard is stored on a single node, but each node can manage several shards. If more nodes are added, the ES cluster automatically rebalances itself by moving a few shards over to the newly added nodes. Each index also has a replica count defined which ensures that for each primary shard there is also N identical “replica shards” that are kept up to date at all times. For example, if you set the replica count to 1, ES writes all data to both the primary shard and the replica and it constantly works to ensure that the replica lives on a different node from the primary. If a machine malfunctions ES can promote a replica to primary or spawn a new replica. All incoming writes are executed on both the primary and the replica shard, but reads can be served directly from a replica shard so other than failover they also improve read performance.

If ES is unable to maintain the configured replica count, the index health status changes from green to yellow (and to red if data is lost). This also means that the overall cluster health status changes. You can see the cluster health status using the _cluster API call from the previous section. Tools like kopf/cerebro make is quite clear that the cluster is unhealthy, for example cerebro renders a the line underneath the main menu using the cluster health status color. When the cluster switches from unhealthy to healthy this line goes from yellow to green. When it’s yellow you typically also see some error message saying that you have for example “15 unassigned shards”. The default replica count is 1 so whenever you add data to ES on localhost for development purposes, the cluster health immediately becomes unhealthy/yellow since the localhost “cluster” only has a single node so ES cannot store the replica on a different node from the primary. You can fix this by selecting the dropdown menu on the index which has “unassigned shards”, then select “index settings” and set the replica count to 0 (NOTE: only do this for localhost dev instances where you don’t want replication).

Rollover Indices

When you store immutable chronological data like logs (and also recorded metrics), it’s quite common to write the data into an index where the name contains a timestamp, for example logstash writes to logstash-yyyy.mm.dd by default. If you want to delete the oldest data periodically you can do this by deleting an entire index at once (which is much faster and more resource efficient than trying to delete documents inside a single index). When you do this, you would typically define an index template in ES that specifies the settings that should be used whenever a new logstash-* index is created (i.e. the number of shards/replicas as well as the field mappings and so on).

People have been using setups like that for years, but there are some drawbacks with it; for example since all write operations target the active index (and for use cases like this often a majority of the read queries target the active index as well) you often want more nodes backing the active index compared to older indices. Also, it’s quite common that not “all days are the same” (e.g. you might be process log data for a stock exchange that happens to be closed during weekends). To mitigate these issues you can use the rollover API instead, which allows you to specify conditions for new index creation on a more granular basis and since the data is immutable you can automatically force merge older indices and move them off hot nodes (assuming the cluster is setup with hot warm architecture).

Memory for ES

Just like other large JVM applications, ES is sensitive to the Java heap limit for compressed ordinary object pointers (oops). If you give a Java process 31gb or less memory it can use 32-bit pointers and multiply them to by 8 to find the real address. If you give it 33gb, the JVM will start in 64-bit pointer mode instead and since ~20% of any Java heap is object pointers, you will have less actual memory availble compared to running with the 31gb limit. Running with limited Java heap is not a problem since you can use more nodes instead. There is also an even more efficient mode called “zero based compressed oops” and to use this mode you need ~26GB or less Java heap. Also, as with other Java applications; resizing the heap at runtime can be quite slow so it’s a good idea to run with identical min/max values, e.g. -Xms31g -Xmx31g. By default, ES starts with 1GB heap which is pretty much always wrong for production clusters, you can configure your JVM heap options in the file config/jvm.options. You can also configure which JVM GC to use in this file, notably the Java 8 G1GC has less stop-the-world GC pauses compared to the default CMS collector.

Because Lucene uses quite a lot of disk-backed data structures, so it’s important to leave memory (ideally at least 50% of physical RAM) for operating system file caches. If you use a smaller Java heap for ES you get shorter GC pauses and more memory for Lucene’s file system caches. If you won’t be aggregating on analyzed string fields you can make the ES Java heap even smaller than 50%. You should also disable swapping, either completely using sudo swapoff -a, and/or by setting bootstrap.memory_lock: true in elasticsearch.yml. Even if you allocate plenty of memory for file system caches, it’s often a good idea to use fast SSD disks (and if you do, double check cat /sys/block/{DEVICE-NAME}/queue/scheduler so you’re not accidentally using cfq which is not suitable for SSDs). You can setup RAID 0 if you want (but skip other RAID variants since failover is handled by replicas).

The Inverted Index vs Doc Values

The main data structure backing full text analyzed fields in ES is an inverted index that maps terms to a set of postings (document IDs of documents containing that term). For example:

    "berry"    -> { freq: 4, postings: [2, 42, 55, 88]            }
    "fish"     -> { freq: 6, postings: [1, 42, 55, 88, 91, 92]    }
    "pasta"    -> { freq: 5, postings: [4, 5, 55, 67, 111]        }
    "red"      -> { freq: 3, postings: [6, 7, 99]                 }
    "zucchini" -> { freq: 1, postings: [121]                      }

Because the terms in the left column are sorted alphabetically, it’s possible to find search terms quickly in that list and hence also the associated document IDs. This is a good data structure for answering queries like “find documents matching ‘red berries’” because ES can just normalize each search word using the configured analysis algorithm (so that “berries” becomes “berry” for example), and then lookup the matching documents. ES tracks various other information like “term frequency” (number of documents in the index that contains the relevant term) and “which character offset the term appeared at inside the field” etc. Also on a lower level, Lucene considers the same string in different fields to be different terms (i.e. a Lucene term is essentially tuple of (FIELD_NAME, SOME_WORD)).

For use cases where you instead want to sort, aggregate or use field values from scripts, the inverted index is not a suitable data structure, but ES can also store the data in another way, namely as doc values. They are essentially an inverted inverted index because given a field name and a document ID you can use the doc values data structure to efficiently retrieve the value of that field in the given document. Doc values are supported for all fields that are not analyzed strings (i.e. all field data types except “text”), and they are enabled by default whenever they are supported. If you are absolutely sure that you will never need to aggregate or sort based on a field you can set doc_values: false in the mapping and save a small amount of disk space and gain some indexing speed. Note that doc values are stored in a disk based data structures and read via the file system cache (just like the Lucene indices) so they don’t bloat the ES Java heap.

For example, imagine we have an ES index with recipes and each recipe has fields for “title”, “cooking_instructions”, “minutes_required” as well as a “recipe_type” field that indicates whether the recipe is a “entrée”, “main course” or “dessert”. It makes sense to store “title” and “cooking_instructions” as text fields (analyzed) so we can do full text search over them, but we will store the “minutes_required” field as an integer so that we can sort the recipes by how long they will take to prepare. For “recipe_type” we will use the datatype “keyword” which is a non-analyzed string type, ensuring that we’ll have doc values available so we can use this field for aggregation (e.g. calculate the number of recipes of each “recipe_type”).

However, we might also want to generate a list of recipes sorted by their “title” but since “title” is a text field we cannot use it for sorting. You could overcome this by indexing the recipe title into two different fields, one with datatype “text” and another with datatype “keyword”) but this would cause some storage overhead in the _source metafield. A better solution is to declare a single multifield mapping that will store the data in both formats at once, so instead of "title": { "type": "text" }, you would use a mapping like:

    "title": {
        "type": "text",
        "fields": {
            "keyword": {
                "type": "keyword"
            }
        }
    },

Multifields like the above are also used by logstash by default which is why you often see both “foobar” and “foobar.raw” when you query data in Kibana.

FieldData

As outlined in the previous section, you should always use doc values for all fields that you will use for aggregation and sorting. Strictly speaking though, it is possible to execute an aggregation query for a field that contains analyzed text, but often it is not a good idea to do this because A) it can lead to memory related performance problems, and B) the result of such aggregation is quite often not what the caller expected.

The memory issues happens because analyzed text fields are stored as an inverted tree (see earlier section) but aggregation would need a data structure similar to doc values to run efficiently. So, when ES is asked to aggregate over a field that contains analyzed text, it will (at query time!) invert the inverted tree and then also cache the inverted inverted tree permanently on the JVM heap. This in-memory data structure is called fielddata. Unlike memory used by Lucene via the file cache, this fielddata cannot be reclaimed by the OS if the machine runs low on memory.

It’s sufficient for a single “one off” aggregation on analyzed text to generate and permanently keep this fielddata in memory, after which Lucene will have less memory to work with and every single GC will take longer to process. Logstash tends to use multifields so you get both “component” and “component.raw” (where the latter is using doc values) and a single innocent pie chart aggregation on “component” could, in theory at least, permanently impair cluster performance. In reality, this particular case is blocked since the analyzed text field “component” would likely be marked with "fielddata" : { "format" : "disabled"} which means that you’d get an error in Kibana if you tried to generate such a pie chart of death. The right way is to aggregate based on “component.raw” instead.

You can see how much fielddata you have allocated for each node, by running:

$ curl -s 'es-hostname:9200/_cat/fielddata?v&fields=*'

It is possibe to clear the fielddata cache using curl -s 'localhost:9200/_cache/clear?fielddata=true' but this will of course cause performance issues if your application relies on this data.

When it comes to B) above (why aggregation on analyzed text often doesn’t produce the desired result anyway), consider this this example:

curl -s -X PUT localhost:9200/bad_recipes
curl -s -X PUT -H "Content-Type: application/json" \
     localhost:9200/bad_recipes/_mapping/_doc --data-raw '
{
  "properties": {
    "title": {
        "type": "text",
        "fields": {
            "keyword": {
                "type": "keyword"
            }
        }
    },
    "recipe_type": {
      "type": "text",
      "fielddata": true
    }
  }
}
' | jq .
curl -s -X POST -H "Content-Type: application/json" \
     'localhost:9200/bad_recipes/_doc' \
     --data-raw '{"title": "first title", "recipe_type": "main course" }' | jq
curl -s -X POST -H "Content-Type: application/json" \
     'localhost:9200/bad_recipes/_doc' \
     --data-raw '{"title": "second title", "recipe_type": "dessert" }' | jq
curl -s -X POST -H "Content-Type: application/json" \
     'localhost:9200/bad_recipes/_doc' \
     --data-raw '{"title": "third title", "recipe_type": "weird" }' | jq
curl -s -X POST -H "Content-Type: application/json" \
     'localhost:9200/bad_recipes/_doc?refresh=wait_for' \
     --data-raw '{"title": "fourth title", "recipe_type": "weird" }' | jq
curl -s -H "Content-Type: application/json" \
     localhost:9200/bad_recipes/_search \
     --data-raw '{"aggs":{"my_agg_name":{"terms":{"field":"recipe_type"}}}}' | jq

The final aggregation outputs the following buckets:

      ...
      "buckets": [
        {
          "key": "weird",
          "doc_count": 2
        },
        {
          "key": "course",
          "doc_count": 1
        },
        {
          "key": "dessert",
          "doc_count": 1
        },
        {
          "key": "main",
          "doc_count": 1
        }
      ],
      ...

That is, because the field contains analyzed text the value “main course” was treated as the field having two values “main” and “course” and the aggregation was computed accordingly (i.e. the counts are “wrong”).

Because it is easy to shoot yourself in the foot with fielddata, it has been disabled by default in recent ES versions, which is why the above example has "fielddata": true in the mapping. In older versions of ES (e.g. 2.x) fielddata was enabled by default, and in ES 1.x doc values were also disabled by default, making the pit of success even smaller.

There are use cases where aggregation on analyzed text does give the correct output; for example if you wanted to make a tag cloud (word cloud) visualization so that words that appeared more often was rendered with a bigger font size etc and you didn’t want to do this for the entire index but rather for a subset of documents. Another such use case is significant terms aggregation.

Refresh

When you write data into ES you cannot find it using a search until after the next “refresh”. By default the refresh interval in ES is 1s, but this can be tweaked to a higher value to give ES more room to optimize throughput. You can see the current refresh interval for a given index using:

$ curl -s es-hostname:9200/indexName/_settings \
      | jq -r 'to_entries | .[0].value.settings.index.refresh_interval // "1s"'

When you write data to ES with curl you can pass ?refresh=wait_for if you want it to block until the data is visible to searches.

More About Elasticsearch

Designing the Perfect Elasticsearch Cluster: the (almost) Definitive Guide by Fred de Villamil
Running Elasticsearch for Fun and Profit
awesome-elasticsearch repo on GitHub

1: https://github.com/kimchy/compass

2: Elasticsearch blog: Welcome Jordan & Logstash

3: Marc Gravell, who works for Stackoverflow, described the reasons why they moved away from MSSQL full text search to Elasticsearch in this comment (from 2013). They periodically post details of their setup, including how they use Elasticsearch.

4: For example, MySQL has support for full text search via ```MATCH() ... AGAINST``` but as soon as you combine such a search with a regular SQL condition performance suffers heavily.

End-to-end testing command line utilities

Sat, 29 Dec 2018 00:00:00 +0000

A quick and clean way to write end-to-end testcases for command line utilities is the cram testing framework. You can install cram using pip install cram (or inside virtualenv), and then create a file called test1.t with the following contents:

The first testcase

  $ echo abc
  abc

  $ echo -e 'a\nb\nc' | wc -l
  3

Launch /bin/false and verify that it exits with exit code 1

  $ /bin/false
  [1]

Launch sub-shell and verify that it exits with exit code 42

  $ (echo abc ; exit 42)
  abc
  [42]

cram will verify that exit code is 0 if no expected exit code N is written as [N]

  $ true

Testcases will be skipped if do you do "exit 80"

  $ echo abc && exit 80

cram can also invoke multiline commands

  $ cat /proc/cpuinfo | grep bogo | \
  > cut -f 2 -d : | \
  > awk '{s+=$1} END {print s}' | \
  > wc -l
  1

You can also invoke multiple commands

  $ export ABC=def
  $ echo $ABC
  def

You can then run the test using cram test1.t. Another nice feature is the interactive mode where you can automatically make the actual output become the expected output (i.e. just like snapshot testing in jest).

Writing tests with cram is extremely powerful because you have all your regular command line tools available. For example, you can curl some API and assert that the response looks reasonable.

Kudos to Brodie Rao for creating this awesome tool!

Detecting unused files with webpack

Fri, 28 Dec 2018 00:00:00 +0000

In a small (9 kloc) webapp I’m working on occasionally I typically use eslint (or similar) to detect unused code and stylelint to find unused CSS. Having zero issues reported is part of the automatic tests that I run on all code before it’s pushed to git. However, I just found a whole unused file in my project which these linters did not warn me about. Luckily, I also found a good way to detect unused files in my pre-push quality check script: the unused-webpack-plugin webpack plugin.

To use it, install unused-webpack-plugin as a dev dependency and then add something like this to your webpack.prod.config.js:

const UnusedWebpackPlugin = require('unused-webpack-plugin')

module.exports = {
  ... ,
  plugins: [

    new UnusedWebpackPlugin({
      directories: [path.join(__dirname, 'src/client')],
      exclude: ['*.test.js'],
      root: __dirname,
      failOnUnused: true,
    })

Episode recommendations for the Functional Geekery podcast

Thu, 30 Nov 2017 00:00:00 +0000

Lately, I’ve been listening to a podcast about functional programming called Functional Geekery. I ended up eventually listening to the entire backlog of episodes, but if you just want to see what the podcast is about, below you’ll find a list of good episodes. The first few minutes usually lists a bunch of functional programming conferences, just skip past those if you’re listening to old episodes.

Bartosz Milewski. Category theory.
Brian Lonsdorf. Category theory, Frisby videos, JavaScript, PureScript.
Hardy Jones, Haskell, Elm, PureScript.
Mark Seemann, F#, Haskell, testing.
Susan Potter, Haskell.
Jose Valim, creator of Elixir.
Matthew Butterick, racket, typography.
Edwin Brady, idris.
Scott Wlaschin, about Railway Oriented Programming
Scott Wlaschin, state, behavior, immutable data.
Rahul Muttineni, Haskell on the JVM and the Eta language.
David Nolen, Clojure, ClojureScript, Om, React, GraphQL and Falcor.
Phil Freeman, PureScript creator.
Adelbert Chang, scala implicits vs type classes.
David Koontz, F#, Haskell, LambdaCast.
Stian Veum Møllersen, Perlis Languages, concatenative languages.
John de Goes, LambdaConf, PureScript.
Matthias Felleisen, Scheme, Typed Racket.

There are several other incredibly interesting episodes. These ones above are just a few ones that came to mind when I scrolled through the episode list again quickly. Also don’t miss the show notes, they are often a treasure trove of links to good papers and videos.

Huge thanks to Steven Proctor for creating this amazing podcast!

AbortController polyfill for cancelling fetch()

Mon, 24 Jul 2017 00:00:00 +0000

While writing tests for a React app I ran into the following warning:

Warning: setState(...): Can only update a mounted or mounting component. This usually
means you called setState() on an unmounted component. This is a no-op. Please check
the code for the BoatyMcBoatface component.

This warning is emitted, for example, when a component starts a REST call (or timer etc) and then gets unmounted before the call completes. Once the call does complete, it typically tries to do setState() on the component, which is not legal after it has been unmounted.

So the correct fix is to abort the REST call when the component unmounts, which is easy if you’re using axios or some other XMLHttpRequest wrapper. The problem was that I was using native fetch() which doesn’t have support for aborting the request (yet). I googled around for the status on cancellation and it turns out that just a few days ago a new API was been added¹ to the DOM specification for this. This new API is a smaller scope compared to cancellation of promises for which discussions stalled a bit early this year.

This new DOM API allows you to create an AbortController that in turn allows you to pass an AbortSignal into the fetch() call. Later on you can call .abort() on the controller to cancel the request. If you used the new API from a React application, it would look like this:

componentDidMount() {
  this.controller = new AbortController();
  const signal = this.controller.signal;
  fetch(`/api/thing?id=${this.props.thingId}`, {signal}).then(res => {
    if (res.status == 400) {
      this.setState({ loading: false, actualThing: undefined });
      return;
    }
    return res.json();
  }).then(actualThing => {
    this.setState({ loading: false, actualThing });
  }).catch(err => {
    if (err.name == 'AbortError') {
      return;
    }
    // It's important to rethrow all other errors so you don't silence them!
    // For example, any error thrown by setState(), will pass through here.
    throw err;
  });
}

componentWillUnmount() {
  this.controller.abort();
}

I wanted to try this new API in my application, so I created a polyfill for it. The polyfill does not actually close the TCP connection on abort, it just throws the AbortError as seen above and then when the response arrives it is dropped silently (the real implementation will close the TCP socket though of course). This allows you to target the new API today and once the API is implemented in browsers you can just drop the polyfill and not modify your code. The polyfill is available here:

If you’re using webpack, you simply npm install --save abortcontroller-polyfill and then import 'abortcontroller-polyfill'; or require('abortcontroller-polyfill'); near the top of your client entrypoint .js or just include 'abortcontroller-polyfill' as an entrypoint itself (before your own entrypoint).

When I was googling around for the status on cancellation, I also noticed that Ron Buckton is presenting a very ambitious proposal (slides) at a TC39 meeting (which happens to be today!) that aims to unify cancellation, not just for DOM but for asynchronous functions, asynchronous iterators, webworkers, animation etc in such a way that they can be both synchronous and asynchronously observed (i.e. asking “was it cancelled?” verus “tell me if it cancels”). This work is based on cancellation in managed threads in the .NET Framework.

1: The final version of AbortController has been added to the DOM specification. The corresponding PR for the fetch specification is essentially done but not technically merged yet, waiting for testcases to be merged into web-platform-tests via PR #6484. Browser bugs tracking the implementation of AbortController is available here: Firefox: #1378342, Chromium: #750599, WebKit: #174980, Edge: #13009916.

Javascript tools for end-to-end testing web applications

Thu, 20 Jul 2017 00:00:00 +0000

Earlier, I compared¹ a few different tools for unit testing Javascript code. This post reviews available tools for end-to-end testing web applications using Javascript (i.e. tests that click buttons, type text etc). Many UI frameworks comes with a tool for this, but I’m looking specifically for a framework neutral tool so I’m not considering for example Protractor (Angular), Capybara (Ruby on Rails), Mirage (Ember) etc.

I want my E2E tests to:

smoke test all layers of the application running together (frontend/backend)
catch browser specific issues (even though browser bugs per se are getting increasingly rare, you typically need to verify that your polyfills are complete and working in each browser for example)
click through parts of the application that are not current being changed/developed to guard against regressions when shared components changes
always report any JS exceptions (if I have assertions verifying correctness for the most important part of a page/screen, I still want a systematic check that no JS exceptions happened for example when some less important “below the fold” part of the page loaded. I don’t want full coverage with assertions in E2E tests, I prefer to push as much as possible to component level tests)

Why should you implement E2E tests in Javascript?

A web application end-to-end testcase will send commands to a real browser and wait for it to complete before sending the next command. Commands are often (but not always) sent to the browser via Selenium or at least the Webdriver API. This means that the testcases are, by their nature, a long string of asynchronous operations. Given the “sort of single-thread” nature of Javascript, there is a potential for “callback hell” issues. Java on the other hand can execute many testcases in parallel using OS threads and block for each asynchronous operation, thus providing a very easy to use API. On top of that Selenium itself is written in Java, so it’s fair to ask why not just write E2E tests in Java?

If your backend is not JVM-based (e.g. Node/Python/golang/PHP), it’s inconvenient to force all developers to install a JRE/JDK, IDEA or Maven/Gradle solely for E2E tests (if you target the webdriver binary directly, you don’t even need a JRE installed, although some cloud testing providers still require you to go via a Selenium server). Even if you have a JVM-based backend already, it’s much faster to iterate on E2E tests written in Javascript (you certainly don’t want to wait for a slow Maven multi-module reactor build) and your testcases doesn’t need HotSpot JIT anyway. On top of that, if your E2E tests execute in Node rather than in a browser, it’s possible to use coroutines implemented by the node-fibers package to create a API that looks and feels synchronous. This allows you to put breakpoints half way into your testcase etc, which is a bit trickier with APIs with chained asynchronous methods (which is also used quite frequently among E2E test frameworks implemented in Javascript).

What are the available options/tools for UI automation in JS?

Among the tools/frameworks that comes up when you google for “end to end testing”, I immediately ruled out a few, like PhantomJS/CasperJS (not a real browser, about to be replaced by headless Chrome anyway). Zombie.js does not test in a real browser and while impressive it looks a lot like a one man project that went into maintenance mode. However, Zombie.js is based on a really good idea (getting rid of slowness and flakiness by targetting jsdom). It would be nice if this approach was combined with full browser testing so that the testcases first ran really fast using jsdom (to make it quick to iterate and develop new testcases) and once they passed there, all testcases would run again but inside all the target browsers. This was attempted in taxi-rank, which is based on cabbie. However, there wasn’t a lot of good articles/examples/documentation about these and NPM downloads are super low so I didn’t spend any time on them, but I like the idea.

Another really nice project is NightmareJS that runs UI automation inside Electron (so it’s much closer to Chromium compared to PhantomJS). It’s much faster than Selenium and avoids a lot of the messy configuration and flakiness that comes with trying to launch a real browser. I’ve used it for a lot of things, but for the purposes of this comparison I’m not including it since it cannot run testcases inside a real browser (and I explicitly want to test in Firefox etc). Dalek.js can test in a browser but the project was never finished and is not maintained anymore. Buster.js and TotoroJS were also abandoned/unmaintained and didn’t pass a 5 second sniff test.

I saw a few people tweeting about Cypress.io so I included it although it’s in closed beta (not technically open source right now but they claim that will change fairly soon).

Which E2E tools/frameworks were compared?

The full list of tools that I ended up trying out was:

	License	API style	Issue Latency ²	Open / All Issues ²
selenium-webdriver	Apache-2.0	fake sync or async/await	n/a	n/a
wd.js	Apache-2.0	multiple	1 day	7%
Nightwatch.js	MIT	async chain	1 days	12%
Webdriver.io	MIT	coroutine sync	12 hours	4%
Chimp	MIT	coroutine sync	75 days	19%
Nemo	Apache-2.0	fake sync	10 days	16%
CodeceptJS	MIT	coroutine sync	30 days	19%
Intern	New BSD	async chained	26 days	10%
TestCafe	MIT	await single chain	7 days	16%
Cypress	?	async chained	49 days	32%

Don’t read to much into the issue latency numbers without checking the issues yourself, for example I opened a bug with Cypress recently and they triaged it and assigned it to a developer the same day and it was fixed 24 hours later.

If you render the monthly downloads for each of these NPM packages (and protractor as well for reference), you get this plot:

The protractor package depends on selenium-webdriver, so that drives selenium-webdriver downloads quite a bit. In general, NPM download statistics should be taken with a grain of salt due to mirrors and download bots, downloads by CI systems and cached installs. And you cannot tell if a package has high download numbers because IT is popular or if it’s because it has a popular package depending on it. Note that nemo.js depends on selenium-webdriver, Chimp depends on WebdriverIO and CodeceptJS depends on protractor, webdriverio and selenium-webdriver (it can execute using multiple backends).

If we zoom into the weeds, we see that during 2017 alone CodeceptJS and TestCafe both enjoyed >100% increase in downloads whereas Chimp and Intern are largely constant. Just for fun, I also included “nightmare” in the chart, it certainly has been gaining a lot popularity during 2017 (currently at 4x of where it was in october last year).

Another interesting thing is the sharp drop in downloads for nemo; but this is partly due to Paypal setting up an internal NPM mirror (they still use nemo).

Another interesting thing to look at is the number of commits in the git repository for each of these test tools. Cypress isn’t open source yet so I couldn’t include it. selenium-webdriver lives in the main selenium repository inside a directory called “javascript” so I did a quick git filter-branch on that directory and measured only commits that touched that directory.

Anyway, if you plot the commit count over time you get this:

The git data goes back a lot longer than the NPM downloads data, maybe wd.js had more NPM downloads during the period when it also had a lot of commits landing. A lot of work was done on wd.js from 2012 to 2014 (FWIW; the main author of wd.js is Adam Christian, who worked for Sauce Labs 2010 to 2015). Webdriver.io started to pick up speed in 2013, and unlike wd.js, is still going strong. It’s headed by Christian Bromann (who works for, you guessed it, Sauce Labs).

Another notable thing in the commit count chart above, is the steep increase for Intern starting in April; this is Jason Cheatham kicking off development of Intern v4 (more on that later).

Below is another plot of the same data; showing commits per year for the three major webdriver implementations and I’ve also added in leadfoot which is an independent webdriver implementation used by Intern (they used wd.js in v1 before developing leadfoot):

Another great way to estimate how many people are invested in a certain technology is to analyze how the number of active developers changes over time. To visualize this, I’ve plotted the number of unique developers seen in the git log during a rolling 365 day window:

This shows strong community momentum for webdriver.io and to some extent CodeceptJS.

I also tried to write a few testcases using each framework. I created a small test page that had a button which started a timer and then eventually made a div visible with some text, and also some other easy scenarios. The test page could also be configured to be buggy, and then various bad things would happen like the element not being added to DOM, some JS error would occur even though the page appeared otherwise to be working, or the element was added to the DOM but inside another div that had overflow:hidden and the child div was then placed outside of the cliprect of the parent making it invisible while still not having display: none. I wanted to see how each framework responded to these “bugs” and how easy it was to understand the error messages / test results etc.

Below are some code examples (in reality you should of course not be hardcoding selectors like this, but instead use the page object pattern or something similar). I’ve also included a few notes for each framework.

selenium-webdriver

For selenium-webdriver, I used regular MochaJS as the runner. I disabled the global promise manager that is used to provide the older “fake sync” API, and I used async/await instead (without transpilation! Yay, Node v8!). The testcases looked sort of like this:

it('scenario 3', async function() {
  await driver.get('file://' + path.join(__dirname, '../../test-pages/index.html'));
  await driver.findElement(By.xpath('//*[text()="button3"]')).click();
  const element = until.elementIsVisible(driver.findElement(By.id('hidden3')));
  const actualElement = await driver.wait(element, ELEMENT_TIMEOUT);
  assert.equal(await actualElement.getText(), 'hidden3 pass');
});

The .get() call blocks until the page has loaded fully so normally you can just do for example driver.findElement(By.name('q')).sendKeys('webdriver') but #hidden3 above was added by a timer after the button was pressed so I needed to add the explicit wait.

Selenium and WebDriver in general doesn’t have protocol support for streaming JS errors via the driver, which is a shame because it would be so useful! However, there is support in the protocol for retrieving the JS console log at least, so I was able to assert that there were no JS errors, by using this clunky hack:

it('scenario 2', async function() {
  await driver.get('file://' + path.join(__dirname, '../../test-pages/index.html'));

  await driver.findElement(By.css('.button2')).click();
  const hidden2 = until.elementIsVisible(driver.findElement(By.id('hidden2')));
  const hidden2actual = await driver.wait(hidden2, ELEMENT_TIMEOUT);
  assert.equal(await hidden2actual.getText(), 'hidden2 pass');

  const browserLog = await driver.manage().logs().get(logging.Type.BROWSER);
  const errors = browserLog.filter(entry => entry.level.name === 'SEVERE')
  errors.forEach(entry => console.log('[%s] %s', entry.level.name, entry.message))
  assert.ok(errors.length == 0, 'JS errors in browser log!');
});

If you’re going to use selenium-webdriver, then you should be using this modern async/await API that I showed above, but there is also an older “fake sync” API that looked like this:

driver.get('http://www.google.com/ncr');
driver.findElement(By.name('q')).sendKeys('webdriver');
driver.findElement(By.name('btnG')).click();
driver.wait(until.titleIs('webdriver - Google Search'), 1000);
driver.quit();
console.log('all done');

This looks synchronous but it is not, instead each of the methods will append their action to a chain of promises that won’t even start until the next runslice. This means that the code will first print “all done” and then it will begin the navigation to google.com. It also means that you cannot stop the testcase halfway through by inserting debugger between two lines. You can call it from inside a then, chained after another line, like this:

...
driver.get('http://www.google.com/ncr').then(() => {debugger});
...

In general, my impression of selenium-webdriver was quite bad because not even the “hello world” example would run due to #4041 and while this fix bug was fixed by Lucas Tierney (kudos!) unfortunately no new release has been pushed to NPM for this critical bug (it’s been >2 months now).

Another drawback for “selenium-webdriver” is the generic name it has. It’s very hard to google for examples and help for this tool because you keep hitting articles about selenium webdriver in C#, Java, Ruby etc etc. This was actually surprisingly frustrating.

wd.js

wd.js offers multiple APIs; one pure async API for those who are really into indentation, one non-chained API based on Q promises for people who enjoy writing .then() a lot, one based on generators and of course one API based on chained async calls (also using Q promises). Their readme doesn’t mention any examples using async/await, though it’s probably not hard to setup if you want it. I thought it was quite interesting to look at all of these API examples in their readme.

Code using the chained Q API looks like this:

browser
  .init({browserName:'chrome'})
  .get("http://admc.io/wd/test-pages/guinea-pig.html")
  .title()
    .should.become('WD Tests')
  .elementById('i am a link')
  .click()
  .eval("window.location.href")
    .should.eventually.include('guinea-pig2')
  .back()
  .elementByCss('#comments').type('Bonjour!')
  .getValue().should.become('Bonjour!')
  .fin(function() {
    return browser.quit();
  })
  .done();

The commits per year plot earlier suggests that webdriverio has much stronger momentum compared to wd.js so I didn’t bother to write my own testcase for wd.

nightwatch.js

Nightwatch offers a async chained API and runs on top of a built-in (forked) mocha version. A test could look like this:

this.scenario3 = function (browser) {
  browser
    .url('file://' + path.join(__dirname, '../../../test-pages/index.html'))
    .useXpath()
    .click('//*[text()="button3"]')
    .useCss()
    .waitForElementVisible('#hidden3', ELEMENT_TIMEOUT)
    .assert.containsText('#hidden3', 'hidden3 pass')
    .end();
};

The chain executes asynchronously so you can’t break it apart and put debugger in the middle, instead you have to inject your code into the chain; by using for example: .perform(() => {debugger}).

You can run tests using nightwatch -c nightwatch.json and use --env chrome if you want to run tests in chrome instead of Firefox etc. In the config file you can set the paths to your geckodriver and to the selenium server jar, and then just set selenium.start_process to true. I also wanted all testcases to finish even if the first one failed so I set skip_testcases_on_fail to false. I was able to retrieve the JS log using the same clunky way I used for selenium-webdriver. The config, command line output and command line switches all just works the way you’d expect them to.

I didn’t try it, but it seems possible to run your nightwatch tests via Sauce Labs quite easily. There is also another cloud testing service called nightcloud.io being developed.

In general, Nightwatch feels like a very competent tool. The only drawback I can think of is that it seems that nightwatch was largely written by a single guy, so it has a low bus factor.

webdriver.io

If you disgard selenium-webdriver downloads (and protractor drives a lot of those), Webdriver.io is the most downloaded E2E package on NPM. Webdriver.io is also the framework with the most active developers working on it, in the last year alone over 130 different developers sent at least one patch to this project.

The code looks like this:

it('scenario 3', () => {
    browser.url('https://www.whatever.com/something/');
    browser.click('//*[text()="button3"]');
    browser.waitForVisible('#hidden3', ELEMENT_TIMEOUT)
    assert.strictEqual(browser.getText('#hidden3'), 'hidden3 pass');
});

This code looks superficially similar to the “fake sync” API of selenium-webdriver, but it is really not the same. If you drop execArgv: ['--inspect-brk'], into your wdio.conf.js and then add debugger somewhere in the middle of the testcase, like this:

it('scenario 3', () => {
    browser.url('https://www.whatever.com/something/');
    browser.click('//*[text()="button3"]');
    debugger
    browser.waitForVisible('#hidden3', ELEMENT_TIMEOUT)
    assert.strictEqual(browser.getText('#hidden3'), 'hidden3 pass');
});

..then the debugger will actually execute the first part of the testcase, open the URL, wait for it to finish loading and then click the button. Only after those things are finished executing will it hit the debugger breakpoint. This is possible because webdriverio uses proper coroutines implemented in native code and supplied by the node-fibers package. This means that for example the, .url() method can send a message to the browser saying that it should navigate to a particular url and then execution context suspends itself in such a way that it’s not resumed until the browser has finished loading the URL. From the outside, it appears like .url() is synchronous even though node is still “single threaded” and we havn’t used any additional OS threads.

Another neat feature for webdriverio is that it has a really good REPL that fires up a browser and then you can experiment with commands and selectors from there. Even better, you can insert browser.debug(); into an existing testcase and have it drop into a REPL at that particular place in the testcase.

The current getting started documentation on the webdriver.io site is not ideal imo. It has lots of steps in the beginning asking you to download specific versions of the selenium server jar, geckodriver, chromedriver etc which is both fiddly and error prone. A bit later on in the guide it mentions wdio, and much later I learned about wdio-selenium-standalone-service which automatically sets everything up. Also, at first I couldn’t find a way to override the configured browser from the command line, but then I realized I could set the config to browserName: process.env.SELENIUM_BROWSER || 'firefox' and override it with an environment variable. The config can also list multiple browsers if you want to test locally in Chrome and Firefox simultaneously, e.g. put this at the end of your wdio.conf.js:

if (process.env.SELENIUM_BROWSER) {
  exports.config.capabilities = [{
      browserName: process.env.SELENIUM_BROWSER
  }];
} else {
  exports.config.capabilities = [
    {
      browserName: 'chrome'
    },
    {
      browserName: 'firefox'
    }
  ];
}

I also ended up setting timeout: 30000 inside mochaOpts. Running with --logLevel verbose was useful for debugging. The watch mode is also very useful.

To assert that no JS errors were encountered, I had to explicitly download the browser log in the end of the testcase, just like I did for selenium-webdriver. Sad!

chimp

chimp is a framework built on top of webdriver.io, an example testcase looks like this:

browser.url('https://www.whatever.com/something/');
browser.click('//*[text()="button3"]');
browser.waitForVisible('#hidden3', ELEMENT_TIMEOUT)
assert.strictEqual(browser.getText('#hidden3'), 'hidden3 pass');

There was a few things I didn’t like about chimp. It talks a lot about Meteor/DDP and Also, chimp --help doesn’t print anything and for every test run it prints “Master Chimp and become a testing Ninja! Check out our course: http://…”

In the end I didn’t think chimp added enough on top of webdriver.io to make it worth the additional dependency.

nemo.js

This E2E framework was developed by PayPal. An example testcase looks like this:

/*global describe:true, it:true */

var basedir = require('path').resolve(__dirname, '..');
var Nemo = require('nemo');
var nemo;

describe('suite using @generic@ nemo-view methods', function () {
  before(function (done) {
    nemo = Nemo(basedir, function (err) {
      if (err) {
        return done(err);
      }
      done();
    });
  });
  after(function (done) {
    nemo.driver.quit().then(done);
  });
  it('should automate the browser', function (done) {
    //login
    nemo.driver.get(nemo.data.baseUrl);
    nemo.view._waitVisible('id:signup-button').click();
    nemo.view._waitVisible('id:cta-btn').click();
    nemo.view._waitVisible('id:email').sendKeys('mynewpaypalaccount@geemail.paypal');
    nemo.driver.sleep(3000).then(function () {
      done();
    }, function (err) {
      done(err);
    })
  });
});

Nemo starts up with selenium-webdriver 2.53.3 which is incompatible with Firefox v48 and all later versions of Firefox (i.e. all versions of Firefox released since mid-2016). Their homepage mentions this and says users should downgrade Firefox or upgrade to a new major version of selenium-webdriver that hasn’t been tested with nemo. It also prints a few other east to fix deprecation warnings like os.tmpDir vs os.tmpdir etc.

CodeceptJS

This framework allows you to run your testcases on top of either webdriverIO, selenium-webdriver, protractor or NightmareJS. Their API uses generators to deal with the asynchronicity:

Feature('Test1.js');

Scenario('test something', function*(I) {
  I.amOnPage('/');
  I.fillField('[name=q]', 'hello world')
  I.waitForElement("h3.r", 30)
  I.see('"Hello, World!" program - Wikipedia');
  const titles = yield I.grabTextFrom('h3.r')
  console.log(titles.join("\n"));
});

When I first tried to install the codeceptjs-webdriverio package it failed to install properly, so I filed issue #1 (yay!) and 25 days later a fix was pushed.

From what I can tell, CodeceptJS doesn’t even have an API to fetch the browser JS log, so I could not implement a check that verifies that no JS error happened on the page.

TestCafe

TestCafe is developed by a company called DevExpress who open sourced it in October 2016. They continue to work on it quite intensely (they are landing about 480 commits per year which is higher than all other frameworks except for webdriverio at >700 commits per year). The most interesting thing about this framework is that it does NOT use selenium or webdriver at all. Instead parts of the test framework itself executes inside the browser.

A typical testcase looks like this:

fixture `example page`
    .page `${testpageUrl}`;

test('scenario 1', async t => {
    await t
        .click('#button1')
        .expect(Selector('#hidden1').innerText).eql('hidden1 pass')
});

If you’re not familar with the syntax near “fixture”; it’s an ES6 tagged template literal.

One really awesome thing about testcafe is that it automatically detects JS errors that happened when the test page was executing. I suppose this is the kind of awesome you can attain when you reach outside of the Selenium box! I also noted that the terminal output when tests failed where quite readable/clean.

TestCafe doesn’t support XPath selectors (#1178) but it has a special API for locating elements by text: Selector('form input').withText('Submit').

Intern

Next up is Intern; a typical testcase looks like this:

'login works'() {
    return this.remote
        .get('index.html')
        .findById('username')
        .type('scroob')
        .end()
        .findById('password')
        .type('12345')
        .end()
        .findById('login')
        .click()
        .end()
        .sleep(5000)
        .findByTagName('h1')
        .getVisibleText()
        .then(text => {
            assert.equal(text, 'Welcome!');
        });
}

As mentioned in conjunction with the “commit count over time” plot earlier, Jason Cheatham is currently working on Intern v4 (the example above is from v4). There are only alpha releases available so far, but the idea is that the final release will be “mostly API compatible” with the latest alpha. Intern v4 will be used to test the new version of the Dojo framework.

While earlier versions required you to declare your test suites as AMD modules, now with Intern v4 you can just babel-register and use require (and the built-in code coverage works even for transpiled code!). Under the hood, Intern itself has been rewritten in TypeScript.

My impression of Intern is that it’s by far the most difficult framework to setup, this will probably change once v4 is finished though (both in terms of reduced quirks and better documentation).

Cypress

This framework is currently in closed beta. Luckily I was able to get an early invite (thanks Jennifer!). Cypress runs via Mocha and its API uses async chaining:

it('scenario 2', () => {
  cy
    .visit(testpageUrl)
    .get('.button2')
    .click()
    .get('#hidden2')
    .should('be.visible')
    .should('have.text', 'hidden2 pass')
});

It does not use selenium or webdriver, instead parts of the test framework is injected into the browser directly. If the test runner needs to do privileged actions like read a file from disk, it makes a request to a local webserver that performs the action for it (or in some cases a browser extension is used as well). For alert popups, Cypress currently auto accepts them but that doesn’t work if they are coming from an iframe (#376).

Cypress also comes with the ability to optionally intercept XHR requests from the browser to immediately return a static result, speeding up testcases where such a trade-off is appropriate. It comes with bundled versions of chai, sinon, moment, lodash and jQuery. The documentation is really good. Finally, Cypress ships with a GUI application based on Electron where you can see your tests execute. For each step in the testcase, it snapshots the browser so you can go back and look at what the DOM looked like at that step.

Since Cypress isn’t finished yet you cannot really compare it to the other frameworks, but it feels really sophisticated and polished. There are major things left to implement though, for example you can only run tests in Chrome and Electron right now; Firefox, Edge and IE is not available.

Conclusion

It’s amazing that there are so many good tools out there to chose from and so many talented people working on them.

In the end I decided to not use wd.js because it’s not maintained actively enough, same goes from nemo (which also isn’t used enough outside of PayPal). Intern was too difficult to setup and in transition between v3 and v4 and I didn’t really see the value in Chimp over WebdriverIO. I also decided to skip Cypress because it’s not finished and cannot test Firefox etc, and even when v1 is out it will require some time to mature. Also it’s risky that there are so few developers working on it (right now), I prefer tools with broad community participation (no doubt they will try to build such a community).

If I rank the remaining frameworks, I get something like this (best up top):

webdriver.io
TestCafe
NightwatchJS
CodeceptJS
selenium-webdriver

There are many things factoring into this ranking; for example on a technical level, NighwatchJS is really good but the community momentum around WebdriverIO seems so much stronger. I’m still pretty interested in seeing what the limitations are of non-selenium E2E testing. Because of this, my plan is to write a slightly larger test suite in both webdriver.io and TestCafe, and have them run in parallel for a while so I can get some more experience of how they work. I will also be checking back with Cypress after they release a stable version that can test Firefox, Edge and IE.

If you have to pick a single test E2E framework today, I recommend webdriver.io.

Epilogue

Did you like this post? Tweet about it or post it on whatever other network / thing you’re on.

Did I miss some important feature / aspect of some framework? Did I miss an entire framework? Tweet at @mo_molsson or drop a comment below. Thanks for reading!

1: Buridan's ass.

2: The median "Issue Latency" and the fraction of open issues among all issues, are both computed using the excellent service Is it maintained?.

Chained APIs with asynchronous functions in Javascript

Tue, 18 Jul 2017 00:00:00 +0000

Quite a few of the web application E2E test frameworks implemented in Javascript have APIs based on method chaining that sort of hide the asynchronous nature of Selenium testing (send command to browser, wait for reply, send command, wait for reply etc). Examples include NightwatchJS, wd.js when it is being used via its chained Q-promises API and also Cypress. Variants of this chaining is sometimes referred to as Fluent Interface.

For example, here is a NightwatchJS testcase:

browser
  .url('http://launchpad.kp/')
  .click('#big-red-button')
  .waitForElementVisible('.confirmation-popup', 1000)
  .assert.containsText('.confirmation-popup', 'fire ze missiles?')
  .click('#cancel-button')
  .end();

These chained APIs make asynchronous code look sort of synchronous even though it’s not. Often people can get by without knowing how it works, until for example they have to step through the code in a debugger; then it really helps a lot to understand how these APIs are built.

Why chained async methods can be unintuitive at first

If you break the chain into two parts and add some breakpoints:

browser
  .url('http://launchpad.kp/')
  .click('#big-red-button')
  .waitForElementVisible('.confirmation-popup', 1000)

debugger;

browser
  .assert.containsText('.confirmation-popup', 'fire ze missiles?')
  .click('#cancel-button')
  .end();

debugger;

When the debugger stops at the first breakpoint, nothing has happened yet. The browser hasn’t received the URL yet, in fact, Nightwatch hasn’t even started the browser yet. However, the same is also true at the second breakpoint. This is because the code doesn’t really perform the actions, instead it just builds a “list of actions”, the first of which the Javascript engine will begin to execute at the next runslice. Technically, this list is stored as a chain of promises.

Actually, there are also non-chaining variants of this where seemingly synchronous code just chains up promises that gets executed later on. For example, early versions of the npm package selenium-webdriver (the “official” Javascript bindings for Webdriver shipped by the Selenium project themselves) had a global promise manager that allowed you to write code like this:

var webdriver = require('selenium-webdriver'),
    By = webdriver.By,
    until = webdriver.until;

var driver = new webdriver.Builder()
    .forBrowser('chrome')
    .build();

driver.get('http://www.google.com/ncr');
driver.findElement(By.name('q')).sendKeys('webdriver');
driver.findElement(By.name('btnG')).click();
driver.wait(until.titleIs('webdriver - Google Search'), 1000);
driver.quit();
console.log('program finished');

But just like the “debugger” breakpoints in the nightwatch example, this snippet would first run to completion (and print program finished) and then after that it would start the actual testing.

Even though I think a more attractive approach (on Node) is proper coroutines via node-fiber, I still think it’s quite interesting to understand what different approaches have been tried, including chaining APIs. And in browsers we only have chaining APIs and async/await to pick from; so in extreme cases like E2E testing maybe chaining is the cleaner option instead of writing await on every API call (for example, Cypress executes E2E tests from the browser itself and uses chaining).

How to implement a chained API with async methods

When you implement a chaining API for synchronous functions, you just put all the functions on an object and you make sure that each function first does its thing and then returns this. Notably the thing that the function “does” can be either A) something that modifies state on this, or B) something that creates a copy of this where the state is slightly different (we don’t have to return this, we are only required to return something that implements all the methods in our API so that the chaining can continue).

For example, in jQuery you might do:

const elem = $('<div>')
  .text('hello')
  .css('color', 'blue')
  .addClass('foo');

When the above API chain is evaluated the element accumulates the text, the CSS and the class.

To implement chaining for asynchronous functions, we do the exact same thing, but the “state” we keep inside this is the chain of promises. We keep appending to this chain of promises as we evaluate the chain in the API.

To implement such a chained API in vanilla JS, one could do:

function someAsyncFunc(cb) {
  console.log('someAsyncFunc() called')
  setTimeout(() => cb(), 100);
}

function otherAsyncFunc(cb) {
  console.log('otherAsyncFunc() called')
  setTimeout(() => cb('otherAsyncFunc() result'), 100);
}

const someFunc = () => new Promise(resolve => someAsyncFunc(resolve))
const otherFunc = () => new Promise(resolve => otherAsyncFunc(resolve))

const MyApi = (previousActions = Promise.resolve()) => {
  return {
    someFunc: () => MyApi(previousActions.then(someFunc)),
    otherFunc: (cb) => MyApi(previousActions.then(otherFunc).then(res => cb(res))),
  };
}

const api = MyApi();
exports.api = api;

console.log("before chain")
api
  .someFunc()
  .someFunc()
  .otherFunc(result => console.log(result))
  .someFunc()
  .otherFunc(result => console.log(result))
console.log("after chain")

If you run the above example, it outputs:

before chain
after chain
someAsyncFunc() called
someAsyncFunc() called
otherAsyncFunc() called
otherAsyncFunc() result
someAsyncFunc() called
otherAsyncFunc() called
otherAsyncFunc() result

Javascript unit testing tools

Mon, 05 Jun 2017 00:00:00 +0000

I have compared¹ few different tools for unit testing Javascript code. The frameworks I looked into were QUnit, Mocha JS, tape, AVA, Jasmine, Karma and Jest. I wrote a few testcases and then ported these to each framework. The testcases I wrote were meant to detect defects in various kinds of functions, for example:

regular function that returned an incorrect value
regular function that unexpectedly throws an error
callback function that calls back immediately (same tick) and returns the wrong value
asynchronous callback function that delivers the wrong value
asynchronous callback function that forgets to callback
asynchronous callback function that accidentally calls back twice
asynchronous function returning a promise and resolving with an incorrect value
asynchronous function returning a promise that is then unexpectedly rejected

Before we dig into the implementation of these test scenarios in each test framework, here is a table summarizes a few basic facts:

	QUnit	Mocha	tape	AVA	Jasmine	Karma	Jest
Total Commits	1,157	2,263	453	1,062	1,584	2,274	2,224
GitHub Stars	3,630	12,480	3,889	10,335	12,502	8,623	10,455
Issue Resolution Time ³	3 days	11 days	1 day	5 days	7 days	58 days	22 hours
Open Issues / All Issues ³	4%	17%	27%	17%	4%	20%	5%
License	MIT	MIT	MIT	MIT	MIT	MIT	BSD-3-Clause ⁴

If you plot the commit count of each project over time it’s clear that QUnit and Jasmine are the oldest projects, and also that there is a very large amount of commits landing in the Jest repository right now (and have been for quite some time now) compared to all the other frameworks:

Another interesting thing to look at is the weekly download count on npm for each framework over time; Mocha is the most commonly used framework by a very wide margin. And if we dig into recent growth numbers (looking at from 2017-01-15 up until today) we see that Mocha weekly downloads increased by 206K (up 19%) while for example Jest added 195K new weekly downloads in the same period (which for them was an impressive 145% increase in less than 5 months!):

Below are some things I noted while writing sample testcases using these frameworks.

QUnit

The first framework that I looked at in-depth was QUnit. This is one of the oldest unit testing frameworks for Javascript. It was originally developed to test jQuery and jQuery UI, but was later extracted into a separate project that does not even depend on jQuery. Unfortunately while writing my first few testcases for QUnit, I ran into a bug! If you have a testcase that calls a function with an asynchronous callback, but the function forgets the invoke the callback; then QUnit terminates with exit code 0 and doesn’t even print the passed/failed summary. I bet many CI systems would interpret that as a PASS, so it’s a show stopper bug imo. The testcase illustrates the problem:

function asyncForgetCallback(str, callback) {
  if (isBuggy) {
    // accidentally nothing
  } else {
    setTimeout(() => callback(str.trim().toLowerCase()), 0);
  }
}

QUnit.only('util test async callback function that forgets to callback', assert => {
  const doneFunc = assert.async();
  util.asyncForgetCallback('   Foo   ', actual => {
    assert.equal(actual, 'foo');
    doneFunc();
  });
});

Later I found that this bug was discovered and filed on Github in March, but nobody has had time to fix it yet: #1132. A workaround for the bug is to set a global default timeout using QUnit.config.testTimeout = 2000 or similar.

On the plus side, QUnit was able to correctly handle the “accidental double callback” scenario (and doing this also without me having to declare the expected number of assertions) by failing the testcase due to assert + doneFunc being called after the original call to doneFunc(). It should be noted that QUnit does offer an “expected assertion count” feature available though, useful for more advanced test scenarios.

There was also some minor nitpicking issues that I noted while writing the rest of the testcases for QUnit:

Normally when you do assert.equal(actual, "expected") QUnit will print both the “actual” and the “expected” value whenever the assertion fails. However, if “actual” happens to be undefined then only the expected value is printed. This is a minor issue, but I filed it as bug #1177. The maintainer asked if I wanted to fix it myself and I figured what the hell, so I did.
In the QUnit documentation, the introduction starts with examples of assert.equal(), but note that the assertion assert.equal(1, "1") holds so (imo) it’s much better to use assert.strictEqual() by default and only resort to assert.equal() when you need it. Imo it would be better if QUnit had a “default assertion” that checked for strict (===) equality. This behavior comes with the default assert module in NodeJS (and also the “CommonJS Unit Testing” specification), so you get the same thing for example in Mocha JS if you choose to use those asserts there; but QUnit doesn’t allow you to use any other type of assertions. Pretty much all modern Javascript testing frameworks default to === and lets you opt-in for == when you need it.
From a UX perspective, I don’t like that QUnit prints its default TAP output as # fail 0 with red color when all tests are passing, while it prints for example # fail 1 as the summary line when one testcase fails. Nothing on screen should be red when all tests are passing. You can use an alternative reporter by passing --reporter but I like good defaults.
Another UX issue is that npm packages qunit and qunitjs both install a binary called qunit but only the latter can be used to run tests using the method described in the documentation. Confusing for newbies.

Mocha JS

Mocha was started in 2011 by Tj Holowaychuk, the same guy that founded ExpressJS, co, commander.js, koa and a bunch of other highly successful NodeJS projects, and who then switched to golang three years ago. After its inception, the project quickly grew into a comprehensive test framework that covers a lot of different testing needs, and a large number of people have contributed to its success over the years (roughly 15 unique authors shows up in the git log each 60 day period). Lately though, the Mocha JS project has been having some problems finding developers to keep up with maintenance, their GitHub page current says “we’re currently unable to merge most pull requests due to lack of maintenance resources.”

That said, this test framework is quite mature. It allows you to customize which type of assertions you use and which test result reporter you use etc. There is also plugins for IDEs/editors like emacs and IDEA and so on.

When I tried porting my testcases to Mocha JS I wasn’t really satisfied with how it handled the “accidental double callback”. For example, if I did:

// someCode.js
exports.asyncDoubleCallback = (str, callback) => {
  if (isBuggy) {
    setTimeout(() => callback(str.trim().toLowerCase()), 0);
    setTimeout(() => callback(str.trim().toLowerCase()), 0);
  } else {
    setTimeout(() => callback(str.trim().toLowerCase()), 0);
  }
};

// myTests.js
describe('#asyncDoubleCallback()', () => {
  // Mocha JS passes context object to it() via "this".
  it.only('is working', function (done) {
    util.asyncDoubleCallback('   Foo   ', actual => {
      assert.equal(actual, 'foo');
      done();
    });
  });
});

Then the default reporter (from Mocha 3.3.0) shows this result:

Note that Mocha does not report the number of passed/failed assertions, it reports the number of passing and failing testcases. In the above example I’m running a single testcase using it.only(), but Mocha reports 1 testcase passed and 1 testcase failed?!

tape

The tape GitHub page starts with testling-ci badge that hasn’t run since 2014, proudly claiming to support “Chrome 29”. There also hasn’t been a lot of commits made in the tape git repository lately. It was slightly annoying having to t.end() or t.plan() all testcases. Also I didn’t like that tape aborts the entire test suite if a single testcase throws, but it was easy to “fix” by switching the require('tape') to require('tape-catch') instead.

Here is how tape handles the accidental double callback bug scenario btw:

Note that tape, unlike Mocha, prints the number of assert passes/fails in the summary so everything is technically correct here in some sense.

Update 2017-07-12: As mentioned by Stefan Maric in the comments below; there is also a test framework called node-tap which is similar to tape in a lot of ways but with a few key issues fixed. Even though node-tap has fewer NPM downloads, npm uses node-tap for testing. If you like tape you should also have a look at node-tap before you decide which framework to use.

AVA

AVA describes itself as “futuristic” and unlike the other frameworks it runs tests in parallel, supports (and promotes) testing using ES2017 (and even post-ES2017 features like Observables). It also has extremely good assertion output, much better than the previous test frameworks. For equality of strings and deep equality of objects it shows diffs to highlight the difference between actual and expected. When my buggy callback function delivers the wrong value causing a t.is(actual, 'foo') assertion to fail, then it immediately highlights the source line where that bad value came from:

While porting my testcases to AVA, I did hit one bug (filed as #1377) in AVA related to timeouts (for the test scenario where a function forgets to call its callback).

Jasmine

Jasmine is a very old framework (first commit is from 2008). It was influenced to some extent by Ruby “RSpec”. Offers test doubles using spies built-in (in other frameworks you need to import something like sinon.js).

When I ported over the testcases to Jasmine, I found that, unlike the four previous test frameworks, it happily reports PASS in the “accidental double callback” scenario even if done() is called twice:

describe('#asyncDoubleCallback() naive', () => {
  it('is working', function (done) {
    util.asyncDoubleCallback('   Foo   ', actual => {
      expect(actual).toEqual('foo');
      done();
    });
  });
});

Jasmine also does not offer any assertion planning feature. At some point in 2014, somebody opened a GitHub issue (#626) asking for this to be added but the issue was closed by the maintainers because “they’ve not heard about this request before” which is unfortunate. They offer call counting for test double spies with callThrough, but using those for this scenario would be cumbersome and inefficient.

Karma

Karma is not a test framework per se, but rather a test runner that can take a test suite written in Mocha, QUnit or Jasmine and then run that suite in parallel on multiple browsers (desktop or mobile, and also including JSDom, PhantomJS and headless Chrome). All browsers report back results that are then presented as terminal output. It can also watch your test directory and automatically rerun the tests each time you update a testcase. Karma is the test runner used by Angular JS and was developed by the Angular team at Google. If you want to run your tests in browsers, then you should probably be using Karma regardless of which of the frameworks you pick (except for Jest, more on that below).

Jest

Jest is the test framework and test runner that is used by the React project and consequently by many libraries, tools and web apps implemented using React. The fact that it’s used by and backed by Facebook lends it both credibility and likely resources. Among the top 10 contributors to Jest, everyone except two works at Facebook². It seems like a large part of the work is done by Chris Pojer tbh.

Jest started as a test runner wrapped around Jasmine, and the assertions still looks like expect(foo).toBe('bar'), but now there are few traces of Jasmine left. Jest doesn’t have support for running tests inside a real browser, cpojer shared some thoughts around this here and in issue #848. One really cool feature of Jest is snapshot testing, which helps you detect regressions in your output (either React component trees serialized to text, or for example an AST serialized to text if you’re working on a source transformation tool).

When I tried to port my testcases to Jest, I quickly ran into a Jest bug where assertion failures inside callbacks were suppressed (I was left with a test that timed out for no apparent reason and I had to do some digging before I even found the suppressed errors). I originally filed this as #3519 but it turned out to be a duplicate of #2059. If your async code uses promises instead of callbacks, this issue won’t affect you. If it’s possible, a nice way to escape the problem is simply to refactor to your old callback based functions to use promises instead. However, if you absolutely need to test functions with callbacks (e.g. maybe you just inherited a big legacy code base and you want to write testcases before you refactor, or maybe the callback API is used by third-party code) then the status quo for Jest seems to be that for any assertion inside an asynchronous callback you have to wrap the assert inside a try/catch that explicitly calls fail instead (like they do in lerna unit tests here). Another workaround would be to temporarily wrap the callback in a promise inside the testcase:

test('testedFunc calls its callback with the correct value', async () => {
  expect.assertions(1);
  const actual = await new Promise(resolve => testedFunc('some input', resolve));
  expect(actual).toBe('foo');
});

As seen in the commit count chart earlier, there seems to be a lot of momentum behind Jest right now. This is also reflected if you plot the number of active developers in each project (i.e. for every date X, the Y-value is defined as the number of unique e-mails seen in git log from X - 365 days up until X). That plot looks like this:

The number of developers involved in Jest increasing very quickly, they must be doing a lot things right when it comes to community building. I also looked at “the number of people which more than 50 commits” for each framework and Jest comes out highest there too with seven such developers so this is not all “one-off drive-by patching” either.

Conclusion

I was surprised that I hit so many little bugs / issues when trying to use these testing frameworks. Maybe the best choice is to do like TJ and switch to Go? :-P Staying with Javascript, I think it’s hard to ignore the momentum behind Jest. It would be awesome if someone fixed #2059 though. Then again, Mocha seems to work well enough for a lot of people.

1: https://en.wikipedia.org/wiki/Analysis_paralysis.

2: Technically, wtgtybhertgeghgtwtg is anonymous but "their contribution history smells like they work for Facebook" and amasad has left Facebook.

3: The median "Issue Resolution Time" and the fraction of open issues among all issues, are both computed using the excellent service Is it maintained?.

4: Contributors need to sign a Contributor License Agreement (CLA) before patches are merged into Jest, and along with the BSD-3-Clause there is a patent grant which can be revoked by Facebook under some circumstances, see details on Wikipedia.

AWS S3: Introduction

Mon, 06 Mar 2017 00:00:00 +0000

Amazon S3 (Simple Storage Service) is a file storage web service offered as part of Amazon Web Services (AWS). Files are uploaded using HTTP REST, e.g. via a super basic web UI hosted by AWS S3 or via third-party upload tools. It’s of course also accessible programatically via countless of SDKs available for all languages/platforms. If the files are marked “public” they can optionally be served as a static website. It’s possible to setup logging to track how often each file/page is loaded, custom 404 pages and you can configure custom HTTP response headers including cache control and CORS etc. It should be noted that while the REST endpoint offers https, the static website endpoint is http only. S3 can also act as a BitTorrent seeder, just append ?torrent to any file URL.

S3 first launched in 2006 and since then many competitors has sprung up (often API compatible with S3), for example Google Cloud Storage and Azure Storage. S3 has a 99.9% SLA (i.e it can be down up to 60 * 24 * 30 * (1 - 0.999) == 43.2 minutes per month) but this only means that you get a 10-25% discount the next month if the service has been down more than that. S3 is billed both based on how much you store and how much that content is downloaded. It does not offer any automated cost control (hard quota based on pre-paid amount), but you can configure AWS to send you an alert if your bill exceeds a limit you set. There is a billing dashboard that gives you a quick overview, and if you’re on the free tier it will tell you how many free S3 PUTs / GETs you have left each month etc. It’s also possible to create “Requester Pays” buckets where the person downloading pays for the request, or optionally pays for the request plus some price that is paid out to the bucket owner via AWS DevPay.

Another basic way to interact with S3 buckets are via the AWS CLI utility. If you’re using Ubuntu you can usually just do sudo apt-get install awscli to get it, or if you for some reasons need the absolute latest version you might want to do sudo apt-get install python-pip followed by sudo pip install awscli instead. Once you have that installed, you can list the contents of public S3 buckets. For example, there is a public S3 bucket called “flaws.cloud” and you can list the contents in it by running:

aws --no-sign-request s3 ls flaws.cloud

By default the AWS CLI utility will sign all requests using your credentials, but you can access public buckets without even having your own credentials if you pass the --no-sign-request parameter like above.

To create your own S3 buckets you need to register an AWS account (there is a free tier, that you can use for up to a year, that includes basic variants of both S3 and EC2 and bunch of other stuff). Once you have registered, go to the AWS management console and click your name in the top-right corner. From that dropdown menu, select “My Security Credentials”, and then click “Continue to Security Credentials”. You really don’t want AWS credentials stolen, so before doing anything else, it’s a good idea to enable two-factor authentication (you can use any TOTP device like a smartphone running Google authenticator). To do that, expand “Multi-Factor Authentication” and click “Activate MFA”, select “A virtual MFA device”, scan the barcode etc.

To access AWS programmatically or via the AWS CLI utility you need to create an access key, you can do this directly from the “Security Credentials” screen on your account but such a key would have complete access to your entire account so instead it’s recommended that you create separate sub-accounts using Amazon IAM and that each such account is given only the specific access that it needs. To do this, click the “Services” link in the top left corner and select “IAM”. Then click “Users”, then “Add User”, check “Programmatic Access” and in the “Permissions” step, use “Attach existing policies directly” to give the user “AmazonS3FullAccess”. For a production system you’d probably want to attach the permissions to a group instead of course. Once the user has been created you get an “access key id” (starting with “AKIA…”) and an “Secret access key”.

Once you have an “access key id” and the corresponding “secret” you can configure these in the AWS CLI utility by running aws configure. This will make the specified key the default key, so all commands submitted to AWS using the CLI utility will be signed by this key.

Once you have a default key setup, you can find S3 commands described in aws s3 help and more advanced S3 API calls described in aws s3api help. For example:

# Try to create a bucket called "test" (hits BucketAlreadyExists since name is taken)
aws s3 mb s3://test

# List all your buckets
aws s3 ls

# Upload a file to a bucket
aws s3 cp file.txt s3://bucket-name-here/

# Upload a file to a bucket ... and also make it world readable
aws s3 cp file.txt s3://bucket-name-here/ --acl public-read

# Show the contents of a text file stored in a bucket
aws s3 cp s3://some-bucket-name/object -

# Remove a file from a bucket
aws s3 rm s3://my-bucket-name/file.txt

# Remove a bucket (only works if bucket is empty)
aws s3 rb s3://bucket-name

The above commands all operate on the default region you specified when you ran aws configure but you can pass --region foo to use another region. Also, if you want, you can configure several keys in the AWS CLI tool, just use a --profile profile-name-here parameter to select the key you want to use (both when you run “aws configure” and then every time you invoke a regular command).

When interacting with the S3 REST endpoint, you can two different URL styles, called “Virtual Hosted-style” or “Path-style”. In the former, the bucket name is part of the domain, and in the latter it is the first part of the path. For example, if you have uploaded a file to stuff/items/file.txt in the bucket situla-deadbeef in region eu-west-1, it can be referred to using either one of the curl command URLs below:

$ aws s3 --region eu-west-1 mb s3://situla-123
$ echo hello | aws s3 cp - s3://situla-123/stuff/items/file.txt --acl public-read
$ aws s3 cp s3://situla-123/stuff/items/file2.txt -
hello

# First curl the file using "virtual hosted-style URL"
$ curl https://situla-123.s3-eu-west-1.amazonaws.com/stuff/items/file.txt
hello

# And now curl the file using "path-style URL"
$ curl https://s3-eu-west-1.amazonaws.com/situla-123/stuff/items/file4.txt
hello

One thing to keep in mind is that you might run into SSL certificate name mismatch when using “virtual hosted-style URLs” for buckets that have . in their name.

$ aws s3 --region eu-west-1 mb s3://fail.pail
$ echo hello | aws s3 cp - s3://fail.pail/file.txt --acl public-read

# Beware that bucket names that contain "." gives you a SSL name mismatch error
$ curl https://fail.pail.s3-eu-west-1.amazonaws.com/file.txt
curl: (51) SSL: certificate subject name (*.s3-eu-west-1.amazonaws.com) does
      not match target host name 'fail.pail.s3-eu-west-1.amazonaws.com'

# But loading the same file over HTTPS using "Path-style URL" works fine
$ curl https://s3-eu-west-1.amazonaws.com/fail.pail/file.txt

And you are not allowed to create buckets with names that cannot be used in a hostname, for example foo..bar, foobar. and .foobar are all illegal bucket names. Bucket names may consist of 3-63 characters.

It’s also possible to make an S3 bucket listable even for anonymous users (for people that knows the bucket name). However, just like it’s a good idea to keep directory listing turned off on webservers, most of the time it’s best to not allow anonymous users to list your bucket contents; this is true even if many or all objects (files) in the bucket are marked with ACL “public-read”. If you want to make a bucket listable for anonymous users anyway, you can do it like this:

# Try to list bucket contents anonymously
$ aws --no-sign-request s3 ls s3://situla-123

# Make the bucket contents listable for anonymous users (previous command now works)
$ aws s3api put-bucket-acl --bucket situla-123 --acl public-read

Other useful things you might need to do are:

# Upload (or update if it already exists) a whole directory to S3
aws s3 sync ~/mydir s3://my-bucket/

# List files inside a bucket recursively
aws s3 ls --recursive s3://situla-123

# Make a public file private again
$ aws s3api put-object-acl --bucket my-bucket --key stuff/hello.txt --acl private

# Create a presigned URL for public access (valid for 30 seconds then stops working)
$ aws s3 presign s3://situla-42/private.txt --expires-in 30

To access your bucket as a static website you need to enable “Static website hosting” for it. The easiest way to do this is via the AWS console web UI, just select the bucket and switch to the “Properties” tab and you’ll find it there. The main difference compared to HTTP GET operations issued to the API REST endpoint is that the static website endpoint will serve a HTTP 403 Forbidden that contains HTML code instead of an XML error message. For URLs that point to files that doesn’t exists, the website endpoint returns a 404 document that you can configure when you enable “Static website hosting”. Unlike the REST endpoint, the website endpoint can also return redirects. You can access the website endpoint via:

http://foo-bucket.s3-website-REGION.amazonaws.com/prefix1/prefix2/file.txt

When talking about the file s3://my-bucket/subdir/blah/file.txt, the subdir/blah is a prefix to the full key name which is subdir/blah/file.txt. There is no concept of folders or subdirectories etc even though the simple web UI that S3 hosts kind of makes it look like S3 has folders. The full key name is a string of Unicode characters for which the UTF-8 encoding is at most 1024 bytes. While all Unicode characters are allowed, certain characters like ?, & and + needs URL encoding so they are a bit messy (especially, if your application has a bug and you need to do some trouble shooting using curl etc). The AWS S3 documentation has a list of characters that you should avoid in key names if possible.

In the properties page of the bucket inside the AWS web UI you can also enable versioning of all objects in that bucket. Once you’ve done this, the change history of each file is saved. You can then do:

# List all versions of the object specified by --prefix foo
$ aws s3api list-object-versions --bucket bkt \
        --prefix private.txt | jq -r '.Versions[] | .LastModified + " " + .VersionId'

# Use .versionId printed above to retrieve the old version of the file
$ aws s3api get-object --bucket bkt --key private.txt --version-id VERSION_ID foo.txt

Finally, I would like to recommend the flAWS challenge, it’s a fun little challenge where you need to fiddle around with AWS CLI tool a bit to gain access to the next “level”.