Featured image

As per tradition, the image of this post is generated by DALL-E, and it well represents the content of the article: an investigator (aka Hugo) with his magic lens (LyraSearch) searching for interesting content in a galaxy, far, far away.

What I love more about the Christmas holidays 🎅, apart from the tremendous amount of food, is the spare time I have to draw a line from the year that just passed and, finally, go through the most exciting stuff I encountered along the way, one of them is LyraSearch, a javascript search engine, which is a: “🌌 Fast, in-memory, typo-tolerant, full-text search engine written in TypeScript.”

I knew the project better when we invited Michele Riva, the inventor of this project, to our company Podcast, it was a great episode with many interesting discussions, sadly it is just in Italian but with the help of the AI and the automatic translations you can try to follow it anyway, In any case, you’ll find many contents and talks in English from Michele and for sure you can see him speaking to the next conference around you.

What is Lyra? Link to heading

Taken from the website:

“Lyra is an immutable, runtime-agnostic, edge, and in-memory full-text search engine that works both on client and server.

Through implementing an optimized prefix tree and some clever tweaks, Lyra can perform searches through millions of entries in microseconds.”

And it only requires a Javascript runtime to work; this means that any browser can be super-powered as a complete full-text search engine; if you combine it with the incredible market rising of frontend frameworks and static generated sites, you can imagine how this kind of project is needed now.

Which static site today doesn’t need a simple and integrated search engine? And what, if you can have it without paying for server-side computing power? This is precisely the space of projects like Lyra; you can self-host your site along your search engine in the same space, hopefully for free.

I liked Lyra’s attention from the ground up to the performance and the algorithms’ design, which is Michele’s passion, as he told us in the podcast.

Lyra is an open-source and modern Typescript project, easy to extend thanks to its plugin architecture; you can already find several plugins on the Lyrasearch organization.

It is still in its infancy, so it is easy to contribute, extend, and adapt it, and why not, dig into the code to learn how to build a search engine.

At the moment is still quite limited in features (no facets, no data shards, no search DSL…), and the data structures are not yet stable, but for the most simple use cases, it works out of the box, and considering the traction that is having I bet a bright future for the project, maybe one-day feature compelling with big commercial products like Algolia or Doofinder.

A project worth mentioning very similar to Lyra is Lunr (”A bit like Solr, but much smaller and not as bright” from the GitHub project description), it is open-source and Javascript based, and it shares with Lyra most of the search limitations I mentioned before, and it seems to be not very active nowadays; the last commit is dated Aug 19, 2020, that can also mean that the project reached the desired feature state and it will not advance more, or that is just more or less abandoned to its destiny. Anyway, this project works well too, and it is battle-tested in thousands of installations and easy to adopt and use in production. (Ed: We use it on our company website).

What is Hugo? Link to heading

“Hugo is one of the most popular open-source static site generators. With its amazing speed and flexibility, Hugo makes building websites fun again.”

And it is also the engine of this blog and many, many others.

As the name suggests, it is written in Go, so to use it, only a single binary is needed, which is responsible for serving the development environment and building up from the sources to the static files.

What I like more about Hugo:

  • No dependencies, just a static binary (multi-arch)
  • Very fast (serve, build) and intuitive
  • Native support for CMS-like concepts like content types, taxonomies, menus, etc.
  • Hugo Pipes - a very powerful way to process assets (including Javascript transpiling)
  • Many free themes to use (if you, like me, are CSS-🐐) - I use Hugo Coder; big kudos to the team.

Well, as it can be easy to imagine, a static site is just a bunch of HTML, CSS, and Javascript files, and Hugo comes without a native client-side search engine solution; the documentation point to external resources to implement by yourself what you need, most of them are based on Lunr.

And we have finally arrived straight at the point 🎉

Hugo + Lyra Link to heading

Let’s start by sharing the project’s repository: https://github.com/paolomainardi/hugo-lyra

You can also test it on this blog; not really needed because, at the time of writing this post, there are just 4 contents (including this one), but I thought that was better than creating a fictional demo site, the blog source code is also hosted on Github, so you can also see it how is integrated.

It is implemented in Typescript, licensed under the Apache-2.0 license, and I tried to reuse many of the practices used on the original project (QA tools, test runner, coding style, etc.)

It comes with server-side Node implementation, also exposed as a CLI and a client-side library, which exposes some helpers to getting started quickly.

This time I invested more than a few seconds to write a decent README, so instead of writing how it works again here, you can rely on it; it will hopefully be helpful.

Instead, I want to share what I learned building this project:

  1. My feelings on Typescript are not yet clear; I like the idea a lot but am scared of the pile of layers on top of the bare language; sometimes, debugging types or finding them (if any) is frustrating; maybe it’s also a matter of being used on it, as my daily work does not involve anymore writing a lot of code in TS.
  2. Esbuild is just awesome, I never had the chance to try it in a real-world project before, and I guess we are finally over the Webpack configuration monstrosity; much respect for it, but DX is all on the Esbuild’s side.
  3. Node-tap is the test library to use with Node; easy, straightforward, with no frills, and the maintainers are awesome and responsive. Another project I didn’t know before starting building.
  4. ESM - CJS is still a big giant mess, and the pain of trying to build isomorphic code, while wrestling between CJS, AMD, UMD, ESM, System, and IIFE formats and the combination of stuff to put on package.json.
  5. Release-it is a lovely project to release NPM projects and taking in charge of all the tedious tasks involved in the release process, like tagging, Changelog file updates, release on NPM, and release on Github; here is the GitHub workflow.

Conclusions Link to heading

I had a lot of fun building it!

I finally had the chance to refresh my knowledge of a rapidly evolving Javascript ecosystem, not only by reading articles or quickly testing things but building from scratch a new one.

Nothing is more rewarding than envisioning building and releasing a product with your hands, even the smaller one, contributing to the open-source ecosystem and being part of a community.

My advice is to find a topic or a project you like, you want to improve yourself, or have always wanted to learn and start building; we live in such a wonderful era for developers; we can build unimaginable things and transform thoughts into living things potentially used by thousands, millions, or people in a matter of seconds.

There are still tons of things to do here, like multilingual and dynamic indexes; come to say hi, any contribution, even the typo, will be welcome.