Knowledge Drop: In-Browser DuckDB

A taste of Vanilla JS topped with DuckDB all in a Vite shell

We’ve been keeping tabs on DuckDB since the early days of The Drop, and I finally indulged in a recent experiment (GH) that is not-at-all (really) tied to WebR. This means I can justify pointing to it, along with some extra commentary here, and “double-dip”. I’ve been publishing one Drop and one “experiment” almost daily for a few weeks now. A reason for both is that I’m learning/dissecting WebR and trying to pave the way for others, so the road is not as bumpy for them; also, so there’s no excuse to be locked into any single vendor or idiom. I’m calling them “experiments” b/c that’s just about what they are. Anyway, I’ve avoided double-dipping since WebR is super niche, and I try to make the M-S Drops general purpose. I think this one counts, especially if I help you get started a bit more than I do in the experiments themselves.

What you’ll find in the aforelinked resources is a mini-tutorial on how to use SQL operations on data in a JavaScript context. The context is specific to the browser, in this case, but it looks like the library I’ve picked to demonstrate WASM DuckDB will also likely work in Node or other non-browser JS contexts.

NOTE: I’ll likely do a normal Drop on Friday vs a “Weekend Project Edition” since this one most certainly counts as a WPE IMO.

What You Need To Get Started

Photo by Arisa Chattasa on Unsplash

The experiments make some assumptions that folks have at least been following along for the past few weeks. I shall make no assumptions here, and provide some foundational information to help you work through JS DB ops.

The core of the project is Vite — a modern build tool and development server designed for web applications, primarily focused on improving development speed and experience. We’ve covered such tools before, but no Drop has focused on Vite-proper.

It was created by Evan You, who also created the Vue.js framework. Vite aims to address some limitations and performance bottlenecks of earlier build tools and development servers, such as webpack and Parcel. Vite leverages features like native ES modules and optimistic updates, allowing for faster development cycles by reducing the time spent on bundling and refreshing the page during development. Vite supports a wide range of web development features, including hot module replacement, CSS pre-processors, asset management, and framework-agnostic support for JavaScript libraries and frameworks like React, Vue, and Svelte.

In addition to its development server capabilities, Vite provides a highly efficient production build system. By leveraging the Rollup bundler under the hood, Vite ensures small and highly optimized production bundles. It supports features like code splitting, lazy loading, and tree shaking, which help to minimize the application size and improve performance. Vite's plugin system allows developers to extend its functionality and integrate with other tools and libraries. The simplicity, speed, and versatility of Vite make it an attractive choice for modern web development, especially for developers working with JavaScript frameworks and libraries.

Vite has excellent docs, which I encourage you to keep in a dedicated Space (you are using the Arc browser, right?).

My experiments also, now, use Lit components. Some recent-ish Drop topics may be starting to make more sense — now. We’ve discussed WebComponents before in the Drop, and there are no crazy complex ones in this example. Two message boxes and something that draws an HTML table. No complex interactions or messaging, too. Just enough to abstract the complexity away from the main page.

There is a touch of WebR in this experiment, but only to prove to the WebR folks that it can coexist with DuckDB.

Finally, there’s DuckDB WASM, but not by itself.

The folks at Observable make the standard library that powers their platform freely available to all. In it, they have a powerful abstraction over DuckDB, which makes it super easy to use in-browser.

I encourage folks to hit all those point-resource links before reading further here, or going through the experiment.

Getting Up And Running

Photo by Miguel A Amutio on Unsplash

Hopefully, you did hit up the Vite site and get Vite going on your system.

After cloning the repo, you should be able to npm i in the project directory to get all the libraries and resources installed.

There are a few things I’ve added to vite.config.js that are worth mentioning. First is that the “entry” point to this “app” is “index.html”. This lets Vite discover more resources to bundle for us.

We let it know, via “formats” that we’re working in vanilla JS bu tare also targeting new-ish browsers, since we expect support for ES6 modules.

It uses “rollup”, something I avoided in early experiments, but I think Vite reduces the complexity enough to warrant putting it in. The “terser()” is a fancy word for “minify”, so our code is going to get mangled before we publish it. We’re also telling Vite that we are not necessarily just using local “node_modules” (something I also avoided early on to reduce complexity), and asking it to resolve all resources specified with URLs.

There is a fancy “topLevelAwait” plugin that fixes some problems caused by some JS modules, and we also have a plugin to ensure any webfonts we may specify are shipped with us at production time.

There’s also a fancy “server” section because WebR really needs those headers to work well.

The Drop has covered the “Just” tool (a “make” alternative), and I highly suggest poking at it, since it shows the use of variables and command execution hiding. I usually type “just render build rsync github” (all one command) when ready for publish time. It saves tons of time.

The “src” directory has all the core content, and anything referenced there or in “public” get copied to the “dist” folder that appears. after a “just build”.

I recommend using Visual Studio Code, and installing the Vite extension. You get superpowers when you do. Microsoft’s guide for doing JS work in VS Code is also pretty darned good.

Finally, while I was using my own custom hack to write in markdown, you can just edit “index.html” to tweak/play. If you do want to work in the land of markdown, then edit “index.js” (it pairs directly with “main.js”) and run “just build” when you want to see your changes come to life.

REMEMBER: it’s a git repo, so commit early and commit often! Branch all the things, too! You can’t break my repo, and you can recover from any major edit misses by good use of what git is good at.

What You’ll Learn

Photo by MD Duran on Unsplash

My hope for the WebR experiments and this experiment is that any puzzle pieces surrounding how modern browser-targeted development works that are still all over the floor and table get put together into a cohesive picture.

Now, there is plenty of “work” (the bill-paying kind) to be had in server and desktop land when it comes to data science-y stuff. But, there’s quite a bit going on in WASM/browser-land in the real of data science, and I really don’t want it to be an exclusive club of a few uber-talented folks. This tech is truly accessible, but many resources seemed aimed at folks who have been doing JS for ages.

Trust me when I say you can do this if you would like to learn it.

FIN

So, once more, you’ve got the experiment (GH) to poke at, some resources to set up to do so, and hopefully the right about of boilerplate to get you started. If you’d like a copy of this project with no “WebR” bits in it, drop me an issue or comment, and I’ll get another one up ASAP. ☮