Letting the moose loose

Our mission is to bring incredible developer experiences to the data stack. Today we’re excited to share our first step in that direction with the Alpha release of MooseJS. MooseJS is an open-source developer framework designed specifically for your data & analytics applications.

Why might the world need another developer framework?

For decades, software developers have leveraged front-end, back-end, and full-stack frameworks to build their software applications. These frameworks abstract away middleware and complexity, and let you, the developer, focus on the unique and interesting aspects of your application. Modern web development frameworks like NextJS from Vercel can get you up and running with a full-stack web application in minutes!

On the flip side, the data and analytics ecosystem is notoriously complex, specialized, and difficult to develop in using modern software development best practices (like version control, CI/CD, automated testing, local-to-cloud development, etc). With MooseJS, we wanted to take the best parts of modern application frameworks and bring those to bear on the world of data and analytics applications.

MooseJS is designed to power any application where collecting data and extracting insights from that data is at the heart of the application. For example:

• When you're using LLMs to automatically generate queries and insights for your users
• When you need to aggregate batch or streaming data to power interactive dashboards
• When you want to curate data and analytics products in a data lake or data warehouse

Some core principles for Moose

In order to bring the best of software development frameworks to the world of data engineering, we’ve leaned on a handful of key, core principles for Moose:

1. The developer experience should match, if not surpass, the developer experience for web and mobile apps
2. Developers should get up and running in seconds to minutes, not hours to days to weeks
3. Adopting software development best practices should be dead simple
4. The infrastructure and middleware complexity of the data stack should be abstracted away
5. The abstractions should be intuitive, easy to use, and available in popular developer languages
6. The underlying data stack should have sane, batteries-included middleware built in, and should be architected to the highest standards of modern data engineering

So what exactly is Moose?

So what exactly is Moose, and how does it work? Using a simple set of primitives—coded in Typescript, using your IDE of choice—you develop the core structure and business logic of your application or service. Then, as you build and deploy your code, Moose interprets these primitives and configures your entire production data stack: typed and coordinated automatically end-to-end, from data capture through data consumption.

For example, you can use the Data Model primitive to define the core schema of your data with a Typescript interface. Based on this code, Moose will automatically configure middleware and infrastructure that might otherwise take hours or days of developer time, including:

• Spinning up an API endpoint to ingest data typed to that schema
• Creating a streaming buffer behind that endpoint to handle peak load (Moose uses Redpanda by default)
• Configuring the database tables and views needed to land, store, and surface that data - again typed to your schema (Moose uses Clickhouse by default)
• Spinning up an API endpoint for downstream applications to consume the data
• Automatically generating SDKs in the languages of your choice to embed in your upstream and downstream applications to facilitate sending/receiving data to/from your Moose data stack

You can then use the Flows primitive to map and transform your data row by row with automatically orchestrated Typescript functions. You can use the Aggregations primitive to manipulate data across rows and tables with reusable SQL or Typescript functions with dataframe-like capabilities. You can visualize your data with auto-generated React components that can be easily embedded into your React framework of choice, like NextJS or Remix.

In addition to this core, end-to-end data stack functionality enabled by the Moose primitives, some of our favorite bonus features with MooseJS are:

1. Hot-reload local dev server - MooseJS lets you run your entire data/analytics stack locally, and see the impact of changes in real time as you edit code, just like developing a web application
2. The automatic migrations and change management - Moose automatically manages schema evolution across your data stack, so you don’t break upstream and downstream dependencies as your application evolves
3. Moose templates - this library of pre-configured, plug-and-play applications built with MooseJS and NextJS make it super easy to get up and running with your first application powered by Moose, and they’re still fully customizable to meet your specific needs

Under the hood, MooseJS is built with Rust, with an architectural eye towards speed, scale, and security. And while Typescript is the first language Moose supports for development, we expect to support more in the future (eg. Python).

Wrapping up

Ok, that’s Moose in a nutshell ;) We’re tagging this release as alpha. MooseJS isn’t ready for production primetime just yet. We’ve been using it to build our own internal data-intensive applications and partnering with design partners who share our vision to simplify the developer experience when building data-intensive applications. We’d love for you to get your hands on Moose too - please check out the docs and let us know what you think by joining our community . If you’re interested in enterprise solutions, commercial support, or design partnerships, then we’d love to chat with you. Simply get in touch here.