It has been a year since I open-sourced NunDB. That happened when https://www.teknisa.com invited me to give a talk about real-time processing. At that time, I worked on NunDB for over six months, and I thought it would be an excellent opportunity to announce the project. It was called “Freira-DB” back then, and the goal of this post is to update the NunDB and what has happened during this one year.

What was the goal for the year?

The goal for the year in my yearly presentation to myself was “Release the NunDB 1.0 and use it in e-belle/gerir-me” I still if you do not know E-belle/Gerir-me are the products of my company (http://www.vilarika.com.br) they are SaaS where we provide a management system for small business in Brazil (That is why the side is all in PT-BR). For a long time, we had the necessity of using a real-time database. Still, I never wanted to use Firebase or other alternatives, so my goal was to introduce my new database to solve that problem. I needed it to be stable before doing that (cause you to know the product is used, and I did not want to cause downtime with the new changes). Before the end of this post, I will share how this plan went. If you cannot wait to see it, go to the “Going live with NunDB” subtitle.

Renaming

The first big thing that has happened was the renaming from “Freira-DB” to “NunDB” I loved the Freira-DB name, but talking to people out of brazil and it was hard for them to get the name and even was hard pronounce it. So I decided to take a advise from a friend @Iucasazevedo and rename it to NunDB, more straightforward, cleaner, and much easier for the people to get the same name and made simple for the people to get it right. After renaming, it was time to start trying it out in some places other than in my examples, so I decide to make an NPM package out of the JS code I had made already.

The npm package and the JS version

The fist public lib I publish was to npm(https://www.npmjs.com/package/nun-db) that for me was clear the first step, once most of the use cases I had was in the front end, and an npm package was naturally the decision to start, I also needed a scala lib so I could integrate my backend (pure scala) with NunDB but it still not ready to publish, so I keep it only internal ( Let me know if you need I can work and get it out there), as I think the scala community may not be so interested in this kind of integration at the moment.

After getting it to NPM, I wanted to make it “deployable,” so I decided to build the Docker image and get it out there.

Release docker image

One of my big goals was to create an open-source alternative to Firebase (the database). I want to be just as simple as it, but 1: Free, 2: Open Source, 3: Small and direct. A docker container is a way lots of databases providers make their versions available to the community, which seemed the correct direction. By the time I did the talk back in 2019, I had a docker image already running, but today’s one is 30x smaller and uses 80% less memory for the same data, so I didn’t even consider that a valid docker version at that time.

So there was I with JS version running and an easy runner docker, so what is next? Get it to my own company’s product needs usage. The first real-world requirement came to life when I would not be able to use the WebSocket or TCP Socket communication seems less because of my product’s legacy code and the absence of great support for such protocols in the pre-existent code, so I came up with new http drive.

Implement http drive

Before going live with NunDB, I needed to add the http drive, as of the integration (with my legacy scala system), and it did not have native support to WebSocket, the http connection interface was needed to do it.

Security updates

We needed to add passwords for different databases and make it possible to hide the admin password only in the backend. Those changes were introduced after I did the initial model to integrate with my SaaS solution. At first, we required all commands to auth with the admin password. That was not a good idea, mode commands can require only a DB auth to execute, and we can save the admin auth from being used only from the backend to the NunDB engine. Therefore one user cannot disturb others (by calling the NunDB API manually). This helped a lot to make NunDB easy to make safe.

Snapshots

Initially, NunDB was all on memory, but soon after starting the POC, I saw I would need to have some more durable data because I would restart the process often (while doing some changes on the fly). My idea for that was to add a command to snapshot the database to the disk with a command. Therefore, I would not need to worry about going to the disk for each command but instead only when the DB user wants the data to be stored in some hard disk to be restored if needed.

Going live with NunDb

The big thing to the year was going live with NunDB in gerir-me. All happened when I release a new mobile app that the users use to control their commission and calendar individually. Of course, the calendar needs to be real-time. That was the perfect integration point. I put out a blog post how it was implemented on here

Add command lines utilities

It turns out that to work in the day to day with NunDB, I needed a command-line tool I could use to save time, I started to create sh files to do stuff, and so I decided to make at last a small subset of command built-in, so any new user can use it for the same things I do.

For that, I have come up with the following commands.

Bug fixes

There were only a few bugs I needed to deal with until now. I credit this to Rust been such a good language and environment to create stable software, the one I had was fun to fix.

1) Database stop responding to the http request. That was the only time I had an unexpected behavior, and it turned out to be a bad error handling. (At some point, I even want to go back end get rid of all the unwrapping of my code) here

2) Using too much memory here

For a js developer with more than ten years of experience, writing this thing for the first time and going live with it in my real business and only having a few problems were quite surprised, to say the least, after one week of it running, I was looking at the logs and guessing I had forgotten to release the new version with NunDB included or that no users were using the latest version, I was quite a positive feeling when I noticed that it was running all the time. There was no fatal, no error, and no user has reported performance problems so far (and at that time, there had been +800 different users that had used that piece of code).

Horizontal replication

It sounded clear to me that Nub-DB needed to be horizontal scalable at the heart if I want to get anyone to use it, that is not only me. Horizontal replication can help me get high availability of what is vital to anyone operating in a real-time database.

I decided to implement this feature as soon as I start using it in my personal SAS product, it is not a must-have to my product itself, but it is a nice to have and would be again from the ground up the other services I use.

It took me more than three months to make it “talk,” there are several problems when you decide to implement a horizontally scalable database. How to create a cluster? Where do you write goals to? Where reads goals to? And many others. There are great books that help with the decisions, and there is no perfect approach to any implementation. Leader election is a big enough problem that will take too much time if you don’t take a pragmatic approach.

I have recently merged to branch with horizontal replication (even tho it is not fully implemented yet). I think I have enough that I can start using it in my own product and start fixing the problems I find when using it.

I need to improve the fail tolerance and implement some operation log (current operation only replicate data to the live nodes when the change occurred). This implementation must not be that hard once we don’t need to store the operation itself, only the keys that changed and use the primary as the source of truth and send the data to the new node once it comes online (soon, I will be working on this :) ).

Today, we can replicate data horizontally, which is already colled, and help with the initial proposal of making it possible to update without downtime.

What is next?

It is always hard to say exactly what is next when we build something like NunDB (an experimental project), but I can try to predict what is next to come. I intend to make NunDB became more popular and get others to use it, and help me to make it better. Therefore a few things are needed. Next, I am presenting some of my take on the next steps to NunDB.

Helm chart for kubernetes?

Deploying any distributed database is complicated, an helm chart would be a great way to simplify it and not scary people that wanna try it in many different ways.

Improve documentation (a lot)

Documentation may be boring to do, but it is such an important thing we all need to get used to doing it. We build things for others to use and to make it successful, and we need to make it so they can understand how to use it, and how to set it up and how to try it, and how not to try it. Today NunDB, documentation is a basically nil. I think next year will be basically about working on examples, documents, tries, helping people to try it out in many ways possible.

Examples examples examples

Working on examples, I think, will be the next big thing. I want to show how NunDB can be used in many different problems and solutions. Therefore the best way to do it is by doing it. I have already done it to react, vue, and node.js, but the example is never enough. I love learning from examples because it quickly allows me to see what I want to experiment in action with some good portion of code.

Monitoring

Observability is such a big thing for any infrastructure solution, and I bet it will be for NunDB. I care a lot about metrics and performance improvements; therefore, I want to instrument NunDB much more to become observable in many different ways, maybe implement a Prometheus collector to store some metrics. I would call it a success if so.

Early adopters?

Let’s see, and maybe I can bring someone building something new to try it out. Perhaps I can give some consulting and deliver it as part of the infrastructure. I am not sure how it is going to happen yet. I know I want to get there. Soon I will figure out how.

Today I have an cluster running on linode where I can easily add instances and start bringing some users to see how it scales, but this is subject for a future post, how do we make this simple DB a platform others can simple use trust and scale.

Reviewers

@danielcn