In this blog post, I document the decision to create a command called Debug to Nun-db. As we scale on usage and features, more complex problems are showing up, and it is about time to start adding features that will help manage Nun-db as Database administrators. If you unintentionally got to this post and want to read more about Nun-db, go to Github repository will be much richer information about the Database. The rest of this post will be focused on people familiar with Nun-db code.

What is the debug command?

The debug command holds admin queries for Nun-db, like, for example, checking the messages that are pending replication from a specific node in the cluster.

Why do we need a debug command?

We may need to check some debug information while developing or managing Nun-db in production. The initial command Debug implementation will help find and fix a problem that only happens in production clusters with multiple nodes and several concurrent users.

Like in the next print, I am showing the number of pending messages growing over time, that should not happen since all messages should be replicated without any problem:

The implementation is simple, there will be a command called Debug to be used only by Nun-db database administrators, and this command will be extended to support all the debug cases we find needed.

Why create a command for debugging instead of a command to check the pending messages directly?

One of the core values of Nun-db is simplicity; therefore, I want to keep the external command surface small. But, to be honest, I think there are too many commands already, and I want anyone to learn the NunQuery language in 2 minutes, which is only possible if there are only a few simple commands to learn.

Oh, the other hand, people who debug Nun-db are not beginners, so the debug subcommands don’t need to be simple.

Auth level?

Only admins should be able to execute the debug commands.

Possible Consequences

There may be several consequences because of this decision.

1. The debug command became the do everything command: I can see that happening. It may become a place to add all sorts of workarounds. To prevent that, I will restrict sub-commands to read-only. No write or change must be done in the debug command. Here is the comment on code.
2. Too many subcommands may lead to a new language: I don’t really care about this problem now, as far as the external API used by users is simple, these subcommands commands can grow.

Context

We are facing the problem mentioned (number of pending messages growing where it should be 0 all the time, and we have no good way to see what messages are failing to replicate) in prod, and we need a scalable way of debugging and solving this kind of problem. So I decided to add a command to help build solutions for them while at the same time keeping the external API simple.

Sample usage

debug pending_ops
#result
pending_ops #list-of-pending-ops#

How to implement debug subcommands

//In process_request find
Request::Debug { command } => apply_if_auth(&client.auth, &|| {
//...
            match command.as_str() {
                "pending-ops" => {
                    let pendin_msgs = dbs.get_pending_messages_debug().join("\n");
                    log::info!("Peding messages on the server {}", pendin_msgs);
                    match client
                        .sender
                        .clone()
                        .try_send(format_args!("pending-ops {}\n", pendin_msgs).to_string())
                    {
                        Err(e) => log::warn!("Request::pending-ops sender.send Error: {}", e),
                        _ => (),
                    }
                }
//... change the pattern match to add your command

Why not create a debug module or add this method to the Database and only call them from the process request?

Keeping in the process class, for now, is an excellent way to protect it from growing too fast, and if we feel I need to refactor the code any time soon, it is a sign that I probably went too far and should stop adding commands to Debug.