Welcome to my yearly post about how to become a better developer; here, I share what I have learned in my last 16 years working as a Software Engineer and what I do to make myself more productive in my full-time job as a Principal Architect in a fast-growing American Startup, running my own SaaS business serving over 6k users monthly, teaching in-person Data Structures at the FasF Faculty (in Brazil) and working on my Open Source project Nun-db in my free time, I have to learn how to do things efficiently and that is what I share in this post yearly.

This year’s post is divided into three major categories: Automate, Coding Like a Pro, and Productivity Tricks,

• Automate
  • Learn and Use Makefiles: they are awesome
    • Why are Makefiles awesome?
    • Simple example
    • More complicated example
• Coding like a Pro
  • Latency is not 0! Learn to deal with it.
  • Learn how to write compilers
    • 2022 Compiler: POC migration from Mongo to Elasticsearch in 2 days:
    • 2023 Compiler: Compile Cypress.js tests to Cucumber
    • Closing the subject
  • Use AI Copilots
• Productivity tricks
  • Try Demo Driver Development
  • Teach
  • Read
  • Take care of your sleep quality
  • Conclusion

Automate

Learn and Use Makefiles: they are awesome

Makefiles are a great way to automate your workflow. Every day, you may do repetitive tasks, and you do not even notice that they are eating your time. It can vary from pushing updates about your tasks to running all tasks of your project or as simple as creating a file to write your next blog post. For a while, I used .sh files to do this automation since they are nearly everywhere, and for most simple tasks, sh is good enough. But you know what is even better? Make it “GNU make utility to maintain groups of programs,” initially made to help C programmers automate compliers tasks that are still very handy nowadays.

Why are Makefiles awesome?

Because it’s seamlessly integrated with the bash commands, and it is simple to define dependencies from commands to other commands. They are extensible and easy to compound. Allow me to show you a few concrete examples.

Simple example

In this very blog, every time I have a new idea, I want to immediately start writing and not having to worry about the file name the folder or even what main parts do I have to add to each port, so I automated that using a simple make file as follows.

new-post:
	@echo "Creating new post..."
	@read -p "Enter post title: " title; \
	file_name=`echo $$title | tr '[:upper:]' '[:lower:]' | tr ' ' '-' | tr -cd '[[:alnum:]-]'`; \
	sed "s/#title/$$title/" _posts/draft_base.md > _posts/draft-$${file_name}.md \
	echo "Opening the file $$file_name.md in nvim..."; \
	nvim _posts/draft-$${file_name}.md
.PHONY: new-post

That will take care of creating the file and opening a new vim editor already editing the file, ready for me to start typing my new ideas.

More complicated example

In Nun-db, I have +20 common commands I use very frequently, and for those, Makefiles are just amazing; here is an example.

Most of the time, I want to test Nun-db in a cluster with at least three processes, so I make sure it will perform as I expect when running in production-like environments. For That, I have to run the next three commands.

1. Build the last nun-db code cargo build
2. Start a primary process server in the background ` NUN_DBS_DIR=/tmp/dbs RUST_BACKTRACE=1 target/debug/nun-db –user $user -p $user start –http-address “$primaryHttpAddress” –tcp-address “$primaryTcpAddress” –ws-address “127.0.0.1:3058” –replicate-address “$replicaSetAddrs” » primary.log &` replicas

3. Start two other server processes to act as secondary replicas NUN_DBS_DIR=/tmp/dbs1 RUST_BACKTRACE=1 target/debug/nun-db --user $user -p $user start --http-address "$secoundary1HttpAddress" --tcp-address "127.0.0.1:3016" --ws-address "127.0.0.1:3057" --replicate-address "$replicaSetAddrs" >> secoundary.log &

4. After I am done, kill all processes using the pid files I created in the start cat .primary.pid | xargs -I '{}' kill -9 {}&&cat .secoundary.pid | xargs -I '{}' kill -9 {}

Imagine having to remember all these commands or having to go to my notes every time I needed to do some of those operations. Totally not adequate. Putting all that in a make file, all I need is to call two commands: make start-all and make kill once I am done. You can see how simple these commands are in my GitHub repo Here, But I am also showing them next picture.

I often combine the command line make clean kill-all start-all create-sample create-test, which will clean up the data files, kill all instances running locally, start all two replicas, and create the sample and test databases for me, all with a single command.

Minor optimizations may not sound like a good idea at first, but they add up over time, and you will get faster and faster at doing the same tasks in the same pass. This will set you to a new level of productivity and will make you shine among others.

Coding like a Pro

Latency is not 0! Learn to deal with it.

Latency is no 0ms! When working locally, you may think and act like it is, and when getting your system to production, you will notice that your code does not behave the way you think it should. In many cases, that is caused by latency differences between working locally and working in production environments.

When working locally with everything running on your machine, Db, application, and Cache, the time for the information to travel from your app code to the database will be virtually 0, and adding 100 individual calls to the database may not sound like a bad idea, in an environment with latency this may kill your system altogether. Check out this fake example I am showing in the next code.

const data  = [..]; //100 items;
const productData = [];
for(const item of data) {
	productData.push(await db.query(item.produce_id));
}
// Do stuff

Now, let’s suppose there is 30ms of latency between the app server and the database server. When this code runs locally, it will run in 2ms and may not seem like a problem. However, when running in production, for example, 100*30ms = 3s, depending on the interaction, 3s is a no deal for most users.

That is why you have to learn how to deal with latency from the ground up, from devs working locally to production environments, I suggest working locally with something at least equal to prod and recommend with a greater (maybe twice as much) latency, and if possible set up test deployments in CI to run with introduced latency so you can find problems earlier.

There are great tools that can help you easily emulate those kinds of environments. In the example, I am going to show how I used toxiproxy, and it will intercept all connections to the server and the connections between servers to introduce latency like I am showing in the next diagram.

This topic became too complicated to fit into this post, so I published a post dedicated to that. If you want to see it in more detail, please take a look at The post Nun-db Chaos Testing with Docker and Toxiproxy, where I shared the details of the tests I made on Nun-db’s Election process.

Learn how to write compilers

It may sound extreme at first, but knowing how to write compilers will be a superpower of yours; it is incredible how quickly some problems can be solved with a simple compiler. Just to be clear, writing a compiler here, I don’t mean creating a new programming language; in fact, most of the time, I had the opportunity to write compilers to compile code that already existed in a famous language like Javascript or some query language, that means one of the most challenging parts of the process is already done.

In the last two years, I wrote two compilers in different circumstances.

2022 Compiler: POC migration from Mongo to Elasticsearch in 2 days:

Imagine you are working on a system where there is a component where the user can search for any of the over 100 available fields, with many different kinds of fields and groups and orders and operators, and the screen and backend are modeled to support this need. The team has done a great job of abstracting and creating strategies to support such a complex use case.

Mongo is having performance issues, and you, the lead, need to decide where to go. Proposing using Elastic Search seems like a good idea, but that will mean you need to dedicate months of work to only then see if it is viable at all. You have to prove to management that that investment is worth it. The code that generates queries for Mongo is distributed into +50 different classes with very complicated rules and extensions.

Easy: Write a compiler to compile Mongo query to Elasticsearch query; the harder part is done already since there is this mongodb-query-parser, so all you need to do is to walk the generated tree and re-write the same query Elastic dialect. In fact, that is what I did in less than 4 hours, and it could take care of 95% of the existing queries with Mongo; of course, I did not use the full capacity of Elastic search and chose each datatype correctly I piked the most obvious ones like making all string fields keywords and did the most basic translation of the mongo operators to Elasticsearch operator.

This is a rather trivial implementation since all you have to do is walk the tree and create the equivalent command in Elasticsearch. It is easy to automate the tests, too.

2023 Compiler: Compile Cypress.js tests to Cucumber

Imagine you inherited 100k lines of code from a legacy project and +400 end-to-end tests that cover a big part of the project you work on, and because of regulation, you need to record on your test record system not only the results of each test cycle but what steps, checks and specks each test do in plain English. You can either put your entire team to work on it for several weeks while they would be in hell trying to describe in text what each text does.

Another option is to create a compiler that compiles function operators and checks to plan English; Cypress tests are written in JS, so you can use a Babel parser to parse it and later walk the tree looking for the branches you want to write as text, in this case, I decided to do it in 2 phases, to make it easier to implement I decided to compile it to Gherkin (Cucumber language) because that would be a bit better structured than plain English and would be easier to map one to the other. Nevertheless, it took less than a week to have more than 98% of tests compiling to some acceptable form of text that we could submit and not programmer humans can read and parse easily.

Closing the subject

Once you learn how to write compilers, they will become a new tool under your Tool Belt, and the opportunities will arrive. I am sure they can save you from spending time on boring activities or investing too many developer hours in some POC that may not be fruitful. I mention the two opportunities I had in the last two years, and they saved me and the company a lot of time and money to overcome some challenges that would be long otherwise. If you are not a CS major and don’t know where to start, I would recommend this book, Crafting Interpreters. But more importantly, do not consider it an impossible task; compilers are just code, and it does take years to build mainstream and polished ones, but that is rarely the case; simple, direct, and hacked ones are quick and fun to build and can accelerate you a lot.

Use AI Copilots

This is probably on many other lists, so I won’t spend too much time on it. I use GitHub Copilot, and it helps when I am prototyping, POC, and experimenting quickly. It also helps when coding something that you are not super familiar with; it may speed up the learning process. It is worth mentioning that the proper way to use it is not to comment on something and expect it to code for you but rather to start coding and let it guess/complete what you have already started doing. It does hallucinate sometimes, so check that it does.

ChatGPT, in the same way, can help; I use it sometimes. For example, if I have something written in bash or nodejs and want something similar in Makefile or another language, it is pretty good at doing this kind of conversion. Again, check the code before using it to make sure you are not doing something absolutely stupid.

Productivity tricks

Try Demo Driver Development

Demo your work frequently; demoing your work to others frequently will help you share whatever you are working on and even gain a better understanding of what you have been working on. Many times, when working on a problem and implementing it.

I have been following this mantra for a couple of years, and it has really paid dividends. Doing demos is stressful and gets most of the developers out of their comfort zones.

It also helps you to sell whatever you are making for the correct price. There are core tips to succeed in doing demos, and they are:

1. Showing the real things running is much better than showing PPT; taking is cheap.
2. “Code is a liability” [1], very few people care about the code. Show the code results and not the code. Feature running, and metrics improvement sell much better than showing blogs of code.
3. Keep your camera on while doing a presentation and show people your expression and what you are proud of.
4. Prepare for the demo and run it before making it, if possible. Mastering the demo and listening to the podcast “Even the Best Rides Come to an End” made me even more sure about this point, and watching Kelsey Hightower’s demos gave me a new understanding of what doing a demo means.

Get used to presenting your work, and do it more often, not only in video form but also in text form. It will do great for your career.

Teach

Teaching is the best way to learn; whether it be mentoring senior developers or teaching someone how to code from scratch, it will make you rethink a lot of how you see things and help to consolidate what you have already studied.

This year, I started teaching algorithms and data structures to first-year students of analysis and systems development at a local faculty in my city. It was a refreshing activity, making me go back and re-read the books I read during my graduation and master’s, and put me in a very uncomfortable situation of having to explain the basics to a group of young and energetic aspiring coders who are trying to find their way into our field.

I spend 4 to 5 hours a week implementing, demonstrating, and discussing data structures with my students, which keeps me up to date with the subject and forces me to stay sharp so I can explain the details and whys behind each data structure and algorithm.

If you want to learn and get better at something, find a way to teach it to someone, whether it be in a class (very dependent on the situation), a blog post, a YouTube video, or a live call; just do it, and you will see how much you learn.

Read

Just reading is a refreshing activity, and it gets you out of the digital world. This year, I did not read as much as I had wished; too much was changing in my working life, and teaching at the university took away a lot of the time I used to put into reading. Nevertheless, reading is one of the activities that gives the most pleasure, and you should learn how to enjoy it too. Every year, this is part of my recommendations, and I still feel like adding it this year again.

Take care of your sleep quality

I am convinced that sleeping well improves the quality of your work overall, but it took me several years to realize I should take care of my sleep quality. This year, I went to a doctor who helps others improve their sleep.

I used the snorelab app to watch how much I would snore while sleeping, and I realized I snored much more than I would have expected. The “normal” for the app would be 15 (The snore score), and on the first night, I got 32, meaning I snored twice as much as the average person. There was the extreme day that I was drunk, and my score was 62, meaning I snored +4x as much as an average user of that app would, and I noticed the days I snored the most were the days I woke with less energy.

I started using an anti-snoring bruxism mouth guard, and my sleep improved a lot. I wake up feeling much more energetic and happy. This is probably a problem that only impacts a small fraction of the population. The lesson to learn here is to track and try to improve your sleep quality. Sleeping at least 7 hours a day and making sure the hours you are sleeping are well used will pay off a lot.

Conclusion

Being a better developer is a daily task, and you have to look for opportunities to improve your day-to-day work and automate the tasks you do repetitively. You will see your life getting better and better and you being able to handle the same amount of load with much less friction and start. Being better and more productive is not only about making more money but also finding smart ways to handle the same task with less stress and less time. That means a more peaceful mind and more time to spend with your loved ones or having fun outside of work.

This was a great year. I started the year working on one of my former employer’s customers and drove the largest migration of my career there, bringing over 700 terabytes of medical imaging data and 300 customers from Azure to GCP with no downtime. I started the company’s first SaaS product from scratch (It used to be a consultancy) and delivered it to general availability with the initial customers and integrators working back in September. Finally, at the end of the year, I joined a new company called Vida (AI-powered lung intelligence) as a Principal Architect to help them scale their business, team, and stack to the next level. It was too much change for one year, and I am happy to have some time to rest at the end of the year as I polish this text that I planned to publish in September. It’s still fine to publish it now, and I look forward to 2024 and what I will learn to share with you next year in this blog; stay tuned.