Have you worked on a large codebase recently? A scary one, maybe? Maybe even polyglot and split across multiple services? It is no surprise today that software is eating the world. Which, to be honest, can be a good thing for us software craftsman but also means that software becomes more and more complex. Just looking at current tech stacks in various companies, they include up to 100 different technologies, tools, libraries, and frameworks. That combined with the fact that software is both a science as well as art, there is often no right or wrong on how to design our software. You may call in architecture, you may be influenced by clean code and other programming techniques, but the hard part is also in between. Implementing a feature in an existing codebase is a delicate interplay of knowledge (of the existing architecture and actual implementation), the test approach, technologies at play, and, not to be forgotten, how it ties into the business needs. Sprinkle some tech debt on top, and you got the perfect complexity cake.

So with all this complexity at play, nobody can reasonably hold all this into their head. Some people fall back on using pair or mob programming (which is an excellent method btw), but that doesn’t reduce the complexity. It merely distributes it. In the last years, I’ve seen quite a few closed and open-source projects of varying complexities. While they’re all different in their own regards, they all suffered from a common theme. The teams behind them were outstanding software engineers, tested their software properly and took care of their codebase. Yet, entropy doesn’t stop the software from rotting. And this isn’t a bad thing. It’s just the way entropy works. And we all know too well (but may ignore it from time to time) that we need to take care of it. Having established that no codebase is perfect, how do we best approach implementing a new feature (or adding value in general) in such an environment?

When designing systems - people tend to fall for one of the two traps: we try to advance a problem analytically by reading a lot of documentation (shall it exist), make assumptions about the parts that are not documented, talk it through with our peers and form a plan of attack. The alternative approach is to go in, guns blazing, start coding the feature at the most convenient place, and somehow pull through, whatever it costs. But there is an alternative approach that we can take: We cheat!

If you’ve played games on PC or a console in the last decade, you may remember one of the more famous cheat codes, the Konami code. It goes like this:

↑ ↑ ↓ ↓ → ← → B A START

Feel free to try this in your IDE. You might get lucky (actually, Opera does implement the Konami code to access advanced settings). So how can we use this to our advantage when sketching new paths through our software. For that, let’s join Susan.

Susan recently joined a new company as a Senior Software Engineer. She has experience with different programming languages, is an avid Practioner of test-driven development, and embraces refactorings.

The codebase of the company she joined is quite complex even though well thought out. Many architectural and implementation approaches are not documented, so she has to rely on tribe knowledge. Her team is friendly, yet, with many new people, there are always aspects of the codebase that no one really knows how it came about. Susan and her team decided to take on a new, non-trivial feature for one of their customers, and Susan decided to take the lead.

Approaching the codebase wasn’t easy, but she remembered reading a blog post about “Cheating in Software Design using the Konami code”. So she gave it a try.

↑ Identify the outer-most interaction surface

One of the first steps towards implementing a new feature is thinking about how an end-user would first approach it in its most minimal incarnation. This might be a single button in the UI, a single argument on the command line, or a single method within an API if you’re developing a library.

↑ Write a high-level system test

Once we have identified a minimal aspect of our feature, start with a system or integration test. Feel free to ignore how you’d actually implement the feature itself. Think about how it will be consumed by the end-user. What is the interaction pattern the user will experience? Is it a button that fills a table with data? Will it produce no observable state except for confirmation but send an event somewhere? Try your best to sketch out a test case for your feature. It doesn’t need to compile yet, let alone run (if it runs and passes, there is no feature to implement anymore, but we’re getting ahead of ourselves).

↓ Build a mental map

With the customer-centric picture of a feature in our subconsciousness, let’s do a quick rehearsal. Think of it like Hansel and Gretel leaving breadcrumbs behind. Try to imagine how you would implement the feature but hold off doing so. Just trace the path through the codebase and try to get familiar with your surroundings. Think it through (even better in a pairing session if you’re up for it). Don’t touch the code yet. Think of what you’d need to do. Does your UI need to call a backend service?

Feel free to make assumptions about things you don’t know yet but feel free to explore those areas. Do you need to maybe change the database schema? Don’t worry about how you’d store the data but try and find the place where you’d need to change the schema. Do you need to talk to a 3rd party API? Have a look at their API documentation, check how you’d authenticate and what kind of data this offers. Is your system already capable of dealing with such data, or is it something you’d need to do? Feel free to sprinkle in some TODOs in the code to remind yourself of your open questions before hitting our cheat code’s next key.

↓ Go the path. All hacks allowed

Now comes the exciting part. Now we go “down” into the implementation. At least sort of. Before we do, one piece of advice. Forget everything you ever heard about clean code, test-driven development, or poor design patterns. The key is to get enough code cobbled together to make the high-level test pass in this step. Don’t have API keys for a 3rd party API yet? Don’t worry. Hardcode it to some reasonable responses from their documentation. Need to get data from one subsystem to the other in a thread-safe way? Feel free to use a static variable. Proper synchronization is nothing we need to fix at this stage. Not sure how to properly style that UI? Use a red button and a green table. Do the cheapest thing possible to getaway. But there is a catch. A single rule so to say:

Don’t fake anything you don’t know yet.

What do I mean by this? Depending on your experience and the infrastructure in your code, certain parts of the implementation are “straightforward” yet require a decent amount of work to make them handle the complexity of the real world. A good example is calling an HTTP endpoint (easy, right?). Something that can be done with quite some ease in most languages nowadays also holds a lot of potential edge cases in how they affect your product (rate limiting, retry behavior, timeouts, …). If you know these things and have a good idea of how to do them, skip it for now. Hardcode the response. Focus on your feature’s path, how it flows within the architecture, which submodules need to interact, what protocol/API/handoff has to happen.

The same goes for algorithmic problems. If it’s local to a particular data structure (e.g., the fastest way to walk a specific tree of data), you can skip this for now and use something that just works, performance aside. But be aware whether this is actually a local aspect or if solving this may require a different data structure and whether that changes the surrounding design for your component.

Your goal is: get our system test to pass while optimizing the time spent there. And keep our single rule in mind. If you’re not sure how to get from A to B, dig “down”.

Got your test passing? Great! You’re almost done with your feature. You did the hard part. You did the exploration. Yes, correct. This isn’t your implementation. This exercise is solely to understand the complete picture, to explore the uncertain parts of our map. And by hacking it together, it is natural to stumble upon those “oh I didn’t see that gap how to get from A to B”.

← Identify bottlenecks/hurdles

  Now that our functionality is “somewhat” working take a step back. You just gained a lot of experience doing this spike. You learned more about the system, you gained insights into which parts fit into the architecture and which parts stood out like a sore thumb. Using our newly acquired knowledge, we can collect two lists of things to further explore:

• Refactorings that are blocking the feature
• Refactorings that support us to fit the feature in more naturally
• Prerequisites that we need to start implementing this for real (got that API key by now?)

“I doubt any of these
are blockers, but there's a certain amount of un-fun grinding here to
clear out the underbrush” is so much better than “Refactoring”. Thank you @BrianGoetz for this beauty https://t.co/dDdMjyZrot

— Benjamin Muskalla (@bmuskalla) February 18, 2021

→ Reset Hard

And with items on the lists above, there is the only thing left for us to do. Stash your system test, mark it as @Ignore, or @NotYetImplemented if your test framework supports that. And let’s do it for real:

git reset --hard HEAD

← Left - Do all refactorings

Now, this step is actually significant for you - as well as other people reviewing your code. Given we’ve already spiked the path through the system, we know which parts are in the way and require some attention. Doing those kinds of refactorings in the middle of implementing a feature often leads to added complexity as refactoring contributes to the number of things we need to handle in parallel. Your tests for your new feature might not be green yet, and that’s the worst time to start refactoring. By learning about the path beforehand and identifying (at least some), refactorings can help us to go faster in the long run. We can now tackle this refactoring independently, which makes them easier to do and easier to review. It also makes it easier for our feature to fit into this. Imagine we have some colossal class that handles multiple alternative implementations at once and got really messy. Refactoring that towards a strategy pattern improves the current situation and clearly indicates that we add a new strategy when we implement our lovely feature. Try to make it easy for the feature to fit in.

Avoid Shotgun Surgery.

→ Implement the steps, one by one

With the whole picture in your mind now, it’s time to pull up the sleeves and do the work. It actually doesn’t matter where you start, and different people have different approaches. Start with something that fits your current mood and get the first part of your implementation in place. It doesn’t need to be connected to anything else yet, but it should be adequately implemented. The good thing is that we can actually work on pieces individually as we already know how we’ll put them together in the end.

The process of writing your second draft is the process of making it look like you knew what you were doing all along.

—Neil Gaiman

“Properly” should be defined by you and your team. Personally, it means I have unit tests for the unit in question, proper documentation if it’s an API boundary, and maybe integration tests depending on the nature of the thing. One by one, those can also go into your main branch or go up for code review as they should be independent of the rest (maybe not connected to everything yet). If you and your team have a good track record working together, you may actually go to the next level of defining an interface between two components and implement them in parallel.

A Introduce your system test (again)

Once you’ve retraced your steps and implemented all the different aspects correctly, it’s time to get back to our initial test and see how it works.

B Connect the dots

Your system test should pass by now. If not, you may have forgotten to integrate some of the newly added functionality. Feel free to add some more system-level tests that span the full feature to see if you have any loose ends.

START The feature is “Done Done”

Congratulations. You’ve delivered a new feature. That’s a tremendous short-term technical win. But you have accomplished a lot more. You’ve delivered a feature. You’ve done the refactorings to support and adequately integrate that feature and have proper test coverage on different levels. These aspects contribute significantly to the long-term goals of your projects and keep them maintainable.

Interested in reading more about some of the strategies mentioned here?

• Book “Growing Object-Oriented Software Guided by Tests” by Steve Freeman and Nat Pryce
• Definition of “Done Done” in “Art of Agile Development”
• Look up the various articles around “Outside-In Test Driven Development”. A lot of them give you hints depending on your language of interest

If you liked the the post, I’d happy to hear your thoughts on Twitter.