Modern Software Engineering Book Reflections
Recently, I read Dave Farley’s Modern Software Engineering publication which outlines a modern, sensible approach to engineering. It articulately describes what it means to be an engineer, and how to tackle engineering challenges.
Dave’s thought process and views on many topics felt familiar and similar to my own, not because we’ve worked together, but because I have the scars of trying many things that didn’t work. Reading each chapter was akin to catching up with an old colleague who’s been there, has seen what works and what does not. It was a refreshing read that I highly recommend.
Reflecting on Dave’s great work, I decided to write my own views after being inspired by Dave, and other thought leaders in our industry.
⚙️ Building Incrementally
Performing heavy upfront analysis and design slows progress; in simple software this may be OK. However, in the current era of building scalable, reliable and fault-tolerant systems, simple is rare. Complex systems aren’t produced by a single mind on the first iteration; instead they are built incrementally, by learning what works.
Some ideas which looked great on paper don’t satisfy requirements because of unexpected unknowns. If you’ve ever worked using waterfall methodology this was its biggest flaw, with many notable systems turning into a big ball of mud, despite hundreds of millions of investment.
We’re in a unique position as engineers, because there are many disciplines that can’t just pivot like us, for example, take building a bridge that spans several miles: a change of plan may be extremely difficult, and could be impossible due to constraints. As software engineers, we can delete code and remove resources.
🔬 Performing Engineering
Being a software engineer is far more than just writing code. Getting architecture and design right for the component you’re working on is important, iteratively reflecting on approaches, learning what works, creating hypotheses, experimenting and drawing conclusions is the value proposition of a good engineer.
As engineers, we need to select approaches and be aware of what our choices really mean, what are the tradeoffs we’re making in our approach? A simple example would be considering CAP theorem. We are trying to organise complexity all the time. It may scale linearly but, ideally, not exponentially, and it is up to engineers to try and tame it.
Code is the output of good engineering, not the driver of it
There’s not great value in clever code, we want code to be simple, expressive, and easy to comprehend. In systems you’re building as part of a team, you want to use code as a communication tool, things that aren’t clear can lead to misunderstandings and therefore bugs.
Compilers are very good at finding optimisations, so whether something can be written in one or three lines shouldn’t be a concern of performance, and if in doubt, test!
🌻 Change
Change is a constant, therefore enabling change should be a first-class consideration in our builds. Following an agile approach of small, incremental, iterative change forces us to build in ways that adhere to principles we’ve already mentioned (modularity, SoC etc), the payoff from this is that it greatly helps to flatten the cost of change in our system, not only in its infancy, but across its entire lifespan.
As you’ve likely experienced, tightly coupled systems become more difficult, more expensive and more dangerous to change. Releasing software like this becomes a scheduled event; because of this, things take longer and releases are delayed. This is far from ideal: we’re slowing down the rate at which we can build, measure, and learn. Meanwhile a competitor that can release more often has a competitive advantage that compounds over time. They could learn faster, and accelerate past you.
So, we agree that change is important, and changes often require having confidence in our releases. If we are confident that each release is bug-free to the best of our knowledge, then the time it takes to release working software (compile, test) should be the only blocker. We can develop this confidence by following a test-driven approach (TDD); following this forces good practices into our design and code.
Furthermore, having a deterministic method of releasing software with automated testing that increases confidence and can exercise our code at different levels is only a good thing. A continuous integration/deployment system is the tool for this; we can include processes such as creating the binaries for our software, performing automated tests from unit to E2E, scanning for library vulnerabilities and so on.
An effective continuous integration process that runs in a timely manner is a strategy that enables some of the largest companies in the world to release thousands of times a day across hundreds of engineering teams.
🧚 Dogma
It can be easy for a team to fall into a trap of doing what they know, why? Well, quite simply it’s worked in the past, or it’s been seen in other projects. A trivial example of this could be installing half a dozen different ruling configurations. These rules dictate how we write specific idioms of code such as loops, types, order of function, documentation, and so on.
Whilst the intent is good, many of these dictator configs are pulled from large companies with specific goals, reasons and styling choices. Do these choices help our team in the specific context of what we’re trying to achieve? Are we as engineers in agreement of following these conventions because they’ll help us? It’s for us to decide, not to blindly follow what has been done before and assume everything applies to everyone.
“Have no respect whatsoever for authority; forget who said it and instead look what he starts with, where he ends up, and ask yourself, ‘Is it reasonable?’”
— Richard Feynman, What Do You Care What Other People Think? (1988)
It’s our responsibility as engineers to make decisions based upon data, this evidence helps us navigate the sea of possibilities and acts as our compass to guide towards a desired outcome. We can also evaluate if we are closer, or further away from our end goal too. Basing our technical decisions on who was the loudest, or had the most conviction, isn’t how we should work.
🏁 Conclusion
Software engineering is difficult, period. Whilst tooling and technology are ever advancing, so is the complexity of our software. We rely on it for mission-critical systems, financial markets, and piloting all manner of vehicles, and its capabilities are ever increasing. The expectation even for general systems such as ecommerce is that they should be available, fast and fault-free regardless of where they are accessed.
Things do go wrong, meeting expectations for customers, or complying with legislation, presents many engineering challenges, and we should recognise and build resilience and fault tolerance into what we do.
How we design systems is critical to facilitate change. Good engineering teams build great software on all frameworks and languages. However, it’s difficult to overcome the wrong design choices, especially if they’re cemented in. It is also not enough for our software to be modular; we need modular organisations too.
If your ability to release software quickly in order to learn is blocked by constraints of depending on other teams, you’re not agile. Being decoupled is a main metric of high-performing teams and organisations; you can only go as fast as your slowest constraint.
Finally, we need to internalise the fact that systems impact real people such as you and me. An error or mistake could have a devastating impact on a person’s life; it could lead to a refusal of a loan, visa, mortgage, healthcare and much more.