Modern Software Engineering Book Reflections
I recently read Dave Farley's Modern Software Engineering, which sets out a modern, sensible approach to the discipline. It explains clearly what it means to be an engineer and how to tackle engineering problems.
Much of Dave's thinking felt familiar. Not because we've worked together, but because I have the scars from trying things that didn't work. Each chapter felt like catching up with an old colleague who'd been there and seen what works. I highly recommend it.
What follows are my own views, prompted by the book and by others in the industry.
Building incrementally
Heavy upfront analysis and design slows progress. For simple software that may be fine, but systems that need to be scalable, reliable and fault-tolerant are rarely simple. Complex systems aren't designed by one mind on the first attempt. They're built incrementally, by learning what works.
Ideas that look great on paper fall short of requirements because of unknowns nobody expected. That was waterfall's biggest flaw, and many notable systems ended up a big ball of mud despite hundreds of millions of investment.
Software engineers are in an unusual position. Many disciplines can't pivot the way we can. Change the plan for a bridge spanning several miles, and it may be extremely difficult or impossible. We can delete code and remove resources.
Performing engineering
Software engineering is far more than writing code. Getting the architecture and design right for the component you're working on matters. The value of a good engineer is in reflecting on approaches, forming hypotheses, experimenting, drawing conclusions and learning what works, over and over.
We need to choose approaches knowing what those choices mean. What trade-offs are we making? The CAP theorem is a simple example. We're organising complexity all the time. It may grow linearly, but ideally not exponentially, and taming it is our job.
Code is the output of good engineering, not the driver of it
Clever code has little value. Code should be simple, expressive and easy to understand. On a team, code is a communication tool. Anything unclear leads to misunderstandings, and misunderstandings lead to bugs.
Compilers are very good at optimising, so whether something takes one line or three shouldn't be a performance concern. If in doubt, test.
Change
Change is constant, so enabling it should be a first-class concern. Small, incremental, iterative changes force us to build with the principles above, such as modularity and separation of concerns. The payoff is a flatter cost of change, not just early on but across the system's whole life.
As you've probably experienced, tightly coupled systems get harder, more expensive and riskier to change. Releasing them becomes a scheduled event, so everything takes longer and releases slip. That slows how fast we can build, measure and learn. Meanwhile, a competitor that releases more often has an advantage that compounds. They learn faster and pull ahead.
Changing often needs confidence in each release. If every release is as bug-free as we know how to make it, compiling and testing should be the only thing between a change and production. Test-driven development (TDD) builds that confidence, and it forces good practice into design and code.
A deterministic release process, with automated tests exercising the code at different levels, adds to it. Continuous integration and deployment is the tool for this: building the binaries, running automated tests from unit to end-to-end, scanning libraries for vulnerabilities and so on.
A fast, effective continuous integration process is what lets some of the largest companies in the world release thousands of times a day across hundreds of teams.
Dogma
It's easy for a team to keep doing what it knows, simply because it worked before or was seen on another project. A trivial example is installing half a dozen lint configurations, each dictating how to write loops, types, function order, documentation and so on.
The intent is good, but many of these configs come from large companies with their own goals, reasons and style choices. Do those choices help our team, in our context? Do we follow the conventions because they help us? That's for us to decide, rather than assuming whatever was done before applies to everyone.
"Have no respect whatsoever for authority; forget who said it and instead look what he starts with, where he ends up, and ask yourself, 'Is it reasonable?'"
- Richard Feynman, What Do You Care What Other People Think? (1988)
As engineers, we should make decisions based on data. Evidence is our compass through the sea of possibilities, and it tells us whether we're getting closer to the goal or further away. Technical decisions shouldn't go to whoever was loudest or most convinced.
Why it matters
Software engineering is hard. Tooling and technology keep advancing, and so does the complexity of what we build. Software runs mission-critical systems and financial markets, and pilots all kinds of vehicles. Even an everyday system like ecommerce is expected to be available, fast and fault-free wherever it's used.
Things go wrong. Meeting customers' expectations and complying with legislation are real engineering challenges, so resilience and fault tolerance have to be built in.
Design is what makes change possible. Good teams build great software in any framework or language, but wrong design choices are hard to overcome once they're cemented in. Modular software isn't enough, either. Organisations need to be modular too.
If releasing quickly to learn is blocked by dependencies on other teams, you're not agile. Being decoupled is a key marker of high-performing teams and organisations, because you can only go as fast as your slowest constraint.
Finally, systems affect real people, like you and me. One error could be devastating for someone: a refused loan, visa, mortgage or healthcare, and much more.