Software systems are complex beasts. Years of layering on intertwined business capabilities, extending with new technologies, and the special touch by "that guy" (who was trying something 'cool' he read in a book) can make the best architectures difficult to understand and extend after years of productive use.
When the day comes to make some substantial changes, or replace, this workhorse, you'll need time, money your best folks and "fun sized units of work". Entire books are written on the practice of software architecture and design but I wanted to touch on the importance of decomposition in design.
I've watched skilled technicians collaborate and decompose complex problems and chunk them into "fun sized", digestible units of work so that everyone in the room has a good understanding of each piece. Some times the pieces will even end up with names that the group will begin to address them as.
Digestible composition is also one of the axioms of OO Design. The human brain can only hold a finite amount of information in working memory so large procedural programs were very hard to understand and digest. Once you understood what an object/component/service does, you can leverage it based upon its interface and move on to higher value activities rather than crawling code to recall what happens on line 10,000.
Decomposition Revisited:
I learned this skill at some point in my career and have applied it to the point where I haven't given it any thought until recently I heard an interview with Johna Lehrer regarding his book called "How We Decide". In short, the book is study of decision-making from a neuroscientific perspective. While neuroscience may have your mouse wandering to the next Woot.com deal, I reconnected with this old approach on decomposition of larger problems into a set of smaller, digestible ones.
Emotional vs Logical Decision Making:
Lehrer's book outlines a study (among others) where subjects were asked to memorize a series of numbers. One group received small numbers and one group received large numbers. The subjects were then tempted with some food choices and the study showed that the subjects who had smaller numbers made better, rational decisions. Subjects who received larger numbers, made poorer, emotional decisions. The conclusion is that when you brain is overloaded with detail, you tend to make poorer, emotional decisions.
The number of items in the overloaded subject group is important here. Princeton Professor George Miller established what is often referred to as Miller's Law which theorized that the human memory span was limited to approximately 7 related items. He calls these items "chunks" which represent "largest meaningful unit in the presented material that the person recognizes". So, a chunk could be a number or...software component.
Decomposition is more than just a good idea:
All of this made me realize that decomposition is not just a good idea, it impacts our decision making and how well our systems are designed. It probably also explains the poor decisions made when we're overloaded with information which, according to Miller, has a threshold of around 7 related items.
So, if you find yourself struggling with the right approach or having a hard time moving toward the right solution as a group, make sure you think: 6-pack and fun sized!
All of this probably also explains why I always gain weight when working on long-hour projects.
More on Miller's Paper here.
photo: Fun
This idea of chunking is relevant in so many errors. I've experienced people taking on too much at once and in most cases the attention to detail is diminished, and the quality of the output of the project suffers. As you state, recall of the projects details are vague and recording of process, steps, and most importantly issues or bugs are overlooked in the rush to complete the project at hand. In every instance the project was delivered on-time, but with a substantial number of bugs, errors, and exhausted personnel. It was not fun! It was overwhelming and it was not fulfilling. If a culture inside a company or business unit is based on this kind of development or project management over a long period of time you end up burning out your talent, delivering a poor product and no one wins.