meaningful software: on pattern languages and design forces

introduction

Pattern languages were introduced into software development in 1980s and popularized in 1990s. They have been an important part of software design since.

It is often emphasized that the solution offered by a pattern “must be applicable in lots of different situations” [Fowler].

As the result, software engineers are expected to recognise and apply well-known patterns, but rarely encouraged to develop skills and conceptual tools to produce pattern-like designs, unless ‘pattern-writing’ is in some way a part of their job description.

I would like to look past the notion of “pattern recurrence” and focus instead on other aspects of patterns, especially on the design forces and their balancing.

First, I will give a very brief overview of software pattern languages.

Then, I will consider the question: what is a pattern without recurrence? Just as the grammatical categories of common and proper nouns, the patterns with and without recurrence could be respectively called “common” and “proper” patterns (there is no impropriety implied for the recurrent patterns, of course).

I will outline some properties of “common” and “proper” patterns and the skills a software designer needs to have to deal with both. I also would like to counter the reduction of a great deal of design to sheer pattern recognition like in the following:

Suppose one needs to design something using available patterns; pick those that are most relevant to the problem at hand, then choose not more than about a dozen related patterns from an existing patterns catalogue. Identify a vertical dimension (e.g., time, space, or group size) appropriate to the process that generates the end product, and study how the generative process develops as one moves up the levels of scale. [Salingaros 2000]

Unsurprisingly, the skills about which I will talk are methodically identifying and balancing forces, as well as good naming and namespacing.

The last two sections contain notes on both skills.

This article offers an overview rather than claiming to present anything novel, aside from a few minor points.

software pattern languages

There are dozens of online overviews of pattern languages and software design pattern language in particular. So, I will introduce only a few facts, definitions and details that would lead to the discussion in the rest of the article.

Here is a typical explanation of what a pattern language is:

“A Pattern Language is a set of flexible, reusable solutions to common design problems. These patterns evolve naturally within systems, and are found rather than crafted. Identifying and recognising these patterns gives us a framework to craft solutions with. (https://maggieappleton.com/pattern-languages)

The concept of pattern languages comes from the works of an Austrian-American architect and urban planner Christopher Alexander. In his seminal 1977 book, A Pattern Language, he wrote:

“Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in a way that you can use this solution a million times over, without ever doing it the same way twice.” [Alexander et al 1977]

The theory of pattern languages was the second out of three design theories Alexander developed over the years [Dawes, Ostwald]. Although his thought expanded beyond pattern languages, only his pattern language-related ideas seem to become a mainstay in software engineering seem.

In very coarse terms, since 1990s, the pattern language movement in the software engineering, computer science, and education has been preoccupied with “writing patterns” in the circuits of conferences, groups, pattern writing workshops and publications:

The engineers, researchers, and working groups would pick specific aspects of software engineering, identify a tapestry of recurring problems or needs in a certain context, and distilling recurring, reusable solutions for them (e.g. patterns for class instantiation or behaviour in OOD [Gamma et al]; or patterns software team lifecycle [Coplien, Harrison], and — not unexpectedly — patterns for pattern writing [Meszaros, Doble 1997]).

They would build pattern repositories such as The Hillside Group Design Patterns Library, Portland Pattern Repository and others with the intention expressed on https://hillside.net/patterns/website:

“The goal of patterns within the software community is to create a body of literature to help software developers resolve recurring problems encountered throughout all of software development. Patterns help create a shared language for communicating insight and experience about these problems and their solutions. Formally codifying these solutions and their relationships lets us successfully capture the body of knowledge which defines our understanding of good architectures that meet the needs of their users. Forming a common pattern language for conveying the structures and mechanisms of our architectures allows us to intelligibly reason about them.” Brad Appleton, The Hillside Group, https://hillside.net/patterns/

Hundreds of patterns forming dozens of pattern languages have become an important part of conceptual toolkit and best practices of software engineering. They also delivered on the promise of a new method of knowledge- and experience-packaging.

On the other hand, exploring or searching the collections of patterns has been as exciting as learning a foreign language by reading a dictionary. Moreover, pattern repositories offer solutions based on someone else’s experience, but do not leave much agency to the reader.

It is true that they encourage to “write one’s own patterns” or to “use [a] solution a million times over, without ever doing it the same way twice“ and teach how to develop a solution to a recurring constellation of forces (see e.g. [Meszaros, Doble 1997]). However, something still is amiss: How do we spot, investigate and express forces that need to be balanced? How do we deal with novel or unique situations?

(Anecdotally, all the questions on the topic I ever got in job interviews go as something like: please name some design patterns other than Singleton. Fair enough, but it would not probe in any way the candidate’s design ability or even the capability of applying the existing patterns “a million times over”.)

Lastly, it has been emphasised throughout the literature that patterns are interconnected and form pattern languages. However, it has never been made clear how a pattern language is a language; how “a pattern language in terms of the properties of pattern combinations” [Salingaros 2000] goes beyond just “the profession’s vocabulary” [Fowler] of interrelated solutions and what makes a themed collection or an interwoven network or fabric of patterns a language in the same sense as English or python is a language? Either the expression “pattern language” has been used as a metaphor or terminological nickname, or it is deemed to be literal, meaning that a pattern language qualifies as a language in the linguistic sense — which is far from obvious. If pattern languages are languages to an extent, then it opens the door of applying to them various linguistic concepts and tools.

In this article, I will not try to answer this question, but until proven otherwise, I will tentatively assume that pattern languages are language-like phenomena (they certainly are semiotic systems, but then what isn’t?); and therefore sometimes, I will try to apply to them some linguistic concepts at least in the form of hypotheses.

what is pattern? a closer look

Martin Fowler, a software practitioner and author, writes:

“A common definition of a pattern is that it is “a solution to a problem in a context”. That’s a definition that’s always struck me as being rather unhelpful.”

…

“Patterns should have recurrence, which means the solution must be applicable in lots of different situations. If you are talking about something that’s a one-off, then it’s not worth adding the name to the profession’s vocabulary.” [Fowler]

As Fowler points out, a software design pattern applicable in many situations becomes an entry of the conceptual vocabulary, which then would be used in design process and incarnated in the actual software system in interaction with other design patterns. (The same is true for organisational and other patterns.)

A great deal of problems and situations arising in design, team work, business, society, or one’s life have happened and will happen over an over. We need patterns, recipes, best practices, or role models to effectively use them as templates, shortcuts, or building blocks to cope with design, business, or life’s challenges.

What happens though if we forget for a moment about the recurrence and reuse and walk through the meaning of “a solution to a problem in a context”?

We face all the time situations that may be unique or novel in some respect and forces acting in them uncertain or not obvious. The existing pattern vocabulary may be insufficient, inadequate, or too generic. It may not be readily possible to express the solution to such a situation — balance its forces — using purely the terms from the existing pattern language.

If patterns of the pattern language fall under the category of “common nouns” (like the “city” pattern), then we could say that the unique, “non-recurrent”, “one-off” solutions correspond to “proper nouns” (resolutions of unique circumstances like “Paris”).

Here is a classic example of such a one-off design situation (highlights mine):

“[In] Berkeley at the corner of Hearst and Euclid, there is a drug store,
and outside the drug store a traffic light. In the entrance to the drug store there is a newsrack where the day’s papers are displayed. When the light is red, people who are waiting to cross the street stand idly by the light; and since they have nothing to do, they look at the papers displayed on the newsrack which they can see from where they stand. Some of them just read the headlines, others actually buy a paper while they wait.

“From the designer’s point of view, the physically unchanging part of this system is of special interest. The news rack, the traffic light, and the sidewalk between them, related as they are, form the fixed part of the system. It is the unchanging receptacle in which the changing parts of the system — people, newspapers, money, and electrical impulses — can work together. I define this fixed part as a unit of the city. It derives its coherence as a unit both from the forces which hold its own elements together, and from the dynamic coherence of the larger living system
which includes it as a fixed, invariant part.” [Alexander 1965]

The context or forces here are:

• position: corner of two streets
• the traffic light making people stop and wait
• however, the wait is not long enough
• to browse in the store
• or to pay by the counter
• people are honest, mostly
• the drug store wants to make an extra buck

The force-balancing solution, “this fixed part”: place the rack and a money jar on the pavement next to the intersection so that there is enough time for the passersby to glance through the headlines, pick the paper, and drop a coin in the jar.

The store owner sees the pattern in the force interaction: business interests, location, human movement, traffic control, level of trust in a given society — and drops in the contraption (the newspaper stand and coin jar) that fits.

This solution is location-specific and not necessarily generic. It is unlikely and somewhat irrelevant whether the store owner thought of standard patterns like ‘street corner’ or formulated his solution as something that could be applied over and over in various contexts. Yet, it is hard to deny that his thinking was very “pattern-like”.

The newsstand scenario is neither strikingly novel nor complex. Every part of it is mundane and simple. It is so mundane that many people would not even discern it as a problem — and so they would miss a minor opportunity to slightly improve their lot. Yet, the problem and its solution is unique and singular. When designing a software system or its parts, such idiosyncratic circumstances routinely arise at all scales and they cannot be solved by just combining existing generic patterns.

One may argue that a “one-off” solution is not a ‘pattern’ but just an implementation of a specific system. However, the design applied to a unique constellation of forces is distilled and conceptualised first even if it gets implemented only once.

The one-off solution not necessarily emanates from a generic template. The newsstand “pattern” may represent only the given situation and thus may not necessarily have a yet undiscovered underlying abstract pattern or archetype.

Does the “one-off” solution come about differently from the way a design pattern gets discovered, distilled, and resolved? In the rest of this article, I would like to make a few points:

Firstly, the analysis and design process is similar if not the same for patterns (which are like “common nouns”) and one-off cases (which are like “proper nouns”). Secondly, it is more important for an engineer to be capable of seeing the “one-off patterns” and coming up with balanced solutions — whether one-off or not — than apply the acquired wisdom of documented patterns — as valuable as the latter might be. Thirdly, the recurrence makes its way back into the “one-off” solutions in a somewhat different form.

Now, let us get back to the software engineering.

“proper” and “common” patterns

The grammatical common names (“city”, “person”) vs proper names (“Paris”, “Johnny”) are more than an analogy in respect to the design patterns because of the strong claim that design patterns are not scattered — albeit interconnected — concepts or phenomena, but form cohesive pattern languages.

So, as the grammar of design pattern languages goes, it would be natural to call reusable patterns “common patterns” and the one-off ones “proper patterns”. I will use the latter term as a shortcut to a “solution of one-off problems in (somewhat) unique contexts at conceptual level rather than as a concrete implementation”. (I.e. I see a pattern in the latter formulation, which I’d name a “proper pattern”). I will keep the quotes to make clear that there is nothing “improper” about the “common patterns” or “uncommon” about the “proper” ones.

The design process starts with seeing how various “common patterns” manifest and interact given the forces of a specific situation and then the solution gets expressed using pattern names as part of the design description vocabulary. So, the “sentences” in pattern languages have semantic/expressive and pragmatic side: they give a vocabulary to express designs plus the solution elements they devise and combine actually do something in the actual systems — the patterns do what they say.

(It is interesting that the “proper patterns” often are given arbitrary names or even no name at all unlike the “common patterns”, which require very distinct and carefully chosen names to make their way into the pattern language.)

The scale of common patterns does not need to be a grand framework, product or system scale. A basic user story (“As XXX, I want YYY, so that I could do ZZZ”) and its specialised conceptual resolution form a “proper pattern”. Such small-scale “proper patterns” commonly remain unnamed.

(I find lack of good naming a design deficiency: if you cannot aptly name something, it means you are not sure or cannot articulate what is it. It also makes the ‘thing’ — especially one-off ‘thing’ — hard to refer to and therefore the meaning and purpose of it gets lost in conversations with customers or between engineers.)

Here is another example of a one-off situation that calls for a “proper” pattern. Assume we have a system failure, a common situation that can arise at any scale. There is a “common” pattern to deal with it. It is Toyota’s “andon cord” pattern: stop, find the root cause, fix it, and make sure it does not happen again in a similar manner. We could add test cases to our regression test suite that would guard our code base from this class of problems (which may be logical defects, missed corner cases, system idiosyncrasies and so on — depends on the root cause). The solution has the structure of a pattern. Its intent is to guard against the found class of problems. Some of its forces are:

• Even in moderately complex systems, it often is unrealistic to maintain 100% test coverage. The test coverage often is much, much lower and ramping it up may be very expensive. This is a generic force.
• The regression test acts more like a sparse fishing net rather than a solid wall: nicely crafted 30% test coverage may catch 70% of failures. This is a generic force, too.
• We deal with failures on a very short notice and always dig to find its root cause (andon cord). After the fix, we protect the given class of problems with a dedicated regression test suit. This is a generic force, too.
• We have a failure with a root cause ABC and its fix XYZ can be very specific to our system. This is a specific force.

Our new test cases need to mend our regression test “fishing net”, making it “denser” at a given spot. We need to cover our system’s behaviour and potential failures in the “neighbourhood” of the mode of failure ABC and its fix XYZ. Devising such a solution is a pattern, which may be “common” (e.g. defining how to deal with special values) or very system-specific, i.e. a “proper” pattern.

Both “proper” and “common” patterns are force-balancing solutions in context. There is no difference in their emergence apart from the generic nature of the “common patterns”:

The designer identifies context and significant interacting forces and then devises — by design or from observation — a solution balancing forces in the desired manner.

Those needs and desires are a part of the interacting forces and thus may also need to be balanced through trade-offs and concessions.

As a matter of fact, reuse itself is a strong design force that is always present in the design thinking and needs to be balanced again the forces in the situation(s) in question.

By definition or rather by the nature of “common patterns”, reuse is an essential force informing the outcome. On the other hand, “proper patterns” may gain a lot of leverage by sacrificing reuse.

note 1: rule of two extremes

The tipping of the reuse force in one direction (toward reuse or toward one-off) corresponds to the rule of two extremes — or rule of no golden middle — I came up with both names: When designing a software artefact, make it as generic as possible or, if not, then make as specific as possible.

If I design, say, a class, I first probe whether I can generalise it: can I move it from a specific application to the project library so that it still makes sense for other users in other contexts? Can I further generalise it and put in the generic library? I stop once further generalisation starts becoming clumsy, over-engineered, or too much too early.

On the other hand, if — at least for now — the class does not generalise gracefully at all, I try to move it to a more specific namespace and location, as close as possible to the point(s) of its actual use: to the specific project, to the specific executable, to implementation details — so that the scope of its visibility and use is clear and limited and thus its implementation does not have to be burdened by the needs of reuse. As specific as possible, but not too specific.

Just like over-generalisation may lead to over-engineered solutions or unaffordable amount of work (too much too early), making the solution more specific than it needs to be adds too much rigidity (e.g. lots of hard-coded definitions or frozen interfaces), too many implementation details, and too much of code/effort duplication.

So, there is no golden middle between generic and specific: When balancing reuse you either choose to press toward more generalisation or more specialisation as far as other forces permit so that you get as much leverage as you can: better design and practice, standardisation, development cost saving through reuse; or (potentially) less mental effort, quicker delivery and postponing design decisions through specialisation.

Lastly, even though the “proper patterns” usually do not fit into the ‘vocabulary’ of a pattern language, their expressive naming is as important. The unique solutions still are a part of a conversation in various contexts: in the process of design, integration or interaction with other systems, business development, marketing, or operations. The one-off names still are used more than once and thus, “proper patterns” require apt, meaningful names just as the “common” ones do. Also, well-designed unique solutions may become reusable one day — as is or after a few rounds of distilling them — which may be hard to know in advance.

note 2: “proper patterns” and metaphor

One way the “proper pattern” may become generic not so much through the principled structured reuse but through becoming symbols, memes or metaphors like Venice in “Venice of the North”, which happens to refer to several dozens cities. Somehow, the technological “proper patterns” often enter into a wider generic use as anti-patterns — recurrent unhealthy constellations of forces like “spaghetti code”. For example, I would mention Ariane 5, which was a very specific accident (one of the most expensive software bugs with a loss of over $370 million), if I want to highlight a cluster of certain maladies in a product: dead code, untested corner cases, and lack of value validity checks. Manhattan Project — also a unique one-off — serves as an instant metaphor of a certain kind of a large-scale endeavour when a design or project is likened to it.

Manhattan Project “proper pattern” probably could be distilled into a “common” organizational pattern, but we are using it as a design meme or metaphor as is.

Talking of metaphors in the context of patterns, in 1970s, Groupe μ, a group of Belgian semioticians, offered a useful formal tool for analysing meaning [Groupe μ, 1970], which bears quite a bit of structural similarity with the force analysis in pattern languages.

In particular, they proposed that the meaning of a word is composed of elements or traits of meaning, which they called semes. Say, in some contexts, a few semes of the word “lion” could be “power”, “courage”, “nobility”, etc. Among all, Groupe μ suggested that metaphor operates by transferring some of its semes to its object. For example, when hearing the name of Richard the Lionheart, we would assume that he probably was strong, courageous and noble (but not necessarily red-haired).

So, when we apply a cultural or technological metaphor like Manhattan Project to an enterprise, we transfer its semes onto our enterprise.

Since the “pattern” does mean a constellation of interacting forces, the forces are the elements of meaning of a pattern, or you can say, its semes. If we want to be less poetic and more technical, we can outline the forces underlying Manhattan Project as applied to and balanced in an enterprise, establishing the relationship — even equivalency — between a metaphor and a more formalised (but not necessarily generic) pattern.

The designer’s ability to address a unique situation and work on a “proper patterns” to resolve it may be more important than the skill of spotting and applying ready-made “common patterns”; just as both the ability of independent thinking and knowledge of best practices are essential.

Both “common” and “proper” patterns share two core groups of core skills:

• Identifying and balancing forces based on the pattern intent and participants (intent of the pattern is a part of its forces). The intent tells how we want to use the solution. It has two consequences: First, since meaning of software is in how it is used (its usage semantics) [Vlaskine 2013], the intended use (intent) is the meaning of the pattern. The use implies the pattern participants/actors including the users of the pattern. Also, the intent can accommodate various concessions and thus ends up being a bunch of the forces to balance with the rest of the forces: different use trade-offs may lead to different solutions.
• Naming pattern including naming the context(s) at play (I will call them “namespaces” with the same meaning as e.g. in C++) is a surprisingly rare skill. It is a must for the “common patterns”, so much so that there is a pattern language for pattern-naming described in [Meszaros, Doble 1997]. This paper talks only briefly about naming contexts, though. Context-naming/namespacing is even less explored for “proper patterns”, where it is more important since the “proper” patterns almost by definition occur in unique or specialised contexts. It’s a part of design to identify those contexts multiple contexts (client’s needs, tech ‘stack’, budget, team, hardware, regulatory, site specifics and so on). It is essential to identify those multiple contexts and make sure that they and their relationships are remembered and reasoned about in the design. To do so, one needs to be able to refer to them and their relationships. Without thought-through naming, those get forgotten, missed, or misunderstood.

In the rest of the article, I briefly address some aspects of both. The section on naming will be brief since the topic is too broad and I hope to revisit it another time.

But first, a quick note on recurrence.

back to recurrence

The word “pattern” means a discernible regularity in the world or ideas, making the phrase “one-off pattern” sound like a contradiction in terms.

In this section, I will briefly show in what form recurrence still is present even in a “proper pattern” and is its essential part.

The “proper pattern” balances forces in a (relatively) unique or specific situation, one that does not lend itself to a generic solution. The recurrence still is possible: Firstly, the situation persists rather than being a fleeting fluke. This constancy is a form of recurrence (also see below). The situation may as well arise again, however when it happens, it is more or less the same (e.g. flooding in the same region, or bloated memory use in the same computation); however there is not enough variation to call its resolution generic over diverse situations.

So, the “proper pattern” solution is not entirely one-off: when we design, our solution has to anticipate and factor in the possibility of the situation happening more than once (see the regression test example above).

Secondly, our situation or context is represented by a bunch of interacting forces. Moreover, if the situation is more than a bunch of significant interacting forces as we are concerned, we should analyse what forces those extras represent. Thus, interacting forces are the situation. (We may be able to abstract away some forces, for example the implementation language, but that may not make the situation or solution more generic.) For the situation to be something concrete and recognisable as opposed to total chaos, its interaction of forces has to have some degree of constancy. This persistence could be compared to standing waves is a form of recurrence. As a matter of fact, such a discernible constellation of forces is called “a pattern” (“the physically unchanging part of [the] system” [Alexander 1965]) both colloquially and technically, e.g. Chladni patterns. So, it could be called “pattern of forces”. In this sense, it is not the solution that recurs, but the situation that persists or reoccurs. That’s what anti-patterns are: they are patterns of forces left to reach point of equilibrium on their own, without any balancing intervention!

Lastly, one of the essential ingredients of good software design is the second use case. With just a single use case, the design is almost guaranteed to lack a second leg to stand on, lack of “semantic triangulation”, and will very likely be too overfit to provide a good solution even for the problem at hand. In the absence of an obvious second use case, the unit or regression test always can (and should) serve as another use case: The test exercises both the usage semantics and testability of the solution — lack of either is a big red flag: If the use is not well-designed, tests will be hard-to-implement, clunky and unreadable — a very strong indication that the main use case would suffer from the same problem. If the solution does not easily lend itself to direct testing (makes its behavioural internals inaccessible, requires test fixtures such as a fiddly mock database passed through a back door, etc), the validation and maintenance become nightmare. So, even “proper patterns” usually “generalise” to or recur in their second use case and thus have a certain degree or potential of being more generic and reusable.

Thus, even if a situation is one-off, just by the merit of being a situation rather than a shapeless soup of random events, it possesses forms of recurrence: persistence of the force interaction (pattern of forces) and multiple applicability (re-use as in the second use case).

naming

After reading this section, one may say: “This all is just semantics.” This seems appropriate since we are talking about language, though.

I have not managed to find much literature on giving names to various bits of pattern languages. If anything at all, the attention usually has been directed to giving a name to the whole pattern language and to individual patterns. The former is usually barely mentioned (“The Pattern Language for XYZ”). The latter has received more focus so much so that the Pattern Language for Pattern Writing has a section Naming & Referencing among a handful of other sections in its structure [Meszaros. Doble 1997, 1.0].

namespaces in pattern languages

Pattern language descriptions often are structured into the sections or chapters.

For example, in a Pattern Language for Pattern Writing:

Section A, Context-Setting Patterns, …

Section B, Pattern Structuring Patterns, …

Section C, Pattern Naming and Referencing Patterns, …

… [Meszaros. Doble 1997, 1.0]

Or in the classical Object-Oriented design pattern language:

• Creational design patterns
• Structural design patterns
• Behavioral design patterns [Gamma et al]
• Etc

There is a dissonance in this picture: the pattern language, patterns in it and their interconnections appear as language phenomena, while, the overall structure of the language is represented not as a language mechanism, but is more or less reduced to a bullet-point list.

Instead, it is natural to think of a pattern language as a namespace (in the same sense as namespaces in C++): an object-oriented design namespace, or a UX pattern language as a user-experience namespace, and so on. Its subsections really are pattern languages (namespaces) on their own: hijacking C++ syntax, it looks as something like: design_patterns::creational_design_patterns, pattern_writing::pattern_structuring, and so on.

In the examples above and in the rest of this section, I am neither attempting to formalise pattern languages nor preparing patterns for “implementing” them in C++. Rather, I am using the C++ notation as a prop to achieve greater semantic clarity.

Now, I can further decouple the composite names and repetitions: e.g. pattern_languages::object_oriented::design::creational; pattern_languages::writing::structuring; and so on.

The name pattern_languages looks like a ‘global’ namespace, but it really should not be there just like C++ namespace in C++ code — cpp::vector etc — would carry no useful meaning apart from limited cases, e.g. for cross-language bindings: cpp::vector vs python::vector or alike. Otherwise cpp namespace is semantic chaff — and so are pattern_languages. Instead, object-oriented design itself is a part of other broader pattern-language namespaces, something like: engineering::software::object_oriented::design::creational.

Patterns populate a pattern language just like a natural language has nouns, verbs, sentences, idioms etc. So, for the OOD pattern language, we could tentatively outline the following namespace: engineering::software::object_oriented::design::creational::patterns. This seems as redundant as patterns::engineering::software::object_oriented::design::creational, but it is not since now we have obtained the following semantic placeholders:

• engineering::software::object_oriented::design::creational::patterns
• engineering::software::object_oriented::design::creational::forces

As I attempt to show in the section on forces below, the forces acting in the context of a pattern language or individual patterns often stay under-articulated or implicit, which has much stronger impact on the one-off cases whose context may be unclear and requires bespoke structuring. The naming below provides explicit semantic placeholders for forces and thus reduces the chance they will be missed or ignored:

• engineering::forces
• engineering::software::forces
• engineering::software::object_oriented::forces
• engineering::software::object_oriented::design::forces
• engineering::software::object_oriented::design::creational::forces
• engineering::software::object_oriented::design::creational::patterns::forces
• engineering::software::object_oriented::design::creational::patterns::factory::forces

It is common design defect for namespaces to be insufficiently structured, both in pattern languages and in the actual code. For example, it is usual (and painful) to see just one top-level namespace, or no namespaces at all, or engineers getting rid of namespaces (e.g. via the using expression in C++, from abc import xyz or import a.b.c as x in python). It leads to monstrous composite names, conflated design and usage semantics, and code that is hard to understand because one cannot figure out where a type or function is coming from.

As in the quotes in the beginning of this section, the pattern languages often get partitioned in a few high-level bullet-point chapters. Not only the finer semantic granularity — deeper namespace hierarchies — would create a clearer (and more “searchable”) picture of a pattern language, but it also will be a natural solution to a common problem:

E.2 Pattern: Common Problems Highlighted

Context:

You are writing a pattern language that provides several patterns that solve the same problem.

Problem:

How do you make readers aware that they should choose one of the alternative solutions?

Forces:

- … Most pattern forms currently in use do not lend themselves to sharing a problem section amongst several competing patterns without taking some liberties with the form.

- Repeating the problem in each pattern that provides a solution may confuse readers …

- Having the problem repeated in each pattern … increase the effort … to maintain each pattern.

- … All patterns which solve the same problem should include the same set of forces …

Solution:

… You can capture the common problem and forces in one place using Separate Problem Description or Referenced Problem Description…
[Meszaros. Doble 1997, E.2]

The forces and the suggested solution in the quote above feel verbose and inconclusive, especially in regard to the description duplication — as if one is describing a duck without mentioning the word ‘duck’. Finer-granularity namespacing offers a natural solution to this pattern:

Each namespace articulates forces acting in it and its sub-namespaces. Then, instead of the “Referenced Problem Description”, the solution is to simply have a new sub-namespace with its common forces acting in it and multiple patterns resolving them — this relation of containing is much simpler than the one of referencing. A pattern residing in this new namespace and resolving its forces may or may not have extra forces specific to the pattern itself.

The resulting namespace can be very ‘small’ in the scope of the whole language, but that’s the point: if this ‘mini’-namespace looks too idiosyncratic, it is likely that the whole language is under-structured and lacks ‘middle-level’ namespaces — which represent smaller-scale pattern languages — as opposed to just high-level bullet point list of sections.

Based on all that, tentatively, we could say that a pattern is a namespace with a single solution. Or even: a pattern is a (mini-)pattern language with a single solution. Then instead of:

engineering::software::object_oriented::design::creational::patterns::factory

we could have:

engineering::software::object_oriented::design::creational::factory

I feel inconclusive about this naming:

• …::creational::patterns::factory emphasises that the namespaces are populated by a variety of semantic entities: at least forces and patterns. It also reads somewhat better.
• …::creational::factory emphasises that patterns are the first-class citizens of namespaces and forces are not. The good thing is that it homogenises the namespace semantics: …::design::forces, …::design::creational::forces, …::design::creational::factory:forces. It also fits well into the existing pattern formats with their section for forces. The drawback is that in this case forces start looking like second-class citizens, whereas they are the underlying reality, the space in which balancing solutions need to be found.

a city is not a tree

…and neither is the namespace structure.

In his seminal article “A City is Not a Tree” [Alexander 1965], Christopher Alexander demonstrated how organising a city into hierarchically nested spatial, temporal, or functional zones planned in top-down manner leads to the urban spaces that are unlivable, inconvenient and lifeless. Instead, various parts and aspects of the city and its life interact and interpenetrate.

Continuing Alexander’s newsrack example:

“In the case of the drug store example, one unit consists of the newsrack, sidewalk, and traffic light. Another unit consists of the drug store itself, with its entry and the newsrack. The two units overlap in the newsrack. Clearly this area of overlap is itself a recognisable unit. [Alexander 1965]

These units interact across domains of traffic and retail each governed by its own emergent forces, and the units themselves a held together by their own constellations of forces:

The set of particles which go to make up a building; the set of particles which go to make up a human body; the cars on the freeway, plus the people in them, plus the freeway they are driving on; two friends on the phone, plus the telephones they hold, plus the telephone line connecting them …

Each one of these is a set of elements made coherent and co-operative by some sort of inner binding forces. And each one, just like the traffic light — newsrack system, has a physically fixed part which we think of as a unit of the city. [ibid.]

Alexander’s theory of pattern languages is an extension of this line of thought.

(We will not discuss here on the specific formalisation proposed by Alexander that the nature of those relations could be adequately modelled in terms of set theory as a semilattice rather a tree, the latter being purely hierarchical.)

In the same manner, the namespaces — i.e. semantic domains — in the software design (or any other) pattern languages form tree-like nested hierarchies, but the forces and patterns in them interact across the boundaries of pattern languages. Forces of reuse, scalablility or convenience in object-oriented design interact with the forces of the design or implementation unfolding in time, with team dynamics, with market forces and so on, but team dynamics and market forces are never mentioned in the GoF object-oriented pattern language, not because those forces are ignored (see next chapter, though), but because they are not part of the OOD namespace and instead are implied as more or less self-evident in the intent section of the pattern description (while I argue below that the forces need to be explicitly outlined in the context/namespace descriptions of the respective pattern languages).

The “units” in pattern language namespaces are somewhat stable constellations of forces, “patterns of forces”. Force interactions eventually stabilise in some dynamic balance, whether or not we are actively trying to balance them. To do so, we inject force balancing structures and contraptions, which are themselves “patterns of forces” and call the whole thing a “pattern” as in “pattern languages”.

Even without our intelligent design intervention, a ‘natural’ constellation of forces may be naturally “good” — or “bad”. As a few examples, anti-patterns like “spagetti code” are those “bad” constellations; and so, I think, was the romantic idea of the self-organising software team [Vlaskine, 2019, 1].

note on self-organising teams — just cannot help

The self-organising team meant to be naturally “good”, but in all my experience, it never magically happens. It is a fallacy to think of ‘self-organising teams’ as something that happend by chance. The team members need to be very experienced and committed and even then without a very busy team lead the entropy in a ‘free-range’ self-organising software team has a huge budgetary price tag due to the lack of communication, lack of pulling in the same direction, lack of quality assurance, longer development time, and so on.

Building efficient self-organising teams is meticulous. For example, it takes three-four years for Toyota subsidiaries [Liker]. Toyota has been highly pragmatic, methodical — and highly successful — at it. Of course, car design and production are very different from software engineering — but in my experience, the time scale is about right. Smaller software companies with shorter time horizons and higher staff turnaround simply may not have enough time to build and retain such a capability. Many businesses also look at their software team as a liability rather an asset — something they are forced to deal with [Vlaskine, 2019]. The common alternative is just to go lax on team coordination, call themselves ‘self-organising’ and live with the ensuing mess presented as .

Of course, this Brownian motion may still work as long as the organisation is prepared to pay the price.

The remaining challenge is to indicate what such emergent “units” — “natural” or “designed” — have to do with language. We use namespaces to organise our way of referring to those “units”. The “units” relate to each and interact with other. However, these names and relations are not enough to make our collection of “units” a language. Say, there is a very elaborate classification and naming system of animals and a whole discipline studying their interactions (ecology and eco-ethology), but zoologists don’t necessarily use this system as “a language” in a proper linguistic sense — even if there have been developments in that space originating from various sources, e.g. from Uexkuell’s concept of Umwelt.

I give here only a quick sketch and the next section approaches it at a different and yet closely related angle: In pattern languages, the way the semantically significant “units” interact, and connect into language-like structures, and produce other “units” is done through their use (their pragmatics), i.e. deployment and interaction of actual forces (meaning-use relationship [Brandom]). This relationship can be illustrated on the example of software proper: Unlike natural languages, software expressions — even declarative ones — are executable, so, each expression says what it does and does what it says (i.e. it is performative in Austin’s terms [Austin]). The meaning of software (its semantics) and its use (its pragmatics) are two sides of the same: when we run it, the software executes exactly what it expresses, no more no less. On the higher level of abstraction, that’s how pattern languages operate, as well.

language games

As I mentioned before, the overall structure of a pattern language is well described through nested namespaces. Such a nested namespace itself is a subordinate pattern language (e.g. the language of creational patterns for OOD). The granularity of the nested pattern languages can (and should) be quite high: we can have a small relatively self-sufficient subset of situations or force constellations and their solutions.

For example, an architectural pattern language deeper down can have a “signs-of-life” pattern sub-language concerned just with ways to check whether the system or its parts are still alive. Watchdog is one of the patterns of this higher-granularity language.

Devising such smaller-scale namespaces gives a pattern language a much greater semantic clarity. It also is very much aligned with the concept of language games introduced by an Austrian philosopher Ludwig Wittgenstein in the middle of the 20th century. One way to see a language game is looking at it as a pattern in which a natural language is used in a certain context. There are “rules” or circumstances that make a certain use of words and phrases “do their job” of shaping the situation in a specific “game” — which sounds awfully close to how the patterns are described.

Just as pattern languages, language games are not just about saying things, but about doing — and hence probably the most well-known example of Wittgenstein’s language game is about a construction site:

“The language is meant to serve for communication between a builder A and an assistant B. A is building with build-stones: there are blocks, pillars, slabs and beams. A has to pass the stones , and that in the order in which A needs them. For this purpose they use a language [my highlighting] consisting of the words “block”, “pillar”, “slab”, “beam”. A calls them out; — B brings the stone which he has learnt to bring at such-and-such a call. — Conceive this as a complete primitive language. [Wittgenstein, Part I, 2]

Wittgenstein’s language games do not necessarily aim at achieving best gains possible or any gain at all. In a way, their purpose might be just to perpetuate themselves. Language games are about use of the language in given circumstances. E.g. aimless small talk certainly is a language game with distinct ‘rules’ of what is appropriate and what is not. Its purpose is social cohesion and avoiding uneasy silence, but it is hard and mostly unnecessary to evaluate how ‘good’ or ‘bad’ an acceptable small talk strategy is.

The language use can cause and effect material consequences as in Wittgenstein’s construction language game or Austin’s performative utterances [Austin] like “I declare war on England”, which do what they say by saying it — provided that the actor is, say, King of France and not a disgruntled tourist. The same is true for the programmatic statements: they do what they say; e.g. running python code that says socket.connect() and connection will be established, unlike my saying out loud: ‘Socket, connect!’ — I could as well be exclaiming: ‘Beetlejuice!’ (See also [Vlaskine 2013])

A pattern is a solution to a problem in context and so it is a language game where the intent is important and explicit — and so it is in a combination of patterns. It is not just a “word” in “the profession’s vocabulary” [Fowler]. Seeing patterns as language games this way helps to start answering the question: what makes a pattern language a language?

Lastly, ‘pattern’ bears the connotation of repetitiveness, which I attempted to analyse above in the context of “common” and “proper” patterns. ‘Language game’ does not have such a connotation. (Moreover, repetition and ‘rule-following’ in language are a subtle matter and easily lead to paradoxes [Kripke].)

Let us finish with a less serious example of a one-off situation and a “proper” (at least initially) pattern arising from it:

Assume that in the previous example, the builder A gets a new assistant C, a foreigner who does not speak the language. All he has managed to learn so far is that when A shouts, he wants something from C. So, C brings something from the pile of the build-stones whenever he hears A shouting. There is a three in four chance that C would bring a wrong item. The builder A has no choice but to live with it: he still calls for the build-stones as before, except if a wrong stone is delivered, he calls again and again until C brings the right item by chance. Moreover, A leaves wrongly delivered stones next to him. So, if the desired item is already there, A just picks it and does not call for it.

In this example, the use (or pragmatics) of the building language has changed although its vocabulary and even utterances stay the same: keep calling repeatedly for the right items. The situation may be one-off (as they find a better way to train their workers instead). And yet its solution is a new language game and so it’s structurally conformant to a pattern, which is rather “proper” than “common”.

namespacing: conclusion

To summarise, this section attempts to refine naming in pattern language structure. This section does not deal directly with the network of connections between the related patterns. Naming in and linguistic character of pattern languages is too broad a subject to comprehensively cover in a single article and I do not make such attempt above.

I suggest that:

• Many of pattern language descriptions are lacking semantic structure. Seeing pattern languages as nested namespaces, where each namespace corresponds to a pattern (sub-)language, lets representing hierarchies of pattern languages in a homogeneous self-similar way with their granularity naturally scaling up and down. For example, such scaling naturally resolves the “logistic” predicament of the same constellation of forces having multiple good solution patterns. The namespaces are the semantic receptacles of both forces and patterns.
• Despite the namespace hierarchy, the relationship between pattern languages and patterns inside and across namespaces is not just hierarchical: forces and patterns interact beyond tree-like hierarchies (as semilattice elements). The way patterns connect into ‘sentences’ is not just ‘semantics’: it happens via pragmatic deployment of patterns, i.e. through what patterns do, not just what they say.
• Lastly, in the context of “force languages”, patterns as linguistic entities are a special case of Wittgenstein’s language games rather than just something like ‘words’ of a ‘vocabulary’.

forces

Spotting and resolving patterns of the force interaction usually is just briefly explained in the software design pattern literature. The engineer is meant to learn to spot the standard situations and pick suitable standard solution elements (factory, facade, watchdog, policy, etc). It is important to have an idea about well-known patterns, but it is a much more important skill to analyse and resolve specific situations in a structured way as the force interaction. This is something much less articulated in the pattern language literature.

“situational” and “contextual” forces

A pattern description based on a typical pattern template gives the pattern intent or motivation and a list of forces, often quite informally. Such informal descriptions are called “prose patterns descriptions” in literature. “Prose pattern descriptions can be very pleasing to read but may be hard to use as a reference because the forces are buried in the prose.” [Meszaros, Doble 1997]

“Prose descriptions” may be better at leading the reader from the problem to the solution. However, they are harder to use when designing a pattern. The situations resolved by the “common patterns” are recurring and thus often the engineers immediately can relate to them, feel the pain, and appreciate the solution.

On the other hand, the intents and forces of one-off scenarios can be vague and new to the designer. Thus, it is important to distill and explicitly list the forces for a good design outcome, whether the latter is formulated as a pattern or an actual implementation.

For example, below I separated the intent and forces described for the Bridge pattern since its documentation does it only informally:

Intent

[D]ecouple an abstraction from its implementation so that the two can vary independently.

Forces

- When using subclassing, different subclasses implement an abstract class in different ways

- [A]n implementation is bound to the abstraction at compile-time and cannot be changed at run-time.

- [A] class varies often.

- [C]hanges to a program’s code [should] be made easily with minimal prior knowledge about the program.

- An abstraction and its implementation should be defined and extended independently from each other.

- A compile-time binding between an abstraction and its implementation should be avoided so that an implementation can be selected at run-time.

- [There may be a single abstraction but more than one implementation.]
[Bridge pattern]

Some forces are implied in the intent of the pattern. It may not matter too much for a “common pattern”, since it is already assumed to be situated in a broader context of software design geared for reuse, high performance, ease of change and maintenance, ease of deployment, etc.

Let us list some of those implied forces: user’s convenience, development efficiency, frequency of code change, code complexity, code readability, code modularity, code dependency management (e.g. not making the class user dependent on the library the class implementation depends on), performance, quality assurance, maintenance, language, reuse, and many others.

These forces are implied, because supposedly they always are on the mind of software designers. To an extent, they are — or should be — part of their intent. However, while it may be true for the author of the pattern, the user of the pattern may be less steeped in the domain, less informed, distracted, or very junior and thus not entirely comprehend the constraints and degrees of freedom of the solution.

Of course, the power of “common patterns” partly lies in their brevity and focus on the significant structural loads, tensions and constraints of a recurring situation. The implied forces represent the overall context of a pattern language, e.g. the object-oriented design pattern language in the example above. Explicitly repeating them in each pattern would bloat the documentation.

From that point of view, the broad context, domain, or “namespace” (a collection of nested namespaces really) of a pattern language exists before the patterns themselves: “object-oriented design pattern language”, “web design pattern language”, “business modelling pattern language”. The practitioners usually are already well-versed in the domain and its major interacting forces. They also know what they want and thus they naturally grasp the intent of a “common pattern”.

On the other hand, there may not be a “pre-existing” pattern language to which the situation or solution belongs. Its context may need to be discovered first and the implied forces may not be “obvious” until articulated explicitly. The situation may have uncommon priorities and constraints and less common trade-offs may offer a big leverage. Higher performance may be often achieved by reducing code readability and modularity. Sacrificing reuse to a specialised design leveraged on particulars or even a quick and dirty approach may sometimes simplify and speed up implementation. (Way too many software professionals overlook ‘may’ and ‘sometimes’ in the previous sentence.) Thus, the design would benefit from the forces explored and articulated first.

In the list of the explicit and implied forces of the Bridge pattern above, we notice their various qualities:

Some forces represent tensions in their interaction. Tensions are what the solution is meant to reduce or balance.

E.g. the force “[A]n implementation is bound to the abstraction … cannot be changed at run-time” represents tension of a “straightforward” class implementation vs modularity and decoupling.

The one-off cases may have lots of specific tensions that are implied or completely out-of-scope of the “common patterns”: tensions between idiosyncratic user’s needs and technology available, between development time and reuse, etc.

Some represent positioning of the structural elements.

E.g. the Bridge pattern talks about classes, sub-classes, class abstraction, implementation, as well as compile- and run-time. Those are the ones the “common patterns” usually outline explicitly, even when written in “prose”.

The one-off cases may be dealing with very specific structural elements or actors. For example, the rollout of a product may be staged in a certain way, which may dictate a specific order of iterations or delivery milestones. A “common” design pattern language, e.g. object-oriented design pattern language, naturally says nothing about the order of implementation. However, its intent clearly implies convenient, decoupled, modular design process. A solution often has a clear time dimension: how its minimum-viable form accommodates later increments, how it preserves backward-compatibility, and so on.

Some represent structural constraints.

E.g. the Bridge pattern imposes the constraints of ‘minimal prior knowledge about the program’ and avoiding ‘compile-time binding between an abstraction and its implementation’.

The one-off cases may be constrained by specific forces like the low compute power or memory footprint, tight workforce or time budget, and so on — the constraints that may not be so pronounced or specific in generic solutions.

Some represent “structural loads”. Those forces impact or ‘stress’ the pattern resolution like the loads gravity, wind, or moist impose onto the structural elements of a building. Forces of this sort may not change the structure of the solution, but they may change it’s form ‘topologically’: you still may have a gable roof both in Cairo and in Oslo, but surely in Oslo you will see steeper gables to withstand the weight of snow in winter.

E.g. in the Bridge pattern, number of class implementations is one of such loads: the pattern remains the same, but if there is only one class implementation, the “instantiation” of the pattern becomes simpler and more like a plain-vanilla pimpl idiom.

The solution may not be able to hold structural load beyond a certain point and we may have to bifurcate to a different solution altogether. Say, if the consumer in the real-time data processing stream becomes too slow, the plain pipeline pattern will not hold anymore and we may need to parallelise data processing or introduce discarding mechanisms. For example, in the one-off cases, specific consumers may be particularly slow (e.g. dependent on an external communications) and thus require a very tailored approach.

Some accentuate priorities

E.g. modularity and decoupling are priorities of the Bridge pattern, at the expense of code simplicity of a plain-vanilla class.

The priorities of the one-off cases usually are strong and vague at the same time. Take the needs of the artefact user: those needs have high priority — that’s how the special case often comes about, e.g. in the form of a user story. On the other hand, the actual user (not just a stakeholder) often needs to be carefully identified — and their actual needs can be further negotiated and balanced against other forces. Instead, the engineers tend to guess or assume what the user needs — and be very wrong about it half of the time.

Some present opportunities of leverage or degrees of freedom. “Reducing” or “cranking up” such a force may lead to a spectrum of solutions.

E.g. “there may be a single abstraction but more than one implementation”: reducing the number of implementations to one would lead to pimpl-like solutions, while multiple implementations may be achieved through the use of policies, templating, or alike — all of those could be formulated as sub-patterns rather than language-specific solutions.

The one-off cases may be very constrained in some respects (e.g. time budget), but (often by the very merit of those strong constraints) have much more latitude along other dimensions. E.g. sacrificing reuse (without sacrificing quality) may lead to a simpler and quicker design; or sacrificing modularity may improve performance.

Obviously, a force may be characterised by more than one of those qualities.

Some forces are more “situational” as they represent the disposition of elements and tensions in their interactions.

Other forces such as priorities and leverage opportunities may be more “contextual” . They often do not even appear in the pattern description as they rather act across the whole pattern language context or one of its namespaces.

The “contextual” forces mostly are implied in the “common pattern” descriptions and often are only barely touched upon in the introduction to a pattern language as “self-evident” due to the optimistic assumption that the reader or user of the language would be familiar with the domain, keep its context in mind all the time and take its problems at heart, while — as it happens way more often than not — she or he easily may be junior, have gaps in their knowledge, limited scope of exposure, entrenched views, or just forget or ignore certain “contextual” forces in a given situation or in general.

So, the following sections will cover some of those forces in more detail.

assessing trade-offs

A “common pattern” is a well-balanced, well-established solution that may not require much of further reasoning about trade-offs. A “proper pattern”, on the other hand, almost always does, since by its definition it is not one size fits all:

Does the solution need to be entirely monolithic to achieve desired performance? Does the user really want full automation or he/she would accept a sensible human-in-the-loop outcome? Does it really need to be done by close of business on Friday?

(Those sound like questions about the specific implementation, not a pattern, but my point is that any situation should first be analysed in conceptual terms and thought of as a pattern that characterises the system or its aspect like partitioning, automation, performance, or time management.)

Once the “situational” and “contextual” forces are listed, we can start assessing priorities, constraints, available degrees of freedom and trade-offs they permit. To make sure, when we list forces that we would like to balance, we do not just enumerate them as they are but formulate how they manifest in the given — generic or specific — situation.

Of course, a design force is not a unidirectional variable, but rather a metaphor for a factor that often is complex, structured and hard to measure, such as “convenience”, “simplicity”, “flexibility”, “importance”, “quality”, and so on. Those factors unlike physical forces are not acting independently, either. Nevertheless, design forces to be balanced create a good mental picture that is conveniently vague. Keeping this image in mind in any design process is a great remedy from the black and white thinking in terms of true or false, right or wrong, good or bad, yes or no, which is so common in client requirement collection, boardroom meetings, and design discussions.

The last paragraph sounds like repetition of what was said above a number of times and thousands times before by others. That’s the point, though: It may be enough to prove a theoretical statement once. However, a good practice is only good as long as it is practiced and walked through repeatedly.

In reality, the design conversations at any point tend to constantly and entirely slide either into implementation details or into black-and-white decision-making depending on whatever specific shiny object that happens to be on the speaker’s mind at the moment. (If you feel offended by this statement, I want you to be because I hope it would make you more reflexive in such discussions.)

The remedy for sliding into implementation details — the implementation bias [Vlaskine 2013] that plagues software engineering profession — is a constant reminder to everyone to speak in terms of the user or usage semantics of the designed artefact.

The other problem — black-and-white thinking — often comes from desires (“right now, I really want this feature — in a week from now I won’t”; “if I don’t win the argument, I’ll feel diminished — I’ll forget it after lunch”; etc) and power relationships (“this is what we agreed upon with the customer — the customer may be flexible but engineers have no access to the him”; “do what you are told — even if it is not right thing to do”; “use this framework because everyone does, everyone cannot be wrong — they can“; etc). These one-factor-in-mind arguments are rationalised as the classical logical reasoning with its sentences assuming the values on the scale between true or false, whereas the multi-factor analysis rarely has a single right answer.

Whenever the conversation veers off into the black-and-white argumentation, it is worth to constantly restate the problem in terms of forces. (Having the list of forces displayed in front of everyone during the meeting could help.) Provided that participants listen to each other, it may be very non-confrontational too since by nature the forces cannot be qualified as just right or wrong. When balanced in a certain way, they produce a certain structure and consequences.

However, it often creates a different problem: Since structured consequences are all that forces produce, you may not be able to “prove” your point in the discussion, but only by demonstrating a working solution, e.g. a working prototype. In the original discussion counterparts may not see the value of the solution and it will become an endless and pointless religious war. So, whenever possible, it is better to run a tight cycle of small iterations between discussions and prototype demonstrations — something that I call ‘performative negotiation’ oscillating between saying and doing/showing [Vlaskine 2014] — which happens to be the structure of patterns themselves with their semantic and pragmatic sides.

Now that we keep in mind the vague picture of forces in a context, what does it mean to balance them?

We would need to assess “magnitude” of each force weighted by its “significance” and its “direction”; as well as their “sum total” for a given solution, which may be “better balanced” for one solution and “worse” for another. We cannot do any of it: our forces are just a metaphor. Our factors range from memory footprint to market forces and from a compilation unit to human desires. They do not have “magnitude” or “direction” and cannot be “summed up”. All of those are metaphors for qualitative assessment of the solution.

There is a strong evidence that humans including top domain experts are extremely inconsistent, almost random at such multi-factor ranking — for example at ranking a dozen job candidates or patient treatment schedules — however people are very good at comparing things pairwise (see e.g. [Kahneman et al] for heaps of striking examples).

The same goes for the forces: unlike in physics, there is no formula to just sum up the design forces as vectors with certain directions and magnitude. Instead, we can say: Client satisfaction is more important than time to market; by slightly reducing client satisfaction we can release our product much sooner. Or memory footprint is less important than computational complexity; so, we can afford to waste a lot of memory on pre-computed tables to somewhat boost CPU performance.

The pairwise comparison allows us to rank the significance (“weight”) of each force and assess the variation of its “magnitude”, even if those “magnitudes” are incompatible across the forces (time, desirability, ease, speed, memory usage, quality, etc).

Firstly, let’s call the “positive direction” of a force its preferred “direction”, the direction that represents less cost of some sort: higher reuse, higher quality, less computational resources, shorter development time, smaller team size, shorter time to market and so on. (To make sure, it is not a utilitarian view. E.g. in the ethical dimension, the “good” direction is less costly in terms of trade-offs with one’s conscience and many people are prepared to endure substantial harm to their interests rather than do something they consider evil.)

So, in a pairwise manner, we can assess “directions” of two forces: Two forces pull roughly in the same direction if cranking either of them up contributes to a similar effect.

For example, increasing (to an extent) the team size and increasing the development time produce a similar effect in terms of the team’s output. Thus, development time and team size (to an extent) point in the same direction. Everything else being equal, cranking up one (making it less costly) also allows to reduce the other. On the other hand, everything else being equal, increase in quality (“positive direction”) would require increase development time (“negative direction”) — the directions of these two forces are opposite.

Elasticity is another useful tool — or at least a mental image — for the pairwise comparison of which force is “stronger” and how much of leeway to “budge” each of them has. In economics, price elasticity of demand is the change of demand for a product when its given price goes up by 1% the rest being the same, i.e. how much people reduce their consumption of the product if the price rises. The greater is the demand drop the more elastic the force is. There are elasticity metrics on other parameters: income elasticity on demand, wealth elasticity on demand, and so on. Needless to say, demand happens to be one of the “market forces”.

We can use elasticity as a qualitative analogy and, sometimes albeit rarely, as a quantitative metric to characterise our forces against each other: convenience, quality, speed of development, reuse, etc.

For example, increasing the team size first speeds up development, but quickly reduces the gains due the the overheads of hiring, coordination, and limits to how much you can parallelise the work. We could say speed of development quickly loses its elasticity relative to the team size as the team size grows.

Sometimes, even pairwise comparison is not straightforward without more indirect reasoning in small cohesive groups of forces rather than pairwise. Varying regression test coverage has no effect on computational performance and vice versa. These two forces look independent or “orthogonal”. However, regression test coverage has a strong effect on quality: better coverage leads to higher quality, and accepting lower quality allows to save time on testing. Quality and test coverage “point in the same direction”. On the other hand, squeezing all computational performance you can beyond a certain point demands idiosyncratic tricks, which sacrifice code simplicity, readability, uniformity and reuse and make code brittle. Thus, enhanced computational performance may reduce quality and these two forces pull in the opposite directions. The higher regression test coverage could balance that: when tests protecting the slower reference implementation pass on the faster variant, it gives more confidence that the faster but more cumbersome and thus more error-prone code still works as expected. Thus, regression test coverage and performance pull in the opposite directions: cranking up the latter may require to crank up the former.

The balanced forces mean that the “magnitudes” of competing forces “weighted” by their priorities “equalise” in a dynamic but stable equilibrium.

Assessing priorities/”weights” sounds easy: just talk to your stakeholders and they will tell you. What they will tell you, though, is that ABC, DEF, and XYZ all are essential and it is crucial to work on all of them — a typical ranking problem at which the humans, even experts, are patently inept [Kahneman et al] not to mention the extra biases of all-or-nothing reasoning and wishful thinking. However, you could make the ranking easy by unrolling it into the pairwise comparisons along the costs axis, e.g. along the timeline: what if by the end of the month, we deliver DEF, but not ABC? Or the other way around? Will it work if you get but not DEF? An so forth.

some of the engineering::software::forces

…representing priorities, loads, and leverage opportunities.

There rarely is anything inherently good or bad, or right or wrong about a force. There is only a price tag attached to each of them in terms of their consequences, i.e. their effect on other forces: costs, time to market, quality, etc.

The leeway of each force is currency to spend and spending it without a gain is a waste, which can be huge. Engineers often dispense with decoupling and reuse really early just because they are rushing through the design effort. The business development team can be aggressively hostile to any time budget allocated to testing since the latter does not create new product features. In both cases, the missed out capability and lost quality become wasteful hidden costs that snowball downstream and can bring down the product and even the whole company.

So, it is important to make those price tags explicit and — especially in the one-off cases — to continuously reassess priorities, directions, and magnitudes of all forces.

The cost transparency itself is a force on the spectrum between hidden and explicit. Hidden costs (of any sort: quality, time, effort, money, human capital, capability retention, and so on) undermine or severely cripple products, timelines, trust, knowledge circulation and morale. There are myriads of examples of projects going tenfold over their budget or duration and colossal amounts of unnecessary miscommunication and waste. Hiding or ignoring costs has been a very unfortunate part of business culture as the only way to get job commissioned. On the other hand, shielding stakeholders from certain costs improves the work flow and relationships. For example, early in one of my projects, the external stakeholders requested us to file with them our sprint burndown charts. We refused and instead agreed to judge our sprints and milestones by their demonstrated deliverables, which worked really well for the next five years until the project was successfully delivered. We shielded our client from our internal planning forces and price tags so that the client could monitor our progress based on the content and scale relevant to them.

One should try to satisfy each force as much as possible as long as it can be done by acceptably small concessions from the other forces. For example, when prioritising work, it might be natural to always pick the most urgent things. The problem is that the items that are not immediately required never bubble up to the top of the priority list and thus never are worked on. Instead, you could commit to always spend 5–10% of time on slowly building the capability for the future. Most of the time it is an acceptable trade-off, which balances prioritisation force vs time-budgeting force.

With some day-to-day practice, the engineer will start to almost automatically apply those mental “sliders” to any development problem, large or small.

The rest of this section highlights (in no specific order) some well-known forces that are almost always present but often remain implied, hidden — and eventually unaccounted for.

The more experienced engineers understand these forces and incorporate the respective trade-offs in their design decisions — sometimes through their muscle memory. The less experienced engineers and many stakeholders tend to overlook, ignore, or be totally unaware of these forces and their consequences.

Also, even when recognised, the forces often are treated as maxims: reuse — always good; low test coverage — always bad; and so on. As the result, the trade-offs may look like deficiencies rather than points of leverage and thus be treated as such. Treating each force as a spectrum helps to correct this misperception.

The next sections give examples of some overarching design forces. The following enumeration of forces, once again, may sound like a platitude: everyone knows (or should know) about reuse, decoupling, or separation of concerns. As mentioned before, knowing is not enough — it is about continuous repetitive practice: When you design or code, do you always keep each force in mind? Do you have the visibility of its consequences? If not, who does? Do you involve that person in the design decision process? Do you see each force as a spectrum, not just something unconditionally good or bad? Do you make it explicit how the pairs or small groups of forces interact? Do you tentatively match forces against each other to establish their relative “weight”, “magnitude” and “direction”? When you budge on certain forces, do you assess the consequences, price tag of your move and the gains it brings? Habitually applying this due diligence to one’s design process is a non-trivial skill that needs to be learned, beefed up, and maintained. Thus, the following sections portray well-known design forces as spectra of their considerations, consequences, priorities and trade-offs.

usage vs implementation

This is one of the strongest design forces, which always gravitates to usage.

The engineers tend to plunge straightaway into how they would implement a certain feature and to pay little attention to how and by whom (or by what) it will be used. Starting any design with a dedicated stage of settling with usage semantics corrects this common implementation bias.

“Strong” here and below means “high priority” or “high weight”. Qualitatively, “gravitates” could be interpreted as quickly becoming less elastic as it slides from usage toward implementation. It means that if we sacrifice some usage coherence and convenience, we may settle for a moderate gain e.g. slightly shorter development time. If we sacrifice “twice” as much of well-designed usage, we need “non-linearly” much more gain on development time — which is rarely justified.

One way to probe usage semantics to avoid the implementation bias is to always speak in the user’s language. At any point of design, the engineer should be able to say in the user’s language what she/he is going to implement. As simple as it sounds, it takes a sustained effort, self-awareness, or external coaching to break the habit of constantly sliding into implementation details — especially if the whole team or group is prone to it.

User stories are one among good tools to design usage semantics.

The pattern languages actually are another one since they represent a way of thinking and speaking of design that naturally distances itself from implementation: by the merit of reuse you cannot formulate a “common” pattern without consistently abstaining from implementation details. The one-off cases lack such a mental safeguard: since they are about specifics, it is tempting to conflate their specifics with technicalities and plunge into the latter. If in the one-off scenario, we keep the mindset similar to the mental effort of working on a “common” pattern, we are more likely to come up with a solution as a “proper” force-balancing pattern from which a concrete implementation will follow later.

reuse: generic vs specific

This design force is strong, but offers a clear-cut heuristic, which almost always works. It is the already described rule of two extremes:

Do not try to balance this force and meet somewhere in the middle: either be as generic as possible or as specific as possible. See the section above for details.

This force gravitates to the points closer to both ends of its spectrum, where it produces much more leverage than somewhere in the middle. Going more extreme beyond those two sweet spots sees the leverage sharply droping as the item becomes either over-generalised or over-specialised. Over-generalisation means over-engineering. It requires disproportionally more time and effort and results in clunkier usage semantics. The getter/setter (anti-)pattern serves as an example of it among thousands of others. Over-specialisation tends to unnecessarily sacrifice reusable capability, reduce usability, increase coupling and reduce modularity.

On a grander scale, the literature on design pattern languages primarily emphasizes pattern reuse, suggesting that only “common” patterns are worth considering. As a result, the pattern-style methodology for addressing one-off scenarios is oddly overlooked, to the detriment of the design process.

On the other hand, marketing, project/product leads and engineers all are famously and strongly biased toward the most specific end of the spectrum. If they rationalise it at all, they might quote lack of resources, utmost client focus, and not see much value in generic stuff, since to them, reuse does not seem to contribute to their specific project goals.

As a result, several projects or engineers keep re-implementing the same solutions, each of those has its own bugs and requires its own testing; there are missed opportunities across company’s products, workflows, usage semantics, and interfaces; there is more code and data formats to maintain; the finished or cancelled projects bury all the knowledge and capabilities within them, etc. These price tags outright annul and outweigh by far the perceived development speed gains.

This bias is a manifestation of the tragedy of commons. A more structured way to look at it is distinguishing between building capability and deploying capability [Vlaskine, 2019] and seeing value in both. The need of capability-building commonly remains hidden: when it is not there, everyone just wants to tick their own box; once it is there, everyone is happy to use it for free; and so, either way, no-one sees the need to invest into it.

unique vs duplicated

This force is very strong. It seems to gravitate to unique. It is related to reuse, but is not the same: a design, artefact, piece of data or documentation can be unique, but not reusable and vice versa.

Any code, data, or information (e.g. in documentation) duplication poses a very high price tag, an overhead of preserving integrity and danger of losing it: The data gets updated at one place, but not the other. The code behaviour is changed in one library, but not the other. Two behaviours being “almost the same” is one of the worst nightmares in software engineering.

On the other hand, spawning multiple instances of the same item is one the centrepieces of industrial production. The whole point of code or data in their distribution: releases, deliveries, installations, multiple points of use, views extracted from databases, etc; i.e. there is a strong need in their duplication. While there is extensive data integrity methodology, software design discipline is much less formalised.

That’s why automated delivery is so important, but it does not guard against the information duplication on its own. So, accounting for this force in the design phase is crucial. Duplication should not be there at any scale unless duplication provides real leverage rather than lets one to spare extra few minutes of mental effort. If we decide to pay this price, then we need to put together a counterbalance to it, for example an automated code or data integrity check or a test assuring that two duplicated bits match each other.

I am talking not just about automated installation and release processes, but also about the design decisions at very high level of granularity. For example, assume we need a definition of speed of light in a C++ header and in a python module and it is too expensive for us at this stage to encumber our code base with python bindings. As any duplication, it should be a red flag (unfortunately ignored by far too many engineers). To address it, we could add a regression test that compares our C++ and python constants. Therefore, we probably need a C++ application that prints all our physical constants, which may be useful in other context. Then, if we defined it as 300e6 first, but later our python engineer decides to do the right thing and set the python constant to 299792458, the discrepancy will be caught by the automated test suite. This example is deliberately trivial, but triviality bias — “it’s trivial therefore it can be ignored” — is a scourge in software engineering.

coupling: modular vs monolithic

This is a strong force massively gravitating toward decoupled and modular. Coherent decoupling is one of the most fundamental software design principles expressed in many forms and flavours, e.g. as single responsibility principle, separation of concerns, and so on. It has a huge price tag to dispense with. It also is the one violated at the drop of a hat.

Modularity somewhat correlates with reuse, but obviously is different: a class may have single responsibility and yet serve a very specific one-off scenario. Moreover, if it a one-off, it usually requires more thorough testing of its idiosyncratic elements, which is exponentially harder to do on a monolithic piece: in rough terms, if each individual functionality requires 10 unit tests and there are 3 of them all spliced together in, say, a single class, then instead of 30 tests you might need to implement 10*10*10=1000 of them for the same coverage.

The ubiquitous and strong bias in the one-off situations is as following: The solution serves just one scenario, therefore, the solution sits at one place, therefore it all can be a single piece, monolithic with poor separation of concerns — with the obvious consequences of poor testability, effort duplication due to low reuse, poor scalability and being hard to change. Unfortunately, these price tags tend to stay hidden for those who don’t feel the pain — e.g. by having the opportunity to throw things over the fence or to move on — or don’t look beyond the next milestone or release.

There are very few cases, like quick throwaway prototyping, which may very conditionally benefit from monolithic designs.

simple vs complex

It is a very strong design force leaning toward simplicity, but balanced by the desired sophistication of a product or software artefact: ‘as simple as possible but not simpler’.

Complexity is very expensive, but it is pervasive in software industry to stop simplifying too early.

The need of simplicity is so strong that the persistent question should be: Can we do less or even nothing at all (provided that it does not blow up costs elsewhere)?

This force is strongly related to the development timeline. If we have to do something, can we do it later (when we have more use cases, more clarity, or we decide we don’t need any of it at all)? If we do it later, can we just provide a conceptual or actual placeholder right now instead of actual implementation so that we have the door still open when the time comes?

Anything has higher complexity than nothing. Any extra complexity, even minor, has a high price tag in terms of development and especially of maintenance and scaling. Anything is much harder to maintain than nothing. (That’s why code duplication and dispensing with reuse have such high price tags.)

‘Doing nothing’ above is not a hyperbole. Trivially and weirdly, there are many ways of doing nothing. For example, in data modelling and management, choosing simpler and more uniform data formats eliminates the need to implement any project-specific tools since the existing generic utilities suffice.

Any use case analysis should start with the question: “Can we do nothing? How?” Often, ‘no’ is the clear answer. But it is not a good reason to ignore this question: always ask it explicitly. If ‘no’ is too obvious, this consideration will take you a few minutes or seconds. Because in software design, the answer often is ‘yes’ and doing nothing has huge cost leverage.

I mention this trivial practice in attempt to correct the ‘sophistication bias’ (which is a flavour of ‘triviality bias’): “we are too sophisticated to bother about trivial things”.

It is worth revisiting design decisions over and over throughout the design and implementation cycles to see in hindsight if anything can be further simplified. These repeated passes and revisions may require a bit of extra time and effort invested into design, but it is minuscule compared to the cost savings even in the short run.

As a trivial anti-example, at some point in time, the obsession with getters and setters made class definitions literally three times longer, drowning the class meaning in a repetitive wall of text. Brevity is a proxy of simplicity.

One more thing: something conceptually simple does not always mean it is quick to implement. Yet, the payback of simplicity usually outweighs this extra effort by far.

clean vs dirty

It is a very strong design force leaning toward clean. In a way, clean is a step toward simple (just like brevity is a proxy of simplicity).

The common bias: clean design is time-consuming, more expensive, hard, and is eventual rather than urgent — as opposed to dirty seen as cheap, easy, quick and the place to start. The quick-and-dirty bias suggests that instead of cleaning up things, we should find a way to kick it down the road.

As the result, the price of dirty keeps compounding. The common outcomes (which usually become obvious in the matter of weeks or months): The development slows down and eventually comes to a standstill so that things require constant debugging, reverse engineering and have to be periodically redone from scratch. The capability development does not progress since the code or data is hard to reuse and thus the things get re-implemented over and over with new bugs and new mess. Eventually, the products lose race to market, market share, or whole businesses close doors because of sleeker, tidier competitors.

Clean is seen as pay-upfront, while dirty get-now-pay-later. Except, the upfront — or rather ongoing — price almost immediately pays back since the development progresses smoothly, scales well and is fun, while the ‘later’ price of dirty implementation is large, keeps snowballing and makes development process stressful and depressing. Choosing the dirty solution regardless is an instance of the preparedness paradox bias as we tend to ignore hidden benefits and hidden costs. Exposing those benefits and costs is a part of the informed force-balancing.

What does ‘dirty’ mean and how it happens? It habitually happens because of the problem at hand may look too hard or too easy.

• Too hard: The engineer may want to fully focus on the problem and ignore everything else including proper design effort. Also, the problem may look open-ended. In this context, the consequences are ignored and thus the solution ends up overfit and coupled.
• Too easy: It looks like nothing there to see: E.g. the engineer may write her/his own class for Cartesian coordinates instead of using (or implementing) a generic class that would be consistently utilised across the company. It ends up in a lot of duplication and therefore lack of testing and lots of data and type conversions required between two pieces of software or two datasets.

Many things still have to start dirty. A lot of legacy code or data is dirty, too. If it is inevitable, at least we could guard ourselves by corralling the mess in clear enclosures: wrapper functions without side effects, self-contained data, and so on. One of the missions of the software engineer is to manage the world’s mess through cleanliness and order.

easy vs hard

We mostly want it easy. However, novel research or things that sell often are hard.

The relatively common bias (for which I don’t have a name since “complexity bias” is taken) we would like to balance and correct against is: We are dealing with complex stuff therefore it won’t be easy.

To balance this force, we separate concerns and identify constraints:

For example, a given algorithm may be complex and open-ended in terms of research, but we still can implement all the simple and well-known parts: the interfaces, library-building, data conversions, etc and localise the hard part(s) in a given class(es). As another example, an ML training problem can be hard and open-ended, but we still can separately develop a toolkit for training dataset manipulations and insights. By chipping away all the easy bits, we localize the hard parts. Once distilled that way, the remaining hard parts may even be not as hard as we first thought.

As for constraints, a general or theoretical problem can be hard, but if we think through all the constraints afforded by our use cases — especially in one-off situations — we might greatly reduce complexity of the problem at hand. As one of myriads examples, graph-colouring problem is NP-hard in general, but if our graph is planar, the colouring can be done in polynomial time.

novel vs well-known

This force is closely related to the easy-vs-hard force and is balanced in the similar ways.

Can we solve the problem, using existing methods and practices or we need to come up with something never done before?

This force does not gravitate to either end of its spectrum but rather needs to be balanced via separation of concerns. It is a major design force in the innovative context, e.g. in R&D:

On the one hand, by definition, pre-existing solutions may not entirely cover novel problems. On the other, one often hears from researchers that the system they are trying to come up with has no precedents and therefore is totally open-ended: we cannot tell upfront how we will deal with it, we cannot produce a road map, time estimates or deliverables for it and so on. It cannot be made less fuzzy and clients will have to accept high risk and high uncertainty. It is a surprisingly common cognitive bias and saddling one’s client with uncertainty of 1–5 years (or maybe never) is rarely is justified.

Because of the novelty, it is important to identify any aspects and constraints of the system under design that are well-defined and can use existing solutions and best practices. Chipping off the problem this way reduces its “dimensionality” and number of moving parts; and thus makes its difficult open-ended part more articulate, tractable, constrained, and specific.

urgent vs eventual

In a large part, balancing this force is about correcting for biases.

Eventuality bias (I made up the name): “Since we eventually need this feature, let us work on it now”. (No-one will use this feature for the next two years. Until then, we will make many design decisions and time management by speculation. By then, people who worked on it will move on, many design decisions will make no sense in hindsight, and either elements of it or the whole thing may not be required anymore.)”

Urgency bias (I made up this name, too): “It is urgent because I want it very much at the moment. (If you tell me it’s less urgent compared to other work, I’ll want it even more. Especially if I am your boss, I don’t want you to implement it. I want you to implement it. What happens to it later is none of your business.)”

In both cases, the pervasively common danger is to spend resources on low-priority or unnecessary things and eventually run out of time or budget.

There are many simple good practices that correct for these biases:

For any feature, identify who is its intended user and when they would use it.

Even if the feature is a must-have in a product, if we implement it too early (meaning it won’t be immediately used or plugged in):

• We will spend resources on something that clearly can wait instead of working on urgent stuff
• When the time comes to use the feature, we may gain new information and experience — it’s better and cheaper to wait until then
• It is healthy for the requirements to change as the client and team gain more information and experience in the course of the project. The feature may be dropped altogether later, thus don’t do it now
• In the meantime, we will have to maintain the feature in a workable state and keep its documentation consistent until someone actually uses it — which may not ever happen

However, there are a few forces that push for early implementation even when the artefact won’t be immediately used:

• High risk: If we don’t have the feature, the consequences for the project or its part are catastrophic
• High uncertainty: e.g. we are not sure what methods to use, how long it will take, what exactly the client wants, etc
• Large amount of development: if we don’t keep chipping off it, its sheer scale may become a high risk later
• High impact: One common theme: “If that utility existed, I’d use it now.”

All in all, urgency management needs to be crafted for specific situations as in Toyota-style and lean development flow management. A few powerful rule-of-thumb recipes sound like:

• Low risk, low uncertainty, no immediate user: don’t do it now! work on something else
• High risk: devise and do something to reduce risk now or soon! e.g. do an early feasibility study — not necessarily implement the artefact itself yet
• High uncertainty: devise and do something to reduce uncertainty now or soon! (e.g. do more breakdown to reduce timing uncertainty or experiments that help to select one of several technologies)
• Low effort, high impact (lots of immediate use cases): do it now!
• Lots of effort: start chipping off

As I mentioned above, comparative judgement and pairwise ranking are necessary tools to assess urgency.

One common problem is a low-priority trap: Assume, we have very good urgency ranking. We may have useful items on the backlog (“If I had them, I would use them right now, but with a bit of effort I can live without them” — that kind of stuff). If we always work on the items of the highest priority, these very-nice-to-have pieces never bubble up to the top of the queue. The planning technique of dealing with it: consistently allocate 5–15% of resources to lower-priority items, cleaning, technical debt, etc — in the spirit of continuous improvement.

expensive vs cheap

Its direction is toward cheap: everyone wants lower costs albeit not by any price.

It is a sort of meta-force since it characterises all other forces. For a given force, the cost may come in the form of monetary value, development time, effort, time to market, human costs (e.g. stress, morale, job satisfaction — which imminently convert into human productivity), ethical and societal good or harm and so on.

When assessing the price tags of an item, hidden costs are a huge problem in general and especially in the software realm. Wishful thinking and reality denial are common human biases. The desire of getting it cheap or selling it expensive also encourages the systemic neglect or coverup of hidden costs. Further, software is not material and thus its scale and impact are even harder to assess with the errors commonly being of orders of magnitude.

note: factor-of-five rule

E.g. from years of observation, I’ve come up with an empirical factor-of-5 rule for time estimates: When giving honest development time estimates — until proven otherwise through a proper breakdown and analysis — it is safe to assume that one’s sizing of a software item is underestimates the actual effort 5 times even among experienced specialists. Typically, their gut-feel estimate takes into account the mainstream functionality and leaves out of picture usability, corner cases, testing, debugging, deployment, follow-up, and maintenance. If you think a software-related piece would take 10 minutes to write, it will take an hour; 1-hour estimate will result in a day of work; 1-day in a week of work; a week of work really will take a month, one month really is half a year, etc.

Thus, initially, the honest time range estimates for items on the backlog should look like:

X: 1–5 days

Y: 1–5 weeks

Z: 1–5 months and so on

Clearly, when planning a year-long project, the uncertainty of X is acceptable, while reducing the uncertainty of Z is a high priority item (see urgency recipes from the previous sections).

Since normally you do not want to blow your budget five-fold, exposing hidden costs is one of the first steps of balancing forces. Thus, leaving the “contextual” forces of a pattern language implicit (because they may seem too trivial, obvious, habitual, or boring) bears the danger that they will not be adequately shared across the team or organization, or will be ignored in the design altogether. It is especially important for the one-off cases, since the context of the “common” patterns is more or less settled.

Overcoming the hidden cost bias requires a constant conscious effort. The bias is a part of human nature and won’t go away (just like many optical illusions persist even when you know that your eyes deceive you). However, building the habits of the conscious balancing act will correct for it:

• Decomposing the problem and evaluating its parts and aspects
• Making Definition of Done and following through on it for each item
• Making an explicit list of forces, going through it and putting price tag for each of them on any given design move: what do we sacrifice and what gain is this sacrifice paying for?
• Gauging against past experience: There are formal methods to do so, e.g. development velocity tracking, but at least it is useful to make it explicit in the review or retrospect what the actual costs are against the original projections.

“objective” vs biased

This force leans toward objectivity since biases create costly errors of judgement.

As software design is concerned, when choosing what the actual state of affairs is, it is very hard to avoid cognitive biases — hundreds of them. So, it helps to continuously be on lookout for biases, make them explicit, and methodically correct for them individually and in the team.

It requires lots of trust in the team and organisation, making sure that people can speak up without fear and the imbalances that are brought up are perceived not as criticism, but as honest reflection that is worth discussing and acting upon.

Of course, pushing too far toward objectivity and excessive rationalization in itself is another bias to guard against with the examples like rigid design practices such as the “classical” object-oriented design, inflexible coding standards, quantitative human performance metrics like infamous lines-of-code as an extreme and so on. There is no single “correct” or “objective” way.

Rather, the “actual” state of a piece of software is not what is there “in reality”, but what that piece means (and to whom). This may sound just as subjective as a cognitive bias, but it is not: software is a language enterprise and its meaning is in how it is used [Vlaskine 2013]. Its use or usage semantics is not the same as objectivity of material things and expecting it to be the latter is a strong bias — or rather a fallacy — to be aware of and avoid. So, software that is used and works, tests that pass, definitions of done that are fulfilled and other use-oriented practices give the necessary grounding in reality.

The other aspect of balancing this force is that lots of biases are rooted in human emotions, just as our motivation is. The deliberate and methodical elimination or neglect of emotions — including biases — in software enterprise operation has negative impact on motivation, sense of direction and decision-making.

It is true not only for the personal motivation. For example, it is the rolling joke that the sales live in a fantasy land in terms of features or timelines. However, when you are coming up with a new product or trying to entice a client, you may need the freedom to dream and think big — as long as you eventually consult with the specialists on the ground before making binding commitments.

It is easier to take a bias correction input from an acceptable figure charged with just that. That’s why some hire a fitness instructor just to shout at them, I guess. In a team, this bias-exposing and bias-correcting roles can (and should) be played by the empowered owners of various parts of the product or area, ideally by everyone — the culture similar to the Toyota Production System (see e.g. [Liker]). Also, in many cases, just a second pair of eyes on the problem at hand can help, as long as people feel safe to speak their mind.

transparent vs opaque

This is a strong design force leaning toward transparent.

Black boxes like remote systems, binaries with limited interfaces, or structured binary data formats tend to have a high degree of opaqueness. Most of the time, opaqueness has a high price tag: opaque stuff is harder to develop upon, generate, convert, debug, edit, or maintain. One has to explicitly keep this cost in mind to weigh against the gains. The gains typically would be performance (CPU, i/o, power consumption, etc), data specifics (e.g. mp4 videos as data are opaque), intellectual property protection, or security. (Sometimes, opaque stuff may make things simpler just because the legacy system already supports it. E.g. if everything already is in C++, thus there is a low overhead of adding more compiled binaries.) If none of those gains is essential, opaqueness is a waste: for a set of language features, python may be a better choice than C++, JSON better than BSON or Google protobufs, etc.

Continuing with the data examples, transparent vs opaque as data is concerned may have two meanings:

• It is easy to get to the bottom of things: e.g. if data is human-readable (plain text, CSV, JSON, etc)
• Data manipulation is so easy and reliable that we never need to look at the data at the byte level (e.g. for the standard image formats)

However, most of the time, keeping data transparent (e.g. human-readable) makes things too cumbersome or intractable: no-one expects image data to be easily human-readable and thus balancing this force is resolved via trade-offs. We would like as little executable code or utilities as possible to interpret or manipulate data (ideally, well-known standard tools or just your own eyes). E.g. you can open and edit a JSON configuration file in any text editor. Image files offer some shades-of-grey examples:

• Images represented as binary array-like binary dumps can be easily loaded, read and manipulated interchangeably as numpy arrays, C++ vectors, OpenCV images, etc at the price of disk space or data throughput . But then, they can be reasonably compressed and read on the fly using e.g. zlib.
• JPG images require relatively standard conversion tools.
• Accessing a specific frame in an MP4 file can be fiddly, but there are sensible libraries for it, too.
• TIFF files can be a nightmare and often require specialized tools for various use cases, a high price to pay for having everything in a single file instead of keeping the image and its text-based metadata in two separate files — which could be zipped together if a single file is a must.

immediate vs indirect

This is a strong design force gravitating to immediate: plain old data is strongly preferred to a class with getters and setters; computations in a single thread to multi-threading or multi-processing; collocated team and face-to-face communication to email and phone; and so on.

This force is related to simple-vs-complex force, but is not the same: indirect implementation may be simpler than immediate — that’s the whole point of the software design.

The fundamental theorem of software engineering (FTSE) says: “We can solve any problem by introducing an extra level of indirection.” However, a really pervasive design fallacy goes as: “I have a problem. FTSE solves any problem. Therefore, it will solve my problem. Therefore, I’ll use FTSE.” — and to take it further: “FTSE solves any problem. Therefore, I introduce indirection anyway even if I don’t have a problem at hand.” I’d call it “the FTSE inversion fallacy”.

Any indirection has the cost of new complexity and potential major loss of transparency: how often do you find yourself jumping from one function, class, or method to the next, trying to reconstruct what is actually going on in the code? This cost vs gain needs to be made explicit and reasoned about.

self-evident vs documented

This is a strong force that needs to be balanced.

Even if a piece of code is self-documented, we still need to convey its usage semantics: intended use cases and scenarios, its limitations and so on: even if you have a manual for a saw, hammer, nails and wooden planks, it is not cannot be possibly self-evident how to build chairs or tables using those.

The cost of documentation is a form of information duplication: almost inevitably the documentation goes obsolete or out of sync even if it is comments next to the code.

The documentation integrity is hard to enforce. Working examples help, but unless the examples are tested after each change, they get broken as easily as the free-text documentation. That’s where the tests as the second use case can (and should) be of help: the test suite should not just cover the functionality, but also have a set of test cases representing and demonstrating the usage semantics of the software piece. Essentially, as a recipe, the usage examples including step-by-step tutorials should be a part of the test suite.

Last but not least, good choice of conventions reduces the amount of documentation to simple statements: “all our data is in SI units“; “all our key-value data can be represented as JSON”; etc.

incremental vs disruptive

This is a very strong design force leaning toward incremental.

A significant feature of the incremental changes is that they do not break backward compatibility (e.g. manifested in O in SOLID design principles). It means that the old software still works as before and others’ ongoing work is not disrupted.

Because disruptive (non backward-compatible) changes do not require much consideration and are quicker to make, this force often gets unnoticed or ignored altogether, especially at a fine-grain design level. However, more often than not, this time saving is a false economy, since it almost immediately requires huge amounts of follow-up fixes downstream, which are extremely expensive and painful especially if the change was made by stealth.

For example, migration from python2 to python3 was more or less planned and orderly, but still required lots of patching for anyone using python in their products. On the other hand, C++ has introduced a plethora of new syntax and features over the last forty years and yet C++ code written in 1980s still happily compiles and runs as before.

Additionally, the incremental design tends to be of much higher quality since from get-go it is done with multiple use cases in mind (legacy ones, current ones, and semantic space for future ones), which makes the designs much more convenient to use and maintain due to their well-grounded and well-”triangulated” usage semantics.

(It also allows the team to work on the development branch in git all the time instead of feature branches with potentially painful merges and push their code or release their data without fear — as long as the regression tests pass.)

Collective design of usage semantics, testing, reviews, and strong design ownership are major factors that make incremental development possible.

The disruptive development usually has a high price tag, but as long as this price tag is made explicit, the disruption may be justified, provided that the owner takes it upon him/herself fixing of whatever he/she breaks elsewhere (“you break it, you fix it”).

scalability: small vs big

Is the design conceptually scalable?

Does it scale up in terms of size, numbers, deployment, complexity, or performance?

Does it scale down, meaning that it should be easy to do simple things or service mainstream use cases without being slugged with massive configuration, set up, distribution, or maintenance effort?

disposable vs non-throwaway

This force leans toward disposable, meaning that we can discard an artefact without consequences and at the low cost. It also means we can discard one thing and seamlessly replace it with another: e.g. throw away an early prototype and replace it with a proper implementation. So, this force points in the same direction as modularity and uniformity.

Disposability is different from modularity: Modularity assumes the extra ability to rearrange modules into a different assembly. Disposability of an item means that it is cheap to produce and cheap to throw away without much pain — even if it is not designed as a “module”.

It is strong force: if I remember to keep my item under design as disposable as possible, it means by design that I am trying to minimise the production, maintenance, disposal and replacement cost. I am forced to think how to make my design modular and decoupled, too. Disposable designs also allow to contain ‘dirty’ stuff and gradually clean it up.

In all my experience the most disposable software artefacts tend to serve much longer than the monolithic designs. The latter often are overfit to the specific use case and as the use cases change, the monolithic designs are left behind with all the work invested into them.

manual vs automated

The full automation is rarely achieved. Automation, especially in terms of data, usually means instrumentation of the design and process incrementally while balancing urgency, development time, etc. This force is strongly correlated with the openness (O in SOLID), modularity and uniformity forces: they pull in a similar direction and enable each other — in other words, any sacrifice in modularity or uniformity comes at the cost of making automation more difficult in future; and giving up on automation may make the uniformity or openness less necessary.

Higher automation produces lots of leverage, however the cost of it may be higher complexity and reduced flexibility. Depending on the problem, this cost may grow very non-linearly beyond certain point and small trade-offs of having human interventions may provide massive leverage: augmented sensing and driving hit the road long ago while full self-driving cars still are not part of daily routine.

considering consequences

This is a very strong — and yet often ignored — design force. It is very expensive to miss or ignore consequences of a design decision.

A clunky class written by a junior engineer, a software framework parachuted upon the team because of someone’s personal convictions or a glossy sales pitch, or automated test effort undermined as non-productive by the accounting department are a few examples among many.

Since software is a language enterprise, its leverages and factors of speed or scale are of many orders of magnitude, which is counter-intuitive to human mind used to think in terms of the physical world. A wrong choice of a default setting in a class spreads like a weed across libraries, the software that uses them, and its deployments. The design choice might take seconds, but fixing it everywhere later can take days or weeks or months of labour, frustration, and harm to the user base. When most of the design is done that way: do it now, fix later (likely elsewhere by someone else), minutes-turned-days snowball into months-turned-man-years.

Considering consequences may not make a design more complex most of the time, but it certainly makes the design process more expensive: more complex, requiring more thoughtful consideration, broader consultation, and better skills. This cost is explicit and upfront, while — for a fleeting moment — the consequences have not happened, yet, and the humans have enormous temptation to ignore what ‘is not real’ — no matter how big and costly, especially if they are not the ones to feel the coming pain: engineers who are not the intended users of the code they write; bosses who are not the engineers dealing with the ensuing ugliness; salesmen who only interested in striking the deal, but not in the delivery; or those who move on later anyway and leave cleaning the mess to others. (The opposite attitude could be called “technical empathy”.)

On the other hand, it is impracticable and intractable to follow all the possible consequence in the world — and it may be difficult to go too far down the causal chains due to limited knowledge, experience, or information availability and foresee everything that has not happened, yet. It is better to make sure instead that there are no consequences to start with with plenty of software methods to do so whenever possible: open-closed principle (as in SOLID), disposable, modular, and more transparent designs (see respective sections on it above), stateless instead of stateful, “linear” instead of event/callback-driven, processes that do their job and exit rather than run as servers, guarding the existing functionality with regression tests (so that we are not afraid to push our code), and so forth.

Applying those makes our design less consequential and affords us with the bliss of technical ignorance. After we make our design as inconsequential as possible, we still have to explore and mitigate the remaining consequences or otherwise pay high price in near future.

reproducible vs non-reproducible

Preferably, execution flow or data manipulation should be deterministic, i.e. we should get exactly the same result logically, procedurally, or numerically (with acceptable floating point errors) on each run.

The determinism allows to easily build robust regression tests, debug, perform controlled experiments, do historic data analysis, etc. especially if the solutions is not numerically stable and therefore non-determinism may lead to vastly different system behaviours.

For example, if time synchronisation is built into a real-time system by design (by synchronising clocks, timestamping all the data streams, and synchronising the streams even in real time by timestamp, not just by co-occurrence), it is easy to replay scenarios as they happened in field or process the time-synchronised data offline. Additionally, depending on the system/data needs, we may leave the door open for approaches like reliable multicasting, bitemporal modelling, etc.

The other aspect of reproducibility is the ability to track how the results were produced: what configuration parameters were used; what software was run; sufficient insight into the intermediate results and so on.

This may incur extra design and implementation costs, but those cost may be low if we take care about the reproducibility early and by design, or at least leave the design open to introducing more determinism as we go.

In general, for reproducibility we should be able to move up the causal chain that produced a given result and make sure that it does not have gaps and minimises indirect reasoning. Thus, the force of considering consequences and reproducibility complement each other: the former calls for moving down the causal chain whereas the latter follows it upstream.

data vs code

This is a very strong design force. It is about separation of concerns: any static information should not be expressed by the executable code. Instead, it should be represented as data files, database entries, or code-wise as constants, data structures or class declarations in headers, and so on.

Any conflation of data and code has a high price tag and thus every time we need to make sure it’s worth paying.

E.g. when implementing a binary communication protocol in C, it’s common to write a function that would populate the protocol packet buffer with values or parse it byte by byte. As the result, you would need to read the code line by line to understand the packet layout — or read the documentation, which would commonly have typos, omissions, or diverge from the code. Anyone who has done such work knows how tedious and unpleasant it could be. The alternative would be to write static definitions, e.g. as plain C++ classes or plain-text descriptors, which represents data as data, while the executable code generating protocol packets would sit among generic serialisers and visitors agnostic of the packet structure.

As another example, take dataclasses package in python: In the “native” python, there is no good way to easily work out what members a class has, since they can be arbitrarily added as self.some_member=… in __init__() and thus, again, static information is dispersed in the executable code. dataclasses solve this problem (using decorators as their mechanism).

As an example of the opposite problem, ansible configuration YAML files inject executable commands and computations via variable substitutions into data. Without rigorous structuring, it can be difficult to determine what is set, what is executed, and in what order. However, this price tag is outweighed by the current status of ansible as a de facto industry standard for deployment.

homogeneous vs mixed

Homogeneous code (or data) is easier to manipulate, easier to split, merge, process in parallel, use interchangeably, compare, use with the same toolkit, interoperate between systems, etc. Homogeneity is strongly correlated with reuse.

Breaking uniformity always costs some of those opportunities. Those losses need to be justified by gains elsewhere.

For example, uniformity brings a huge value in using as few communication and data exchange protocols as possible across the system. Otherwise:

• a massive amount of the development and deployment time is spent on bridging different comms systems and data conversions
• the code gets bloated because often each message type needs dedicated converters and some message do not match one-to-one across the communication protocols — and if there are more than three communication protocols involved, the complexity of number of conversions may become higher than linear
• the computation overhead grows and communication throughput drops as the conversion bridges introduce performance bottlenecks

Striving for uniform communications to unblock development and communication flows and shed waste is literally the lean development textbook.

On the other hand, when overdone, uniformity stifles flexibility. Stipulating code linters is an example: linters may produce better code formatting in 70% of cases, but turn the remaining 30% into garbled mess while humans would adjust and align code so that it looks just right. What would a linter do to the following code?

std::vector<float> my_square_matrix = {   1,  2.2,   3
                                      , 0.1, 3.33, 0.1
                                      , 0.1,  1.1,   4 };

Attempts at uniformity may also lead to really expensive over-engineered designs: For example, as mentioned above, TIFF images may be a nightmare to work with. If we sacrifice the opaque one-size-fits-all binary representation and put the image metadata in human-readable files (e.g. JSON), breaking uniformity (everything being binary with each image+metadata as a single file) allows us to use simple standard tools.

Sometimes breaking uniformity is inevitable, but it is really expensive and has to be traded for commensurable gains.

forces: conclusion

To sum up, one of the key design skills is spotting how design forces interact, how they form persistent force patterns (or “units”), and how we improve their balance in reusable or unique ways.

Considering design forces a metaphor of physical forces, I gave a brief informal classification of forces: “constraints”, “loads”, “leverages”, and so on; as well as ways to characterise and balance forces in terms of their “direction”, “weight”, “magnitude”. Given that the design forces are just a metaphor, evaluating and balancing them requires approaches different from the physical systems. We cross the boundary of the metaphor by just “summing” the forces up. It has been shown that human mind deals very poorly with comparing, combining, or ranking many things at once, but is pretty good at matching things pairwise (or in very small cohesive groups) — and this is my suggestion of dealing with the design forces at every level of software design down to an individual line of code. This level of granularity may seem unreasonable, but with practice it becomes a muscle skill and requires only a small time/effort overhead that pays back almost immediately and many times over.

Lastly, I emphasised the importance of always making forces explicit in any design considerations and then gave an overview of some overarching forces in software design since the existing documented pattern languages don’t seem to do so.

The forces I mentioned may look like a commonplace. However, there are two ways to see it:

We can either succumb to the triviality bias: trivial means unimportant, thus we establish that something is trivial and conclude that we can forget it, or ignore, or assume that it is obvious to anyone. As I tried to show throughout the article, there are three fallacies in it and all of them are wide-spread in software engineering with damaging consequences. Trivial can be important; trivial (as in ‘nothing to see here’) may not be so trivial upon a closer inspection; and it also may not be obvious at all to junior team members, engineers with entrenched views, people dealing with a different problem domain, and so on.

Or we can follow the common analytical way of solving problems: break them down to the ‘trivially’-looking pieces and deal with them explicitly instead of hiding or abandoning them: “Quality — oh, of course I know”. Knowing is not enough — it is about continuous repetitive practice in one’s design process: Do we always keep each force in mind? Do we see each force as a spectrum, not just something unconditionally good or bad, right or wrong? Do we make it explicit how the pairs or small groups of forces interact? Do we tentatively match forces against each other to establish their relative “weight”, “magnitude” and “direction”? When we budge on certain forces, do we assess the consequences, price tag of our move and the gains it brings?

References

[Alexander 1965], C. Alexander, A City is Not a Tree

[Alexander et al 1977], C. Alexander, S. Ishikawa, M. Silverstein, A Pattern Language

[Alexander 1979], C. Alexander, The Timeless Way of Building

[Austin], J. L. Austin, How to Do Things with Words

[Brandom], R. Brandom, Between Saying and Doing: Towards an Analytic Pragmatism

[Coplien, Harrison], J. Coplien, N. Harrison, Organizational Patterns of Agile Software Development

[Dawes, Ostwald], M.J. Dawes, M.J Ostwald, Christopher Alexander’s A Pattern Language: analysing, mapping and classifying the critical response

[Fowler], M. Fowler, Writing software patterns

[Gamma et al], E. Gamma, R. Helm, R. Johnson, J. Vlissides, Design Patterns: Elements of Reusable Object-Oriented Software

[Groupe μ, 1970], Group μ, A General Rhetoric

[Kahneman et al], D. Kahneman, O. Sibony, C. R. Sunstein, Noise. A Flaw in Human Judgement

[Kripke], S.Kripke, Wittgenstein on Rules and Private Language: An Elementary Exposition

[Liker], J.K. Liker, The Toyota Way: 14 Management Principles from the World’s Greatest Manufacturer

[Meszaros, Doble 1997], G. Meszaros, J.Doble, A Pattern Language for Pattern Writing

[Salingaros 2000], N.A. Salingaros, The Structure of Pattern Languages

[Vlaskine 2013], V. Vlaskine, Code as two texts, first published in ACCU C VU as Two sides of the code

[Vlaskine 2014], V. Vlaskine, Staying in Touch: Performative Negotiation, ACCU C VU, 26(1):15–17, March 2014

[Vlaskine, 2019], V. Vlaskine, Meaningful software: Preserving capabilities

[Vlaskine, 2019, 1], V. Vlaskine, The Myth of Scrum Team

[Wittgenstein], L. Wittgenstein, Philosophical Investigations