The Flow Dialectic: The Key to Creating Healthy and Effective Organizations
There is a strong correlation between organizations that struggle to adapt the people and process elements of Agile and those where the foundational technical practices of Agile software development, like Continuous Integration/Delivery (CI/CD), Pair Programming, and Test-Driven Development (TDD) are absent. It is a dialectical relationship where the presence (or absence) of one set of conditions tends to encourage the presence (or absence) of the other.
For the software engineer, once one is accustomed to it, TDD can be quite a joyful experience. It reminds me of making a really delicious steak dinner.
When I was a teenager, I would often help my mother cook dinner– and not always voluntarily. She rarely consulted a recipe book. She would just have me stand beside her and “chop these onions”, or “peel this garlic”, or “stir these beans every few seconds or so”. I had no idea what I was doing and I could scarcely follow what she was doing. But I would just struggle to follow her instructions and hope for the best. You could call it a literal trial by fire.
Now and then, she’d tell me to add something I’d chopped or otherwise prepared into the meal while it was cooking. When I would ask her how much of this ingredient should I add, her answer would invariably come as something like, “start putting it in, and I’ll tell you when to stop.” The whole process was unscripted, completely intuitive. She just knew what to do, and she knew it so well it was sometimes difficult for her to describe in words what she wanted.
At UMASS Amherst, where I attended college a few years later, there was a rule that students had to live on campus for the first two years. After that, if one wanted, one could seek off-campus housing if you could find it. I was thrilled when in my junior year, finally, some classmates and I scored an apartment about a mile down the road from the campus. We were all excited about having our own place, but I was particularly excited about the kitchen. Living on campus, we’d always just eaten in either the cafeteria (not a particularly memorable experience) or at the various restaurants in and around the campus. I couldn’t wait to try cooking for my friends in my very own kitchen. And most of all, I really wanted to cook a steak.
For a while, it was a disaster. I ruined many expensive cuts of meat. But over time, I realized that lessons my mother had drilled into me years before were still there, somehow present in my muscle memory. There were actually a lot of tasks that I remembered perfectly, dormant for the years spent on campus, and yet suddenly readily accessible as soon as I needed to use them. Eventually, I nailed the steak and got it just as I wanted it (rare, of course, with a nice red wine deglaze and everything).
My friends were duly impressed. One of them remarked one evening, standing in the kitchen watching me chop, that I seemed really “in the zone” when I was cooking. Over the years, I have heard that same observation from time to time from guests and loved ones alike. There is something about chopping onions or herbs that just completely relaxes me. The great Mihaly Csikszentmihalyi would call it a state of flow.
Csikszentmihalyi spent his career studying the concept of flow in the workplace, and developed some very compelling ideas about what types of environments enhance a state of flow in employees and what types inhibit it. The widely acclaimed book by Daniel Pink, “Drive”, which I’ve referenced many times before, is essentially based on his work.
Only a few years after college, I would experience a similar state of flow whenever I would sit down to code. After I learned the Test-Driven Development (TDD) style of programming, that is. If you have not written software programs for a living, this may be hard to appreciate, but please bear with me. I think you’ll get it.
Before I learned TDD, coding was exciting, but also a bit of an emotional roller coaster. Programming computers is a complicated business, and there are many things to keep track of, and you can’t possibly keep them all in your head. Most software developers first learn to code by hacking away, trying different things, not entirely sure of why something works or doesn’t. One minute you’re sure you know what you’re doing, and the next you’re totally crest-fallen as your program fails to run properly, then elated once again a few minutes later as you find the bug and fix it. Then on to the next step, and the whole cycle repeats.
Too many times, I would hack away at an algorithm for a while just hoping that when I finally ran my program it would work as expected. If it didn’t, it would throw my heart into a pit of despair. And it didn’t most of the time. That rare triumphant feeling when things worked the way they should was sweet indeed, and it was enough motivation to drive me to continue programming for a living.
Then, a few years into my career as a software developer, one of my engineering friends introduced me to TDD. For the layperson, here’s the basic idea. You write another program, called a test program, at the same time you’re writing your main program. The only job of the test program is to run your main program every time you change something in it and see if it still works. There were testing programs before TDD, but in TDD you write the two programs simultaneously, line by line, switching from one to the other. First you write the test, and then you write the code necessary to pass the test.
For example, if I want a program that adds two numbers (you’d never do that, but I am using a really trivial example to keep things focused), I might write a test program that runs my main program, feeding it two numbers and verifying its result is correct. Now, if I run the test program before I complete my main program, the test will fail and alert me. We always do this first to guarantee we have a null hypothesis, as it were. Then, once I add the code to my main program to add those two numbers and return the result, I run the test program and it passes. Yay!
From there, every time I want to make an addition to my main program, I will add a new test to the test program first, and then fill in the code to make it pass. Gradually, the test program forms a protective scaffolding around my main program. I can never accidentally add a bit of code that breaks one of the earlier features, because the minute I do the test program will catch it immediately and alert me.
This approach to coding transforms the emotional rollercoaster into a high powered jet engine. No longer are we hacking away in the darkness, hoping our code will work. The freedom and confidence this generates when you’re coding is pretty indescribable. It is not dissimilar to that feeling I have when I am cooking a steak dinner. The TDD method just takes away all of the anxiety so that you can focus on the creative aspects of what you’re doing in the code.
In short, you fall into a state of flow.
When I am in the kitchen now, cooking up a steak dinner, I move with total focus and confidence. I don’t have to think about the basics, they are just fundamental. Similar states of mind occur when working with code using TDD as a practice. One can create software with total focus and confidence as well, and the basics are all just abstracted away.
There is another meaning for the word “flow” in the technology industry. Proponents of Lean product development have long used the word flow to describe the steady and regular movement of ideas and material through the value stream in an organization, starting with the initial concept and ending up as a finished product in the hands of the customer.
There are many good histories of Lean, so I won’t attempt to create another here. James and Womack wrote the definitive text, “The Machine that Changed the World”, in 1990, so if you’re interested in the history, do check them out.
Briefly, the idea of flow in product development largely originated from the work of Taiichi Ohno at Toyota, inspired in no small measure by earlier work by W. Edwards Deming. The key innovation of the venerated Toyota Production System is that it is pull-based as opposed to push-based, and that makes all of the difference in the world. In a push-based system, work is pushed downstream from one workstation to the next (often starting from the CEO or the board). If work is pushed into a workstation that is already busy, it has to wait in a queue until the workstation downstream has the capacity to work on it.
These queues of partially completed can build up very quickly, creating piles of excess inventory that represents investment that cannot be realized as profit until it is moved through the system and sold to customers at the other end. The longer work sits in the queue unfinished the more money is wasted.
In a pull-based system, workstations only pull work into their queue when they have the capacity to work on it. The workstations upstream only start work on previous phases when workstations downstream have emptied their queue, indicating that they now have capacity to take another batch of work. Workstations with capacity downstream send signals upstream when it is ready to accept more work. There are queues of work waiting between stations, but they are kept very small. Thus the descriptive term “lean manufacturing.” This method has been staggeringly successful for the manufacturing industries and is now the norm.
In the early 2000s, the tech industry began adopting principles from Lean manufacturing into software development operations. Technology teams employ this same pull system by enforcing limits on queues of work waiting between workstations. The only difference in technology product development, as pointed out by Lean guru Don Reinertsen, is that the inventory in product development work is physically invisible.
The work items are only visible in a visual control system, such as a Kanban board or a task management software. And yet, the same principle applies. Downstream workstations set the pace of the whole system by pulling work into their process only when they have freed up capacity from previous jobs.
Despite the fact that high-performing tech organizations utilize a pull-based system to realize smooth and steady flow of work through their processes, most organizations still don’t use this method. The vast majority of organizations still follow antiquated process control and management methods from the industrial age of the early 20th century, in a word, a push-based system.
A quick way to tell whether or not your organization uses a pull based system is to look at the way that new work requests enter your product development system. Does work just arrive, handed to your team by a stakeholder who already has a deadline worked out for it? Or does work wait for your team to finish what they’re working on and accept new work requests when they have capacity to begin working on it immediately? Think carefully in answering this question. Your boss need not be a tyrant or a command and control freak for you to have a push based system in your organization. It’s not a personality problem, it’s a systems problem.
The feeling of flow in individuals identified by Professor Csikszentmihalyi is not exactly synonymous with the state of flow that is referred to by practitioners of Lean methodologies. But there is a relationship between the two. Creating a flow of value in your organization by enacting a pull-based system leads inexorably to enabling the feeling of flow in the people in your organization. And it's the people’s personal state of flow that will produce the highest quality business outcomes that leaders are ultimately after.
When organizations that do not practice TDD, CI/CD, or Pair Programming are asked why they don’t, they often give answers like “we are too busy” or “those things are a luxury and we’re trying to run a business here.” It’s thus a sad irony, since those techniques that have proven time and time again to create ideal outcomes in digital product development are held just out of reach by the very lack of flow-based pull-systems in the organization.
When an organization is operating as a pull system it is much easier to build in practices that encourage a state of flow in the individuals doing the work. There is an increased focus on few work items at a time, gradual development of standards and automation that clear the way for the creativity and flexibility required for knowledge work. And all this allows individuals and teams to develop and master skills that produce the best possible business outcomes for the organization.
When you create a work environment that optimizes for flow in the workstream you enable the conditions for flow in the workers themselves. When you put attention on creating the conditions for flow in the workers, you’ll inevitably foster an environment that prioritizes flow in the workstream. Whichever entry point you choose, and ideally you’ll choose both, the transformation is worth pursuing.
If you get stuck, we’ll be here to help.
