MBSE Success: Keeping the architects ahead of the construction crews

One of the interesting problems of justifying Model-Based System Engineering (MBSE) tools to support Systems Engineering (SE) is product development work is going to proceed with or without Systems Engineering. I know of no other discipline that even making no decision is a decision as everyone moves on in their development effort with or without SE direction—so MBSE justification needs to include the time value of doing systems engineering fast enough to stay out in front of the downstream product developers (see the previous discussion on see-saws/systems engineering leverage ). Put another way, if the SE leadership is missing (for whatever reason) there’s no technical guidance/direction making the project predisposed to be out of control and go runaway (runaway: a project that is out of control, behind schedule, and over budget).

You can think of these system engineers/product architects as performing the same task as building architects; the architect understands the customer needs, then defines an architecture that drives what the downstream disciplines (structures, HVAC, electrical, plumbing, etc.) will be doing to meet those needs. Without that architect leadership it’s equivalent to the construction crews building something without the architecture spec/blueprints—so we shouldn’t be surprised when unexpected things/interactions happen late in development causing expensive redesigns and heroic integration efforts (per forensic accounting runaway analysis at TI you have a 70% probability of a runaway in the making if the architects are documenting what the construction crews are building)—so MBSE tools are supposed to accelerate the SE process to keep the architects ahead of the construction crews so they can lead product development to a successful design.

Out of college, I went to work in high-tech, where we were facing a daunting project backlog with the forecast telling us we needed 3x more SE’s then we currently had to do the front-end product architecture work. Given the difficulty in finding more of these types of people, we needed to make the ones we had 3x more productive than they currently were. We studied where SE’s were spending their time; the results came back:

• 50% of their time messing with documents

• 25% of their time entering the same information in different places

• 14% of their time ‘futzing’ (getting documents to print right)

• 33% of their time in meetings communicating design decisions (often the same meeting over and over)

…you’ll note it adds up to greater than 100% because the average SE workweek was >60 hours, which meant we couldn’t get any productivity improvement by working them longer hours

Hammer/Champy Reengineering the Corporation teaches us to look for processes that are no value to the customers, those are subject to elimination/streamlining. Looking at the amount of time spent on documentation by SE’s should be of incredible value to customers, but we quickly found out that no one reads the documents they produce, no one trusts them since things have changed since publication (leading to ignored design spec’s) and more. So, documents seemed like a perfect target for improving SE productivity—eliminating something like 60%-70% of SE labor/time to work on the real value architecting a solution vs putting the information in document form for delivery to others.

Of course, you know what happened to that kind of thinking…we found out document deliverables are how you get paid, how you know when you’re done (via a set of document deliverables), carry the review/approval process, keep many people busy managing, archiving, etc. and even found out that the organization structure often matched the documentation (an interesting derivative of Conway’s Law).  

So rather than fight corporate culture, we decided to use the models to generate the document artifacts as a by-product of the modeling process–just like the 3D modeling guys generate 1D/2D drawings from the 3D models—in this case generating documents from the models. This created another set of problems: people editing the documents vs the models, control processes were built around managing documents not models, massive development/support efforts to build and maintaining document templates, and the list went on–chewing up our much of our productivity gains.

These types of hidden processes do the same thing in your businesses, costing you far more than tool investment and disabling the full value/benefits you get from tools—all contributing to the hidden costs, lost productivity, and more. So, the question might be, “How do you nail down these hidden process costs so you can use them in your MBSE ROI calculation?”

In my spare time, I teach MBSE in graduate school, many of my students work full time while working on the Masters and Ph.D. programs. I teach a class on implementing systems engineering processes, tools, and techniques dealing with these types of process/cultural problems. The class requires a term paper on how they could apply what they’ve learned about SE processes, tools, and techniques in their work or daily life. You can imagine the variety of papers applying SE/MBSE from standard applying x to my project to some others branching out in dealing with more personal systems problems like systems engineering applied to a bad boss, raising teenagers, planning my summer vacation, to actual work situations/projects they worked/interacted with—resulting in some embarrassing insights into organizations you’d never see in the outside world.  

One student writes about an engineer dealing with managing test execution needing to manage test cases and the results. Given the number of test cases, he went looking for a tool, found one that would do the job for ~$150k and submitted it to a capital acquisition committee. Since it wasn’t budgeted, it was rejected. Rather than give up, he sat down with his Access database (included in Microsoft Office already installed on his computer) and wrote up some forms and started managing his test cases using Access. It wasn’t too long before his cube-mate saw what he was doing and wanted to use it as well—and so it slowly spread across the project. Of course, the users would come and ask questions, ask for enhancements, etc. and soon our engineer was spending a large amount of his time taking care of his creation. Management found out about what was going on after discovering that it had wormed itself onto the critical path of the project and they couldn’t just shut it down, so they allocated money/time from the project schedule to maintain/take care of the tool. Of course, over time project engineers began to migrate to other projects and soon the test management database began to show up on the critical path in other projects across the division–budgets grew, support systems were needed, a helpline/enhancement tracking system was put in place, development resources were added, a multi-user/distributed database was put underneath since they needed to collaborate on test cases and reuse test cases across the division. Over time the engineering support staff continued to grow taking on a number of developers with a large support staff needed to support the solution with large amounts of overhead dollars being spent annually to maintain the homegrown solution. Eventually, the organization acquired one of its competitors and the reconciliation process began as they merged the groups. The acquired group had purchased an OOTB external test management solution (much like what our engineer had wanted to buy), but as they began to merge the groups a tool conflict surfaced between the home-grown solution and the off-the-shelf solution. By now this was personal with careers and turf to protect, having invested so much time and effort in developing their baby and the conflict went all the way to the CEO to resolve. When the CEO found out what happened he summarily sacked everyone involved with developing their own solution and the board put a new policy in place with consequences for anyone that builds their own solution in the future.

That’s one way to implement a process/culture change! A better way would have been to find a tool that was close to what you needed and use your influence/dollars to enhance the COTS product to what you want. But according to Dr. Robert Halligan that would still fail since you still started with the tool rather than the process. You should start with a process, then use tools to support/enforce a standard based process and let it help you with the culture change in a systematic way to be successful—after all complex organizations are complex systems (I recommend a great read on this topic by Dr. William Donaldson: Simple Complexity). Both authors suggest you begin with policies (such as define the problem before you solve it), then work the processes (i.e. V&V best practices) and then look for tools that support the processes, not the other way around. In our example case, a well understood/vetted V&V process should have come first rather than looking for a tool to manage just test cases, stepping back we would have noted we could easily use standard tools and practices that guide/encourage/enforce best practice—like our PLM System or Requirements Management solution.

We’ll talk more about implementing best practices and changing processes with tools in our next blog…

Mark Sampson.