Agile development and embedded systems

[mgeerts]

I work for a company that produces custom hardware/firmware solutions for customers in ANY industry. One day i may be working on a plasma cutter, another day a hospital bed, another day farm equipment.

I constantly find that our projects run into the same problems:
-We didn't understand the requirements when we wrote the quote
-The customer didn't even understand their own requirements when asking us to quote
-As the project continues, the customer wants changes weekly/daily either providing incomplete descriptions or contradicting what they have previously requested.

These things make completing a project in the quoted time within the quoted budget impossible. We regularly lose money on engineering time in the hopes of making it back in production profits (we reserve the right to produce what we design so customers are locked in with us).

Not only is this a waste of time and money, but it is massively demotivating to me and the other developers.

Agile project management has been the buzz of the 21st century. The premise being that you take a budget and a timeline and chop it up into segments, then make only promises for the first segment with the intention of re-evaluating the customer needs before proceeding to the next segment.

I presented this method to my Engineering Manager and the response is that it is impossible to do Agile development because there is hardware design involved. I shake my head in disbelief.

As I see it, collecting hardware requirements and producing a board, then proving that the board can work in the environment is simply the first iteration. The next iteration would be, for example, a basic user interface outline (customers never know what the UI is to look like), then work on features from there in stages. Seems simple to me. Less risk, more opportunity for change, smaller tasks.

Please reply back with your experiences in Agile design in this sort of field (short design cycles [6-16mo], different customers/industries).

Also, please reply back with any resources that I may find to convince my manager that "waterfall" is not the only method in the world that works with hardware (the evidence tells me it does not work!).

 

[MacGyverS2000]

Before I begin a project I have laid out a rough spec of what the system will do, as well as the manner in which it will achieve that goal, including the interface. If you cannot define these basic things before a single component is designed, your project is hopelessly doomed to cost/time overruns. If you cannot define every single piece of the puzzle, at least in a rough enough sense that major errors can be caught, you shouldn't start.

The spec will change as the project progresses, but hopefully to a relatively minor degree... if you change the spec, you pay, if they change the spec, they pay. Be flexible, but don't equate flexibility with designing on-the-fly.


Dan - Owner

http://www.Hi-TecDesigns.com

 

[mgeerts]

Our customers are not in the tech industry whatsoever. They have a machine that does something and they want to make it "high tech" for their market image. These types of customers have literally no clue what they want when it comes to things like GUI development. They have little clue what they want for things like actuators and I/O. They don't even know what exists. I have had more than one customer describe what happens between the interface and the actuators as "magic". These people are old-school salesmen, machine shop owners, farmers, etc.

INVARIABLY when the customer does not know what they want, they will accept what you propose, then tell you they don't like it when you've created it.

It sounds to me like you're sticking with the ol' standby of waterfall development. I don't see that as a bad thing, it just doesn't apply to our business type.

Perhaps the weakness is in our sales and contract-writing. Regardless of whether or not the customer ends up paying for changes, they will be displeased by the extended time that changes require despite their indecision being the cause of these changes. They have business oriented leaders saying "But you said it would be ready for our tradeshow, now change everything".

Does anybody in this business work on a stricly Agile development method?

 

[IRstuff]

Seems to me that much of your problem is simply wussy management. Presumably, you've bid this as firm, fixed price, in which case, any errors on your part have to be sucked up, but any changes on the customer's part must be fully documented, costed, and approved by the customer, BEFORE you incur additional costs.

Obviously, as a good sub, you can suck up some of that in your management reserve, but everything else is strictly a program management fubar, for failing to document and charge the customer. How would you possibly defend yourself against a customer's charge that you made changes that they didn't authorize? Would you suck up those costs as well?

How do you document your basis of estimate (BOE)? Where do you document your assumptions and conditions? How did you develop your bids from previous, historical performance? Your cost accounting for previous projects should be able to shed light on what your bids should look like. This would be tempered by any assessment of what a "winning" price might be, and it's usually less than what you bid, but that's something that your company's management needs to grapple with.

Obviously, few companies are able to do this routinely and rigorously, but the closer you can come to that ideal, the less pain you're going to have in the future.

Your proposal for "agile" management is more or less on the right track, but the key issue is really to rein in your customer, and to rein in yourself. With a thorough understanding of your own costs, you can make in-situ decisions about whether to let some change ride, or to charge the customer for it.

TTFN

FAQ731-376

 

[mgeerts]

IRstuff:

I will let your questions remain rhetorical. I really appreciate your input and think that your assumptions and advice are bang-on.

Few of our customers require any "bidding" as there is little competition for our company. I think that our pricing is more of a "what can we get from them" model. Some customers pay by the engineering hour, some on a "not more than quoted" hourly basis and others on a flat rate. Either we give them something they can afford, or they just don't bother with the [automate the machine, add bells and whistles, etc].

So it seems to me that an agile approach may kill a flock of birds with one stone. We won't ever find outselves neck deep, the customers can pull the plug any time or spend as much as they want on features, and the initial price tag looks low.

Primarily in the project that i'm working right now that has sparked this research into agile, I think the issue comes down to, as you put it, wussy management.

 

[IRstuff]

The only issue I have is the time scales for your decision points. In a classical program structure, ala aerospace, there are already built-in decision points:
SRR
SFR
PDR
CDR
PRR
etc,

In many cases, these line up roughly on 3 month increments. But 3 months is long enough to get you into trouble, particularly if your billing is laggy. Any time scale shorter than that doesn't correlate with anything that the customer can recognize as obvious decision points. Clearly, it would be desirable to have a monthly cycle, and in many aerospace contracts, there is a monthly program review, where the technical, schedule, and cost statuses are presented.

A bigger issue may be that your company isn't doing "earned value" accounting on a monthly basis. Yes, that's tedious, but knowing monthly, what your position is, gives you the flexibility to make the hard decisions.

My recommendation is that regardless of what the customer might want or care about, your company needs to do monthly reviews. Every cost account needs to be updated according to their progress, schedules need to be updated, technical progress needs to be updated. Your cost accounts should be broken up into man-month sized chunks; do not give your team leads a man-year's worth of budget, because everyone lies, or, at least, sees the world with rose-tinted glasses, and you won't know until month 11 that a team lead is choking to death.

I think that even with old-school, classical methodologies, you can run efficiently, effectively, and profitably. If your management can't swallow "agility" because it's new-fangled, just bear in mind that most of these methods are not new, they're simply renamed. I would simply recase your proposals in terms of more classical management terms, like earned value, monthly program reviews, etc. I would bet good money that they'll still get into trouble, because it's not the method that's the problem, its the dedication and motivation of the management that's the problem.

Design for manfacturing, design for test, concurrent engineering, etc, were all the same basic approaches. They were renamed and repackaged, not because they provided anything new, but because management never wants to actually the grind work that they should be doing.



TTFN

FAQ731-376

 

[mgeerts]

Thanks again.

So nevermind my problems - Does anybody have a success story about moving from a failing BRUF (Big Requirements Up Front) design/management method to an agile or other iterative method?

 

[analogkid2digitalman]

I ran across this article in Control Engineering a few months ago- "How to avoid Project Failure" that has a few relevant points in it (not agile:

http://www.controleng.com/article/367270-How_to_Avoid_Project_Failure.php

-AK2DM



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"It's the questions that drive us"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 

[MikeHalloran]

For a time, I was a {Cowboy Coder| eXtreme Programmer} inside of a a BRUF outfit. I got called when the BR were months or years away, and an important show was weeks away, and the hardware had just been changed in a major way, stuff like that.

It worked best when I could just interface with the hardware guys and the testers, who could serve as an analog of the end users. I was cranking out four or more iterations per day, each a little better than the last, all documented (only, and thoroughly) in the source code.

( We used printers on an OEM basis, and since I printed out a complete set of source with every iteration, I got to test the printers too. I was chewing through a couple boxes of fanfold every week. )

It turned out badly when the BR bureaucracy got involved, and demanded a Formal Design Review, with flowcharts and more for everyone in attendance at a huge meeting that I was expected to organize and run. I have little tolerance for meetings.

A fair amount of my code ended up in the BRUF product, after the BR had been generated from it, and the source had been stripped of all comments.


Back to your question.
I think an iterative method can work,
IF:
- You produce a (barely) functional prototype as soon as possible.
- You chain an end-user representative to it, and have him test every iteration and sign off on it, or bitch about it, as continuously as possible.



Mike Halloran
Pembroke Pines, FL, USA

 

[mgeerts]

Mike, thanks for your input. I'm amused that your code would be stripped of comment, "design" documents created from it, then pushed into the project - seems an obvious waste of time at no benefit. That must have been painful to watch.

 

[Noway2]

I have spent the last two years working for a company that, on the surface, does projects in a BRUF format. In that time, I have watched how the engineering manager handles this, knowing full well that in reality the BRUF method won't get the product out the door on time.

This manager routinely would make decisions based on the knowledge available, admitting that some of them will prove wrong but focusing on the fact that most of them will prove right. In the absence of marketing (customer) input, this person would become marketing and make the decision. The fact that the customer doesn't know what they want is / was not an excuse. Development went forward because it had to or it would be engineering that didn't make their commitments.

It looks to me like you need a combination of this "damn the torpedoes" attitude combined with a process that factors in an enhanced requirements gathering where you have (marketing) iterations working on figuring out the requirements. It also sounds reasonable that your (billing) strategy should spell out time and material for these iterations. If the customer objects to this and insists on a set of requirements then you clearly have cause for making them pay and schedule slippage when they change their mind.

To answer your question regarding, experience with "Agile" - it is a buzzword. The only people that value buzzwords are the recruiters and those that aren't in the know.

 

[IRstuff]

Just one last interjection from me.

Someone who has failed BRUF and moves onto (insert fad here) WILL fail at that too, if they don't understand WHY they failed in the first place, and do nothing to fix the systemic problems. If they do fix their systemic problems, they WILL succeed at BRUF, without resorting to switching to (insert name here).

TTFN

FAQ731-376

 

[mgeerts]

"To answer your question regarding, experience with "Agile" - it is a buzzword. The only people that value buzzwords are the recruiters and those that aren't in the know."

Right, just like the buzzwords "lean", "5s", "six sigma" that are allowing the Japanese auto market to dominate ours. Just buzz words. I contend that someone that claims that a new way of thinking is a buzzword, is either a stubborn mule or not in the know - I hope you're the latter. I've worked in places that value "buzzwords" and they work amazingly well and profit massively. Now i'm in a place that calls them "buzzwords" and we do fun stuff like record sales with 0 profit.

IRStuff
It seems to me that through observation of our past failures and near-failures, the one thing that remains constant is BRUF on projects that have customers who don't really know what they want.

When I refer to "customers" I'm not talking about a market segment of consumers, I am talking about a guy that owns a business that makes a machine and wants it automated or gadgetized. There is no level of "market research" that can protect you against that customer saying "No, I don't like that, make it different".

 

[IRstuff]

> Ignorant customers are nothing new. Buzz word solutions aren't really going to help that, particularly if nothing changes internally.

> 6-sigma isn't what's killing American business, it's simply the lack of desire to improve one's product. Oddly, the lone survivor of the Big 3, Ford, was, at one point in time, completely unable to use Mazda build-to-print Ford transmissions, because the US engines were too sloppy to mesh properly with the Mazda transmissions that were all built basically dead-nuts to the drawings. Ford's MO was to mix&match engines and transmissions, which result in neither system every coming close to being on nominal spec.

Note that the Japanese domination is simply the result of a 30-yr progression, when 6-sigma didn't even exist, but Deming did. The basic philosophy is simple, "continuous process improvement." That's something that few US companies do, and when they do, it's usually for 6 months every decade or so.



TTFN

FAQ731-376

 

[mgeerts]

"continuous improvement"? That's just another buzzword!

Seriously, though... I'm looking at what we're doing, it isn't working and I think the philosophies of agile project management are the cures to what we're doing wrong. I'm not looking at agile project managment, thinking it is cool, then trying to find ways to make it fit our business.

When I first started here from University I thought, as I was taught in school, that projects could be completely designed before the work began. This just isn't true in this market. Maybe it is in Aero or Auto or in larger companies with larger budgets and more market research ability, but it just doesn't work with what we do.

 

[IRstuff]

No, that's not the case in aerospace, although, we attempt to do that. I've been on a program where there were over 500 major change notices at the major prime level, where we were on the 3rd tier.

I wish you luck in your endeavors; I can only say that with 110% commitment from management on down, NO philosophy, new or old, will succeed. I can tell you that in my last company, we KNEW the correct way to bid programs for more than 20 yrs, yet NEVER attempted to implement that approach even once.

Continuous process improvement is one of the few buzz phrases that have survived, precisely because someone, primarily the Japanese, took ownership and have stuck with it for more than 50 years. On reading the Wiki article, it states that Ford was the only US auto company to seek and follow Deming's advice, which included a heavy dose of management changes.

http://en.wikipedia.org/wiki/W._Edwards_Deming

TTFN

FAQ731-376

 

[mgeerts]

Sometimes I feel like I should describe the business of engineering as "Watching everything go wrong around you and having nothing to do about it."

Thanks for the discussion, all. Seems like a not-well-loved methodology... just like the rest of them

 

[Noway2]

The last company I worked for was really big into "Continuous Process Improvement" and held Kaizans all the time. I think it was a vendor who once put it best: isn't Kaizan Japanese for "waste a bunch of F'ing time?". Of course THEIR idea of a kaizan was get a bunch of office people to come up with a layout for a (buzz-word warning) production "cell" and ultimately decide to take away the shop worker's radio and chair. Hell, they went as far as hiring two people simply to plan and hold these damned events.

Stubborn mule or not in the know. I have enough years to no longer be not in the know. But stubborn I am - and I can even be a real ass too. Why? Because I have seen enough managers sit around, pow-wowing in a conference room, stroking each others egos while it is dumb grunts like me that have to actually come up with a way of meeting the quarterly figures that Wall Street seems to value so much.

 

[mgeerts]

Don't confuse poor managers with poor management methodologies. Any good concept can be screwed up. Brushing your teeth in the mornign is a good plan, but don't tell your managers that or they may come in to work the next day with an eye patch due to a toothbrush wound to the eye and fire you.

My last job took these "buzz words" more seriously than the pope takes the bible and their production was seemless! It was beautiful. My current job likes to jump on a new one every year, put 4 people on it for 3 weeks, then forget about it. I know how you feel!

 

[IRstuff]

Well, given what you've just said, your only hope is to make small, invisible, incremental changes, one at a time, small enough that no manager will notice, until the overall process has improved.

Anything else, WILL fail, because THEY KNOW all such things fail, because they don't believe in them, and they have no commitment to them.

TTFN

FAQ731-376