Story Time: Learning the value of over-communicating

In the early 2000’s I began working as the product manager for an instrument that had its most common application in industrial coatings. It was a stylus profilometer, and it was a rather simple concept. We used a sharp diamond tipped stylus (think LP needle) that was dragged across a surface and measuring the height variations.

The product I managed was first sold in 1968, so the company had a LONG history, and a great reputation.

But, being a simple product, with a respectable (but not really growing) market, it wasn’t exciting. However, any product that you had been building and selling for over 35 years, and you still are doing about $14M year in revenue ($10K at a time) is by any measure a YUGE success.

That said, it was an amalgam of clever engineering, some software magic, and a cadre of customers that kept coming back. Plenty of stories around that fact, but this is a different tale.

The Trouble

About 8 months after I joined the company and took over the product line, we got our first whip of an issue. A circuit board that took the position measurement of our LVDT and converted it to digital values that we would then translate to heights and then report thicknesses of films had been rock stable, but began failing our incoming tests.

We literally shipped hundreds of units per quarter, and the incoming yield on these boards was in the high 90s for passing. Suddenly, they were failing in large numbers. 30, 40, or 50% were being rejected on incoming test, and we didn’t know why. We had been using this exact same board for all our computer controlled versions of our systems for about 18 years at that point with no issues.

Turns out that that 60% rejection rate was good, shortly we were rejecting virtually 100% of the incoming boards. Our production line was effectively shut down.

Yikes.

My first big test as product manager for this product. We had virtually 0 yield, we were beginning to miss committed shipments, and we didn't know why.

How I responded …

Recognizing that this was not a transient, I did what I thought was right, and I sent emails to our sales team, our agents and our Japanese distributor. I also briefed our management.

I explained what the problem was, how it was found, and the manifestations of the issue. I didn’t spare any details. I was open, honest and unstinting. I explained how a critical component that had never been a problem in the past was failing, and that we didn’t know why. I also committed to communicating at least two times a week, and that once the problem was addressed we would be shipping our growing backlog of instruments.

I sent emails on Tuesday mornings, and Friday afternoons. I updated on the diagnoses, the progress, and what we were doing.

It worked. It bought us the time to figure out that the original design of the board specified a single resister in an LRC circuit with an almost impossible specification (1% tolerance) and when the supplier of these resistors terminated production of then, they subbed in a 5% resistor, and bam, no yield. Turns out that our original design was far too sensitive to this component’s tolerance.

A redesign to the circuit (relatively minor) reduced this sensitivity, and yields suddenly returned to mid 90’s in the percentage.

I communicated this as well, explaining the issue, describing the fix, and that restored the faith and trust with our partners, and our teams.

What I learned

A few lessons. First, open communication is key. Never be afraid to come clean. Hiding it is a recipe for disaster. They can smell a smoke screen, and that erodes trust.

Second, shit happens. No matter how well designed something is, a failure is always a possibility. You can’t prevent them 100%. You can just deal with it head on. No matter how stressed you are about it, it will work out if you don’t lose your head.

Third, I learned that really good manufacturing engineers are gold. They work their asses off, they are smart, and if you listen to them, you will be smarter too. Put your ego away, and sit down with them.

Lastly, good analog electronics designers are worth their weight in gold. Good digital guys are a dime a dozen. You can recruit from the campus, and get great people. But for analog signal processing, clean electronics, and superb sensitivity, those EE’s are becoming rarer than hen’s teeth. When you have one in your stable, treat them right. It is much more difficult to replace them than a whole pack of engineers and product managers.