Engineers weigh in with their stories, possible solutions to Toyota's problems

Let's hear it for -- and from -- the engineers!

My piece from last Sunday, "What it's so hard for Toyota to find out what's wrong," really brought the engineers out of the woodwork, and that's good to see. Engineers rarely find themselves portrayed or their work understood here in the mainstream media. Scientists have better visibility -- whether they're discovering new dinosaurs, holding forth about the climate or opining on stem-cell research -- than do engineers. But without engineers, to co-opt a slogan, life itself would be impossible.

Engineers take science out of the lab and put it in the real world, where you use it every day, even if you don't understand it. Or want to understand it. Or sometimes flout your ignorance of it, like a member of Congress. Too often, highly educated individuals, who have no math or science training, demean engineers to the status of geek while raising doctors to sainthood, though there is less difference between the two professions than you'd realize.

Here is a sampling of some of some of the e-mail I got in response to my story, starting with a terrific story from a former aerospace engineer, talking about failure testing during the Cold War:

"A classified payload was scheduled for liftoff from Vandenberg AFB in the late 60's. About two weeks of pre-launch tests continually failed for different reasons. An expert in electromagnetic interference from Wright-Patterson AFB was brought in to help. He took a simple doorbell ringer powered by a small battery and merely walked around the payload ringing the doorbell as tests were being conducted. Needless to say, the tests failed precisely when he rang the bell. He then designed some filters which were installed and the launch proceeded smoothly. I have never forgotten that experience since it forcefully demonstrated the absolute need for exhaustive testing of failures followed by knowledgeable design fixes."

"Working with computers for many years, I have found that anything that transmits a radio signal can affect the operation of a computer. What could be a more random signal generator than a cell phone ringing in your glove compartment, purse or pocket. A cell phone in the glove compartment is close to the car's computer. I suspect the problem could be solved by properly shielding the cars computer."

"Not quite thirty years ago, my electrical engineering professor father and a fellow civil engineer duplicated in a courtroom the failure of a Ford vacuum-modulated cruise control that resulted in serious injury when the engine went full-throttle in a residential neighborhood and the vehicle struck a house. The cause was poor quality control and an infrequently changed boring bit that was to drill a very precise hole, but when neglected, the over-sized hole changed the desired air passage past the stepped needle. An analog failure to be sure, but one engineers ferreted out nonetheless. They combed junkyards looking for similar and found them."

"As one who also has degrees in mechanical engineering and worked in development of automation, I think your article was right on target. When I started, computers were not used in controls and now they are extensively. I believe most are coded in C++, which is not a naturally self-documenting software. It is a lot more exciting to write code than comments, so it can be really hard to debug. Also, the software guys can't get down to business until the hardware is built. Then everyone is breathing down their neck. In the various programs I use on Windows, its clear that there is a lot of non-standardization and it takes a while to figure out some of what would seem obvious problems.

"One of the problems engineers have is communicating to non-engineers what there problems are. The American people look so quickly for easy answers and their eyes glaze over when an engineer tries to explain a complexity. Colleges require the same English and foreign languages plus some social sciences for all graduates but they can select "Baby Math" and Baby Science". Rather than concentrating on getting more engineers it might be easier to get the rest of the population more attuned to technology and not expect simple answers. Then the engineers can spend their time engineering rather than spending time making power point presentations to sell their work to the uninformed."

"Blaming carpets and faulty shims in the gas pedals sounds too far-fetched. I have over 40 years of experience in electronics and quality management. I have witnessed countless electromagnetic problems, from faulty designs through faulty handling on the production floor. I have also witnessed major ESD (electrostatic discharge) damage due to improper production procedures. Instead of all kinds of assumptions and crosstalk from Toyota, why doesn't some responsible government agency take a few cars, or better still, reported defective cars, to an authorized [electromagnetic compatibility/electromagnetic interference] lab and bombard it with electromagnetic interference.I have no doubt that they will find what they are looking for - a glitch in the computer system, faulty electrostatic discharge measures during production, and/or a combination of both."

"The preliminary consensus here in Silicon Valley is that there is some kind of error recovery problem in the embedded software. Specifically, what it looks like is the software incorrectly responding to one of the following unusual error conditions:

1. A bad connector, which causes a sensor or control input to read 'off scale.'
2. A bad sensor, which causes the wrong RPM reading, for example, to be input.

3. A 'bounce' condition on an input, which is usually the result of corrosion or dirt on the contacts.

These errors occur at the rate of perhaps once every few million hours of operation, which would produce problems at pretty much the rate reported. The bad news is that there is essentially no way for a lab to reproduce these problems, given that labs use gold-plated test connectors and known good engine sensors. The really bad news is that, if the above is correct, the rate of sudden acceleration incidents will increase as the fleet of 'drive by wire' cars gets older."

"The software issue is very pertinent and worrisome. Having worked 30 years in defense electronics, my initial reaction to [Toyota president Akio] Toyoda's assertion that they had never experienced a failure in this type of electronics rang hollow. Many years ago, as part of the program management division of a major aerospace firm, I went through the late design period and fleet introduction of the a new Navy fighter-attack aircraft on which my company supplied the radar, the first such unit to be software-controlled. Today's automotive electronics easily rival the complexity of that system (which sported a two-kilobyte hard drive) and the problems encountered were many and subtle, even after the aircraft entered fleet service -- and they certainly included software errors. But, in a pretty short time the radar achieved reliability levels previously not attainable. But no one truly familiar with software-driven complex electronics could have made a statement like Toyoda's, in good conscience."

And here are two very interesting e-mails from drivers' first-hand experience with runaway acceleration and what they did to fix it:

"About two weeks ago the driver’s floor mat on our 2001 Toyota slipped and covered the accelerator while my wife was driving. I removed the floor mat for about three weeks until I discovered why it had moved up to cover the accelerator. The car wash attendants remove the floor mats to clean under them before sending the car through the car wash. After washing the car, they replaced the driver’s mat incorrectly. The rear of the mat has a hole that fits over a hook that resembles a fishhook, to hold the mat in place. When they replaced the mat, they did not place over the hook, causing the mat to slide. The hook had been turned 180 degrees from its correct position, and this could only have been done manually."

"I owned a Lexus GS300 for 10 years. Shortly after purchasing it, it started to have problems with unintended acceleration. In my car, it occurred when I was trying to maneuver into a parking space. Shifting between forward and reverse, in a small area, required two feet on the brakes at all times. I gave up on Lexus because, whether they knew anything, or didn’t have a clue, they were stonewalling me. My mechanic and I devised a plan. I firmly believed that the on-board computer was storing commands and retrieving them at random. He agreed and we proceeded to take out the battery and let the charge run out. We left the car for two days without a power source. It cleared all the programming that was in the computer. Once he put the car back together; we never had another problem. I am nearly positive that Toyota’s problems reside in the on-board computer and that the accelerator pedal is just a visible, tangible object. By changing something tangible, people can see Toyota’s intent to repair damage."

Follow me on Twitter at @theticker

By Frank Ahrens | March 10, 2010; 10:30 AM ET
Categories: Autos , Congress , Corporations , The Ticker | Tags: Lexus, Runaway acceleration, Toyota problems, toyota, toyota recall model and years, unintended acceleration
Save & Share: Previous: Today marks the first anniversary of the stock market bottom
Next: In New York, another runaway Prius

Comments

1. Take out the trash. 2. I have a headache 3. Wet it before it sets.

Is that blood? No grandma it's ketchup.

Posted by: tossnokia | March 10, 2010 12:28 PM | Report abuse

I saw one post that may be indicative of the problem but not as described. Since the acceleration problem spans many lines of Toyota the problem should be part of a feature/function that would be in each.

Has there been significant investigation of the Cruise Control? Cruise control has the acceleration function and though it may function, connect or react differently in each model it's purpose it to accelerate when that function is activated. If it is dysfunctional, it could both cause unlimited acceleration (depending on the previous setting it thinks it has (for "resume")) and the problem may have also disabled the "shut off" from the brake system.

The cruise control is probably a servo system (of some sort) with a (or several) feedback loop. These are very difficult to troubleshoot because of the feedback system itself. Feedback systems are used in both mechanical and electronic systems to control the output. If my memory serves me right from courses (many) years back, a feedback system that gets positive feedback continues to increase output further increasing positive feedback further increasing output, so on and so forth. Correct feedback is used to limit output as desired.

Sounds similar to the toyota problem. This isn't a full analysis/diagnosis by any means but perhaps a start or will stir further discussion.

Posted by: bill_me | March 11, 2010 9:24 AM | Report abuse

I am not an Engineer, although I am an Amateur Extra Ham Radio licensee.

Why have Toyota not implemented the low tech solution of installing emergency ignition "kill switches" in their troubled vehicles?

I realise that this could disable other systems.

The question becomes:

Is it better to coast to a gradual halt or to fly along at 95MPH calling 911 on the hands free hoping not to run into traffic?

Posted by: williambanzai7 | March 11, 2010 10:28 AM | Report abuse

I am an engineer with 30+ yrs of design experience. Electromagnetic interference has a chance of being reproduced in the lab. What has little chance of being duplicated is Semiconductor Upset. I only know of one type, which has been a non-issue since the 70's but it was with CMOS technology which is what is in use today. If semiconductor upset is the cause for some of these events I believe current fail-safes can not guard against it. Additionally, software won't find them and there won't be any trace of the event once the units power has been turned off and back on. Everybody will just be standing around going, "what the...???", you know like what we are all doing right now.

Posted by: jim_fuller | March 11, 2010 11:30 AM | Report abuse

TOYOTA ENLISTS JAKE AND ELWOOD BLUES:
http://williambanzai7.blogspot.com/2010/03/toyota-enlists-jake-and-elwood-blues.html

Posted by: williambanzai7 | March 11, 2010 12:10 PM | Report abuse

I sure hope that an individual(s) did not intentionally sabotage the Toyota computers to assist with the economic recovery of the other auto manufacturers. The difficulties faced by Toyota at this point most certainly will assist the other companies in succeeding.

Posted by: DebraLRice | March 11, 2010 5:12 PM | Report abuse

The comments to this entry are closed.

Network News

Engineers weigh in with their stories, possible solutions to Toyota's problems