In a dark, windowless room in a Helsinki basement, Kjell Ingman slams a small plastic bag down in front of me, like a detective presenting a suspect with damning evidence of their guilt. He opens the bag and starts emptying its contents – dozens of tiny cubes rattle out onto the wooden table.
They are multilayer ceramic capacitors – a key component of the drives that engineering giant ABB builds here, in a series of vast, connected buildings in the Finnish capital. Earlier in the day, we’d walked around the factory floor where giant motors were being assembled by workers in blue overalls. They tightened bolts, fitted magnets, and wound huge, glistening reels of copper into the precise arrangement that makes motors spin.
Variable frequency drives control the motors by altering their electricity supply. For ABB’s motors, which can power huge machinery such as ship turbines or cranes, these drives are housed in large cabinets filled with electronics – hundreds of different components and interfaces.
The cubes now lying scattered across the table form part of IGBT (insulated gate bipolar transistor) modules, which act as a bridge between the drive and the motor. They are about 2mm on each side, and completely identical to the naked eye.
They don’t seem like much, but they go through much more stress than other parts of the drive, particularly in situations where the power is constantly being cycled on and off.
If the drives fail, it can be disastrous. “If you have a failure in a steel mill and you lose a slab, or in a wafer plant and you lose the entire wafer, it’s a huge downtime cost and a huge recovery cost,” says Steve Ruddell, head of global marketing and sales in ABB’s Drives division. “On an oil rig you can be looking at $400,000 an hour in outage costs.”
That’s why ABB has invested a lot of money in making sure its drives don’t fail. It’s assembled a team of technicians and engineers to track down problems and identify the culprits.
Ingman is an engineering detective – it’s his job to tell these supposedly indistinguishable objects apart, and investigate why they fail. To do that, he has a range of cutting-edge forensic tools that would be the envy of most crime labs, all held in a series of connected rooms at ABB’s Motors and Drives factory. They call it the Customer Supplier Investigation Lab. Welcome to CSI Helsinki.
Understanding failure
Like most engineering companies, ABB routinely carries out quality checks on products as they roll off the production line. Unlike most, when they spot something wrong with one of the circuit boards, it goes through an unprecedented battery of testing and interrogation.
For new products in development, every single board is checked at CSI Helsinki, which opened in 2015, while others will only be sent in for testing if the system triggers a fault during production, or if a customer reports a problem with one of the drives out in the field.
Ingman leads us into a room full of expensive imaging equipment. In the centre, there’s a computer console with a screen mounted at eye level, and a black keyboard with glowing red letters. An IGBT module featuring some of the small ceramic capacitors has been sent into the lab for investigation, and placed into the €700,000 X-ray machine.
A young technician taps away at the keyboard, bringing up a two-dimensional image of the sample similar to that taken with an optical microscope. Nothing appears amiss, but with the X-ray machine they can go deeper, as Ingman explains. “If we want to go much deeper into details we can change the mode to 3D, put the sample inside and then rotate it through 360º.”
By taking up to 1,000 images as the sample completes a rotation, this technology can be used to build up a complete 3D model of the component, and spot defects as small as 40 microns. In about 10 minutes, the technicians can get a good idea of the soldering quality on a particular part, but it takes between one and three hours to get a complete picture. In this particular component, there’s an internal crack in the ceramic that wasn’t visible in the 2D scan – it shows up as a thin white line against the grey of the X-ray.
In one case, as a cheery video voiceover outside the CSI lab explains, a printed circuit board was flagged up during production and sent for a “root cause failure analysis”. It was found, using X-ray scanning, that the issue was a low-quality counterfeit component from one of ABB’s suppliers. “The result,” explains the video, “is that nearly 2,000 units were blocked from distribution, and no suspect components ever reached our customers.”
Ingman also shows us a magnification of the delicate copper tracings that make up the circuit board, like a network of roads. At one place, contamination from the manufacturing process has created a short-circuit. This is something that would be impossible to see normally without destroying the board, explains Lasse Makelin, ABB’s senior vice-president of Drives.
The lab has other tools for even more detailed analyses. In one corner, the Sonoscan booth bubbles away like an aquarium crossed with a 3D printer. It consists of an airtight chamber surrounded by water, and it uses ultrasound to create images of components, layer by layer.
So instead of just seeing the external surface of an object, or any defects revealed by X-rays, ABB’s technicians can create up to 100 virtual ‘slices’ through an object. “With this machine, you can see layer by layer and you can check exactly which layer is defective,” explains Makelin.
Wearing down of the chip soldering layers is evidence of power cycling breakdown, while wear of the system’s soldering results from thermal cycling. A smoking gun.
Tin whiskers
At about 6.30pm on 28 August 2009, police in San Diego, California received a frantic phone call from inside a speeding car. Off-duty highway patrol officer Mark Saylor was behind the wheel of a Lexus ES 350 saloon car, with his family, when the accelerator jammed.
The car was out of control, travelling at more than 160km/h. Saylor’s wife called 911, but moments later the Lexus slammed into the back of an SUV and careered off the road, bursting into flames. All four occupants were killed.
The incident was one of a number involving Lexus and Toyota cars. The Japanese firm which makes both brands had already instituted a mass recall of rubber floor mats, which were thought to make the accelerator stick, and was soon rolling out replacement accelerator
pedals too.
Some have argued that electrical problems were to blame, although Toyota maintains that the issue is mechanical. A study into two Toyota vehicles from 2003 and 2005 by NASA’s Engineering Safety Centre found tin whiskers growing on the circuit boards of both the cars’ accelerator pedals. Tin whiskers are thin protrusions that grow out of the solder on circuit boards, like tiny hairs – although they’re much thinner than human hairs. They’ve become more prevalent as a result of the reduced use of lead in solder owing to environmental concerns. If they grow long enough to touch other parts of the circuit, they can cause a short, with potentially devastating results that are almost impossible to detect.
Unless, of course, you’re in the CSI lab at ABB. The technicians use a scanning electron microscope to bomb samples with electrons, creating an image that demonstrates exactly which elements the sample contains, and where. Ingman brings up a picture of a resistor and shows us the layer of silicon on the top of it, just seven microns thick, but in the image it’s as clearly defined as a layer of jelly in a trifle.
The electron microscope can take things down to the nanometre. Ingman shows us the tiny aluminium balls that make up the heat sink for a chip, and the tin whiskers protruding from them like tiny cracks on a window pane that caused a short-circuit inside a prototype drive. “Now we can investigate, and make new designs so that they don’t grow any more,” he says.
Accelerated life
The next room on our tour hums with the noise of fans. If the first area was the forensics lab, then this is more like an interrogation suite. Fridge-sized chambers constantly heat and cool, as ABB’s technicians try to simulate the stresses and strains that chips will go through out in the real world. “Here we find the limits, where the boards break down, where the components break down,” says Ingman.
They can change the humidity, temperature and voltage – a technology that’s currently unique to ABB. In some cabinets, the chips are constantly switched on and off, and the leakage current is measured to see when they start to break down. Others rapidly switch between hot and cold – one goes from -80ºC up to 185ºC, with the temperature rising by 2ºC per second.
They call this Highly Accelerated Life Testing. Four months of it is the equivalent of 10 years in the field. “If they break down at 100ºC then we need to do some changes and make sure they go up to 150ºC,” Ingman continues. “Then we compare the old design with the new design and make sure it’s better.”
Sometimes, non-destructive testing isn’t enough. Next door, in what I like to think of as the autopsy room, there are protective masks that look like riot helmets, unmarked containers full of white powders, a set of digital scales and a hairdryer. Compared to the high technology of the main investigation room, this looks more like the trailer park laboratory of a meth cook – like something from Breaking Bad.
They start by cutting the area of interest out from the circuit board in question, and then set it in epoxy – that white powder is weighed and melted, and then set with a hairdryer. Then they grind the sample down to just the right level to bring to the surface the layer they want to look at. A similar technique is used by neuroscientists to make slices of the human brain for examination under a microscope. It takes a bit of practice, says Makelin, “but after 2,000 samples you know what you’re doing”.
Then they can use an electron microscope at up to 300,000X magnification – it generates images that are a gigabyte in size that start to reveal the secrets of defective electronics in fine and sharp detail. We can get close enough to see a crack running all the way through a component.
“We still don’t know the root cause,” says Makelin. “Now we work on it and try to find out. We suspect that it’s come from our supplier – when they pick and place the component maybe they used too much force and broke it, or maybe there was humidity in the package.”
It’s partly about trust, explains Makelin, back in the dark meeting room where we started our journey. “The component business seems to be very varied,” he says. “You cannot trust the datasets.” He points out that even components with the same part numbers could have been made in different factories, in different countries, and may vary in quality. Whenever ABB changes supplier – or even if one supplier starts making a component in a new location – they run through this battery of tests. “Understanding failure is the key to our success,” is how the video voiceover explains it.
“It’s almost a fanaticism here,” says Ruddell, which is why we leave Helsinki feeling slightly fearful for one of the factory workers. An IGBT board has failed during the production process.
The technicians at CSI Helsinki were able to use their expensive technology to pinpoint the problem, and identify the culprit. There’s a tiny crack in the epoxy – just seven microns thick, impossible to see by eye – and humidity has entered the chip, causing a short-circuit.
“Something happened in the production,” explains Makelin. “When they installed the board, the screwdriver operator slipped and broke the component.”
At some point soon, a busy worker in ABB’s sprawling Helsinki factory is in line for a tap on the shoulder and a visit from the engineering crime squad. “We’ll teach him for next time.”
Content published by Professional Engineering does not necessarily represent the views of the Institution of Mechanical Engineers.