Electronics Conspiracy - Techno geeks ?

Okay, just warning folks in advance; this is a conspiracy which will likely only be of interest to techno geeks and electronics but may be of interest to others as well

As I've noted previously, I work in the electronics field with some pretty advanced systems. Virtually all of these systems require UPS power backup for power stabilization and battery back up. Consequently, we have literally hundreds if not thousands of UPS's ranging in size from 7-8kVA all the way to units in the mVA category. Because many of these systems are mission critical the UPS's are monitored via network monitoring.

Recently we've been experiencing a number of failures of network interface (NIC) cards in one specific variation of Eaton UPS's we use. The counts of failures suggests an endemic problem or manufacturing defect. Eaton, the manufacturer, of course, denies such a defect exists as this card is used in a wide range of their products. Just to give you an idea of the scale of the problem, we have roughly 50 of these specific UPS units and as of right now we have had over 35 NIC failures (just under a 75% failure rate).

When the UPS's were under warranty the manufacturer would replace them free of charge. Now, however, they are no longer under warranty and the manufacturer wants several hundred dollars each to replace them all while denying they have a latent problem.

So yesterday and today I had my technicians go out and pull all the recent failures from the field, and gather up a couple of known working units and we put them all on the bench to see if we could figure out what's going on. We found a couple interesting things:

1. The problem units all almost exclusively have a part number ending in a "rev. 2". Our known working units actually have the same part number, but it ends in a "rev. 10", "rev. 17" and "rev. 19". Now the simple fact that the manufacturer of the boards has found a need to revise the design of a simple NIC nineteen times suggests a problem! It's actually almost stunning that a manufacturer would revise such a simple part so many times, and suggests something more is going on here.

2. We also noticed that all of the cards which have failed have longer pins at the DIP switch protruding from the bottom of the circuit board. Immediately under the DIP switch is a grounding bracket which grounds the board ground to the case of the unit. The pins appear almost long enough to touch the grounding bracket. On all the working units the pins are trimmed off flush with the bottom of the solder point.

The board in question has an Eaton part number of 116750221-001 (rev. 'x') on the board itself. In discussions with the manufacturer they claim there is no difference between the boards, but if you look at them you can see the different revisions are considerably different in many respects. The components on some are even different. Eaton claims they can't tell what revision they send out if you order the part (and even allege that the rev number is meaningless). Yet, we've got a whole bench full of bad "rev. 2" boards, and they claim revision 2 doesn't even exist!

We've determined that this particular product is used in a large cross section of their products from low to high end models. That it is used in some of their more expensive UPS units might be why they don't want to acknowledge the problem, but yet they cannot assure us they will ship a replacement that isn't "rev. 2" because, again, the rev number doesn't exist. Frustrating...most definitely, but we will eventually get to the bottom of it.

Because this is kind of a unique problem, I thought it might be interesting to see what some of the bright folks here on ATS thought.

Personally, I think they're ducking a larger problem. What do you think?

sounds like software updates on the chips. that could be anything.

could be they sourced a new supplier ..could even explain what seems like the multiple software updates on the chips? then again, it's not unheard of that chips have exploits introduced into them..

i'm sure head office will sort if out in 2-6 weeks and we'll never hear anything about it again

..how soon until someone posts bible verses from revelations?

Ask them how they can have a revision 10, 17, and 19 without a revision 2. They don't start a revision at nine

Originally posted by rickymouse
Ask them how they can have a revision 10, 17, and 19 without a revision 2. They don't start a revision at nine

They deny there are any revisions.

reply to post by Flyingclaydisk


Interesting, I have never had an issue with a particular board.
But we had a large problem with a set of lithium batteries that had a fail safe programed into them, so that if they discharged below a level of 32v the became inoperable and nonchargeable.

Eventually they came up with a battery recovery system, but it was only like 40% effective. And those batteries cast $10,000 a piece. We were irate to say the least.

Those types of companies will do anything to make a buck.

reply to post by UNIT76


We are the head office.

Do they manufacture the network cards themselves or are these components sourced from an external manufacturer and simply installed in their units?

Other than that it sounds like typical left hand not knowing what the right hand is doing. All too common in tech support as you'll be well aware.

seems to me that the manufacture might be trying to pull some woll over the eyes. If it were up to me personally I'd be switching to a different manufacture, to me it seems that the rev.2* cards are causing the problems and the manufacture are not sticking by their product.

To me, I'v seen this type of behavior in electronics before too and its not worth it to keep giving those companies the money, its a well known marketing scheme to design parts to have a "shelf life" if you will...

Originally posted by scoobyrob
its a well known marketing scheme to design parts to have a "shelf life" if you will...

Ah yes, thank you Henry Ford. If it wasn't for planned obsolescence we'd save ourselves a fortune, but then they'd be broke.

reply to post by Flyingclaydisk


Revisions could be minor changes and tweaks to software or hardware. Even changes in suppliers might necessitate some change in the board.
Any number of reasons.

Maybe they switched suppliers because of sub standard parts. And don't want to say anything to avoid liability.

I myself find that most of the APC ups units we have are failing after years of use. Nothing lasts forever.

You say the card is in use in much of their range. Consider the cost to replace all those cards world wide. You are not only looking at the cost of the cards, but the cost for them of swapping them as well.

It is this sort of thing that can send a company broke.

I doubt they make the cards themselves, they are more than likely made in Taiwan or some similar place.

Have your lawyers send them a letter threatening both a court case and exposure in the industry press.

It is exactly the same when motor cars need a recall. Some companies are good about it and then there are the rest who have to do a lot of kicking and screaming.

You may want to offer "You provide the cards and we will install then without cost to you."

P

Parts and writing on them are meaningless from the perspective you're talking.

You'd need to talk to the manufacturer of the component, not the entire thing. The people who set out specifications don't care who supplies the parts as long as they meet the specs and are within the tender budget.

The company might not have made any changes to the specs required, so have not made any revisions. But the people who make the components might be constantly using new chips, manufacturing processes, different boards entirely.

And if you're the only place that is getting a 75% failure rate, and the company you deal with says they have no problems elsewhere, and you can't find evidence of problems elsewhere with the same devices, then it implies something could be awry within your infrastructure.

The revision # of the chip is probably a firmware upgrade or revision. I would take digital photographs of each revision and send them to the manufacturer to prove that there is indeed a different revision number on the chips. The people that are shipping.ordering parts are not ET's. I am an Electronic Tech myself, but that is not my profession. I acquired the degree back in 1984. Anyway, they probably don't have a revision for their part number, or boards that they know of, however, the manufacturer of the NIC is probably ordering chips from a chip maker that has made these firmware changes to use in other electronics components. It doesn't mean the whole NIC is revised.

It is interesting that the NICs that have failed all have the Revision #2 chip onboard though. It could be doing many things that are causing premature failure. You said you are an ET, have you identified which components on the board have actually failed? Or do you only know the board failed? Do you have the schematics for the board that failed? I normally see capacitors fail alot because of cheaply manufactured caps, but I would like to know if the Rev.2 chip has failed or if another component failed. I had a lot of a particular motherboard fail on some POS computers and it was cheap capacitors on the boards that the manufacturer was installing on the boards for a period of time before changing caps and buying more quality caps. I simply replaced the caps with quality caps and they were up and running without buying new mother or main boards.

If it is the Rev. 2 chip, then I would also report that these indeed were the failure points and not some other component. Also, the fact that a ground plate is so close would be concerning in an APS if surges or some other environmental condition generates static electricity in the area. Network cards are extremely sensitive and any static could damage them. I would place and insulator in between the card and ground plate. You could use some non-conductive tape on the ground plate just to ensure you aren't getting any arcing from static build up.

Basically, you as the repair tech want to stop the problem. If they are using them in more expensive units and they are not failing, then I would think some problem with the units structure that you operate. Insulating the NIC from any static would be my first step. I don't know how much heat they are producing, but you could use some pink poly and wrap them to insulate the NIC from static. Other than that, you need to identify the failure point on the card itself. I am guessing that you have to use their card and can't use another manufacturers NIC. Anyway, good luck with figuring out the source of the problem and proving it to the manufacturer. Like you said, they aren't going to admit anything. Especially after their customers have had to spend so much capital repairing or replacing the offending cards.

Given your description of the pins/soldering job, I'm thinking its a manufacturing failure & cover up.

I'd not be concerned about the revision/version number over much, except as a way to possibly track a bad lot.
I see a ton of software revisions on my hobby, and well some patches or versions are better than others.

I think you should try asking around on the more techie forums like Tom's Hardware and or others in your industry for a better idea on how this brand is doing for other groups.

It sounds like a bad factory / parts run, but thats just my opinion.

M.

reply to post by spirit_horse


Great comment!

Actually, I am an electronic systems engineer and program manager for the entire enterprise (it's actually more than that, but I digress). I had my techs go retrieve the chips and then myself and another engineer who works for me went to examine them.

The 'rev. 2' is not a firmware rev, it is the actual board rev. The firmware (which we also checked) is a rev 4.3.4. Our issue when they go out is basically that they just lose their mind and stop communicating. We've thought (and checked) things like surges (all surge protected), differential ground potentials, lightning, thermal stress...virtually everything. In the end my OP is a summary of the only commonality we've been able to find.

When the boards die we are unable to even console in to them to do anything. All rev 2, and the longer leads on the DIP switch by the grounding strap.

Yes, the manufacturer would likely have to spend countless millions if they admitted a latent defect, but the boards are cheap enough that they can replace them under warranty then deny there's a problem when the unit is no longer under warranty. In the end, it costs money; money to replace them, money to troubleshoot them, money to have myself and others have to even get involved in this problem. That money is a lot more than the $300+/ea. cost of the board itself, but when you multiply the problem by dozens you can see the issue.

As it turns out this same board is used in some of their higher end UPS's. Changing out those units to accept a new board would cost millions alone. This is why I'm thinking they are in denial over the problem. Our units are relatively small units (just 10kVA units), but they're scattered over a 53 square mile area (some in very difficult locations to get to).

reply to post by winofiend


Also a great comment / observation!

As noted, we've checked the infrastructure for problems (that's where we started initially). This has been an on-going problem for almost two years with these units. This is also why I directed our techs to go pull every bad board and get them in the same place at the same time so we could look at the bigger picture for clues.

Yes, I'm aware of part numbering from sub-suppliers, and this may well explain the rev numbering. However, that the manufacturer refuses to acknowledge anything of the sort (including the rev numbering which is plainly embossed onto the main board) suggests to me the mfr. is acutely aware of the issue and consciously denying it.

As much electronics is today, many components are manufactured by a supplier other than the manufacturer of the completed unit. Take PC's for example; they are a collection of parts from numerous manufacturers. However, in this case, the particular board is proprietary to Eaton. I can't just go to another supplier of NIC cards and buy one like I could for a PC for example. Thus far the manufacturer will not acknowledge different suppliers, different revs; they only use a part number that is the embossed part number without the rev #.

As I noted earlier, there are actually different components on the boards of some of the revs. Literally! Some of the values of the components are also different (transistors, capacitors, etc.)

We've asked Eaton for the schematic to the device, but of course they refuse to release it to us because they know we'll likely go have one built (no comment there). Initially, we'd just like the schematics so we could bench level test the circuitry to find out where the fault is. This might help us understand the problem better.

Originally posted by spirit_horse
I normally see capacitors fail alot because of cheaply manufactured caps, but I would like to know if the Rev.2 chip has failed or if another component failed. I had a lot of a particular motherboard fail on some POS computers and it was cheap capacitors on the boards that the manufacturer was installing on the boards for a period of time before changing caps and buying more quality caps. I simply replaced the caps with quality caps and they were up and running without buying new mother or main boards.

Some day I am going to either write an expose or do a documentary about a conspiracy regarding electronics manufacturers and electrolytic capacitors.

I am a computer/electronics technician, been doing it for most of my life i.e. 30+ years. And I can say without heistation that the entire electronics industry is using substandard electrolytic capacitors in order to enhance planned obsolescence.

I see it on a daily basis, with PC based equipment. The substandard caps in the power supply start swelling and failing, then the substandard caps on the motherboard start to fail because the power supply caps no longer filter the DC properly. We have equipment 10+ years old, almost 100% of which are still running. Equipment purchased in the last 5 years? Out of 60+ pieces of equipment, 100% have failed.

Here's a good example - my 50" LG plasma TV. It died, almost exactly after the warranty ran out. I pulled it apart, and probably 50%+ of the caps were swelled and blown. I replaced them, it ran for another 2 years, then the rest failed, I replaced those and it is running to this day.

Sorry to highjack this thread, but I had to vent.

reply to post by FosterVS


It's not a hi-jack, you're right and this is probably related to the problem we're having. It may not be a cap, but there are clearly manufacturing differences in these components with some significantly better than others from a QA/QC perspective.

I'm still amazed at 19 (that I know of) revisions on a simple thing like a NIC card!

reply to post by FosterVS


Could not agree more.

I have audio equipment manufactured in the 70s and 80s that is still functioning quite well. That is 40 years and still going. Replacing all the caps sometime in the future would give it another 40+ years but then you have the problem of trying to buy good caps and that is very hard.

OP. You must know who your industry competitors are that use much the same products. Give them a ring! Ask if they are having the same problems.

Also, have you tried typing into google the brand and model number and the phrase 'known issues' or 'problems'

If you haven't, try it. I am fully aware that we are not talking about consumer products, try it anyway. It works quite well! Yes I have been in the electronics industry.

P

Sounds like a bad revision to the LRU you identified. It is hard to t/s from afar though and without first hand knowledge. In my experience (10 years) Eaton was crap; all the time. But that was me.

ETA:

Although not that common anymore, do you have anyone that has bench experience that can test the card down to component level?