F-35 software overrun with bugs, DoD testing chief warns

F-35 software overrun with bugs, DoD testing chief warns

I like that. whining.

I guess the I told you so crowd is going to say i told you so.

But I guess with something that's so freaking complex there will always be software bugs. No matter what.

It's interesting to see how they are struggling with development.

And I hope by they time they get to block 3 code it will be better.

The F-35's flight plan appears to have delays written all over it. A previously unreleased memo from Michael Gilmore, the Department of Defense's director for Operational Test and Evaluation (OT&E), details a list of problems that will likely hold up the testing of the final configuration of the aircraft—and will mean the "Block 2B" aircraft now being delivered to the Marine Corps soon will continue to be full of software bugs for years to come. But officials with the F-35's Joint Program Office (JPO) have downplayed the seriousness of Gilmore's concerns, with one military member of the office taking to the Facebook page of a defense publication to call the memo "whining."

It did read a bit "whining". These all appear to be his suspicions rather than facts and concerns that he may be valid in raising internally which should not have found its way out of the project? In any case, it appears its his job to raise potential issues with test programs.

Its a pretty unique program so its not unlikely

tripling the rate at which weapons delivery events have historically been conducted,

I dont know the program but if Italy, the UK and Australian birds (for example) take part in weapons delivery then it wouldnt be unthinkable for the development program to beat historic records, its planned to smash many historic production records!

If thats an official document, either its formatting has either been stuffed up by .pdf or he needs a new PA.

Computer bugs are usually assigned a priority:
showstopper/critical - causes the system to crash for no obvious reason. This has to be investigated immediately.
major - the applications keep running but some functionality is lost, maybe a particular menu window becomes unusable after a being used a couple of times
minor - A particular men option isn't set when the window is opened, but can be read by clicking the update button
trivial - wrong error code is returned in a particular module, or there's a spelling mistake in a message. It's more important that some information is provided than nothing at all

There are other classifications: Bohrbug, Heisenbug, Mandelbug, Schroedinbug, Fractal bugs
thebeautifullmind.com...

The number of bugs found and number of bugs fixed usually form a normal distribution curve. So that at the start more bugs are found than fixed. Then at the end more bugs are fixed than found.

A project that big and complex is bound to have freaky bugs in it.
For other reasons, see my signature.

As for the schroedinbug.
Well. In my career I only once had to hunt down a supposed "schroedinbug".
In the end, it was poor name convention and lazy "search and replace".
He must have hit "search and replace all" and his poor naming convention added to the problem.
I could bring evidence in the form of earlier file versions and the history of the search and replace actions and I was able to recreate the bug.

So he search and replaced, went away, troubleshooted his other bug and started to investigate, then found the other bug(that he also made) by accident. He was adamant it´s a schroedinbug. But since he was telling everyone (as an excuse for his mistake, admit it and it´s ok, mistakes happen), the head of the office tasked me hunting down this anomaly.

That´s why a good piece of code is always pre planned and not "let´s figure this out while we do it". Not saying on the fly coding sessions are worthless. It can be fun at times but the planing method yields better results in a shorter time.
Not saying the F35s source isn´t. But in general.

This is interesting:

The software is still a work in progress, and testing of potential security vulnerabilities—which could potentially keep aircraft from being able to take off—has largely been deferred for now while Lockheed Martin and the JPO focus on getting the software to actually work as intended.

Emphasis mine.
I consider testing security vulnerabilities to be highly important in any software development process, military or otherwise.

And rather than fix Block 3i, the JPO made a "schedule driven decision,"

Schedule driven decisions are sometimes necessary. I had a boss who had a saying: "Time to shoot the engineer and ship the product." We are perfectionist sometimes. But, I have also tested First Customer Ship (FCS) software in the past, that was schedule driven, and all of the critical functionality wasn't even in place! A whole test plan scrapped because the UI had nothing but stubs behind it.

I'm not saying that this is the case here. But since this program is so far behind schedule and over budget, I do question the validity of this decision.

-dex

a reply to: DexterRiley

The issue is ALIS. The unclassified side of it has been giving them fits.

a reply to: Zaphod58

I have to admit, while an interesting idea, ALIS seemed like it was going to be fraught with problems from the moment they announced it. At least to me.

a reply to: anzha

The classified side is coming along. They're slowly reducing the errors down. The biggest problem is that if someone can get into the unclassified side they may be able to backdoor into the classified side.

a reply to: DexterRiley

Because one reason is you start to do security checks (of course take security into consideration from the start), when you´re seeing the finish line. Before that, there are too much changes.

Edit: Not saying there do not seem to be much problems but with millions of lines of code... It´s forgivable to a degree.
And consider, a "bug" can also be wrong text etc.

a reply to: verschickter

Because one reason is you start to do security checks (of course take security into consideration from the start), when you´re seeing the finish line. Before that, there are too much changes.

True. But there should still be at least some limited security testing even in the Unit Tests. And, if you consider the finish line as the point where some actual features are becoming available for functional testing, that is a perfect time to start some security testing related to those available features. It's better to find any major security flaws while the system is still somewhat fluid.

ETA: This notion of not being refined enough for security testing would also imply that the product is quite far from the finish line.


-dex

a reply to: Zaphod58

The biggest problem is that if someone can get into the unclassified side they may be able to backdoor into the classified side.

This was the first thing that I thought of when I read your reply.

I have no idea how this product works. But, presumably if there is an unclassified section, there has to be some data that is exchanged between the trusted and non-trusted domains.

-dex

a reply to: DexterRiley

The system was designed so that one computer could access both sides of the system. One side deals with logistical systems, while the other deals with aircraft systems. They don't actually talk to each other.

a reply to: DexterRiley

Absolutely spot on. But we don´t know the details, how much had been done until now etc.
I get the impression -could be wrong of course- that the consens is like:

"Oh it´s software, we can fix that any time. Let´s do the hardware stuff first."
When we developed microcontrollers for machine learning back a decade, it was a hand in hand process until the end. Security issues were considered but more on a side line. What could go wrong?
What went wrong was that on a certain software condition, the hardware-reset lines jumped high, sending a "wave" of resets that propageted through the entire cluster of controllers, effectively making the whole batch useless.

The result was a orgy of debugging, both on soft and hardware side for weeks. When we found the bug, it was a capacitor on the bus line. Somehow, a comma shifted. They didn´t notice, everything went fluent but on certain conditions you could start an avalanche of resets.

But then, I suppose there are capable people there, they should know what they´re doing. Oh wait, we were too

a reply to: verschickter

"Oh it´s software, we can fix that any time. Let´s do the hardware stuff first."

Yep, been there. Then there is usually the option of "fixing" the hardware bug in the software.

The result was a orgy of debugging, both on soft and hardware side for weeks.

Years ago, one of my mentors referred to that as "a memorable debugging session." There are bugs, and then there are BUGS. The BUGS are the ones that you remember for the rest of your life.

But then, I suppose there are capable people there, they should know what they´re doing. Oh wait, we were too

I have no doubt that the folks designing and building those systems are quite capable. I've worked with a few of those folks who come from the defense and aerospace sectors, and they are quite "high-end" engineers. But as you said before, with several million lines of code even the best are going to come across unforeseen issues.

-dex

a reply to: DexterRiley

Years ago, one of my mentors referred to that as "a memorable debugging session." There are bugs, and then there are BUGS. The BUGS are the ones that you remember for the rest of your life.

So true. Hacking around so intensive that you loose your feeling for time and after those weeks, looking back it was a nightmare of overhours, sleep deprivated induced funny and awkwardness, dark ringed eyes, red eyes, no sunlight, funny ear sounds and unhealthy doses of tobacco, coffee and energy drinks.

One time a younger developer (who you would think would drop the floor as one of the last ones), fell asleep on his keyboard with headphones on (the "i need MIDI music to code type"). So we played Carmina Burana on his headphones. You should have seen his face as he woke up I was directing the music with two soldering irons standing on a chair in front of his desk with what I considered a crazy wide eyed look in my face, while the other fidgeted with the light and making thunder sounds with thin sheets of copper dancing around him.

Sleep deprivation on it´s best. Back in the good old days when rubber boots were made out of wood. Great times.

originally posted by: grey580
F-35 software overrun with bugs, DoD testing chief warns

for nearly all values of XXX: XXX Software overrun with bugs, XXX testing chief warns.

Source: writer of buggy software, like everybody else.

a reply to: mbkennel

It´s somehow understandable that laymen people think it´s not a problem to write clean code without bugs but the witch hunt they make about is funny.

Source: writer of buggy software, sometimes, like everyone else ;-)
Mistakes happen if not you are not human

originally posted by: verschickter
a reply to: mbkennel

It´s somehow understandable that laymen people think it´s not a problem to write clean code without bugs but the witch hunt they make about is funny.

Source: writer of buggy software, sometimes, like everyone else ;-)
Mistakes happen if not you are not human

The witch hunt isn't really about that. It is that the primes are so afraid of bad press (public or within DoD) that they spin everything but fatal errors in code as "nominal behavior" until it breaks in front of someone important or puts somebody's life in danger. Nobody want to step up and say "We just won't be ready by such and such date". The witch hunt is due to the primes saying we can do such and such for XXX money in XXX time and failing miserably after asking for more and more money since they know they can. It happens all the time in almost every program.

a reply to: Halfswede

Any further evidence other than your word for that?

I worked for Lockheed for four years (software engineer). We weren't writing the most cutting edge software, but there were very few talented engineers that I came across at that company. I went to Amazon after that and basically started my career from scratch because I learned very little at Lockheed about how to actually be a software engineer. I had four years "experience", but people at Amazon with one year experience knew a lot more than me. I look back at my code at Lockheed and realize how #ing terrible it was, and that there was no one on my team who was able to mentor me or give me actual design guidance in code reviews. My designs were terrible, and people with 20+ years experience couldn't even notice. So, while I'm sure there are plenty of extremely talented people working on the F-35, I can see how gov't programs have much #tier software than non-government businesses. However, even the most reputable software companies new products are often wrought with bugs, and it takes several iterations to get things right. So to anyone with any experience in any engineering related field, it's no surprise that the F-35 is having this many issues. But, they will likely get fixed over time.