Why I Left NASA: A Curiosity of Spirit

page: 1
109
<<   2  3  4 >>

log in

join

posted on Mar, 28 2010 @ 01:41 PM
link   
Sorry for the delay in posting this, I had hoped to have it ready last week. Work has been very hectic. I am still not finished typing everything up, so I will post this installment while I finish the rest. I've needed to consult my journal more than expected, and that has been a tedious process from which I've allowed myself to be easily distracted. I find the process of re-reading my own history very emotionally draining.

For those who don't want the suspense of not knowing where this is going, and may feel my posting in installments is manipulation instead of slow fingers attached to a slow brain, you can read my post last week where I tell you in advance what we discovered.

-

This experience marked the beginning of the end of my NASA work, and the end of the beginning of my awakening; they were hiding something, and the mostly transparent organization I thought I worked for was no more.

Last January I was at JPL on a project when a colleague of mine in the unrelated MER program was working a Spirit glitch. Spirit is one of the two rovers still operating on Mars. Spirit's expected mission was for a little over 90 days long, but six years later the rover is still operating. Spirit has performed well beyond expectations, but even the best robot is going to have its off days. One of these off days was in late January 2009. My friend and colleague, who I'll call Rich, was a senior software engineer on the team tasked with maintaining the mobility flight software, the code that controlled the rover's movements and experiments.

Spirit had refused an instruction. It had been sent instructions to move, it acknowledged receipt of those instructions, but it did not move. In and of itself that was not highly unusual, the rover is given license to ignore move requests it does not believe it can fulfill successfully or safely. But not only did Spirit not move, its non-volatile flash memory was missing data about its motionless hours. Through a separate subsystem they were able to estimate that it had been awake for at least an hour during the gap, but what it had been doing or why it had decided not to move was unknown. Imaging before and after the event showed no change, cameras, IDD, suspension values, terrain; all were the same. This was the mystery my colleague and his team were engaged in solving. Why did the rover ignore its command to move? Why did the rover record nothing? Presumably the one question would answer the other.

They ran a battery of system checks and all appeared nominal. A refusal to comply can have its origins in the rover having lost track of its orientation, so they attempted to recover this with an onboard program that uses the panoramic camera and accelerometers to locate the sun and determine its own orientation from that. After an initial unrelated failure in this procedure (the accelerometer package was off) they were able to reacquire orientation. After a bit more testing and investigation, with more nominal results, the rover driver (RP) once again prepared move orders that were vetted and ultimately uplinked. This time the rover moved as expected, and recorded its activities to flash.

The most probable explanation for the initial faults (the failure to move, memory loss, loss of orientation) was cosmic rays. Cosmic rays are energetic particles, usually protons, capable of disrupting electronic systems they come in contact with. With all systems once again performing as they should, it would be hard to explain any other way. And this can happen, cosmic rays have been suspected in the transitory failure of other satellites, orbiters, and landers. The potential for problems has been amply demonstrated in labs on earth, and as a result Spirit's electronics are somewhat hardened, to the degree reasonable for its expected exposure.

(continued)




posted on Mar, 28 2010 @ 01:42 PM
link   
As part of working the problem, before Spirit had reacquired orientation, Rich had done a few things to dig deep into the mystery. He had dumped portions of Spirit's memory to see what he might find through a forensic examination back on Earth.

Spirit has three kinds of memory: 3 MB EEPROM (like a bootable thumb drive, where the VxWorks OS lives), 128 MB RAM (like your computer's memory), and 256 MB flash memory (like your hard drive). He would have loved to dump everything, but that was completely impossible. Spirit's downlink is far too slow and there are limits to what you can safely dump from a running system. Downlink usually comes from a combination of UHF to orbiter to Deep Space Network (DSN) and HGA direct to DSN. UHF is faster than HGA but limited by having only two 10 minute passes of the orbiter each sol in which to transmit data for relay. HGA is much slower but the window is considerably longer, three hours in many cases, ultimately limited by power, temperature, and visibility of earth. Daily downlink capability is only about 9 MB per earth day. So, Rich focused on building the sequences to dump what he suspected would be key portions of flash and RAM. After thorough testing he got the TAP/SIE to send these on the next uplink.

The commands were in fact sent, which was actually a pretty big coup for Rich. The practice of dumping memory for analysis is a common technique for debugging software on earth, but rarely (possibly never) performed on a vehicle in space. In this case it was allowed because the specific commands were deemed very low risk, and the uplink and downlink bandwidths were otherwise unused as the engineering teams worked the problem in simulator.

Rich pulled down about 7 MB. He was after regions of flash which might hold evidence of a damaged table of contents (TOC), perhaps even hints of the missing records. He was also looking for regions of RAM that might contain loaded low level OS code, possibly revealing unusual EEPROM issues. Also included in what he pulled down were small data samples taken across the entirety of the flash memory. He had a pet theory that a cosmic ray memory failure event may leave some sort of signature in the data on the failed flash memory. He was able to slip this 256kb flash sampling past the TAP/SIE censors in the guise of sampling for the evidence of missing records, which arguably it could have been. He received the data, but had no opportunity to examine it before Spirit was deemed restored.

Rich's regular schedule resumed and it was several days before he had the time to dive into the data. A terrestrial memory dump is no special thing; it is a computer's dejecta, something a software engineer wants to deal with only when he must. But a memory dump from a craft on Mars? Now that's something special. So on his first day off he began to examine what they'd pulled down. The first few hours were spent writing code to help the analysis, to separate the single block of data received back into its original and separate parts, and to place them in a proper context so his debugging tools could assist in the analysis.

(continued)



posted on Mar, 28 2010 @ 01:43 PM
link   
A few hours later he had found the flash's TOC and found they contained no evidence of records corresponding to the rover's unresponsive time. This was fine, it was merely a possibility he wanted to check, since the presence or absence would hint at the nature or timing of the initial failure. He looked at the samples he took across the flash drive, and found nothing of absolute significance. He knew that was a long shot, and that he would likely need to collect more data from other similar failures, likely from test labs on earth, before he could see if there was a real case to make. He now turned his attention to the RAM he dumped. He was targeting the region most likely to contain the code and stack of the applications core to OS and flight software. Analyzing this would be painful, and possibly fruitless. RAM is very dynamic and while it's parceled out in a fairly orderly way at first, as each application requests and releases memory it very rapidly becomes chaotic and a useful forensic analysis is nearly impossible. At best he was expecting to find chunks of programs with signatures he could match up with code from the EEPROM he knew to be onboard, as well as the data (the stack) from some of those running programs. What he would be looking at would be machine language, not the original high level language, C. If he found anything interesting he could use decompiling tools to give him a leg up in understanding the machine language he was seeing by turning it back into the more readable C. His days off were over before he'd found anything of interest, and it wasn't until his next time off that he was able to give things another look.

Over the next week Rich had an idea for a tool to help him understand the RAM dump. He wrote a little utility to sample a dump of the version of the EEPROM most recently uploaded to Spirit, generating signatures (context triggered piecewise hashes) for the various pieces of application code on it. The signatures themselves were made of short regions of the applications so in this way he might be able to better match the fragments he would have in the RAM dump. He finished writing the tool and generating the signatures during the first half of that next day off. He then ran it against the RAM he dumped and was able to identify quite a few sections of application code. He began to check (diff) each section of recognized application code against the EEPROM source. All were as expected except for regions of code related to telecom. Only portions of the RAM dumped version matched the EEPROM version. There was nothing yet startling here, perhaps he was comparing RAM to an older EEPROM dump, perhaps this section of the rover's memory had been unloaded and the region reused while he was dumping it, or perhaps there was a bug in the code he wrote to break up the original monolithic dump file. He double checked everything. He was using the latest known rover code. He didn't see anything wrong with his parser. The only thing he could do was to see whether or not the regions which differed looked like they were actually part of the original telecom code. He found some later regions which were, and this strongly suggested that either portions of the original code had been damaged or that the portions were simply different by design, representing a different (older or newer) version of the code. He used his decompilation tools to examine the entire section of application code he could reconstruct and marked the regions which were different. He then began to walk those sections of code. Sure enough the code was different, and very oddly so. The code dumped from the rover included function calls referring to a device accessed via the VME interface (similar to a computer's PCI slot) which Rich did not know about.

(continued)

[edit on 28-3-2010 by AstroEngineer]



posted on Mar, 28 2010 @ 01:44 PM
link   
-

Sorry about the teaser ending. I might as well have a little fun, and it was the best place to break. I'll try to post the next block within 24 hours.

John

I finally created a blog last week and will post this on there as well, since it may be far more readable and easier to follow without the 4000 char limit.

*snip*

 

Personal blog removed per ATS Terms & Conditions Of Use – Please Review This Link.

[edit on March 29th 2010 by greeneyedleo]



posted on Mar, 28 2010 @ 01:58 PM
link   
First s&f...'cuz I'm counting your 24 hours...

Please hurry up. This is like a book I'm being forced to put down.



posted on Mar, 28 2010 @ 02:01 PM
link   
man i read all that for nothing....


teaser endings might be fun for you, but its only going to hurt your reputation on ATS.



posted on Mar, 28 2010 @ 02:05 PM
link   
Great thread.. i want more!!



posted on Mar, 28 2010 @ 02:06 PM
link   
sounds like some hidden code and multifuctionality not listed in the instruction book. heh.

I assume the little rovers are accounted for in every way from concept, to design, every piece, right up until the thing is being launched into space, including backups of all code within the programs...wouldn't this have been noticed beforehand or are you suggesting the program coding is different than the actual on file coding to begin with...and if so, what is the purpose and functionality of the new code anyhow?



posted on Mar, 28 2010 @ 02:08 PM
link   

Originally posted by Alaskan Man
teaser endings might be fun for you, but its only going to hurt your reputation on ATS.


Because as we know, your reputation on ATS matters.



posted on Mar, 28 2010 @ 02:10 PM
link   
Always love a good mystery, John.
I checked out your blog ...
awaiting the next installment.



posted on Mar, 28 2010 @ 02:14 PM
link   
I have been dying for you to post since you joined. I will be following this thread with great interest.



posted on Mar, 28 2010 @ 02:18 PM
link   

Originally posted by Alaskan Man
man i read all that for nothing....


teaser endings might be fun for you, but its only going to hurt your reputation on ATS.


It will not hurt his reputation in the least. The man has a job and a life besides posting his info here for us. Hang in there. This is like a fascinating book and you don't want to read it all in one sitting.



posted on Mar, 28 2010 @ 02:29 PM
link   
Enjoyed reading through that, have set on your my 50 things to do on a daily basis.



posted on Mar, 28 2010 @ 02:29 PM
link   
post removed because the user has no concept of manners

Click here for more information.



posted on Mar, 28 2010 @ 02:30 PM
link   
reply to post by SaturnFX
 


it does is you care about ATS, look at members like phage for a positive example.


otherwise you could be added to the huge list of people who use this site more like a creative writing forum.



posted on Mar, 28 2010 @ 02:44 PM
link   
There might be clues in here as to why the rovers have lasted so long, are self cleaning !?! and have at some times behaved in strange ways including waking up in different positions on Monday morning after being shut down for the weekend.

Looking forward to the next installment



posted on Mar, 28 2010 @ 02:54 PM
link   
reply to post by Mr Mask
 


What are you on about?!

How many pages have you written here thus far, and still present no proof, evidence or even a reason to believe you?

Yeah...count me as one of the many here who think you are full of crap


Proof of what? Evidence of what?

You didn't read it did you.

I thought it was very well written and many technical terms are included(you got me OP if you are joking, sounded like you really did work for NASA). Perhaps you know what the OP was trying to prove?

I read through it, and honestly have no idea because it isn't finished yet...

---

I loved this:

But a memory dump from a craft on Mars? Now that's something special.


Reminded me of the scene in Transformers where the guy from 'Sector 7' comes to NASA to debrief the Secretary of Defense and shows the clip from the mars rover that has a Transformer tampering with the rover. Can't find the clip online...





posted on Mar, 28 2010 @ 03:08 PM
link   


For those who don't want the suspense of not knowing where this is going, and may feel my posting in installments is manipulation instead of slow fingers attached to a slow brain, you can read my post last week where I tell you in advance what we discovered.


Anybody know where the link to this is? THX



posted on Mar, 28 2010 @ 03:13 PM
link   
reply to post by Mr Mask
 
I'm feeling the same vibe on this one. 'Slow path to awakening from NASA?' Sounds like a prelude to the standard ATS anti-climax we've seen before. It's maybe too early to call BS, but time will tell...

I'd enjoy being wrong and having something interesting come of all this. Still, I'm channelling images of a profile page with the legend...(This user is banned and comments have been disabled.) Anyone remember Taymour?



posted on Mar, 28 2010 @ 03:18 PM
link   

Originally posted by treemanx



For those who don't want the suspense of not knowing where this is going, and may feel my posting in installments is manipulation instead of slow fingers attached to a slow brain, you can read my post last week where I tell you in advance what we discovered.


Anybody know where the link to this is? THX


Could be this...




I was at JPL when a colleague of mine was working the Spirit glitch of January last year. He showed me what certainly appeared very strong evidence of faster than light communication. (I hesitate to use the word "proof" here, since as a scientist that term demands independent verification that was not possible. But it was effectively proven to me, by its repeatability, by our exclusion of other explanations, and by evidence we found which suggested it was intended.) Before you get the wrong idea, I am not invoking aliens here! This was surely our "stuff". But the event was significant because my notion that I was part of a largely transparent organization was severely damaged. If this existed and was deployed under everyone's noses, what else was (out) there?
ATS post

Posts by AstroEngineer





top topics
 
109
<<   2  3  4 >>

log in

join