How NASA Repaired Voyager 1 From 15 Billion Miles Away

Engineers have partially restored a 1970s-era computer on NASA's Voyager 1 spacecraft after five months of long-distance troubleshooting, building confidence that humanity's first interstellar probe can eventually resume normal operations.

Several dozen scientists and engineers gathered Saturday in a conference room at NASA's Jet Propulsion Laboratory, or connected virtually, to wait for a new signal from Voyager 1. The ground team sent a command up to Voyager 1 on Thursday to recode part of the memory of the spacecraft's Flight Data Subsystem (FDS), one of the probe's three computers.

“In the minutes leading up to when we were going to see a signal, you could have heard a pin drop in the room,” said Linda Spilker, project scientist for NASA's two Voyager spacecraft at JPL. “It was quiet. People were looking very serious. They were looking at their computer screens. Each of the subsystem (engineers) had pages up that they were looking at, to watch as they would be populated.”

Launched nearly 47 years ago, Voyager 1 is flying on an outbound trajectory more than 15 billion miles (24 billion kilometers) from Earth, and it takes 22.5 hours for a radio signal to cover that distance at the speed of light. This means it takes nearly two days for engineers to uplink a command to Voyager 1 and get a response.

In November, Voyager 1 suddenly stopped transmitting its usual stream of data containing information about the spacecraft's health and measurements from its scientific instruments. Instead, the spacecraft's datastream was entirely unintelligible. Because the telemetry was unreadable, experts on the ground could not easily tell what went wrong. They hypothesized the source of the problem might be in the memory bank of the FDS.

This story originally appeared on Ars Technica, a trusted source for technology news, tech policy analysis, reviews, and more. Ars is owned by WIRED's parent company, Condé Nast.

There was a breakthrough last month when engineers sent up a novel command to “poke” Voyager 1's FDS to send back a readout of its memory. This readout allowed engineers to pinpoint the location of the problem in the FDS memory. The FDS is responsible for packaging engineering and scientific data for transmission to Earth.

After a few weeks, NASA was ready to uplink a solution to get the FDS to resume packing engineering data. This datastream includes information on the status of the spacecraft—things like power levels and temperature measurements. This command went up to Voyager 1 through one of NASA's large Deep Space Network antennae on Thursday.

Then, the wait for a response. Spilker, who started working on Voyager right out of college in 1977, was in the room when Voyager 1's signal reached Earth on Saturday.

“When the time came to get the signal, we could clearly see all of a sudden, boom, we had data, and there were tears and smiles and high fives,” she told Ars. “Everyone was very happy and very excited to see that, hey, we're back in communication again with Voyager 1. We're going to see the status of the spacecraft, the health of the spacecraft, for the first time in five months.”

Throughout the five months of troubleshooting, Voyager's ground team continued to receive signals indicating the spacecraft was still alive. But until Saturday, they lacked insight into specific details about the status of Voyager 1.

“It’s pretty much just the way we left it,” Spilker said. “We're still in the initial phases of analyzing all of the channels and looking at their trends. Some of the temperatures went down a little bit with this period of time that's gone on, but we're pretty much seeing everything we had hoped for. And that's always good news.”

Through their investigation, Voyager's ground team discovered that a single chip responsible for storing a portion of the FDS memory had stopped working, probably due to either a cosmic ray hit or a failure of aging hardware. This affected some of the computer's software code.

“That took out a section of memory,” Spilker said. “What they have to do is relocate that code into a different portion of the memory, and then make sure that anything that uses those codes, those subroutines, know to go to the new location of memory, for access and to run it.”

Only about 3 percent of the FDS memory was corrupted by the bad chip, so engineers needed to transplant that code into another part of the memory bank. But no single location is large enough to hold the section of code in its entirety, NASA said.

So the Voyager team divided the code into sections for storage in different places in the FDS. This wasn't just a copy-and-paste job. Engineers needed to modify some of the code to make sure it will all work together. “Any references to the location of that code in other parts of the FDS memory needed to be updated as well,” NASA said in a statement.

Newer NASA missions have hardware and software simulators on the ground, where engineers can test new procedures to make sure they do no harm when they uplink commands to the real spacecraft. Due to its age, Voyager doesn't have any ground simulators, and much of the mission's original design documentation remains in paper form and hasn't been digitized.

“It was really eyes-only to look at the code,” Spilker said. “So we had to triple check. Everybody was looking through and making sure we had all of the links coming together.”

This was just the first step in restoring Voyager 1 to full functionality. “We were pretty sure it would work, but until it actually happened, we didn't know 100 percent for sure,” Spilker said.

“The reason we didn’t do everything in one step is that there was a very limited amount of memory we could find quickly, so we prioritized one data mode (the engineering data mode), and relocated only the code to restore that mode,” said Jeff Mellstrom, a JPL engineer who leads the Voyager 1 “tiger team” tasked with overcoming this problem.

“The next step, to relocate the remaining three actively used science data modes, is essentially the same,” Mellstrom said in a written response to Ars. “The main difference is the available memory constraint is now even tighter. We have ideas where we could relocate the code, but we haven’t yet fully assessed the options or made a decision. These are the first steps we will start this week.”