Roy Estess was one of the smartest and best that I have ever met. He spent most of his career testing large rocket engines at the NASA installation now known as the Stennis Space Center. In fact, he became the Center Director for SSC later in his career. Everybody who knew Roy came to appreciate his insight, his high level of integrity, and his managerial skill.
In 2001, Roy was appointed acting Center Director at Johnson Space Center where I really got to know him. On one business trip together, I heard him tell the following story, full of wisdom, which you should appreciate.
Roy said that as Stennis Center Director he got a call from the NASA Administrator about once a month. In between these calls, his pride would suffer. ‘Aren’t I a Center Director? Isn’t Stennis a NASA Center? Don’t we deserve more attention than one call a month?’ He asked. Then with a grin, Roy said, ‘Now, I’m at JSC and every day the NASA Administrator calls two, three, four times.’
“And I’ve decided that once a month was just fine.”
Working as a Flight Director on the overnight shift was at once a hardship and a delight. After all the senior managers made their last phone calls, generally before midnight, and before they all started showing up for their morning pre-briefings about 6:30, the Flight Director was master of all. Or so it seemed. From about midnight to 6 AM you could decide what the shuttle team would do without any oversight or interference. Except of course that Mr. Abbey always came in right about 2 AM. But he generally did not direct, he just wanted to say hello. Every morning. You could set your watch. But Mr. Abbey never stayed very long and never ever gave any direction – at least not to me.
Being the master of your own fate is a great thing, until you need advice or don’t know what to do. A very important fact that I learned early in my career: calling people at 3 AM is not a good way to get advice. I did that exactly once. Thereafter I decided that there were no troubling issues that could not wait until 6:30. And in the meantime I could decide to do anything – anything that would be over by 6:30.
For decades, folks at NASA have wished that they could get the attention of the President, like Webb had with JFK. Occasionally that happened, GHWB proposed a Mars mission in 1989. That did not work out well. Reagan finally approved a Space Station, but Clinton almost cancelled it. So on and so forth. Good and bad but nothing spectacular and not like JFK. Probably never happen again.
Now, NASA has the attention of the President, the Vice President, and a whole bunch of Very Important People.
Now to return to a subject I have left pending for too long: The STS-121 launch.
We had three major problems to solve to get approval for the launch.
I hate intermittent electrical problems. It doesn’t matter whether they are on my car, with my home sound system, or on the space shuttle, intermittent electrical problems are the worst. Even when you get the repairman to look at the system when it is not working, sometimes it is nearly impossible to find the problem. As my friends remind me, I’m a Mechanical Engineer by training, I don’t really understand electricity!
So it was with the shuttle in return to flight time. The big orange external tank does not have a gas gauge similar to what you find or a car or boat or airplane; it simply had a few ‘level’ sensors that tell when the tank is filled up to the top (for loading) and when it is empty. There were no measurements in between. Many other rockets are like this. Even telling when the gas left in the tank is at the ‘full’ or ‘empty’ point is not easy when dealing with liquid hydrogen and liquid oxygen because they are so very cold; it takes a special sensor to indicate ‘wet’ or ‘dry’.
Starting with some tanking tests for the first ‘return to flight’ – STS-114 – we started experiencing some problems with the ‘empty’ sensors, called Engine Cutoff (ECO) Sensors because they were there for a critical safety issue. If the fuel – hydrogen – tank ran dry while the engines were still running, the ‘fire’ in the engines would get very hot indeed due to the surplus of oxygen and likely the engine would suffer – a great euphemism – ‘an uncontained failure’. Not what you want.
So, we started troubleshooting: the first suspect was the electronic box (‘point sensor box’) in the orbiter that deciphered the electrical signals from the sensors. I was surprised to find out that this piece of equipment was Apollo heritage! The electronic schematic drawing was signed off in the 1960’s for the upper stage of the Saturn V. We put a team of experts lead by Ed Mango on the investigation. After weeks and many tests on various tanks and orbiter point sensor boxes, the conclusions exonerated this old gear. As the orbiter team members told me: think outside the box. They even had a T-shirt made with that phrase.
Next we investigated the little sensors themselves. A metal cube about an inch on a side; inside was a very fine wire that changed electrical properties depending on whether it was immersed in fluid or not. We found that the electrical connections inside this little sensor could have some issues. Ah ha! Multitudes of x-rays and resistance tests were suspicious but inconclusive. But that had to be it.
Many long hours were spent in meetings and reviews to develop ways to determine if a particular sensor was good or likely to fail. New techniques for manufacturing were proposed. During this time, I elected to make a site visit to the people that build those sensors: the Goodyear aircraft avionics plant in Vergennes, VT. It probably scared the dickens out of the factory technicians to have the Space Shuttle Program Manager come stand at their work bench and watch them make tiny crimp connections on the almost microscopic wires. But we were convinced that was the problem and we were on the road to fixing it. The ET assigned to STS-121 had the ‘best’ sensor boxes we could find.
That was the status as of the Flight Readiness Review in June of 2005. With some reluctance, the FRR board accepted our plans including the wacky logic tree for what to do if more than one sensor failed during the countdown. So, despite all our worry and work, or because of it, ECO sensors were not the reason that there was disagreement over signing off on the CoFR.
I wish I could tell you that was the end of the story, but it wasn’t. Not only did we scrub the first launch attempt for STS-121 because more than one ECO sensor circuit was giving erroneous reasons, but later we found out the real cause: It wasn’t the Point Sensor Box in the Orbiter; it wasn’t the sensors in the bottom of the External Tank. It was the pin connectors on the pass through where the wiring went from inside to outside of the hydrogen tank. Something we thought we had exonerated early on. We had jumped to an erroneous conclusion early in the troubleshooting and spent over a year working on the wrong problem. Somebody from a different program pointed out – much later than STS-121 – that the Delta program had a similar problem which was caused by pin connectors in the tank wall pass through and they had solved their problem by soldering the wires together. Which is what we did. Which solved the problem. After almost two years of work.
I wish I had a nickel for every time we misdiagnosed a problem during our days on the Space Shuttle.
After additional runs in the SMS and SPF and analysis of the data collected on those runs, FSW DR 121227 has been opened on the issue. The SMS AR will be closed in reference to the FSW DR. The description of the FSW DR reads as follows:
“In MM 104-105, 202, and 301-302 when an Item 22 (Load) is executed on the XXXXX YYYYY MNVR display, it is possible for 2 SQRT of negative number GPC errors to be generated. This will occur when the TIG entered is not consistent with the PEG 4 burn targets. The Orbital Altitude Time Task in the DIP computes the inverse of the mean orbital rate and the sine of the eccentric anomaly as a function of the square root of the semi-major axis. Given the inconsistency of the TIG with the targets and the fact that the guidance converged to a solution, the semi-major axis is computed as a negative number.”
So here is my problem: almost two decades later I completely understand EVERY BIT of this email!
How many of my memory cells are taken up with holding on to obsolete and totally useless information? Is this why I can go down the hall to another room in my house and, upon arriving there, wonder what it was I came to do?
I imagine that there are dozens, if not hundreds, of former flight controllers, astronauts, trainers, software programmers, and trajectory analysts who were nodding their heads reading that email and saying ‘Yes! I understand the problem!’
Some neuro-psychologist needs to explain to me how to clean out my memory cells of unwanted and useless information so that I don’t get confused looking for my car keys.
Oh well, its fun to remember the old days.
And my wish for you today is that your semi-major axis is never a negative number.
Two weeks ago, I participated in the NASA Remembrance Day and lessons learned activities which happen every year around February 1. I believe it is very important to remember Dick Scobee, Gus Grissom, and Rick Husband and their crews; to remind ourselves of their sacrifice and what we need to do to safeguard future space travelers. But this post is not about those brave crews or the lessons should remember.
What really blew me away was the realization that fifteen years have passed since the loss of Columbia and her crew. It seems like yesterday.
Isn’t that what all the old folks say; it seems like just yesterday.
That tragedy sparked changes in my professional life that I had neither foreseen nor desired. Reviewing the events in my life over the past fifteen years brings so much to mind: being selected to leadership roles in the Space Shuttle Program, working feverishly to fix the problems, change the culture, and get flying again, making sure that the program would operate safely – or rather, more safely – all that in just the first five years. Changing jobs inside NASA to build partnerships with other organizations, supporting the President’s commission to plan the future of human spaceflight, helping to develop the initial stages of NASA’s commercial crew program – that took the next three years. Then retirement from the government and the start of a new career providing support to commercial spaceflight on the industry side – where I am today. Katrina, Ike, Harvey. And in between it all seeing my son and daughter graduate from college; experiencing the wedding of my son and his beloved; the blessing of grandchildren into our lives; the passing of my mother, the passing of my wife’s parents, and the passing – oh how hard to write this – the passing of my daughter. The last fifteen years have been packed full of happiness and tears.
Just like the song: “sunrise, sunset; I don’t remember growing older, when did they?
Fifteen years earlier, at the end of February 1988, my class of flight directors was selected. Becoming a NASA Flight Director had been my top career goal for years. It proved to be the best job I ever had. There is no feeling like leading a team of highly trained and highly motivated folks to achieve great things in a difficult environment. It was tremendously exciting and equally frightening. There were tremendous possibilities and “always be aware that suddenly and unexpectedly we may find ourselves in a role where our performance has ultimate consequences.” Making tough calls about the weather and priorities and all the little things that go wrong; participating in Hubble and ISS assembly and Chandra and on and on. I got to work hand in hand with astronauts, senior NASA executives, scientists; meet numerous VIPs that were interested in our work – everybody from movie stars to US senators to members of foreign royalty. And through all that watching my children grow, coaching them through all kinds of activities, actively participating in local civic affairs, taking long family vacations. Its hard to see how I could have done so must; I must have been much younger.
Fifteen years earlier, an astounding 45 years ago, I was a college freshman just trying to figure out who I was – even though I knew who I wanted to become. In the following fifteen years I met and married my beautiful wife, graduated from college both undergrad and grad school. I got my dream job at NASA: Mission Control and STS-1 and Challenger, working with Sally Ride, John Young, and working for Gene Kranz. And we bought our first and second houses, and in between had babies. Claudette, Alicia, Allison. How could all of that be packed into just 15 years.
Fifteen years earlier – can it be 60 years ago? It was 1958 and I was three. I don’t remember – who remembers when they were three – but according to family legend I was totally captivated by space; sputnik in October 57, Vanguard (kaboom!), Explorer 1 at the end of January 1958. Then hanging breathlessly on the adventures of Mercury, Gemini, a hundred robot explorers, and the Moon landing. It seems I was destined to participate in the great adventure from my earliest years. Growing up in the 60’s was sometimes a surreal experience and it is filled with memories even of people I knew that never came back from southeast Asia and others we lost. ‘Has anybody here seen my old friend John, can you tell me where he’s gone?’ It was an unbelievable time.
Sunrise sunset, swiftly pass the years.
Fifteen years is simultaneously a long time and the merest instant. So much to be learned. So much to experience. So much more to come.
I just pray the next fifteen years will have more happiness and less tears.
I’ve spent all week at the Langley Research Center in Virginia; there is much good work going on here. We had no small number of discussion about ‘game changing technology’ without a good definition of what that phrase really means or how such a change takes place.
At the same time I am keenly aware of the surrounding area’s historical sites. Jamestown, Yorktown, and Williamsburg are just up the road. There are Civil War sites all around. But the one that jumped out at me may have some lessons for those of us in the space business to consider. Just a few minutes from the Langley gates is the Mariner’s Museum with relics from the battle between the USS Monitor and the CSS Virginia, formerly Merrimack. There is a lesson in game changing technology there.
In the winter of 1861 a technology revolution was brewing in the midst of the U.S. Civil War. Naval technology had progressed incrementally for a long time, centuries actually. Wooden ships with sails carrying broadsides of carronades. Recently steam engines had started appearing onboard oceangoing ships but these were auxiliary power, sails were still the prime mover. In the Crimean war around 1856 some innovative designs were partially tried out but the results were inconclusive. Tradition and the conventional wisdom of the day did not look to radical change.
The Confederacy knew they could never match the number and size of the Union fleet but they were desperate to break the naval blockade that was strangling their war effort. Something radical must be tried to overcome the Union advantage. A plan was developed to use a partially burned frigate hulk – the USS Merrimack- as the platform for something very radically different; a solely steam powered, iron clad floating battery that might just turn the tide of the war. This was the basis for the CSS Virginia; hull and steam engine from a conventional frigate with an upper deck unlike anything seen on the water; sloped and reinforced sides and top covered with iron sheeting.
They didn’t pay much attention to military secrecy; the entire activity was written up in the local newspapers. Maybe that was an early attempt at psych ops – fill the enemy with fear. Construction of the Virginia was widely reported and the news traveled north.
When the CSS Virginia became operational on March 8, 1862, she was nigh invincible. Steaming to the Union fleet she created total destruction leaving flaming and sinking wrecks in her wake as their cannon shells bounced harmlessly off. Only the outgoing tide and drawing sunset abated the Virginia’s destructive path.
The North was in a panic at the report. An invincible terror weapon had been unleashed on them and there was no defense.
Well, not exactly. At the earlier reports, the admirals turned to a truly wild man with crazy ideas: John Ericsson. He was the Elon Musk of his day; promising radical and unbelievable change. His design was unlike anything ever seen; a practically submerged main body of a ship with a rotating iron turret housing two cannons on top. With the southern warship well under construction, the admirals gave him 100 days to build this new vessel which was not much more than a concept. He did it in 118 days. (Note: 18% schedule overrun). It was ready just one day after the Virginia’s first foray.
You probably know the story; the two iron-clads slugged it out without doing any real damage to each other. Military historians call the action of March 9, 1862 a draw. Except, of course, that the blockade was not broken. Neither ship saw action again and within months both were destroyed.
But their encounter changed everything; in the blink of an eye naval design would never be the same. The age of wooden sailing warships was over. Every ship in every fleet in the world was instantly obsolete. “Monitor madness” ensued while nations around the world started building copies of Ericsson’s design.
Game changing technology.
So what is the lesson for us? Here are the first three that I can think of:
There are harbingers out there.
Only truly crazy revolutionaries make game changing inventions
When the time is right, change will happen overnight