RS Classic: Single Point Failures

Written in 2008.

On NASAWatch, they noted loss of a single component brought down the entire NASA email system (this just after they were down for nearly a week because the server goes through JSC only, so the hurricane [Ike] affected all NASA mail). There are few things, technologically, more irksome, more certain to fail, and more embarrassing when they do so, than single point failures.

Failure tolerance is a good thing. It’s the best defense there is against Murphy’s law. At one time, NASA strived to meet the principle of Fail Operational, Fail Safe. That meant that any single failure did not affect function (or at least not critical function). A second failure might leave the hardware nonfunctional, but it would leave it safe - if the function is critical, of course, nonfunctional is not safe. Even more, it ensured that there were at least three failures required before reaching an unsafe condition.

Now, of course, that isn’t always possible. A nuclear reactor containment vessel is unlikely to be redundant. Extra wings on a plane in case one shears off is likely less than useful. Sometimes adding redundancy and additions paths adds so much complexity, weight, etc. that the design becomes ridiculously unwieldy.

But, in general, it’s a good thing. If 12 bolts can bear all the stress necessary, adding one or two other bolts allows for failure without compromising the overall capability. (This is especially important in space endeavors where a stripped or broken bolt may not be recoverable). If one can accommodate redundancy, especially with unreliable equipment, it’s smart engineering practice to do so. This is more true when hazards can result from failure.

So, if a complex or key system fails because of a single component or a readily foreseeable circumstance, that argues poor engineering. Bad in real life…

But it’s damn useful in science fiction. You want to make an interesting story, twist the plot, add hardship, shake things up a bit? Have a critical component fail. You want to add tension, pressure, up the excitement? Make your critical item one of a kind, no spares, hard to come by, made of something scarce and/or requiring interaction with someone hostile to recover it? Do so, and suddenly things are hot and exciting.

Just remember not to use this little trick too often because, if you do, your engineers will look like total morons or it will become cliché. Not that it isn’t a little cliché already thanks to Star Trek and the like. But it’s still good for excitement if you can exercise a little restraint.

Having said that, I'm not a personal fan of changing the two fault tolerance (fail operational/fail safe) philosophy to a more amorphous fault tolerance highly dependent on probabilistic risk assessment (PRA(and design to minimum risk (which can not be applied to all hardware). NPR 8705.2B, the standard for human rating required by NASA, used to dictate 2 fault tolerance for catastrophic failures. By requiring that, vendors and designers were required to provide redundancy or justify each instance where it was not provided [paragraph 3.1.1 in Rev A, 3.2.2 in Rev B]. In the version released in May 2008, this was undone. Rev B is the current version.

Old version: "Space systems shall be designed so that no two failures result in crew or passenger fatality or permanent disability." [Note that there were caveats in the next requirement for exceptions, but the exceptions had to be brought to management]

New version: "The space system shall provide failure tolerance to catastrophic events (minimum of one failure tolerant), with the specific level of failure tolerance (one, two or more) and implementation (similar or dissimilar redundancy) derived from an integrated design and safety analysis (per the requirement in paragraph 2.3.7.1)(Requirement). Failure of primary structure, structural failure of pressure vessel walls, and failure of pressurized lines are excepted from the failure tolerance requirement provided the potentially catastrophic failures are controlled through a defined process in which approved standards and margins are implemented that account for the absence of failure tolerance. Other potentially catastrophic hazards that cannot be controlled using failure tolerance are excepted from the failure tolerance requirements with concurrence from the Technical Authorities provided the hazards are controlled through a defined process in which approved standards and margins are implemented that account for the absence of failure tolerance. "

I'm not objecting because PRA isn't usefuluseful (it is) but because I've seen a great deal of misuse of good models with bad assumptions or limited understanding of the uncertainties and caveats. Not necessarily by the analysts but by those feeding them data and those wanting "an answer" without taking the trouble to understand what that answer really means. And, I have to say, changing a straight-forward requirement into a morass of legalese, well it just gives this old safety engineer the willies. Requirements should be clear and verifiable.

Read more...

There's Always a Price

While we're celebrating the triumph of the moon landing forty years ago, I think it's only fair to remember those that have fallen in the pursuit of human space exploration. These are not just Americans because, to be frank, I think human exploration is more than one country, but for all mankind, just as Neil Armstrong said.

When we've trod on the moon, there were tragedies that paved the way for our success. When we built the ISS, there were losses that taught us (or tried to teach us) important lessons. If we forget those that have been lost to our lack of foresight and the lessons we have had at the cost of their lives, their lives were lost in vain.

• 1930 Max Valier killed by rocket engine explosion
• 1933 Explosion in rocket manufacturing room of Reinhold Tiling (Tiling, his assistant Angela Buddenboehmer and his mechanic Friedrich Kuhr were killed)
• 1934 Ground test engine explosion (3 lost)
• 1944 An A4-rocket crashes at a test launch in a trench. Several soldiers who were in the trench were killed (7 lost)
• 1960 The Nedelin catastrophe; Explosion of R-16 ICBM on launch pad (126 lost)
• 1961 Valentin Bodarenko died in a chamber fire in a 100% oxygen training run from excessive burns. (Information about this death was not released until 1986 and might have helped prevent Apollo 1).
• 1964 Delta rocket ignited in assembly room, killing 3 technicians and injuring 9 others. The ignition was caused by a spark of static electricity
• 1964 Theodore Freeman was killed when a goose smashed through the cockpit canopy of his T-38 jet trainer. He ejected but was too close to land, which led to zero-zero ejection system currently in use with many aircraft.
• 1964 Mail rocket built by Gerhard Zucker exploded and debris hit crowd of spectators (3 lost)
• 1964 Oscar Simmons falls from the 46th floor of the Vehicle Assembly Building during construction
• 1965 Lightning killed Albert J. Treib on launch pad B of Launch Complex 39
• 1966 The original Gemini 9 crew, Elliot See and Charles Bassett, were killed while attempting to land their T-38 in bad weather.
• 1967 A fire claimed the lives of the Apollo 1 crew (Gus Grissom, Edward White II and Roger Chaffee) as they trained in their capsule. High pressure, 100% O2 and offgassing materials may have contributed as well as a multitude of electrical issues.
• 1967 Soviet cosmonaut Vladimir Komarov died on board Soyuz 1 when his parachute malfunctioned during reentry.
• 1967 Clifton 'C.C.' Williams (backup Apollo 9 and likely Apollo 12 crewmember) died after a mechanical failure caused the controls of his T-38 to stop responding.
• 1967 Michael J. Adams died while piloting a suborbital spaceflight in a rocket plane (X-15).
• 1967 Robert Henry Lawrence, Jr. (first African-American astronaut for the U.S. Air Force Manned Orbiting Laboratory program) died when his F-104 Starfighter jet crashed at Edwards Air Force Base, California. He never made it into space.
• 1968 First human in space Yuri Gagarin died when his MiG-15 jet trainer crashed while he prepared for the Soyuz 3 mission.
• 1971 The crew of Soyuz 11, Vladislav Volkov, Georgi Dobrovolski and Viktor Patsayev, were killed after undocking from space station Salyut 1 after a three-week stay. A valve opened during separation and exposed the crew to vacuum during reentry.
• 1973 Launch explosion of Cosmos-3M rocket (9 lost)
• 1980 Explosion while fueling up a Vostok rocket booster (48 lost)
• 1981 Anoxia during preparations for STS-1 (2 technicians suffocated during nitrogen purge during ground ops)
• 1986 The Space Shuttle Challenger was destroyed 73 seconds after lift-off on STS-51-L when SRB blow-by impinged on External tank. Greg Jarvis, Christa McAuliffe, Ronald McNair, Ellison Onizuka, Judith Resnik, Michael J. Smith, and Dick Scobee were killed.
• 1995 Long March rocket veered off course after launch (At least 6 lost)
• 1995 Anoxia; Luc Celle and Jean-Claude Dhainaut died during an inspection in the umbilical mast of the launchpad
• 1996 Long March rocket veered off course 2 seconds after launch, crashing in the nearby village (56-200 dead)
• 2001 Crane operator Bill Brooks was killed in an industrial accident at Launch Complex 37
• 2002 A Soyuz-U exploded 29 seconds after launch, killing a soldier, Ivan Marchenko, and injuring 8 others.
• 2003 The Space Shuttle Columbia was lost as it reentered after a two-week mission, STS-107 due to damage to the thermal protection system. Rick D. Husband, William McCool, Michael P. Anderson, David M. Brown, Kalpana Chawla, Laurel B. Clark, and Ilan Ramon were killed.
• 2003 A search and rescue helicopter involved in searching for debris following the disintegration of the Space Shuttle Columbia crashed, killing Jules F. Mier Jr. and Charles Krenek, and injuring three others.
• 2003 Explosion of an unmanned rocket during launch preparations (21 lost)
• 2007 Explosion during a test of rocket systems by Scaled Composites during a nitrous oxide injector test (3 lost)

(Although I first found these in Wikipedia, I have cross checked almost all of these elsewhere).

All honor to them.

Read more...