The unclassified version of a new Government Accountability Office report, “NNSA and DOE Need to More Effectively Manage the Stockpile Life Extension Program,” sheds a lot of light on NNSA’s problems in getting the W76 life-extension program up and running, with a lengthy discussion of the problems in remaking Fogbank at Y-12 after a lengthy absence of production.
Although there are questions still unanswered, including details of the classified material known as Fogbank, the report goes far beyond previous information released by the NNSA and its contractors. The report talks about over-ambitious schedules that put production of national security goals at risk and the tens of millions of dollars wasted by poor planning.
Y-12 finally completed its first set of production parts for the W76 in August 2008, more than a year later than scheduled, and the NNSA announced last month that the first fully rehabbed W76 (Trident) warhead had been accepted by the Navy and returned to the arsenal.
The new GAO report discusses the problems with the B61 bomb project that, in part, led to the rushed inefficiencies with the W76 program that seemingly fell apart under the pressure and finally resulted in a “Code Blue” effort throughout the weapons complex to try to resolve the Fogbank issue.
The GAO also confirms, circumstantially at least, that the so-called Purification Facility at Y-12 is the production facility for Fogbank. In referencing the 2005 startup of the new facility, the report noted, “The Fogbank facility was the first new manufacturing facility to be built at Y-12 in 30 years.” That coincides, of courses, with the startup of the Purification Facility, the mission of which Y-12 has refused to discuss.
For those interested in the Y-12 aspects of the GAO report, here is a lengthy excerpt from the section about problems with the W76:
At the beginning of the W76 life extension program in 2000, NNSA identified key technical challenges that would potentially cause schedule delays or cost overruns. One of the highest risks was manufacturing Fogbank because it is difficult to manufacture. In addition, NNSA had lost knowledge of how to manufacture the material because it had kept few records of the process when the material was made in the 1980s and almost all staff with expertise on production had retired or left the agency. Finally, NNSA had to build a new facility at the Y-12 plant because the facilities that produced Fogbank ceased operation in the 1990s and had since been dismantled, except for a pilot plant used to produce small quantities of Fogbank for test purposes.
To address these concerns, NNSA developed a risk management strategy for Fogbank with three key components: (1) building a new Fogbank production facility early enough to allow time to re-learn the manufacturing process and resolve any problems before starting full production; (2) using the existing pilot plant to test the Fogbank manufacturing process while the new facility was under construction; and (3) developing an alternate material that was easier to produce than Fogbank. However, NNSA failed to effectively implement these three key components. As a result, it had little time to address unexpected technical challenges and no guaranteed source of funding to support risk mitigation activities.
After determining that 2 years was sufficient time to test and perfect the Fogbank manufacturing process, NNSA set March 2005 as the target date to begin operations of the new facility at the Y-12 plant and worked backward from that date to establish a design, build, and test schedule for the new facility, according to the official in charge of the project. Working from lessons learned from the W87 life extension program, NNSA strove to achieve an early operations start date to allow sufficient time to address any potential problems in manufacturing Fogbank. In 2000, we reported that production problems resulting from such factors as restarting an atrophied production complex and addressing safety and technician training issues led directly to slippage in the W87 life extension program schedule and contributed to increased costs.8 In addition, NNSA’s own lessons learned report on the W87 program identified the need to demonstrate processes early and often and stated that, with limited resources, assumptions such as “we did it before so we can do it again” are often wrong.
NNSA started the new facility’s operations about 1 year late because the schedule for building the facility was unrealistic, disagreements on the implementation of safety guidelines emerged, and the W76 program manager lacked authority to control the schedule. Focused on meeting an operations start date of March 2005, NNSA developed an aggressive construction and operation start schedule with no contingency for cost overruns or schedule delays. This schedule increased risk to meeting the program schedule because any delay would leave less than 2 years to conduct test production runs, which NNSA determined were necessary for perfecting the process. In addition, the Fogank facility was the first new manufacturing facility to be built at Y-12 in 30 years; therefore, a lack of recent experience with construction project management and implementing safety guidelines heightened the potential for problems. In fact, the contractor building the facility underestimated the time needed to complete preparations for start-up, including training and certifying staff to use the equipment and calibrating instruments.
In addition, NNSA and the contractor disagreed on the interpretation and implementation of safety guidelines. A lack of clarity about which guidelines would apply and the proper interpretation of the guidelines caused confusion over the course of the project. At a late stage, NNSA directed the contractor to apply more conservative nuclear facility safety requirements. As a result, the contractor needed additional time to address safety concerns by, for example, installing weather- and earthquake-proof equipment.
When these issues emerged, the W76 NNSA program manager did not have the authority to manage the construction of the project or resolve the dispute over safety guidelines even though a key risk mitigation strategy was the timely start of facility operations. Construction and start-up of the facility was managed by Y-12, which reported to the Y-12 Site Office, a separate organization not under the authority of the program manager. As soon as the March 2005 new facility start date was missed, the program manager raised concerns and elevated them to the Deputy Administrator for Defense Programs, the cognizant management organization at NNSA headquarters, but the issues remained unresolved. Ultimately, start-up of the new facility was postponed by approximately 1 year, leaving NNSA with half the time originally planned to re-learn the Fogbank production process.
NNSA planned to use the Y-12 pilot plant to gain a better understanding of Fogbank properties and to test the production process on a small scale while the new facility was under construction. The pilot facility could only produce a small amount of Fogbank for the W76 program because it had only a few machines. Although NNSA used the pilot plant from 2000 to 2003, it did not have funds to continue the effort because it shifted money from the W76 program to support higher priority programs at the time, such as the W87 and B61 life extension programs.
However, in 2004, anticipating delays in starting operations at the new facility and recognizing the importance of continuing work at the pilot plant, NNSA provided funding to pay for additional work at the pilot plant. By completing this work, NNSA learned that certain techniques significantly affected the quality of the end product and made adjustments to meet requirements. However, NNSA did not conduct as much work as originally planned and missed opportunities to learn more about the manufacturing
process before starting operations.
In 2000, NNSA considered replacing Fogbank with an alternate material that was less costly and easier to produce but abandoned the idea because NNSA was confident that it could produce Fogbank since it had done so before. In addition, LANL’s computer models and simulations were not sophisticated enough to provide conclusive evidence that the alternate material would function exactly the same as Fogbank. Still further, the Navy, the ultimate customer, had expressed a strong preference for Fogbank because of its proven nuclear test record. In response to the Navy’s preference and the lack of sufficient test data on the alternate material, NNSA did not pursue the development of an alternate material until 2007.
In March 2007, however, NNSA again considered producing an alternative material when it was unable to produce usable Fogbank and was facing the prospect of significant schedule delays. Computer models and simulations had improved since 2001, enabling greater confidence in the analysis of alternate materials. Thus, NNSA began a $23 million initiative to develop an alternate material. LANL officials told us that NNSA plans to certify the use of the alternative material by the end of 2009 for the W76 warhead and if NNSA faced additional Fogbank manufacturing problems during full-scale production, the alternate material could then be used instead of Fogbank. Had NNSA continued research and development of an alternate material during the program, it would have had more information on the viability of using the alternate material in the weapon before March 2007. This additional information also might have provided the Navy greater assurance that an alternate material performed as well as Fogbank.
A failure to implement the three components of NNSA’s risk management strategy for Fogbank led to a 1-year schedule delay and a $69 million cost overrun. This cost overrun included $22 million to resolve Fogbank production problems, $23 million to develop the alternate material, and $24 million to maintain Pantex’s production capabilities. Regarding Fogbank production problems, in March 2007, NNSA discovered that final batches of the material had problems. To address the problems and try to meet its September 2007 date for producing the first refurbished weapon, NNSA launched a major effort–“Code Blue”–that made the manufacture of Fogbank a priority for the design laboratories and production facilities. However, this effort failed, and, as a result,
NNSA delayed producing the first refurbished weapon from September 2007 to September 2008, and it began its efforts to develop an alternate material to Fogbank.
Finally, while Pantex was unable to begin assembling refurbished units in September 2007 as planned, it still spent $24 million in fiscal year 2008 to remain in “stand-by” mode, which includes maintaining the skills of the technicians who will assemble refurbished W76 weapons.
The 1-year delay led to logistical challenges for the Navy and an aggressive production schedule of refurbished W76 warheads to make up time. The Navy originally planned to start replacing old W76 warheads with refurbished ones on submarines in April 2008.
However, owing to W76 production delays, the Navy had to replace aging parts of W76 warheads in its current arsenal and has had to delay replacing old warheads with newly refurbished weapons until April 2009. Furthermore, to make up for initial schedule setbacks caused by Fogbank production problems, NNSA has increased the rate at which it plans to produce refurbished W76 weapons. NNSA will produce more weapons per year than originally planned, an annual increment that over time will enable it to still finish production at the originally planned end date. However, a higher rate of production requires more resources and leaves less room for error because any slowdown will have a greater impact on the larger number needed to be produced. NNSA production officials have indicated that they may not be able to meet this more compressed schedule if they do not receive extra resources or if they encounter any production problems, both considered realistic possibilities.