LPD-17 Reliability Issues Surface Again

“Need a tow?”
(click to view full)

Problems with USS New York [LPD 21] reportedly have the US Navy scrutinizing every ship built in the class’ 2 shipyards. Unfortunately, it’s just the latest installment in a long string of basic workmanship issues. In March 2005, “Cost Overruns, Budget Uncertainties Hurting USN and Contractors” noted:

“With the help of a $50 million grant from the state of Louisiana, Northrop Grumman has modernized production at Avondale, and the company is now projecting completion of future amphibious ships at a much faster pace than in the past. Nevertheless, scathing Navy inspector general reviews that detailed shoddy construction and basic workmanship problems at Avondale are cause for legitimate concern in areas that will not be fixed by modernization alone.”

While some teething problems are common for first ships of a new class, The new San Antonio Class stands out for their number and severity. All in a ship whose costs rose from about $700 million when the program was sold, to over $1.7 billion – then stayed at that drastically elevated level through subsequent vessels.

Fundamental Problems

LPD-17 cutaway
(click to view full)

Worse, LPD 17 failed to complete a series of sea trials in late March 2007, and could not be sea-tested during a 5-day inspection period because one of its two steering systems completely failed. Navy inspectors found major defects in 3 of 17 categories, and the ship required millions more in repairs. In August 2008, after 2 failed INSURV inspections and 2.5 years after the Navy had officially accepted LPD 17 from the contractor, the first San Antonio Class ship was deployed on an operational mission. Whereupon it sprung oil leaks, and had to dock in Bahrain.

According to San Antonio Express-News, the San Antonio’s problems began on the drawing board. Specifically, in a computer design program dubbed 3D CAD, which was touted for its ability to give three-dimensional views but was reportedly not up to the task of designing an entire ship. Annual attrition rates of 35% or more during construction were also unhelpful, but the bottom line remains the ship’s quality.

That workmanship has been an issue for some time, and leads to legitimate questions concerning the Navy command’s acceptance of the ship from the contractor.

The ship failed 2 successive Navy INSURV inspections, before it was passed and readied for deployment with the Iwo Jima strike group in August 2008. Virginia Pilot’s “New Navy ship San Antonio found to be rife with flaws” has further details regarding the March 2007 testing SNAFU. On June 30/07, the paper ran a follow-up article: “Navy ship $840 million over budget and still unfinished.” Key excerpt:

LPD-17 construction
(click to view full)

“The highly touted nerve center of the new, $1.8 billion amphibious ship San Antonio is fraught with computer hardware crashes that could cripple operations.

The ship lacks basic safety equipment, such as hand rails and reliable guns to battle close-in attacks.

In all, Navy inspectors found 30 major flaws aboard the San Antonio, according to an internal report obtained by The Virginian-Pilot… The report reflects some of the same problems disclosed by The Pilot in July 2005. Two years later, the San Antonio is still incomplete and $840 million over budget [DID note: for a total of $1.7 billion – the same amount appropriated by the HASC in the FY 2008 supplemental for one more LPD-17 class ship]… Secretary of the Navy Donald Winter criticized shipbuilder Northrop Grumman Ship Systems for substandard work and, in a letter last week, questioned the future of amphibious and destroyer ship programs under contract with the company.

“By taking delivery of incomplete ships with serious quality problems, the Fleet has suffered unacceptable delays in obtaining deployable assets,” Winter wrote to Ronald Sugar, Northrop Grumman’s chief executive officer.

Two years after accepting the San Antonio, “the Navy still does not have a mission capable LPD ship,” Winter wrote.”

The article goes on to add:

“…In March 2006, chief of naval operations Adm. Mike Mullen also attacked Northrop Grumman over its work quality. The average cost per ship has risen 50 percent over original estimates, according to the Navy… The worst problems were in the propulsion, auxiliary and aviation systems. Nearly two-thirds of those serious problems were discovered during an earlier inspection, reported as fixed, but still existed during the later check.

The second ship in the amphibious class, the New Orleans, has fewer problems but was still incomplete when accepted by the Navy, Winter wrote to Northrop Grumman. The company’s “inefficiency and mismanagement of LPD 17 put the Navy in an untenable position,” according to Winter.

He has assigned a deputy to perform quarterly reviews on the shipyard and all ships under contract with Northrop Grumman.”

Subsequent ships of the San Antonio Class have passed inspections, though USS New Orleans [LPD 18] has noted significant issues of its own, and also took more 2 years after its March 2007 commissioning date to deploy on a mission. USS Mesa Verde [LPD 19], which was built in a different shipyard at Pascagoula, MS, sailed through its initial inspections with flying colors. Avondale’s USS New York [LPD 21] also sailed through its initial inspections, until serious quality problems led to a US Navy reassessment of all ships built at its Avondale, LA shipyard. Its fellow Avondale ship USS Green Bay [LPD 20] was then found to have its own set of problems, some of which were new.

Top date, all Avondale-built ships have had serious problems: USS San Antonio, USS New Orleans, USS Green Bay, and USS New York. In July 2010, Northrop Grumman announced that it would be closing its Avondale shipyard, and transferring all LPD-17 work to Pascagoula, MS by 2013. This will raise the costs of LPD 23 Anchorage and LPD 25 Somerset by about $210 million total, and the firm expects to absorb those costs.

A Failed First Deployment

LPD 17, Dockside USA

USS San Antonio continued its long streak of poor performance during its first operational deployment to the Persian Gulf in 2008. On Oct 9th and 17th, leaks were discovered in the pipes that deliver lubricating oil to the ship’s 4 diesel engines. The fault is classified as hazardous, because the leaks drip flammable oil into open spaces. On Oct 31/08, therefore, the ship was forced into to a Bahraini shipyard for at least 2 weeks of repairs. When the ship pulled in, it was greeted by a crew of 30-40 engineers, pipefitters and welders flown to Bahrain from the U.S.

Engineers had to conduct a root-cause analysis and the repair and estimating/ working on fixes, some of which require replacing whole sections of pipe. Something one might expect on a very old ship – but not on a new one. Even the Navy’s INSURV inspectors may not catch those kinds of very basic workmanship issues, unless they’re visually obvious.

Captain Jan van Tol (ret.) has commanded US Navy ships whose cruises were interrupted by major breakdowns, but said that this crew’s size was notable. Gannett’s Navy Times quotes him:

“It surprises me to see oil leaking from such major points. I associate leaks with moving parts… What’s unusual is the sheer number of people who are going out to address what appears to be a wider-ranging problem… Are these systemic problems in one or more of the ship’s systems and physical plant? If they are, that goes to the question of craftsmanship and why did the Navy accept the ship? Are there ship-wide problems of a similar nature of poor craftsmanship and quality assurance? Who made the decisions to allow it to reach this point?”

Former 3-star vice-admiral Rep. Joe Sestak [D-PA] looked at the verified but unofficial photos [pdf] of the problem, and seems to be thinking along similar lines:

“I expected to be handed machines of war that had a certain level of readiness I then had to maintain… Something could break. But I never expected to deploy with a machine of war, particularly a relatively new one, that had systemic problems that would take weeks at a time [to fix]… When it’s something that appears systemic to the construction… we’re giving short shrift to our warriors out there… I’d like to go back to ‘What are the institutional processes that permitted this to happen?’… and find out how this can be done better. I have proposed that we should have hearings on acquisition reform in the new session, with LPD 17 part of that.”

During a speech at the CSIS think-tank, Navy Secretary Donald Winter noted that he remained unsatisfied with the USS San Antonio’s performance, and promised “an appropriate course of action ahead” without mentioning any specifics. Given the Navy’s acceptance of the ship, however, his options are limited.

HMLMS Rotterdam LPD

Winter did make an invidious comparison to the quality he has seen at Hyundai Heavy Industries in South Korea, however, which builds over 70 civilian ships per year as well as South Korea’s destroyers and amphibious assault ships. That competitive status was contrasted with American shipbuilding’s monopsony, in which the government is often the only major buyer and market forces are not really at work.

Which may indeed be part of the larger problem. Meanwhile, the US Navy’s immediate problem involves resolving the quality issues with almost $4 billion worth of amphibious ships, which form a significant fraction of its future amphibious assault capability. Even as other shipbuilders around the world seem quite able to build capable, modern LPD ships at a lower price per ton.

Updates and Key Events

Will they find it?
(click to view full)

May 6/11: MSMO Mayhem. NAVSEA announces that it has terminated Earl Industries, LLC’s multi-ship, multi-option (MSMO) maintenance contract for the San Antonio Class. The move comes in response to:

“…Navy findings of improper work performed and concern regarding Earl Industries’ quality assurance program and the company’s ability to control the quality and documentation of work it performs. Those concerns were triggered by the number and severity of corrective action reports issued… “The company’s performance on this contract was not in keeping with the type of quality work the Navy expects from our industry partners,” said NAVSEA Commander Vice Adm. Kevin McCoy. “These failures are unacceptable, and we have lost confidence in Earl’s ability to continue successfully performing this same type of work… under the MSMO contract.”

It’s the most severe option – a complete termination of all work in process by the Norfolk, VA contractor, as well as all options for future scheduled and unscheduled maintenance work on the class over a 5-year period. In place of Earl’s contract, the Navy plans to compete scheduled Chief of Naval Operations availability, and all necessary Emergent Maintenance/ Continuous Maintenance work for the San Antonio-class ships homeported in Norfolk, among all eligible contractors in the Norfolk area.

The Virginia Pilot’s “Earl Industries’ $75M Navy contract: What went wrong?” has a pertinent examination, which notes that Earl won the contract, despite having a higher bid, on the basis of Navy evaluations of “exceptional” performance on past contracts. The firm retains maintenance contracts involving the USN’s carriers.

April 20/11: Command oversight suspended. The US Navy announces that it has suspended the oversight authority of its Norfolk Ship Support Activity, at Norfolk Naval Station, VA, which is responsible for supervising maintenance work done by private companies on Navy surface ships in the mid-Atlantic region. Investigations are also underway concerning specific repairs to the USS San Antonio [LPD-17].

By suspending the command’s oversight authority – formally known as its “technical warrant” – the Navy essentially said it no longer trusts it to make sure work by contractors is being done properly. The issue is reportedly that the government can’t tell, based on required reports, what work was done and what wasn’t.

Thomas J. Murphy, who had been the command’s civilian executive director since 2004, was replaced in March 2011, and sources outside the Navy said several other officials at the command were also removed. Virginian Pilot | Information Dissemination | UPI.

Oct 29/10: A Lack of Leadership. Based on the Bloomberg report, the naval blog Information Dissemination looks at the DOT&E reports from 2006-2009, and matches them with command histories. The results are enlightening, and the op-ed point following those report excerpts is apt:

“There are clearly issues here that raise serious questions of specific industry companies as to why they have been unable to meet requirements. There are also serious questions for the Navy though, starting with why the recommendations made by DOT&E have gone ignored for several years in a row through at least December of 2009… LPD-17 class features networks with single points of failure that appear to be perpetually unreliable, new weapon systems that don’t meet requirements, and unreliable communication and information exchange equipment – all of which piles on top of the incredible number of HM&E problems identified as a result of poor construction and shipyard practices that have had most the class sidelined.

…Admiral Harvey took over Fleet Forces Command in July of 2009, and if you look over the CRS report by Ronald O’Rourke (PDF) that lists the history of construction problems from pages 17-45 (28 pages!), 10 of those pages disclose problems identified and reported over the 15 month time period since ADM Harvey took over responsibility at Fleet Forces Command… from June 2005 until July of 2009 – 49 months – very few of the major problems that are class-wide and often discussed today were apparently identified, or reported. Why did everyone have to wait for Admiral Harvey to assume command of Fleet Forces Command… Why was ADM Jonathan Greenert, who was in charge Fleet Forces Command from September 2007 to July 2009, unable to uncover any of these issues?

…As a reward for ADM Greenert’s apparent ignorance (or intentional concealment) regarding the depth of the LPD-17 class problems – he was promoted to the Vice Chief of Naval Operations. I would also think there are plenty of questions for VADM Kevin McCoy who was the Chief Engineer in NAVSEA from 2005-2008 until he became commander of NAVSEA in June of 2008 – because all of the problems with LPD-17 took place while VADM McCoy was part of the leadership in NAVSEA over the last 5 years.

Problems with the LPD-17 class are similar to problems seen in other classes of ships built and maintained over the last several years, and these are problems that leadership at the time did not address and have gone on to cost the Navy billions to resolve. Noteworthy, as a reward for their work (and the problems listed in the Balisle Report is basically the resume of failure at Fleet Forces Command under ADM Greenert btw), the current CNO promoted these folks and the Senate approved those promotions… Screw up as a leader at sea – You’re Fired! Cost the country billions while leading ashore – You’re Promoted! That is my definition of a leadership culture that selectively applies accountability.”

Oct 28/10: Survivability, quality questioned by Pentagon. Bloomberg News reports on a classified report sent to Congress in June 2010, outlining Pentagon testing that found serious issues with the LPD-17 San Antonio Class’ ability to survive combat situations. Their report is based on an unclassified summary of that report, and an email response from Michael Gilmore, the Pentagon’s director of operational test and evaluation, who described the ships as “not effective, suitable and not survivable in a combat situation.” The core of those reports is that the ships continue to experience widespread, persistent engineering problems, and couldn’t continue to operate reliably after being hit by enemy fire, in part because of the engineering problems mentioned. From the Pentagon’s DOT&E FY 2009 Annual Report:

“Chronic reliability problems associated with critical ship systems across the spectrum of mission areas reduces overall ship suitability and jeopardizes mission accomplishment… Emerging results from [Navy] trials indicate the ships could not demonstrate the required levels of survivability, largely because of critical ship system failures after weapons effects.”

“…Reliability problems related to well deck ramps, ventilation, bridge crane, and Cargo Ammunition Magazine (CAM) elevators… [and] Engineering Control System (ECS), including frequent failures and high false alarm rates, and the electrical distribution system, including unexplained loss of service generators and the uncommanded opening of breakers… The Navy’s Board of Inspection and Survey (INSURV) identified similar deficiencies in identical areas (propulsion, auxiliaries, electrical, damage control, deck) during both acceptance and final contract trials across all four of the first ships of the class. Catastrophic casualties recorded prior to the Full Ship Shock Trial in LPD-19 and during LPD-17’s deployment revealed serious fabrication and production deficiencies in the main lube oil service system. The ship is capable of supporting [C4I] requirements in an ESG environment; however, reliability problems with the SWAN(Shipborne Wide Area Network) and the Interior Voice Communications System degrade command and control and are single points of failure during operations.

The LPD-17 exhibited difficulty defending itself against several widely proliferated threats, primarily due to… Persistent SSDS Mk 2-based system engineering deficiencies… The ship’s RAM system provided the only hard kill capability, preventing layered air defense [DID: in fairness, the ships were designed this way]… Problems associated with SPS-48E and SPQ-9B radar performance against certain Anti-Ship Cruise Missile attack profiles [DID: also a known design limitation]… Degraded situational awareness due to Mk 46 Gun Weapon System [25mm remotely-operated] console configuration… The survivability of the San Antonio class ships appear to be improved over the LPD class ships they will replace. However, problems encountered with critical systems during testing (particularly with the electrical distribution, chilled water, SWAN, and ECS) and difficulty recovering mission capability may offset some of the survivability improvements and have highlighted serious reliability shortcomings.”

Oct 15/10: LPD 17 can’t go. U.S. Fleet Force Command (USFF) Commander Adm. John C. Harvey Jr. announces that USS Mesa Verde [LPD 19] will replace USS San Antonio [LPD 17] in the USS Bataan’s [LHD 5] Amphibious Ready Group (ARG) in the summer of 2011. Mesa Verde, which was built in Mississippi instead of the San Antonio Class’ primary yard at Avondale near New Orleans, returned from a 7-month deployment to the Persian Gulf in August 2010, and wasn’t expected to deploy again until late 2012.

San Antonio is currently scheduled to conduct comprehensive crew certification and sea trials in early spring 2010, but Adm. Harvey would only say that: “San Antonio will deploy when it is operationally sound and ready to go.” The ship’s overhaul at Norfolk was expected to take about 4-5 months and cost $5 million, but bolts in the foundations of the diesel engines and the main reduction gears were improperly installed at the shipyard. That created vibrations in the drive train that could have completely destroyed the propulsion system over time, and repairs are now expected to take about 11 months and at least $39 million, possibly more. USFF | Defense News.

LPD 20, trials
(click to view full)

July 29/10: LPD 20 problems. Gannett’s Navy Times reports on testimony before the House Armed Service Committee’s readiness panel, indicating unique problems with USS Green Bay’s [LPD 20] steering system. That’s in addition to other problems generic to the class involving metal shavings polluting the lube oil systems and damaging the engines:

“Vice Adm. Kevin McCoy, head of Naval Sea Systems Command… detailed the work that engineers have been doing to Green Bay: After workers discovered “significant foreign material” in the ship’s steering system, they had to make repairs to make sure the fluids in the system stayed clean. Then they had to cut open Green Bay’s deck plates so they could remove and replace its rudder rams, vital internal components that help the ship steer. Technicians also had to do the modifications the Navy is making to all its San Antonio-class gators… Although the litany of problems aboard the San Antonio class gators has become familiar – flawed design, poor workmanship, network, weld and lube oil problems – McCoy’s mention of Green Bay’s steering system troubles was first time that has been made public. It wasn’t immediately clear if the Navy would look for steering system problems on the other ships of the class.”

USS Green Bay was also built at the Avondale shipyard near New Orleans, which Northrop Grumman is moving to close. She apparently finished her final contract trials last week following months of work. The question is whether it will be able to support the deployment of the 13th Marine Expeditionary Unit as scheduled, or if another ship would have to take its place the way the Austin Class USS Ponce [LPD 15] did in 2008 for the first-of-class USS San Antonio. A panel question from Rep. Rob Wittman [R-VA] dd not get a clear answer.

July 13/10: Closing Avondale. Northrop Grumman Corporation announces plans to consolidate its Gulf Coast shipbuilding operations in Pascagoula, MS, and try to sell its entire shipbuilding business. Its Avondale, LA shipyard will close by 2013, transferring all LPD-related work. With the hysteria surrounding Hurricane Katrina and its aftermath a thing of the past, and a new emphasis on financial performance in the firm’s boardroom, these moves become politically possible at both the corporate and national levels.

“The consolidation of Gulf Coast ship construction is the next step in the company’s efforts to improve performance and efficiency at its Gulf Coast shipyards… Since [early 2008] Gulf Coast organization and leadership, operating systems, program execution, risk management, engineering, and quality have been the focus of intense improvement efforts. Consolidating new ship construction on the Gulf Coast in one shipyard will position Shipbuilding to achieve additional performance improvement and efficiency over the long term. Ship construction at Avondale will wind down in 2013. Future LPD-class ships will be built in a single production line at the company’s Pascagoula, Miss. facility. The company anticipates some opportunities in Pascagoula for Avondale shipbuilders who wish to relocate.

…the company expects higher costs to complete ships currently under construction in Avondale due to anticipated reductions in productivity and, as a result, is increasing the estimates to complete LPDs 23 and 25 by approximately $210 million. Of this amount $113 million will be recognized as a one-time, pre-tax cumulative charge to Shipbuilding’s second quarter 2010 operating income. The balance will be recognized as lower margin in future periods, principally on the LPD 25. The company also anticipates that it will incur substantial restructuring and facilities shutdown-related costs including, but not limited to, severance, relocation expense, and asset write-downs. These costs are expected to be allowable expenses under government accounting standards and recoverable in future years under the company’s contracts. The company estimates that these restructuring costs will be more than offset by future savings expected to be generated by the consolidation.”

June 30/10: More Avoidable Avondale issues. Gannett’s Navy Times offers excerpts from a US Navy report, which indicated continued problems with basic workmanship aboard the Navy’s billion-dollar San Antonio Class ships:

“Inadequate government oversight during the construction process failed to prevent or identify as a problem the lack of cleanliness and quality assurance that resulted in contamination of closed systems,” said the Navy report, [dated May 20th but] released Thursday. “Material challenges with this ship and other ships of the class continue to negatively impact fleet operations. Failures in the acquisition process, maintenance, training and execution of shipboard programs all share in the responsibility for these engineering casualties… [With its automated systems] not functioning as designed, the ship was unable to effectively operate and maintain the engineering plant.”

The problems reported in January 2010 were traced to contaminated lube oil systems that were damaging their main engines, and USS San Antonio [LPD-17] and USS New York [LPD 21] remain affected, with San Antonio expected to be in dry dock until late 2010 as engineers attempt to repair a bent crankshaft.

Jan 25/10: Propulsion problems. Defense News:

“Northrop Grumman Shipbuilding, the U.S. Navy’s Office of Research, Development & Acquisition and the Naval Sea Systems Command (NAVSEA) each provided statements concerning problems with the main propulsion diesel plant on San Antonio LPD-17-class amphibious ships and on the ability of NAVSEA’s Supervisor of Shipbuilding (SUPSHIP) to provide proper oversight of construction work on the ship. The statements were too lengthy to include in the story, “We Thought We Had It Licked,” in the Jan. 25 issue of Defense News, but are published here in full…”

Jan 22/10: Following the problems with USS New York, Gannett’s Navy Times reports that:

“Inspectors are rechecking every pipe weld aboard every ship built in the last several years at Avondale, La., or Pascagoula, Miss., including destroyers and small- and big-deck amphibs, after discovering so many problems that all pipe welders and Navy inspectors at both yards had to be decertified and then recertified to work on ships… The disbarring and reapplication took place last summer, when some of the problems were first discovered… A major question was how or why NavSea’s inspectors approved work that subsequent Navy inspections later found inadequate… Inspectors are looking at the entire San Antonio class of amphibious transport docks to determine what has caused systemic lube-oil problems in multiple ships, as well as damage to engine bearings that recently sidelined the newest ship, New York.”

Most LPD-17 class ships have been built at Avondale, near New Orleans, LA – a shipyard that has has demonstrated extensive workmanship problems throughout the program. USS Mesa Verde [LPD 19], which was built at Ingalls shipyard in Pascagoula, is currently at sea, inspected, and will continue its mission to Haiti and the Middle East. USS New York [LPD 21] is dealing with lube oil and engine problems, and a bowed crankshaft that will need to be replaced in an unprecedented procedure. Northrop Grumman will pay for work on USS New York, which is still under warranty. Any problems found in other ships will be subject to negotiation.

Jan 8/10: Propulsion problems. The US Navy announces that a week long, at-sea examination following USS New York’s commissioning has discovered the “premature failure” of bearings associated with the ship’s Colt-Pielstick main propulsion diesel engines. After the damage was found, the ship returned to Naval Station Norfolk under its own power.

The USS New York was built in Northrop Grumman’s Avondale shipyard in Louisiana near New Orleans, as opposed to the Ingalls shipyard in Mississippi. The failed components are under warranty, and will be repaired. It’s currently unclear how long the repairs will take, however, how serious the failures are, or whether the problems affect other ships in the San Antonio class. Virginia-Pilot | Hampton Roads WTKR.

Oct-Nov 2008: LPD 17 breaks down. On Oct 9th and 17th, leaks are discovered in the pipes that deliver lubricating oil to USS San Antonio’s 4 diesel engines. The fault is classified as hazardous, because the leaks drip flammable oil into open spaces. On Oct 31/08, therefore, the ship is forced into to a Bahraini shipyard for at least 2 weeks of repairs. Photos [PDF] | Gannett’s Navy Times | San Antonio Express-News | Virginia Pilot. Gannett’s Navy Times re: SecNav Winter’s CSIS comments | Mississippi Press re: SecNav Winter’s comments.