Rebuilding the National Weather Service

 Weather prediction is a critical national activity.

As noted in various publications, trillions of dollars of U.S. economic activity are sensitive to weather (Dutton 2002), with nearly a half-trillion dollars of variability in our economy due to weather (Lazo et al., 2011).  Furthermore, the U.S. experiences a wide range of severe weather conditions (tornadoes, hurricanes, severe convection, windstorms, snowstorms) that not only results in economic impacts, but the loss of hundreds to thousands of lives a year from intense weather.

The U.S. should have the best weather prediction capability in the world.  We have the world's most extensive weather research community and many key weather prediction breakthroughs have occurred here, including the first numerical weather prediction.  We also spend more money on weather prediction than anyone else and our private sector meteorological community is large and vigorous.

But the sad truth is that we are lagging behind the world leaders, and are even farther behind our inherent capabilities.  The loss to the nation, both in lives and economic value, is immense and unnecessary.   It is time to face up to the problems and fix them.

This year a National Academy of Sciences committee, asked to evaluate the National Weather Service (NWS), completed a report: Weather Service for the Nation:  Becoming Second to None.  The implication of this report is clear:  the National Weather Service is no longer the world leader in weather prediction.

During Hurricane Sandy, much of the media discussed how the best forecasting system for predicting Sandy was not American, rather it was the model of the European Center for Medium Range Weather Forecasting (ECMWF) in Reading, England.  Even USAToday (see graphic) and the NBC Nightly News talked about U.S. inferiority.  When the popular media are talking about U.S. weather prediction inferiority, you know you are in trouble.

But the problems are worse than inferior global weather prediction, which are confirmed by the statistics on the NWS's own web site.  The U.S. is falling behind in the next generation forecasting technology based on ensembles of many forecasts:  probabilistic prediction.  Furthermore, the head of the NWS was recently forced to retire when financial irregularities in the NWS budget were found, and mismanagement of the acquisition of a new generation of polar orbiting satellites threatens to produce a gap in critical satellite coverage.  And there is a lot more that I will not mention now.

It is not that the National Weather Service isn't making a huge contribution today--it is.  NWS forecasts during the days before hurricane Sandy were extraordinarily useful, saving property and many lives.  Few government agencies provide such a valuable, useful product.  And the NWS is staffed with dedicated, talented, and well-educated forecasters, researchers, and other employees.  But it is undeniable that this organization is a shadow of what it could be:  the dominant weather prediction entity of the planet backed by the largest national research enterprise and the largest private sector weather sector in the world. 

The impact of not being the best is costing the U.S. billions, if not tens of billions of dollars per year, and many lives.

In this blog, I will describe some of the problems and how they might be addressed.  This is going to be a very frank evaluation, perhaps too frank. And let's be clear up front:  the issues are both long-standing and aren't limited to the National Weather Service, they extend into NOAA management and higher.  And without fundamental change, the nation will continue to be served by a critical organization not realizing its potential.

Let me now discuss a few of the problems I see, starting with the most easily addressed first.

Problems 1:  the lack of computer resources.  

The National Weather Service efforts in global and national prediction are being undermined by lack of computer power.  This is not just my opinion--ask leaders inside the weather service, National Academy committees, or knowledgeable folks in the field and they will tell you the same thing.  Why are ECMWF's forecasts so much better than the NWS's?  A big part of it is that ECMWF has the computer power to run their high-resolution global forecast at 16 km resolution, while ours is 25 km, their ensemble forecasting system has twice the resolution, and they can use an advanced data assimilation approach (4DVAR) that requires a lot of computer resources.

Consider the difference in the computer resources available to ECMWF versus the National Weather Service's Environmental Modeling Center (EMC), and keep in mind that EMC has a much wider range of responsibility (includes high resolution national and regional forecasting) than ECMWF (which does only global forecasting).

ECMWF has two machines, each with 24546 cores and a computational ability of .75 petaflops (thousand trillion floating point operations per second).   These machines are #37 and 38 on the worldwide list of top 500 computers.  The National Weather Service has two computers that are not even on the top 500 list.   Each has 4992 processors and an ability to do .07 petaflops.   You see the problem?  The NWS has less than 10% of the computer power as ECMWF and has many more responsibilities.

Ironically, NOAA (of which the NWS is part) and other government agencies have some very big machines (NOAA's Fairmont, .38 petaflop, DOE's Sequoia and Titan-20, 27 petaflops), which are NOT being used for weather prediction but climate prediction and other needs.   Kind of ironic we have huge computer power available for climate prediction, but not for predicting the weather today that critically influences U.S. citizens.  A year ago, at the dedication of the new coastal radar, I was having lunch with the head of the NWS and Senator Maria Cantwell.   The NWS head admitted that U.S. weather prediction no longer led the world and that lack of computer power was an issue.  Senator Cantwell could not believe it, noting how many powerful computers Congress had authorized for national needs.

So why is the NWS stuck with such inferior computers?  It is not like my colleagues at the NWS are not clambering for them.  First, the NWS/NOAA computer acquisition managers have made  poor choices, playing it safe with extremely expensive, "heavy metal" IBM hardware rather than less costly commodity chips (like Intel Nehalem cores).  Second, it is clear that NWS and NOAA management have not given NWS computer acquisition sufficient priority, a clear indication of management failure, since they have known of this situation for years.

Recently, NOAA/NWS signed a TEN-YEAR agreement keeping IBM and its gold-plated hardware as the choice for the NWS (IBM is finally going to use commodity chips, I should point out, but IBM is still very, very expensive).  Unbelievable. Why commit to long-term deal with an expensive vendor?  Experienced computer folks I know were just shaking their heads.

How to fix this problem.  NOAA management has to immediately secure funding for a petaflop-class machine for operational numerical weather prediction (NWP).   Another approach would be to immediately reassign the use of the computer resources already in NOAA (e.g., Fairmont) or other agencies (e.g., DOE) from climate to weather prediction.  We need some rebalancing of priorities.  For the price of one expensive military jet, the nation could have far better weather prediction. 

Bu the problems with the National Weather Service extend far beyond computer power.

Problem 2: Structural deficiencies of how the National Weather Services organizes its research and operations regarding weather modeling.

Case in point: the weather modeling and the research that support it are not in the same organization.

Operational weather prediction is found in the National Weather Service's Environmental Modeling Center (EMC, located in DC), while most of the research and development is location in NOAA's Office of Oceanographic and Atmospheric Research (OAR).   So the director of EMC and the head of the NWS DO NOT HAVE CONTROL OF THE RESEARCH THAT SUPPORTs THEIR OPERATIONS.   Much of the relevant research is being done in Boulder at NOAA's Earth Systems Environmental Lab (ESRL), while other research is completed at the Geophysical Fluid Dynamics Lab (GFDL) in Princeton (hurricane modeling) and additional venues.   The folks in NOAA OAR and the NWS EMC often are not well coordinated, have different ideas on how to proceed, and frequently evince a competitive attitude towards the other.

NWS Organizational Chart

 Even within the NWS there are structural problems.   The key area of postprocessing (statistically improving model output) is not in NWS's  EMC but found in ANOTHER lab (Meteorological Development Lab, MDL) at another location in DC.   The result has been duplication, waste, and lack of coordination. 

Possible Solution:  Concentrate government operational modeling and most Federal modeling research in Boulder, Colorado, by moving EMC to that city.  Boulder is the intellectual center of weather prediction modeling in the U.S.  Not only is the NOAA ESRL lab there, but the center of research modeling--the National Center for Atmospheric Research (NCAR)-- is there as well. Boulder is a much more attractive and central location for attracting visitors and employees than the suburbs of DC.  It makes sense for the Environmental Modeling Center (which includes oceans and atmosphere) to be in NOAA, rather than the NWS.  And a formal connection with NCAR should be made as well, including long-term support of research and technology transfer, as well as adding the NCAR Developmental Testbed Center (DTC).  The statistical postprocessing folks in MDL should join the new organization.  In short, a highly integrated and comprehensive group--encompassing weather prediction operations and research should be established in Boulder, Colorado.  The impact of doing so will be substantial and positive.

Problem 3:  Poor leadership in the National Weather Service and NOAA.

Visionary leadership has been absent from the National Weather Service for a number of years.  For decades it has been the pre-retirement home of colonels, generals and admirals, with a number of them never mastering the complex technologies for which they were responsible.  Just to give you an idea, here are the recent heads of the NWS:

Colonel Elbert Friday, 1988-1997 (USAF, but did have a Ph.D. in the field)
General Jack Kelly, USAF, 1998-2003
General David Johnson, USAF (ex-pilot) 2004-2006
Colonel Jack Hayes, USAF (weather officer with Ph.D) 2007-2012

Importantly, these military leaders brought many military folks into middle management positions in the NWS.  The military leaders of the NWS had a varied meteorological background, ranging from virtually nothing to Ph.Ds; most had a priority of keeping the ship afloat, versus scientific/technical excellence.  They had all managed large organizations before, but clearly many of them did not have a clear vision of how to lead a science-based, technological organization.   During their tenure, the NWS slid from number one in weather prediction to secondary status, with problems like those noted here festering and worsening.

A new, high-level position was created recently in NOAA to oversee NOAA operations (including weather prediction).  You will never guess his background:  an admiral.  And he is promoting a failed bureaucratic approach in which all the weather agencies will somehow work on the same models, called NUOPC.

I am not saying that generals and admirals can't be effective leaders for a highly scientific civilian agency.  But as documented in Thomas Rick's bestselling book, The Generals, our military leadership has become strong on tactics and weak on strategy and vision.  It has certainly been true of their management of the National Weather Service.

I should be clear that NOAA management is also heavily indicted in NWS problems, since the issues really encompass the combined NOAA/NWS entity.  The NWS has suffered from lack of resources from NOAA and the U.S. government.  In fact, the previous head of the NWS Jack Hayes tried to get more resources from NOAA management, but was rebuffed.   As the NWS has slid, few in NOAA leadership seemed to have noticed or cared.  Neither has Congress, until recently.

Solution:  The NWS director position is now open after two failed searches (a story in itself).  This time find a civilian leader with strong scientific credentials and a credible vision of the future of the organization.  If possible, someone with fresh insights from outside of the organization.

Problem 4:  Poor organization of other key National Weather Service components

There are some strong components of the National Weather Service, such as the local office structure that allows forecasters to become intimately knowledgeable about their local weather and to effectively interact with regional agencies, governments, and users.  (I have heard rumors that some in NOAA leadership have proposed to destroy this structure, closing offices into regional forecasting centers.  This would be a disaster).

But large amounts of staff and funds are wasted on the division of the NWS into regions, each with its own leadership and bureaucracy (Eastern, Western, Central, Southern, Alaska, Pacific).  I have a lot of friends in the NWS and most of them believe the region system is outdated and wasteful.

There are also national centers whose functions are redundant with local offices.  A prime example is the NWS Hydrometeorological Prediction Center (HPC), which makes all sorts of precipitation forecasts and prepares old-fashioned surface charts.  Nearly all of their functions are redundant with local offices or could easily be done locally.  My colleagues in local NWS offices believe this as well.

Possible Solution:  End the region structure and integrate all functions into an improved national structure. Reduce or eliminate the HPC and other unnecessary centers.

Problem 5: Lack of rational observing system planning

 Recently, the media has been abuzz with the expected gap in polar-orbiter weather satellite coverage, suggesting weather forecasting skill would decline.   The NOAA/NWS polar orbiter acquisition program has been characterized by mismanagement for years, not only delaying the next generation satellites, but costing the nation billions of dollars.  Even Jane Lubchenco, head of NOAA, acknowledged the polar orbiter satellite acquisition had become a “national embarrassment.”  But the problems are deeper than satellite development and acquisition.

JPSS Polar Orbiter Satellite

Numerical weather prediction is based on using a wide range of observations, from billion dollar weather satellites, to surface observations, radiosondes, aircraft observations, and other collection platforms.  The NWS needs to assess the optimal collection of observations, including dropping old data sources that are no longer needed.  There is a rational way to do so through Observing System Simulation Experiments (OSSEs) and Observing System Experiments (OSEs).  In the first, we simulate (forecast) for an extended period and make believe that is reality.  Then we sample from these simulations, taking pseudo observations, and then simulate the impacts of varying observing systems and strategies.  In the OSE approach we run forecasts and drop varying observing systems  to see the impacts.  To decide whether we need a new satellite system, you need to do extensive OSSEs.  Unfortunately, the NOAA/NWS has done very little of this and billions of dollars of hardware is being purchased without a clear understanding of the impacts of new sensors and satellites.  (My discussions with high-level folks in NOAA confirm this, by the way).  There is a lot being said about major negative impacts of losing one of two polar orbiting satellites without enough information to back it up.  Congress needs to insist on this.

Problem 6:  Lack of interactions/cooperation with the meteorological community

The National Weather Service is a major component of the weather prediction enterprise, but only one part.   A large academic research community produces scientific and technological advances that the NWS needs to support and take advantage of.   A large and healthy U.S. meteorological private sector uses NWS model output and observations to provide a wide range of weather products for the nation.  Thus, the NWS cannot act in a vacuum and needs close interactions with the entire community.  Decisions on everything from modeling to data formats need close cooperation between the NWS and the rest of the community.

Unfortunately,  this has not been the case.  The National Weather Service, and particularly its weather prediction entity, EMC, have had a tendency towards isolation, lack of openness to other approaches ("not invented here attitude"), and a habit of putting out plans without much community input.  The fruits of nation's research community are not flowing in the National Weather Service.  In fact, foreign centers (such as ECMWF) have been the eager recipients of U.S. research and development advances, when NWS has shown lack of interest.    A good example of the problem occurred in the late 90s when  a new modeling system (WRF, Weather Research and Forecasting Model) was developed that was going to bring the academic and research communities together, to the great benefit of both.  The NWS then decided it wanted to use its own model, one that was inferior by objective standards.

Another example: National Academy committees and the private sector have recommended and pleaded over the past decade for the NWS set up a formal advisory committee to provide community input; such an advisory board still does not exist.  Recently, Tom Bogdan, head of the University Corporation for Atmospheric Research, is working to set up an alternative advisory group:  the Weather Commission.  Perhaps that might be a fruitful approach to the problem.

You would expect that the NWS would provide substantial grant support to universities working on research with applications to NWS goals. You would be wrong.  Funding for extramural research has been miniscule, and such research is the first thing to be cut when NWS budgets get tight.  Believe me, I know about this intimately.

The insularity of NWS personnel has recently been substantially enhanced by the cancellation of nearly all travel to important meteorological conferences and workshops.  A penny wise, pound foolish policy at best.

Finally, documents describing NWS future plans are generally vague and uninformative, with heavy reliance on catchy, essentially meaningless, phrases such as "weather-ready nation."  (It is amusing to listen to how many times administrators and managers like to mention that line).  If you want to see what I mean, check out their latest "strategic plan."   Where is the specific plan for becoming the number one weather prediction entity in the world, with adequate computers and staff?  That plan you won't find anywhere.

.....................

This blog is getting long, and there is a long list of NWS issues that I haven't gotten to, including profound problems at the National Climatic Data Center, the national repository of weather and climate information (it is extremely hard to get relevant weather/climate data, particularly for someone without an intimate knowledge of NWS datasets).  Or the ill-fated attempt to create a parallel National Climate Service.   And when I talk to middle-level managers in the NWS they complain about powerful unions that slow down innovation and new ways of doing business.

The National Weather Service and NOAA are simply not providing the nation with the quality weather forecasts and weather information they should, and a substantial amount of public funds are not being used effectively.  Which is a terrible shame considering the need, the extraordinary talent within both NOAA and the NWS, and the nation's scientific dominance in the field.

Better, more visionary, management is needed.  It is become more than clear that current NOAA/NWS management are incapable of fixing the problems themselves.  Congress, with its oversight responsibilities, must intervene and the meteorological community must weigh in on needed changes.

This blog is going to get me in trouble with some, I know that, but the stakes are too high and the importance to the nation is too great to keep silent.

Comparison between the U.S. (GFS, black), European Center (red lines), and other modeling systems over the northern hemisphere.  The closer to one the better.  The ECMWF model (red) beats the U.S. model (black) for nearly all days.  The difference is even larger in the southern hemisphere.