Jan 112012

Solar Storm Effects on IT

Each time when there was a strong geomagnetic storm induced by increased solar activity, we had a higher rate of failure of computer and telecom equipment in our data centers and throughout the institution. This wasn’t affecting all systems, but those that had their weeks or days numbered would fail around that time. The failures themselves were not a surprise but their grouping was. Coincidentally (or most likely not), we also had some inexplicable software bugs occur during these storms so we used to blame “gremlins” for all this trouble until we slowly connected the dots and saw the corelation between these events and news about strong CME (coronal mass ejection) followed by reports of aurora borealis (northern lights) appearing in areas much further south than usual.

Even Redundant Systems Are Vulnerable

With our huge systems with thousands of concurrent users and over half a million of named users, failures normally and routinely happened every now and then, but the biggest solar flares tend to group these together instead of making statisticians happy by randomly failing within their normal life expectancy. As one would expect, this makes us IT geeks quite nervous and scared for our jobs, wondering what the heck was going on with our systems, no matter how huge and how redundant they are. Even biggest systems need work to restore their full redundant functionality back to normal. Despite their redundancy and despite the fact that the end users often don’t see a single glitch, there’s still risk involved.

Luckily, my 15 year long IT career is way too short to have experienced the strongest solar flares recorded and so far the contemporary world has been lucky not to have them as strong as they can get. Still, our friend Murphy says that whenever something can go wrong it will, so we should consider how far south these things can go (both northern lights and affected systems) and we should be prepared to stop and smell the flowers until civilization and its progress gets back on track after a fierce solar storm. Joking aside, there is a lot at stake in case of an unusually large solar activity – a very strong solar flare with an unfortunately directed CME could instantly and temporarily disrupt world communications, navigation and travel. However, the worst, slightly delayed effects would be exerted on electricity distribution where the worst affected power grids might take years to fully recover. Institutions and services critical for human life better be prepared for these events. There are ways to protect Humanity from worst effects but they require vigilance and agility of those in charge, starting from power plants to national headquarters. But before we go into protection strategies, let’s see some examples from history to get an idea of the possible problems and their scope.

Telegraph Wires on Fire 1859 – The Carrington Super Flare

The most powerful recorded solar flare and solar storm during the last 450 years occurred 1859 and caused serious problems with scarce electric systems around the planet, mostly affecting the telegraph. The northern lights were visible even in the Carribean and Central America, while according to newspaper articles from that time the night skies in the north east coast were so bright that the newspapers could be read without any artificial light. Gold seekers in California were preparing breakfast in the middle of the night assuming it was dawn. The magnetic field fluctuations induced more power than the batteries were providing for telegraph communication and they were setting telegraph wires, poles and equipment on fire and shocking operators. The telegraph couldn’t function until some smart engineers realized that the system was at times working much better with power cut off, running only on electricity induced by the magnetic storm. This solar storm is often called the Carrington Event after the amateur astronomer who observed and described the flare in details.

The Solar Storm of May 14-15 1921

62 years later occurred the second most powerful solar storm in the recorded history, just above one half as powerful as the Carrington Event of the 1859, but almost twice as strong as the one in 1989 that caused the Canada blackout (described next). The big magnetic storm from 1921 again caused problems with telegraph wires, knocking out telegraph service west of the Mississippi, shocking operators and setting equipment and telegraph poles on fire. This time it also “fried” some train installations, railroad signal stations and control towers in New York, and caused the fire that destroyed Central New England railroad station.

Judging from the effects of these two solar storms on then scarce power and communication lines, we’d have a lot to lose if something as powerful unleashed on our planet nowadays. Although we haven’t encountered anything as strong, a smaller but still powerful magnetic storm wreaked havoc on Canadian power grids in 1989.

1989 – Canada Blackout with Cuban Northern Lights

67 years later, on March 13, 1989, another magnetic storm caused the major power outage that started in Montreal and spread across Canada. The root cause was a “powerful” explosion on the sun (for our referent system, not the sun’s) on March 10, followed by a solar flare that almost immediately caused strong electromagnetic interference in the ionosphere which jammed some long-range shortwave radio stations (e.g., Radio Free Europe) and got the Soviets wrongfully blamed.

However, the real damage happened about two days later when the solar wind (as opposed to the initial electromagnetic pulse) reached Earth’s magnetic field and the atmosphere in the evening of March 12. The northern lights usually visible in polar regions, were reportedly seen as far south as Cuba. The magnetic disturbances reached their peak in the early hours of March 13 when Earth’s magnetic field was oscillating most, shifted by the solar wind and settling back in place. These changes of the magnetic field on a global scale induced very powerful currents in very long power lines, pipelines and in the ground.

What happened next is a classical blackout scenario – these powerful electric currents followed a path of least resistance into Canadian power grids and rendered many big crucial transformers inoperable. The system reacted by shifting the load to other working parts of the grid which under the extra load, possibly also affected by similar currents, overloaded and failed, which caused more load rerouting and more overloads and failures.

All this happened within about 90 seconds, before any human could have reacted and done anything. The resulting blackout affected 6 million people for 9 hours with major disruptions and people stuck in the subway, in elevators and at home or work without heat in freezing temperatures (it was 19 degrees Fahrenheit or -7 Celsius in Toronto), and resulted in many millions of dollars of damage on equipment and disrupt business. Moreover, in the following years Canadians spent billions to convert their longest power lines into smaller sections and on other sophisticated ways to protect their grid from similar ground currents. When a solar storm like this happens again, they may be much better off than us south of their border.

The USA power grids weren’t immune to these effects, but had less problems with geomagnetically induced currents, because the ground in our neck of the woods is more conductive, what allowed more of these currents to move through the ground instead of through our power grids. Still, per NASA there were more than 200 power grid failures recorded in the USA on March 13 that can be attributed to this event with many millions of dollars of equipment damage. These failures didn’t cause any blackouts because there was more spare power available and perhaps also because the currents asserted on the power grids were less powerful than in Canada where there’s higher resistance in the ground. However, if we take into consideration that we have less spare power and many more communication lines today, and that the 1989 solar storm was just above one half as powerful as the one from the 1859, we can imagine what could happen if we get hit by the big one.

Largest Recorded Solar Flares Missed Earth

November 4, 2003

November 2003 Superflare

Apparently, we lucked out twice recently: in April 2001 there was a huge solar flare later classified as level X22 on a scale that maxes out at X20 (“it went to eleven” in the Spinal Tap referent system), but the other one in November 4, 2003 takes the title, being later classified around X40 (some teams of astronomers went as high as X50). Both discharged huge coronal mass and radiation bursts that mostly missed Earth, otherwise by now we’d know the extent and types of damage an extremely fierce solar storm can do to our modern technology.

Radiation Is Not The Cause of Solar Storm Effects on Earth

In spite of the misconception that the the solar storms are dangerous because of the electromagnetic pulses and radioactivity, this applies only to astronauts, satellites and stations in space, and to planes flying very high and close to the Earth’s poles, but it’s actually not the case down here on Earth, where total cosmic radiation decreases during a solar storm.

Humans and systems on Earth’s surface are shielded by her magnetic field and the ionosphere, and also by all the molecules in the atmosphere including even clouds and atmospheric moisture, hence besides slightly heightened rates of heart attacks there never were any reported injuries other than some side-effect shocks or burns of the telegraph operators from electrically overloaded equipment during the worst solar storms of 1859 and 1921.

Before learning more about influence of solar storms on technology I also mistakenly believed that the failures we experienced were happening because of increased radiation, but after learning more, the pattern of our equipment failures makes more sense. Our data centers from the beginning of the post were not hit by any radioactive or electromagnetic pulses, but by the subtle magnetic field changes (subtle on a local scale) and most of all by the power fluctuations in the grid caused by the Earth’s magnetic field shifts during her encounter with the solar wind, hence the coincidence of our failures with northern lights instead with the much earlier initial interference with shortwave communications that hits and bounces off of our planet’s natural shields. What makes this even more plausible is the fact that some of our data centers and some other facilities were even during “quiet times” susceptible to a bit higher than normal power fluctuations, as we were advised by HP experts after they installed a large Unix server farm and did preventative environmental measurements.

Geomagnetically Induced Current – The Big Badass

Most damage occurring after solar flares and CME events is caused by the geomagnetically induced currents (GIC) that either flow through the ground or find the path of least resistance through available grounded power grids, but also induce in long conductors and pipelines laid down across vast areas. This happens because the solar wind pushes back the Earth’s magnetic field which then oscillates back and forth until it settles, and in the process the global changes of magnetic flux excite electromotive force and cause electric current to flow through any very long conductors. These electric currents had an open path into the telegraph wires spreading across the continent during the Carrington Event 1859 because the system had only one wire and used ground as the returning path. These currents also caused most of the problems in Canadian blackout and American power grid failures in 1989.

Magnetic Storms Hit Far from The Equator

GICs are typically stronger in higher latitudes like in Canada, northern parts of Russia and Scandinavinan countries. The closer to the equator, the greater the Earth’s protection from the threat. In the US, a power facility or a server farm in Texas, New Mexico, Nevada, Louisiana and particularly Florida might be almost immune from the effects of a massive solar storm but one in the New England states would be right in the thick of things.

What made things worse in east Canada during the 1989 blackout were its huge igneous rock (solidified lava) areas of lower conductivity in the ground, so the electric current found other paths of least resistance, unfortunately through the power grids, because most of them were grounded, which is usually beneficial, but not in this case.

GICs brought down power grids by overloading and burning key high voltage transformers. Typically each of these high technology transformers costs millions of dollars and it takes about one year to make. So not only does the big solar storm threat to knock off entire power networks, but it also might leave us in the darkness for quite some time if we’re not prepared to act quickly and do immediate preventative shutdowns before the solar wind pushes Earth’s magnetic field and makes it oscillate and induce powerful currents on the ground.

Space Weather Forecasts

There are no ways to accurately predict strongest solar storms and their extent, but fortunately our distance from the Sun gives us a delay of less than a day for strongest magnetic storms, and that time can be used to prepare for an outage. The usual time it takes smaller “solar particle showers” to arrive and hit Earth is 3-4 days, but the 1859 magnetic superstorm particles hit the Earth’s atmosphere in 18 hours due to their extra strength and the path cleared by the immediately preceding flare and coronal mass ejection. Judging by this example, it seems that we have about half a day to prepare for “The Big One”.

Typical Timeline of A Strong Solar Storm

1. A sunspot forms

Solar flare erupting into prominence/CME

2. Days or weeks later, the sunspot explodes and a solar flare occurs as an unusually bright flash of light and all other electromagnetic radiation emitted from that location on the sun’s surface. Lasts minutes to tens of minutes.

3. 8.3 minutes later the flare is observed on Earth (8.3 light minutes away from the sun).

4. Proton storm – reaches Earth minutes to days after the flare has been observed (the strongest recorded proton storm in history arrived 15 minutes after the flare was observed, on January 20, 2005).

5. Solar wind (plasma consisting of mainly electrons and protons from coronal mass ejection) reaches Earth 18 hours to several days later, causes auroras, delivers a magnetic punch to Earth’s magnetic field which shifts and bounces back, creating the geomagnetic storm.

6. Soon after the auroras appear, geomagnetically induced currents form as a result of Earth’s magnetic field shifts across great distances. This induces high voltages and currents in long power lines, pipelines and everything else that is long enough and conducts electricity. This is what can overload and destroy big and expensive transformers and cause blackouts like the 1989 blackout in Canada.

How Bad Could It Be?

When trying to predict the future it’s good to consider the worst case scenario instead of just wearing tin foil hats and screaming “the end is near”. First the electromagnetic pulse and the proton storm would bring down or at least temporary shut down some satellites, possibly also disabling GPS navigation. 12 or more hours later, geomagnetic currents would bring down the power grid by overloading and disabling transformers and generators which then would bring down plethora of systems that need electrical power to run. You wouldn’t even be able to refuel your car because gas stations use electricity to run their pumps. Even heating and cooling systems that operate on natural gas or other fuel rely on electricity to pump water or other heating or cooling media through pipes and radiators, so most people would be without heat or AC. Studies mention possibly 130 million people left without power for days, with trillions of cost and years to fully recover.

After an extreme (once a half millennium) solar storm event, power generators and solar chargers would be in high demand and very expensive, so if you think you may need them, better get them during good times, especially if you run into a special sales offer of some of the units with good reviews. Also it is possible that cell phone internet access may be restored sooner than home connections, so a generator, or solar or car cell phone charger can be handy for this purpose as well. If you can reduce your power requirements to the most essential kilowatt or few, you may save a lot of money when buying a generator (and gas once you’re using it). This is much harder to do with fail-over generators for business use, which may be a great alternative or a complement to a UPS of a limited life.

According to the Metatech Corporation study of a possible severe geomagnetic storm, the USA could possibly suffer a damage of $1-4 trillion in the first year with years needed to recover!

Fiber Optics Limited Immunity

Fiber optics lines will do just fine regardless whether they are the major ones on the ocean floor connecting continents, or local ones used for WAN, MAN and LAN or ISP connections to consumers such as Verizon FIOS. Very long fiber cables should have no problems if they don’t have any copper conductors running alongside all their length.

Still, mostly all fiber switches include electrical circuits and feed off the same power grid everything else does, so the benefits of running fiber could be quite limited without isolated sources of electricity on both ends of the cable, but with them this circuitry might never go down, or otherwise can be the first to recover from effects of a major magnetic storm once the power is restored, with undamaged communication lines and capacities.

Possible Protection from Solar Storms

Tinfoil hats won’t work. Protecting systems from magnetic storms can’t be done overnight and it should be a long term strategy rather than a tactical goal. Learning from their own bad experience, Canadians spent billions on hardening their grid against future magnetic storms and they may be better off than the rest of us if the big one strikes in the relatively near future.

Most modern satellites temporarily shut down when exposed to an overwhelming solar radiation. Some are shielded, but not too well, considering that shielding weighs a lot and tremendously increases the amount of fuel and money needed to launch the unit into orbit. Still, most satellites survived through numerous solar storms in the last decades, but the question is how well they could withstand a Carrington event.

Critical devices on the ground could be unplugged from the power grid and fed from a UPS, an independent generator or any other alternative local source of electricity to avoid damages from varying voltage and outages. Some advocate that critical systems should be in basements and in Faraday cages, but this doesn’t help much because most server and computer chassis are already Faraday cages and the damaging effect of the solar storm down here on Earth is not an electromagentic pulse. Laptops should be OK if they are disconnected from the power grid, and even if connected, they have a built-in UPS: the battery which keeps them up when disconnected. Ideally, to avoid damage from this and similar effects, circuits should be overload safe, reverse-polarity-safe, with auto-shut-down functionality but now I’m getting too technical and into preventative circuit design, which is not the intention of this post.

Only Way Out – Preventative Shutdowns and Disaster Recovery

The biggest anticipated damage could be on overloaded large and crucial transformers and generators. However, this can be minimized by their controlled shutdown and disconnection so that these devices can’t be destroyed during the worst period of geomagnetic currents. The power companies are already monitoring space weather and have proactively undertaken measures to protect their equipment and business from disastrous events.

Still, the question is when the big one strikes, whether they will have enough time and an available strong-minded decision-maker with enough guts to shut down and disconnect everybody to prevent bigger damage in spite of possible lawsuits of the businesses that will lose customers and money. I recently heard of an engineer of an educational institution located near the World Trade Center in New York who during the 9/11 catastrophe acted proactively and shut down the building’s air conditioning system, saving it from tremendous amounts of dust that soon covered the area and penetrated into most other buildings who weren’t as lucky and had to undergo expensive and long decontamination process. Similarly, I hope that some of power network managers who will be in charge when that day comes will have their priorities straight, but judging on the 9/11 experience it seems that not everybody will do the right thing. If we furthermore include our Katrina experience where communication and collaboration was bad and where the levee resistance was estimated only against average storms, I can’t be a total optimist. Just like the once-a-century storm hit New Orleans, once-a-500-year solar storm can happen tomorrow and we should know how to handle the worst.

Our planet needs contingency plans similar to the ones that were made for Y2K. The more critical the business, the more critical it is to prepare it for these circumstances. On the other hand, disaster recovery and business continuity installations can be extremely expensive, so those who can afford a slow or dysfunctional business for hours, days or weeks could actually save because the probability of the big one so far seems to be about once in 500 years, according to the measurements of the indicators in the old polar ice, but this is a very narrow sample of the last 500 years which can be totally off, either against us or to our advantage. Insurance companies might help, but consider that whenever you insure something, you bet your insured asset will fail and the insurance company bets it won’t. In a drastic case like this when you need them most, they would have to pay too many customers (losing too many bets) so they might run bankrupt. Same thing might happen to specialized disaster recovery service providers. In an event of this scale, it might be wise not to count on solutions, providers and insurance companies who would have to serve, accommodate or pay too many customers.

Other than that, I believe that Humanity can withstand and survive a major solar storm with only minor economic disruptions, however, this may vary a lot for individuals with different location, levels of preparedness or degrees of luck. As I keep repeating, don’t believe the false prophets who have been insisting for thousands of years that the end is near. Most of all, don’t give them your money. Invest it in some emergency supplies and solar power or a generator instead. That can be handy not only after a major solar storm, but also after a tornado, major winter storm, flood, hurricane or long blackout. False prophets won’t save you, but that might.

Special thanks to James A. Marusek of breadandbutterscience.com for comments, suggestions and expert opinion.


A Super Solar Flare
by NASA’s Trudy E. Bell and Dr. Tony Pillips, 2008

Sept. 2, 1859: Telegraphs Run on Electric Air in Crazy Magnetic Storm
By Alexis Madrigal, 2011, Wired.

Solar Storm FAQs by NASA

Solar Storm of 1859
(Wikipedia article)

Canada Blackout 1989:
The Day the Sun Brought Darkness
By Dr. Sten Odenwald, NASA Astronomer

Severe Space Weather Events–Understanding Societal and Economic Impacts: A Workshop Report – sponsored by the National Academy of Science, 2008

Metatech study on possible catastrophic effect of big solar storms:

Related web presentation (pre-recorded):

Great in-depth article about causes and effects with nice set of scientific facts:
Solar Storm Threat Analysis
By James A. Marusek, 2007
Solar Storm Disaster Preparedness Plan by the same author.

On possible prevention:
This Week’s Solar Flare Illuminates the Grid’s Vulnerability (June 9, 2009, the New York Times)
by Peter Behr of Climatewire

More on possible prevention:
A future Space Weather catastrophe : a disturbing possibility
By Dr. Jeff Masters, 2009.

Discussion on realistic vs the-end-is-near-so-wear-your-tinfoil-hat approach: :
http://www.futurepundit.com/archives/006079.html (I especially like the comment made by WestHighlander)

Solar Proton Events:
A New Kind of Solar Storm(About the fastest solar proton event recorded) NASA Science News, 2005

Space Weather Status:


 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>