Engineers at Google’s headquarters in Mountain View, California, back in mid-December 2009, began to suspect that hackers in China had obtained access to private Gmail accounts. Those accounts included those used by Chinese human rights activists opposed to the government in Beijing.
Like a lot of large, well-known Internet companies, Google and its users were frequently targeted by cyber spies and criminals. But when the engineers looked more closely, they discovered that this was no ordinary hacking campaign.
In what Google would later describe as “a highly sophisticated and targeted attack on our corporate infrastructure originating from China,” the thieves were able to get access to the password system that allowed Google’s users to sign in to many Google applications at once. This was some of the company’s most important intellectual property, considered among the “crown jewels” of its source code by its engineers. Google wanted concrete evidence of the break-in that it could share with U.S. law enforcement and intelligence authorities. So they traced the intrusion back to what they believed was its source — a server in Taiwan where data was sent after it was siphoned off Google’s systems, and that was presumably under the control of hackers in mainland China.
“Google broke in to the server,” says a former senior intelligence official who’s familiar with the company’s response. The decision wasn’t without legal risk, according to the official. Was this a case of hacking back? Just as there’s no law against a homeowner following a robber back to where he lives, Google didn’t violate any laws by tracing the source of the intrusion into its systems. It’s still unclear how the company’s investigators gained access to the server, but once inside, if they had removed or deleted data, that would cross a legal line. But Google didn’t destroy what it found. In fact, the company did something unexpected and unprecedented — it shared the information.
Google uncovered evidence of one of the most extensive and far-reaching campaigns of cyber espionage in U.S. history. Evidence suggested that Chinese hackers had penetrated the systems of nearly three dozen other companies, including technology mainstays such as Symantec, Yahoo, and Adobe, the defense contractor Northrop Grumman, and the equipment maker Juniper Networks. The breadth of the campaign made it hard to discern a single motive. Was this industrial espionage? Spying on human rights activists? Was China trying to gain espionage footholds in key sectors of the U.S. economy or, worse, implant malware in equipment used to regulate critical infrastructure?
The only things Google seemed certain of was that the campaign was massive and persistent, and that China was behind it. And not just individual hackers, but the Chinese government, which had the means and the motive to launch such a broad assault.
Google shared what it found with the other targeted companies, as well as U.S. law enforcement and intelligence agencies. For the past four years, corporate executives had been quietly pressing government officials to go public with information about Chinese spying, to shame the country into stopping its campaign. But for President Obama or Secretary of State Hillary Clinton to give a speech pointing the finger at China, they needed indisputable evidence that attributed the attacks to sources in China. And looking at what Google had provided it, government analysts were not sure they had it. American officials decided the relationship between the two economic superpowers was too fragile and the risk of conflict too high to go public with what Google knew.
Deputy Secretary of State James Steinberg was at a cocktail party in Washington when an aide delivered an urgent message: Google was going to issue a public statement about the Chinese spying campaign. Steinberg, the second-highest-ranking official in U.S. foreign policy, immediately grasped the significance of the company’s decision. Up to that moment, American corporations had been unwilling to publicly accuse the Chinese of spying on their networks or stealing their intellectual property. The companies feared losing the confidence of investors and customers, inviting other hackers to target their obviously weak defenses, and igniting the fury of Chinese government officials, who could easily revoke access to one of the biggest and fastest-growing markets for U.S. goods and services. For any company to come out against China would be momentous. But for Google, the most influential company of the Internet age, it was historic.
The next day, January 12, 2010, Google’s chief legal officer, David Drummond, posted a lengthy statement to the company’s blog, accusing hackers in China of attacking Google’s infrastructure and criticizing the government for censoring Internet content and suppressing human rights activists. “We have taken the unusual step of sharing information about these attacks with a broad audience not just because of the security and human rights implications of what we have unearthed, but also because this information goes to the heart of a much bigger global debate about freedom of speech,” said Drummond.
Back at the State Department, officials saw a rare opportunity to put pressure on China for spying. That night Hillary Clinton issued her own statement. “We have been briefed by Google on these allegations, which raise very serious concerns and questions. We look to the Chinese government for an explanation,” she said. “The ability to operate with confidence in cyberspace is critical in a modern society and economy.”
As diplomatic maneuvers go, this was pivotal. Google had just given the Obama administration an opening to accuse China of espionage without having to make the case itself. Officials could simply point to what Google had discovered as a result of its own investigation.
“It gave us an opportunity to discuss the issues without having to rely on classified sources or sensitive methods” of intelligence gathering, Steinberg says. The administration had had little warning about Google’s decision, and it was at odds with some officials’ reluctance to take the espionage debate public. But now that it was, no one complained.
“It was their decision. I certainly had no objection,” Steinberg says.
The Obama administration began to take a harsher tone with China, starting with a major address Clinton gave about her Internet Freedom initiative nine days later. She called on China to stop censoring Internet searches and blocking access to websites that printed criticism about the country’s leaders. Clinton likened such virtual barriers to the Berlin Wall.
For its part, Google said it would stop filtering search results for words and subjects banned by government censors. And if Beijing objected, Google was prepared to pull up stakes and leave the Chinese market entirely, losing out on billions of dollars in potential revenues. That put other U.S. technology companies in the hot seat. Were they willing to put up with government interference and suppression of free speech in order to keep doing business in China?
After Google’s declaration, it was easier for other companies to admit they’d been infiltrated by hackers. After all, if it happened to Google, it could happen to anyone. Being spied on by the Chinese might even be a mark of distinction, insofar as it showed that a company was important enough to merit the close attention of a superpower. With one blog post, Google had changed the global conversation about cyber defense.
The company had also shown that it knew a lot about Chinese spies. The NSA wanted to know how much.
Google had also alerted the NSA and the FBI that its networks were breached by hackers in China. As a law enforcement agency, the FBI could investigate the intrusion as a criminal matter. But the NSA needed Google’s permission to come in and help assess the breach.
On the day that Google’s lawyer wrote the blog post, the NSA’s general counsel began drafting a “cooperative research and development agreement,” a legal pact that was originally devised under a 1980 law to speed up the commercial development of new technologies that are of mutual interest to companies and the government. The agreement’s purpose is to build something — a device or a technique, for instance. The participating company isn’t paid, but it can rely on the government to front the research and development costs, and it can use government personnel and facilities for the research. Each side gets to keep the products of the collaboration private until they choose to disclose them. In the end, the company has the exclusive patent rights to build whatever was designed, and the government can use any information that was generated during the collaboration.
It’s not clear what the NSA and Google built after the China hack. But a spokeswoman at the agency gave hints at the time the agreement was written. “As a general matter, as part of its information-assurance mission, NSA works with a broad range of commercial partners and research associates to ensure the availability of secure tailored solutions for Department of Defense and national security systems customers,” she said. It was the phrase “tailored solutions” that was so intriguing. That implied something custom built for the agency, so that it could perform its intelligence-gathering mission. According to officials who were privy to the details of Google’s arrangements with the NSA, the company agreed to provide information about traffic on its networks in exchange for intelligence from the NSA about what it knew of foreign hackers. It was a quid pro quo, information for information.
And from the NSA’s perspective, information in exchange for protection.
The cooperative agreement and reference to a “tailored solution” strongly suggest that Google and the NSA built a device or a technique for monitoring intrusions into the company’s networks. That would give the NSA valuable information for its so-called active defense system, which uses a combination of automated sensors and algorithms to detect malware or signs of an imminent attack and take action against them. One system, called Turmoil, detects traffic that might pose a threat. Then, another automated system called Turbine decides whether to allow the traffic to pass or to block it. Turbine can also select from a number of offensive software programs and hacking techniques that a human operator can use to disable the source of the malicious traffic. He might reset the source’s Internet connection or redirect the traffic to a server under the NSA’s control. There the source can be injected with a virus or spyware, so the NSA can continue to monitor it.
For Turbine and Turmoil to work, the NSA needs information, particularly about the data flowing over a network. With its millions of customers around the world, Google is effectively a directory of people using the Internet. It has their e-mail addresses. It knows where they’re physically located when they log in. It knows what they search for on the web. The government could command the company to turn over that information, and it does as part of the NSA’s Prism program, which Google had been participating in for a year by the time it signed the cooperative agreement with the NSA. But that tool is used for investigating people whom the government suspects of terrorism or espionage.
The NSA’s cyber defense mission takes a broader view across networks for potential threats, sometimes before it knows who those threats are. Under Google’s terms of service, the company advises its users that it may share their “personal information” with outside organizations, including government agencies, in order to “detect, prevent, or otherwise address fraud, security or technical issues” and to “protect against harm to the rights, property or safety of Google.” According to people familiar with the NSA and Google’s arrangement, it does not give the government permission to read Google users’ e-mails.
They can do that under Prism. Rather, it lets the NSA evaluate Google hardware and software for vulnerabilities that hackers might exploit. Considering that the NSA is the single biggest collector of zero day vulnerabilities, that information would help make Google more secure than others that don’t get access to such prized secrets. The agreement also lets the agency analyze intrusions that have already occurred, so it can help trace them back to their source.
Google took a risk forming an alliance with the NSA. The company’s corporate motto, “Don’t be evil,” would seem at odds with the work of a covert surveillance and cyber warfare agency. But Google got useful information in return for its cooperation. Shortly after the China revelation, the government gave Sergey Brin, Google’s cofounder, a temporary security clearance that allowed him to attend a classified briefing about the campaign against his company. Government analysts had concluded that the intrusion was directed by a unit of the People’s Liberation Army. This was the most specific information Google could obtain about the source of the intrusion. It could help Google fortify its systems, block traffic from certain Internet addresses, and make a more informed decision about whether it wanted to do business in China at all. Google’s executives might pooh-pooh the NSA’s “secret sauce.” But when the company found itself under attack, it turned to Fort Meade for help.
In its blog post, Google said that more than twenty companies had been hit by the China hackers, in a campaign that was later dubbed Aurora after a file name on the attackers’ computer. A security research firm soon put the number of targets at around three dozen. Actually, the scope of Chinese spying was, and is, much larger.
Security experts in and outside of government have a name for the hackers behind campaigns such as Aurora and others targeting thousands of other companies in practically every sector of the U.S. economy: the advanced persistent threat. It’s an ominous-sounding title, and a euphemistic one. When government officials mention “APT” today, what they often mean is China, and more specifically, hackers working at the direction of Chinese military and intelligence officials or on their behalf.
The “advanced” part of the description refers in part to the hackers’ techniques, which are as effective as any the NSA employs. The Chinese cyber spies can use an infected computer’s own chat and instant-messenger applications to communicate with a command-and-control server. They can implant a piece of malware and then remotely customize it, adding new information-harvesting features. The government apparatus supporting all this espionage is also advanced, more so than the loose-knit groups of cyber vandals or activists such as Anonymous that spy on companies for political purposes, or even the sophisticated Russian criminal groups, who are more interested in stealing bank account and credit card data. China plays a longer game. Its leaders want the country to become a first-tier economic and industrial power in a single generation, and they are prepared to steal the knowledge they need to do it, U.S. officials say.
That’s where the “persistent” part comes into play. Gathering that much information, from so many sources, requires a relentless effort, and the will and financial resources to try many different kinds of intrusion techniques, including expensive zero day exploits. Once the spies find a foothold inside an organization’s networks, they don’t let go unless they’re forced out. And even then they quickly return. The “threat” such spying poses to the U.S. economy takes the form of lost revenue and strategic position. But also the risk that the Chinese military will gain hidden entry points into critical-infrastructure control systems in the United States. U.S. intelligence officials believe that the Chinese military has mapped out infrastructure control networks so that if the two nations ever went to war, the Chinese could hit American targets such as electrical grids or gas pipelines without having to launch a missile or send a fleet of bombers.
Operation Aurora was the first glimpse into the breadth of the ATP’s exploits. It was the first time that names of companies had been attached to Chinese espionage. “The scope of this is much larger than anybody has ever conveyed,” Kevin Mandia, CEO and president of Mandiant, a computer security and forensics company located outside Washington, said at the time of Operation Aurora. The APT represented hacking on a national, strategic level. “There [are] not 50 companies compromised. There are thousands of companies compromised. Actively, right now,” said Mandia, a veteran cyber investigator who began his career as a computer security officer in the air force and worked there on cybercrime cases. Mandiant was becoming a goto outfit that companies called whenever they discovered spies had penetrated their networks. Shortly after the Google breach, Mandiant disclosed the details of its investigations in a private meeting with Defense Department officials a few days before speaking publicly about it.
The APT is not one body but a collection of hacker groups that include teams working for the People’s Liberation Army, as well as so-called patriotic hackers, young, enterprising geeks who are willing to ply their trade in service of their country. Chinese universities are also stocked with computer science students who work for the military after graduation. The APT hackers put a premium on stealth and patience. They use zero days and install backdoors. They take time to identify employees in a targeted organization, and send them carefully crafted spear-phishing e-mails laden with spyware. They burrow into an organization, and they often stay there for months or years before anyone finds them, all the while siphoning off plans and designs, reading e-mails and their attachments, and keeping tabs on the comings and goings of employees — the hackers’ future targets. The Chinese spies behave, in other words, like their American counterparts.
No intelligence organization can survive if it doesn’t know its enemy. As expansive as the NSA’s network of sensors is, it’s sometimes easier to get precise intelligence about hacking campaigns from the targets themselves. That’s why the NSA partnered with Google. It’s why when Mandiant came calling with intelligence on the APT, officials listened to what the private sleuths had to say. Defending cyberspace is too big a job even for the world’s elite spy agency. Whether they like it or not, the NSA and corporations must fight this foe together.
Google’s Sergey Brin is just one of hundreds of CEOs who have been brought into the NSA’s circle of secrecy. Starting in 2008, the agency began offering executives temporary security clearances, some good for only one day, so they could sit in on classified threat briefings.
“They indoctrinate someone for a day, and show them lots of juicy intelligence about threats facing businesses in the United States,” says a telecommunications company executive who has attended several of the briefings, which are held about three times a year. The CEOs are required to sign an agreement pledging not to disclose anything they learn in the briefings. “They tell them, in so many words, if you violate this agreement, you will be tried, convicted, and spend the rest of your life in prison,” says the executive.
Why would anyone agree to such severe terms? “For one day, they get to be special and see things few others do,” says the telecom executive, who, thanks to having worked regularly on classified projects, holds high-level clearances and has been given access to some of the NSA’s most sensitive operations, including the warrantless surveillance program that began after the 9/11 attacks. “Alexander became personal friends with many CEOs” through these closed-door sessions, the executive adds. “I’ve sat through some of these and said, ‘General, you tell these guys things that could put our country in danger if they leak out.’ And he said, ‘I know. But that’s the risk we take. And if it does leak out, they know what the consequences will be.’ ”
But the NSA doesn’t have to threaten the executives to get their attention. The agency’s revelations about stolen data and hostile intrusions are frightening in their own right, and deliberately so. “We scare the bejeezus out of them,” a government official told National Public Radio in 2012. Some of those executives have stepped out of their threat briefings meeting feeling like the defense contractor CEOs who, back in the summer of 2007, left the Pentagon with “white hair.”
Unsure how to protect themselves, some CEOs will call private security companies such as Mandiant. “I personally know of one CEO for whom [a private NSA threat briefing] was a life-changing experience,” Richard Bejtlich, Mandiant’s chief security officer, told NPR. “General Alexander sat him down and told him what was going on. This particular CEO, in my opinion, should have known about [threats to his company] but did not, and now it has colored everything about the way he thinks about this problem.”
The NSA and private security companies have a symbiotic relationship. The government scares the CEOs and they run for help to experts such as Mandiant. Those companies, in turn, share what they learn during their investigations with the government, as Mandiant did after the Google breach in 2010. The NSA has also used the classified threat briefings to spur companies to strengthen their defenses.
In one 2010 session, agency officials said they’d discovered a flaw in personal computer firmware — the onboard memory and codes that tell the machine how to work — that could allow a hacker to turn the computer “into a brick,” rendering it useless. The CEOs of computer manufacturers who attended the meeting, and who were previously aware of the design flaw, ordered it fixed.
Private high-level meetings are just one way the NSA has forged alliances with corporations. Several classified programs allow companies to share the designs of their products with the agency so it can inspect them for flaws and, in some instances, install backdoors or other forms of privileged access. The types of companies that have shown the NSA their products include computer, server, and router manufacturers; makers of popular software products, including Microsoft; Internet and e-mail service providers; telecommunications companies; satellite manufacturers; antivirus and Internet security companies; and makers of encryption algorithms.
The NSA helps the companies find weaknesses in their products. But it also pays the companies not to fix some of them. Those weak spots give the agency an entry point for spying or attacking foreign governments that install the products in their intelligence agencies, their militaries, and their critical infrastructure. Microsoft, for instance, shares zero day vulnerabilities in its products with the NSA before releasing a public alert or a software patch, according to the company and U.S. officials. Cisco, one of the world’s top network equipment makers, leaves backdoors in its routers so they can be monitored by U.S. agencies, according to a cyber security professional who trains NSA employees in defensive techniques. And McAfee, the Internet security company, provides the NSA, the CIA, and the FBI with network traffic flows, analysis of malware, and information about hacking trends.
Companies that promise to disclose holes in their products only to the spy agencies are paid for their silence, say experts and officials who are familiar with the arrangements. To an extent, these openings for government surveillance are required by law. Telecommunications companies in particular must build their equipment in such a way that it can be tapped by a law enforcement agency presenting a court order, like for a wiretap. But when the NSA is gathering intelligence abroad, it is not bound by the same laws. Indeed, the surveillance it conducts via backdoors and secret flaws in hardware and software would be illegal in most of the countries where it occurs.
Of course, backdoors and unpatched flaws could also be used by hackers. In 2010 a researcher at IBM publicly revealed a flaw in a Cisco operating system that allows a hacker to use a backdoor that was supposed to be available only to law enforcement agencies. The intruder could hijack the Cisco device and use it to spy on all communications passing through it, including the content of e-mails. Leaving products vulnerable to attack, particularly ubiquitous software programs like those produced by Microsoft, puts millions of customers and their private information at risk and jeopardizes the security of electrical power facilities, public utilities, and transportation systems.
Under U.S. law, a company’s CEO is required to be notified whenever the government uses its products, services, or facilities for intelligence-gathering purposes. Some of these information-sharing arrangements are brokered by the CEOs themselves and may be reviewed only by a few lawyers. The benefits of such cooperation can be profound. John Chambers, the CEO of Cisco, became friends with George W. Bush when he was in office. In April 2006, Chambers and the president ate lunch together at the White House with Chinese president Hu Jintao, and the next day Bush gave Chambers a lift on Air Force One to San Jose, where the president joined the CEO at Cisco headquarters for a panel discussion on American business competitiveness. California governor Arnold Schwarzenegger also joined the conversation. Proximity to political power is its own reward. But preferred companies also sometimes receive early warnings from the government about threats against them.
The Homeland Security Department also conducts meetings with companies through its “cross sector working groups” initiative. These sessions are a chance for representatives from the universe of companies with which the government shares intelligence to meet with one another and hear from U.S. officials. The attendees at these meetings often have security clearances and have undergone background checks and interviews. The department has made the schedule and agendas of some of these meetings public, but it doesn’t disclose the names of companies that participated or many details about what they discussed.
Between January 2010 and October 2013, the period for which public records are available, the government held at least 168 meetings with companies just in the cross sector working group. There have been hundreds more meetings broken out by specific industry categories, such as energy, telecommunications, and transportation.
A typical meeting may include a “threat briefing” by a U.S. government official, usually from the NSA, the FBI, or the Homeland Security Department; updates on specific initiatives, such as enhancing bank website security, improving information sharing among utility companies, or countering malware; and discussion of security “tools” that have been developed by the government and industry, such as those used to detect intruders on a network. One meeting in April 2012 addressed “use cases for enabling information sharing for active cyber defense,” the NSA-pioneered process of disabling cyber threats before they can do damage. The information sharing in this case was not among government agencies but among corporations.
Most meetings have dealt with protecting industrial control systems, the Internet-connected devices that regulate electrical power equipment, nuclear reactors, banks, and other vital facilities. That’s the weakness in U.S. cyberspace that most worries intelligence officials. It was the subject that so animated George W. Bush in 2007 and that Barack Obama addressed publicly two years later. The declassified agendas for these meetings offer a glimpse at what companies and the government are building for domestic cyber defense.
On September 23, 2013, the Cross Sector Enduring Security Framework Operations Working Group discussed an update to an initiative described as “Connect Tier 1 and USG Operations Center.” “Tier 1” usually refers to a major Internet service provider or network operator. Some of the best-known Tier 1 companies in the United States are AT&T, Verizon, and CenturyLink. “USG” refers to the U.S. government. The initiative likely refers to a physical connection running from an NSA facility to those companies, as part of an expansion of the DIB pilot program. The expansion was authorized by a presidential executive order in February 2013 aimed at increasing security of critical-infrastructure sites around the country. The government, mainly through the NSA, gives threat intelligence to two Internet service providers, AT&T and CenturyLink. They, in turn, can sell “enhanced cybersecurity services,” as the program is known, to companies that the government deems vital to national and economic security. The program is nominally run by the Homeland Security Department, but the NSA provides the intelligence and the technical expertise.
Through this exchange of intelligence, the government has created a cyber security business. AT&T and CenturyLink are in effect its private sentries, selling protection to select corporations and industries. AT&T has one of the longest histories of any company participating in government surveillance. It was among the first firms that voluntarily handed over call records of its customers to the NSA following the 9/11 attacks, so the agency could mine them for potential connections to terrorists — a program that continues to this day. Most phone calls in the United States pass through AT&T equipment at some point, regardless of which carrier initiates them. The company’s infrastructure is one of the most important and frequently tapped repositories of electronic intelligence for the NSA and U.S. law enforcement agencies.
CenturyLink, which has its headquarters in Monroe, Louisiana, has been a less familiar name in intelligence circles over the years. But in 2011 the company acquired Qwest Communications, a telecommunications firm that is well known to the NSA. Before the 9/11 attacks, NSA officials approached Qwest executives and asked for access to its high-speed fiber-optic networks, in order to monitor them for potential cyber attacks. The company rebuffed the agency’s requests because officials hadn’t obtained a court order to get access to the company’s equipment. After the terrorist attacks, NSA officials again came calling, asking Qwest to hand over its customers’ phone records without a court-approved warrant, as AT&T had done. Again, the company refused. It took another ten years and the sale of the company, but Qwest’s networks are now a part of the NSA’s extended security apparatus.
The potential customer base for government-supplied cyber intelligence, sold through corporations, is as diverse as the U.S. economy itself. To obtain the information, a company must meet the government’s definition of a critical infrastructure: “assets, systems, and networks, whether physical or virtual, so vital to the United States that their incapacitation or destruction would have a debilitating effect on security, national economic security, national public health or safety, or any combination thereof.” That may seem like a narrow definition, but the categories of critical infrastructure are numerous and vast, encompassing thousands of businesses. Officially, there are sixteen sectors: chemical; commercial facilities, to include shopping centers, sports venues, casinos, and theme parks; communications; critical manufacturing; dams; the defense industrial base; emergency services, such as first responders and search and rescue; energy; financial services; food and agriculture; government facilities; health care and public health; information technology; nuclear reactors, materials, and waste; transportation systems; and water and wastewater systems.
It’s inconceivable that every company on such a list could be considered “so vital to the United States” that its damage or loss would harm national security and public safety. And yet, in the years since the 9/11 attacks, the government has cast such a wide protective net that practically any company could claim to be a critical infrastructure. The government doesn’t disclose which companies are receiving cyber threat intelligence. And as of now the program is voluntary. But lawmakers and some intelligence officials, including Keith Alexander and others at the NSA, have pressed Congress to regulate the cyber security standards of critical-infrastructure owners and operators. If that were to happen, then the government could require that any company, from Pacific Gas and Electric to Harrah’s Hotels and Casinos, take the government’s assistance, share information about its customers with the intelligence agencies, and build its cyber defenses according to government specifications.
In a speech in 2013 the Pentagon’s chief cyber security adviser, Major General John Davis, announced that Homeland Security and the Defense Department were working together on a plan to expand the original DIB program to more sectors. They would start with energy, transportation, and oil and natural gas, “things that are critical to DOD’s mission and the nation’s economic and national security that we do not directly control,” Davis said. The general called foreign hackers’ mapping of these systems and potential attacks “an imminent threat.” The government will never be able to manage such an extensive security regime on its own. It can’t now, which is why it relies on AT&T and CenturyLink. More companies will flock to this new mission as the government expands the cyber perimeter. The potential market for cyber security services is practically limitless.
Photo credit: Whatsupic