Opinion piece about the cyber supply chain and how national security should be the focal point over the next decade
A few months ago I wrote an article about the trend with cloud computing and how we are creating a logical single point of failure for humanity. This is because people are depending so much on these centralized services in the cloud, such as business, communications, banking, health care, entertainment etc. But think deeper and more broadly now on how society is depending on electronics and networking in general. The global communication networks that are powering the so called 4th industrial revolution and facilitating these logical services are controlled and functional due to processors and semiconductors that are then operated by software in electronics. We're talking backbone internet routers, hand held mobile devices, desktop computers, aircraft, satellites, and cellular wireless antennas. Think of how many millions of chips will be created and deployed in these devices over the next decade, how many satellites will be launched etc.
For the major part of the 20th century the United States (US) has really been the leader when it comes to manufacturing and technology. Regardless if you believe this has been through pure human innovation or from reverse engineered tech in Area 51, this facilitated the greatest advancement of humanity in the shortest period of time in history. Think of the difference between the way someone lived in 1919 vs 2019. The big differentiation is the electronics and technology that exist in our life.
As many people know most of the assembly of these electronics has come from China (CN) or other Eastern countries. Although the US was still innovating many businesses were moving manufacturing and other forms of sourcing/assembly over seas. This has ramped up since the mid 1990s with warmer intl. relations and then China's inclusion into the world trade organization in 2001. The terms you probably see a lot is designed in the US but assembled overseas.
Part of the reason for this move in manufacturing was the cheaper price of labor but also because of the large amount of capital needed for semiconductor manufacturing facilities (aka Foundrys or Fabs) and other parts of the supply chain. It was economically easier to just design in house and then to out source the creation of the product that utilized the chips, that business trend facilitated the growth of China and the rise of many tech companies we know today. In addition to CN there is also Japan and South Korea who are the other primary countries that have developed this industry. (e.g. Samsung, Sony)
Semiconductors are used here as a blanket statement for the essential parts of electronics from the actual material (silicon) to the components like integrated circuits, CPUs, RAM, boards and the like. Comparably with other industries electronics has a supply chain from the actual design and creation of the chips, to assembly of the components, to the development and implementation of software. The control of this supply chain has slowly moved out of the control of the US over the years which could be considered a troubling trend for them.
The reason that its troubling is if you can control part of the inventory or process, not only could you command the market you could hamper your competition, and also weaken your nation-state adversaries cyber capabilities. Moreover, you could create an environment where it could be possible to change something technical to provide like a back door, or a flaw in the device to make it vulnerable and exploitable - more on this later.
Strategic look
China announced in 2015 a "Made in China 2025" plan which essentially outlines a plan to become the innovation leader and independent when it comes to technology and manufacturing advanced tech. It argues the semiconductor as the foundation of tech where innovation there will lead to other advancements and will create an advantage to whomever controls the chips.
I agree with the reasoning behind this strategy and think there needs to be more policy talk from the US leadership and understanding and cooperation with domestic businesses to facilitate this strategy at home. About 5 years ago the US enjoyed a 51% dominance in the semi industry by market share with 52% wafer (fabrication) capacity, so on the flip side its clear why China has created this strategy out of their own interests. Although, in a release this year it appears that percentage has dropped to 44% for wafer capacity and 47% market share for the United states of America which could confirm the previously mentioned trend.
Part of the 2018 Cyber strategy issued by US DoD stated China and Russia as our top competitors in the digital arena and are adversarial in nature, so why would we allow such a strategic advantage to trend towards a competitor? Profits and lack of foresight probably. I digress.
We have seen recent activity heat up though with the so called US-China Trade War. Such as when the US issued orders about banning foreign (targeted at Chinese) telecommunications equipment from being procured or being used in government IT networks and such. Although we didn't see much of a similar targeted response from China, because I think of the state of their current domestic market (they still depend on US intellectual property and have a few deficiencies), but they are heavily investing in domestic companies like Huawei to help speed up their independence and market share - as outlined in their 2025 policy. Some of the tariffs issued from Beijing and Washington have caused some shakeups, Cisco said it had moved some manufacturing last year, and other companies are or have signaled they intend to bring some business back to the US to Geo-diversify.
Furthermore, you have to think if the tables were turned and the US depended on a foreign entity more so than it is now for sourcing chips and an export ban was ordered, the ramifications would be dreadful.
One scenario that could play out could make this a reality. Taiwan semiconductor (TSMC) was named the biggest semi manufacturer by market cap and market share. This company makes a lot of chips for many companies like Apple and Qualcomm. Taiwan is located off the coast of mainland China and they've been separated since the revolution, however according to the One China principle, Taiwan is a part of China. I think Taiwan will return to the mainland in the future which will in turn capture TSMC. If this is done before the USA can move the cyber supply chain out of the East to itself or allied nations then this could be a problem (not going too deep on Geo-politics here). There is Intel and Micron which are located inside the US, however many large US firms like Nvidia and Apple rely on Taiwan semiconductor for fabrication.
Although just last month TSMC announced a new fab would be constructed in Arizona. The plant will be capable of fabricating the new ultra small nanometer chips of the future, but there are still a number of years ahead before its operational and there are just one piece in the over all industry (although a big one). It is good news for the US though on this topic. There should be encouragement via private investment and subsidies to bring more manufacturing on-shore - similar to Beijing's strategy. We did get a glimpse of how this might look back in June. Clearly from history manufacturing is how countries are made, that is definitely one reason how the USA and CN became superpowers.
Another key point that will play out over the next 10 years is the space race. You could consider the importance of electronic components like a solar panel is to a satellite, pretty much a required item. Boeing and other US based aerospace companies do provide these, but think if that slowly deteriorated each year and then the dependence was shifted overseas, this could be a costly pressure and disadvantage.
Markets are a big driver here, who is buying what and who is making disruptions. So similarly as I talked about in my cloud post with increased outages and impacts to the public, once revenues are hit from security incidents here or there is a need for a certain chip architecture, then possibly we could see no-question investment in that aspect. Furthermore, there is the existence of mergers and acquisitions, as there have been some US companies that have been purchased recently by overseas businesses and even more over time which has weakened the US's position some would say.
A recent example was the possibility of chip maker Singapore owned Broadcom (former US) taking over San Diego based Qualcomm who is a leader in wireless, Ai, and mobile chip technology. The deal was blocked by the current administration, but I've heard of fears that under past or future administrations the deal would have been allowed to proceed. Keeping companies like Qualcomm domestic will ensure leadership in technology - policy makers and business leaders must recognize this. To illustrate the point, below is a statement from the US Treasury Department about why this particular business should be kept domestic:
U.S. national security also benefits from Qualcomm’s capabilities as a supplier of products. For example, Department of Defense (“DOD”) national security programs rely on continued access to Qualcomm products. Qualcomm holds a facility security clearance and performs on a range of contracts for United States government customers with national security responsibilities. Qualcomm currently holds active sole source classified prime contracts with DOD. Additionally, Qualcomm currently holds unclassified prime contracts with DOD. Qualcomm’s partnership with the United States government encompasses efforts to address cybersecurity in the next generation of wireless, 5G, and the Internet of Things. Limitation or cessation of supply of Qualcomm products or services to the U.S. government could have a detrimental impact on national security.
Therefore by keeping innovation and Research & Development protected we can better prepare ourselves for the future evolution and facilitate the necessary means of advancement from a connectivity and cybersecurity perspective. From the procurement side once there becomes some technical breakthrough that everyone will need either from a societal or defense perspective, the last thing you'd want is scarcity created by a foreign entity.
Lastly, South Korea and Japan are key allied countries where semis are fabricated, so keeping close ties with them is another key strategy to maintain.
A look at Networks
As I mentioned before the reason it's important to control the manufacturing of semis and related products is to ensure the quality and security of the product but also to eliminate a dependency from competitor sourced procurement. Although, how can it be guaranteed when you are receiving a product that comes from a place where an adversary could have the control? The rival obviously wants every advantage it can gain, and although its more easy to perform within software, what about from the hardware side?
Now you've obviously seen that there will be hardware vulnerabilities even when sourcing from reliable OEMs. For example we saw the Intel vulnerabilities like metldown - some of which can't be patched I'll add. So if an attacker could somehow engineer their way into the architecture, chip/board, firmware etc. it could possibly create exploitable conditions like back doors, man-in-the-middle capabilities, information disclosure, or denial of service conditions. I argue that its easier to accomplish this with foreign entities or in emerging markets like Malaysia where chip making is on the rise. The reason for emerging markets is because they likely have less mature security controls, and possibly easier to bribe or influence due to their current economic situation vs more industrialized countries.
To continue - In my research I came across this article about exploiting Broadcom WiFi chips from Googles project zero, Much of the article is not my area of expertise, but I understood how it illustrates the types of capabilities that an attacker can gain, see an extract below:
"The first blog post will focus on exploring the Wi-Fi SoC itself; we’ll discover and exploit vulnerabilities which will allow us to remotely gain code execution on the chip. In the second blog post, we’ll further elevate our privileges from the SoC into the the operating system’s kernel. Chaining the two together, we’ll demonstrate full device takeover by Wi-Fi proximity alone, requiring no user interaction."
To sum this up an attacker could chain hardware vulnerabilities to take over a device. The exploits were patched by the vendor in a timely fashion though according to the article. Nevertheless this illustrates the future doomsday scenario in 2030 and beyond of supply chain hacks where if somehow there was a zero-day exploit for a mobile WiFi chip used by millions of peoples' mobile phones, the privacy and security ramifications would be significant.
Now my understanding and maybe yours is that the big guys mostly take security of the R&D to manufacturing process serious with vigorous quality control and assurance (QA/QC), so it would theoretically be difficult to inject something like that into the supply chain. Not that large Nation states don't have the motivation or budgets to accomplish that. It does show we still need to have high quality software and design to avoid these flaws.
Speaking of Broadcom, on the network side I'm sure you've heard about their Trident and Tomahawk chips also known as ASICs used in many network switches. They've really cornered the market - big name vendors like Arista use their chips in their switches (and they're used in white box switches a lot), Arista supplies thousands of switches to the big cloud providers.
Think about if there was some unpatachable hardware forwarding vulnerability for these switches, what if a crafted packet could cause a denial of service condition? The attacker would execute order 66 and quickly the network would be brought down - then consider what if that network facilitates a critical global service.
Additionally examine how a single company is slowly monopolizing the commodity network chip space, they make the decisions and can guide the industry as they see fit, I see this as a problem.
To dive further into this concept I want to bring up the outage Century Link (now Lumen) had 2 years ago to this month. The outage brought down large parts of their network including DSL and e911 services. This outage effected a million people at least. Here is the full FCC root cause analysis if interested. To summarize the devices in question are Layer 1/Layer 2 optical switches from Infinera in the Lumen network.
I will paste the important excerpts below (i bolded for emphasis):
"The nodes in the affected network possess a proprietary internode management channel. This proprietary management channel is designed to allow for very fast, automatic rerouting of traffic to avoid a loss of traffic during a failure in the network. It does this by enabling line modules to send packets directly to other connected nodes without receiving network management instructions about how to route traffic. To prevent management instructions from being sent to other nodes, the proprietary management channel has a filter that prevents packets that are 64 bytes or fewer from using the channel. As the supplier of these nodes, Infinera provides its customers – including CenturyLink in this case – with the proprietary management channel enabled by default. CenturyLink was aware of the channel but neither configured nor used it."
"In the early morning of December 27, 2018, a switching module in CenturyLink’s Denver, Colorado node spontaneously generated four malformed management packets. Malformed packets are packets that, while not rare, are not typically generated on a network and are usually discarded immediately due to characteristics that indicate that the packets are invalid. In this instance, the malformed packets included fragments of valid network management packets that are typically generated. Each malformed packet shared four attributes that contributed to the outage: 1) a broadcast destination address, meaning that the packet was directed to be sent to all connected devices; 2) a valid header and valid checksum; 3) no expiration time, meaning that the packet would not be dropped for being created too long ago; and 4) a size larger than 64 bytes. CenturyLink and Infinera state that, despite an internal investigation, they do not know how or why the malformed packets were generated. "
"Due to the packets’ broadcast destination address, the malformed network management packets were delivered to all connected nodes. Consequently, each subsequent node receiving the packet retransmitted the packet to all its connected nodes, including the node where the malformed packets originated. Each connected node continued to retransmit the malformed packets across the proprietary management channel to each node with which it connected because the packets appeared valid and did not have an expiration time. This process repeated indefinitely."
So to translate this perfect storm. First the management channel was turned on and not configured, and then 4 packets spontaneously were generated with the perfect attributes to be propagated throughout the entire network which eventually caused resource exhaustion on all the network nodes causing the outage. To stop it they had to bring down and reboot devices to stop the loop and propagation.
In my opinion this was probably an attack perpetrated by a malicious actor with targeted knowledge and capability in likely the most efficient DoS attack on a large network ever executed. But that's my opinion and I have no proof to back that up.
Its probably clear to you by now why I bring up this incident. Because think if an optical SFP module was somehow compromised along the supply process with a vulnerability that mimicked this outage's behavior. Then millions of these modules were purchased and implemented worldwide or in a large critical network. It would only take a few crafted packets to then cause DoS conditions on a potentially wide scale. Next level 2030 sabotage?
This is why its important to consider the importance of telecommunications equipment and the need for it to be brought in domestically and protected, or at least more controlled from a supply chain perspective. Networks are vital to the nation's communications infrastructure security and should be treated as such.
Other examples
There was also the Super Micro story where the allegation was that China had somehow compromised SM to get a tiny chip onto their boards which was back door capable thereby providing access into the likes of Amazon and Apple icloud etc. It was very controversial when it came out but it was pretty much disproved to actually have happened. Super Micro defended their process with how much QA they do and how hard it could be to actually do that type of attack. However it doesn't leave it out of scope, and if it was executed, likely no one outside the factory would be able to detect it, which leads to the next example - interception.
We saw the NSA router and server interception for surveillance purposes. It in my thought this probably still going on from both sides, which is one of the motivations for this post. This illustrates the importance of supply chain integrity. Moreover, from China's and others perspective it is also in their best interest to keep assembly in country for their own security.
Conclusion
I believe China has played the supply and manufacturing space ingeniously with foresight beginning in the 90s, guiding the USA freight train by laying down track in the direction of their advantage so to speak. Although the US still leads currently in intellectual property, if "the west" doesn't heed to the current trend and no new action is taken by private businesses or from a national strategic standpoint then I think the United States will eventually be surpassed overseas when it comes to control of the technological supply chain.
Businesses need to recognize that the compromise of their product or intellectual property will hurt the bottom line and consumer confidence in the long run. Its in their best interest to ensure the security and integrity of the semiconductor used in their products. Investment must be made to move manufacturing out of adversarial territories. Likely the argument will be that moving to the US will be too costly, therefore emerging markets will be chosen, however it should be understood that the danger could still exist even after that, so key allied countries should be chosen if absolutely necessary.
From the national security perspective its in the interest of the survival of the future 21st century nation to keep up technologically and to look to lead in that arena. For procurement, supply chains must be controlled in order to maintain an advantage and to avoid lock out in the 11th hour. Electronics from consumer to defense utilize semiconductors, with the way things are moving the need for semiconductors only grows every day; therefore policy and leadership must reflect the significance of controlling the cyber supply chain. Its clear the "control the chips strategy" will continue to play out over the next decade illustrated by competing interests and investment trends.
The technical examples of vulnerabilities and compromises in hardware are generally available to read about. Its important to note how wide spread these problems could propagate in large networks or in popular devices purchased by many people. The ramifications can not be over stated.
I didn't cover software supply chains in this post, as that is a whole other topic entirely. With the recent Solarwinds breach I want to cover that in a future post. I leave you with a quote of my own.
In the past, human power was the driving factor behind the nation, evolving over centuries, measuring its power for miles. Although much of that still remains, the cyber age has revolutionized the way humans clash, consume, create, and communicate, with the integrated circuit as the driving factor. Now that payloads are delivered at the speed of light and integral societal technology dependent on microscopic silicon. The 21st century nation's fate will be decided in milliseconds and nanometers.
Thank you
Would you like to know more?
Dedicated to my grandfather who made a living in those early years in the Northern California semi industry to then manufacturing chips in Santa Ana, CA until 2017.
