When IBM sent Zaphiris Christidis on an international assignment to China to manage the supercomputing division for climate and environment applications, he would be part of an over 50,000 strong force in China managing weather forecasting and research. While some were part of IBM or other technology vendor teams, it was striking to him how many people in the country were devoted to operational weather forecasting, not to mention how complex each of the regional centers that radiated off the main Beijing sites were.
It was surprising to Christidis, but not necessarily unexpected. After all, with such diverse regions with their own weather prediction needs in a country where floods and other climate trends can have a significant impact on the economy, there is always a return on investment for weather forecasting. And what all this means is that the brute force of a large fleet of tightly coupled, multi-million dollar supercomputers distributed throughout the mainland is an expensive, necessary operation. It is one that the China Meteorological Administration (CMA) orchestrates to a great degree from its center inside a massive government and education complex in Beijing.
It was in this complex that Christidis set about his initial work, which was primarily focused on tuning the high performance computing clusters for the various weather applications used by CMA and other groups. In doing so, he became something of a weather supercomputer ambassador, informing the Chinese and the United States via IBM on the software and architectural requirements of large systems.
He oversaw the installation and use of some impressive machines—from one of the first RS/6000 SP2 parallel systems, then through a succession of other large-scale supercomputers scattered at various sites in China. Christidis remembers well the installation of systems that made big waves internationally due to their capability, including 21 teraflops supercomputer based on the Power architecture (a massive beast for the time, in 1994) and more recently, a 1.65 petaflops Power7 based system for CMA. One thing that can be noted about some of these machines, among the largest for their time in all of China, was that they were all from IBM. There were (and still are) others from Chinese vendors, including ShenWei and Galaxy.
As his stay in China extended and the machines built by IBM for Chinese weather modeling increased in size and scope, Christidis found that his position had another element.
“You see,” he explains in reference to last and fastest supercomputer doing weather operations for CMA., “for the U.S. to sell supercomputers to China, they had to get an export license to install the system. As I was in China working on these machines at the time, they assigned me, sort of like a supercomputer policeman, to the system.” Christidis explained how his investigations were collected into quarterly reports on how the system was being used with the intention, of course, being that it was exclusively for weather forecasting.
When the U.S government placed these demands, it required that a U.S. citizen check the logs. In other words, every time an application runs, there is a log entry that indicates things like how long the job ran, how many codes were used, the name of the executable, and so forth.
“Since it’s not like they are naming the files NUCLEAR_EXPLOSION.exe, that meant I had to go through all of this,” Christidis tells The Next Platform. “So, every three months, I spent my time doing statistics from all the logs, putting the applications in different bins, and writing reports and sending them to IBM offices in Washington D.C. so they could submit those then to the government.”
Sometimes Christidis would get a visit at the center from the U.S. Embassy and once in a while, he says, the Secretary of Defense or other people from the Department of Defense, to talk about his sense about what the systems were being used for. “I had all the logs from everywhere. I had all the keys. I would compile the logs according to users and applications and no, there was never a time when anything looked remotely suspicious. I would tell them whether or not things were okay, and they were. If anyone would know, I would know.”
We should not be surprised at this, either.
If the Chinese government wanted supercomputers to work on classified missions, it would certainly not buy an IBM machine—or a supercomputer from a non-Chinese company. Organizations do have choices inside of China, although with very few exceptions, many Chinese OEMs are using Intel chips, just as they do elsewhere in the world. And with Intel chips being placed under a blockade for Chinese supercomputers (or at least subject to extra investigation, depending on who is telling the story about what the block really means), it makes it harder all around. This means that the U.S government is, whether on purpose or not, pushing China to hasten its development of more homegrown architectures and fabs. (But that’s another story for another time.)
Another point to make here is the obvious correlation that the very thing we worry about Chinese tech manufacturers for imported U.S. goods is done to the Chinese. But Christidis is careful to note that this should not be particularly striking either. And indeed, for any IBM system over a certain performance level, there is likely to be ongoing monitoring.
On that note, another interesting bit of information emerged from our chat. While the performance specs for big supercomputers that require a “spy” are not surprising, the way the “threat” of a system is assigned has an interesting variation. Even if there was a machine ordered from a U.S. OEM with four million nodes and incredible compute capability, the interconnect determines to what degree it is red-flagged. And here is where things get weird. Because the real flag is that it is a proprietary interconnect—meaning any IBM or Cray supercomputer, for example, would be subject to export restrictions simply due to the interconnect. But say someone puts together that hypothetical, unlikely, four million node cluster to deliver to China but hooks it together with an off-the-shelf interconnect—supposedly it will get through.
Okay, maybe not to the tune of four million nodes. However, an off-the-shelf interconnect, in the U.S. government’s mind, is not just Ethernet—it also includes InfiniBand. As Christidis shared when asked how this could be possible, “it’s because these policies are developed by people who don’t have any idea how computations actually work.” And for anyone outside the bubble, the background here is that one can certainly build a very high performance, tightly coupled supercomputer that is blazingly fast (and supposedly possibly dangerous) with 10 or 40 Gigabit Ethernet—and certainly with InfiniBand.
To confirm this, we talked with insiders at Cray, who pointed us toward a recent supercomputer deal in Hong Kong and who said that this was certainly the case with interconnect restrictions two or three years ago, but that is changing. Either way, Cray says that this is not a reason for their lack of a major presence there.
As yet another interesting aside, the way the U.S. government looks at the capability of supercomputing systems being exported to China is measured in a unit called “MegaTops” or MTOPS. He said that to go over the limit, a machine had to be at the 2 MTOPs level to warrant restrictions. The petaflopper they installed for CMA was around 40 MTOPS. That, coupled with the custom interconnect from IBM, meant the government watched this machine closely—and as one might imagine, not just on the weather supercomputer front.
As just a speculative aside here, the dominant notion about how the next-generation Intel chips might have been blocked from going to Chinese supercomputers is because of the computer power they promise. But with this interconnect issue in mind, if it is indeed still a factor, OmniPath is not Infiniband—it too is a custom interconnect.
Naturally, if the U.S. government is going to provide Very Big Supercomputers to China, it wants to know what is happening on them—hence the fresh round of export restrictions for Intel. Meanwhile, if we are talking about such machines, recall that the top system on the planet (by a long shot) is the Tianhe-2 system, which is comprised of both U.S. and Chinese-developed technology. It is hard to guess what the monitoring might be on a super like this—or how that would happen given that the front end system is comprised of homegrown gear (with a tailored variant of Linux, no less).
Christidis could not remark on the Tianhe-2 machine. His supercomputer police days behind him, he fell on the Lenovo side of the IBM split, where he is back to tuning weather models, including WRF, for various architectures. Oddly enough, the reason The Next Platform talked to him in the first place was because of a striking paper in which he showed that despite big cost, configuration, and other differences between IBM systems, the de facto standard software in weather modeling, WRF, runs almost exactly the same, no matter which architecture is chosen. We had initially been surprised he published that with his Lenovo and IBM affiliations. He is no longer in China and while it sounds like it was a remarkable experience, is too busy refining a follow-on to the above cited research to present at ISC in Frankfurt where we will, doubtlessly, buy him a beer.
This article addresses the pretext of mis-use of a large computer in China; but doesn’t say anything about the possibility of clandestine reverse engineering. It would be interesting to know if that was addressed also or if there was no reason for concern about that.