I am not a professional in the field of information security, my area of interest — it is high-performance computing systems. The theme of the IB I came by chance, and that is what will be discussed further. I think this true story is much better highlight the problems associated with hardware virtualization, rather than a dry presentation of facts.
Even before the official announcement of the new Intel processors with support for hardware virtualization (in early 2007), I decided to use these chips to create a unified computing system based on multiple servers, which would be for the OS and applications of single computational unit with SMP-architecture. It required to write compact hypervisor with non functional, the main feature of which would not be a single computing resource sharing settings between different operating systems, and vice versa, the pooling of resources of several computers in a single complex, which would be run by the same OS. In this case, the OS does not even have to guess that does not deal with a single system and multiple servers. Hardware virtualization provides such an opportunity, although not originally intended for this purpose. Actually, the system in which the equipment is used to dlyavysokoproizvoditelnyh virtualization computing, is not created until now, and only at that time I did was a pioneer in this field.
Hypervisor for this problem, of course, was written from scratch. It is essential to run the OS was already on a virtualized platform, with the first teams to boot OS all work in a virtual environment. This had to virtualize the real model and all operating modes of the processor and run just after initialization virtualization platform to boot.
As virtualization solution for this purpose turned and looked precarious as a fully autonomous compact program module (code size up to 40-60 KB), the language somehow did not turn the hypervisor call it, and I began to use the term "giperdrayver" because it more accurately conveyed the essence of the functional purpose of the system. Standard equipment with hardware virtualization at the time was not yet, however, thanks to the cooperation with the company "Kraftway" I had access to pre-production models of processors and motherboards with support for virtualization that are not yet officially released (so-called samples, that the firm provides to its Intel kindly business partners). Therefore, the work began to boil at this "semplovom" equipment.
The model was built, giperdrayver written, everything worked as planned. I must say that at the time the equipment was very virtualization "raw", because of what she once refused to work as written in the documentation. We had to deal with literally every assembly instructions and the commands themselves to equipment virtualizatsiipisat a native, because then there was no compiler supports virtualization commands.
I was proud of the results, felt almost master of virtual worlds … but my euphoria did not last long, only a month. By the time I was about to layout based servers with hardware virtualization, the first production samples of which had just appeared, but the layout was not working.
I began to understand and realize that my system hangs when executing hardware virtualization. One had the impression that they either do not work or work outside the box somehow. Freeze occurred only during the virtualization hardware in real mode, but if my system is started from protected mode, after loading the OS, everything was normal.
Professionals know that the first revision hardware virtualization Intel did not support the CPU in real mode. This required an additional layer large enough to emulate a virtual x86. Since giperdrayver run before the operating system so that it can fully believe in the new virtual configuration, a small piece of the OS boot code run in a real mode of the processor. The system was dying just to emulate the real-mode handlers in giperdrayvere. At first I thought it was a mistake somewhere, something is not understood, something forgotten. Checked every last bit in the code, no errors found and began to sin no longer for themselves but for colleagues from abroad.
The first step is to replace the processor, but it did not help. On motherboards while hardware virtualization was only in the BIOS, where it is initialized during the power on the server, so I started to compare the bios in the motherboard (the same type of cards with samples) — all come together to the byte number of the BIOS. I fell into a stupor, and not knowing what to do, used a last resort — "at random." Why have not I done, not thinking, just combining, and ultimately stupid to get the bios from the official Intel website and rewrote them again in the motherboard, and then it worked …
To my surprise there was no limit: BIOS number was the same, the images of the BIOS match bytes, but for some reason the serial motherboard earned only when I filled them in the same BIOS, taken Site Intel. Hence, the reason for all the same on the motherboard? But the only difference was in the labeling: the samples were written Assembled Canada, and the serial boards — Assembled China. It became clear that the boards from China contain additional software modules, sewn in the BIOS, and the standard analysis software, these modules have not seen. Apparently, they also worked with hardware virtualization and thus were able to hide the true contents of the BIOS. Became clear and the reason to hang my giperdrayvera these Chinese boards: two software systems working simultaneously on the same hardware virtualization, which is not allowed to share their resources. I wanted to deal with this malicious BIOS, and without a second thought about the "tabs", "backdoor", "undocumented features", was just of academic interest, and nothing more.
It must be said that, in parallel with the introduction of Intel hardware virtualization radically updated chipset. This chipset, received number 5000h, is available in several versions so far. The south bridge of the chipset, 631xESB/632xESB I / O Controller Hub, which is connected to the flash chip with BIOS, virtually unchanged since 2007, is available and is used as the base for almost all chip in a two-socket server performance. I downloaded the datasheet for the south bridge, read the description and was blown away. It turns out that this new south bridge connects three devices flash memory: the first is the standard BIOS, the second program of the CPU allocated for the network controller, and the third is designed for integrated into the South Bridge block the Navy.
Block Management System (Navy) — is based remote control unit and computer monitoring. It is indispensable for large server rooms, where, because of noise, heat and drafts impossible long to be.
The fact that the Navy units have their own processor and thus, flash memory for its programs, of course, not new, but until now, such a processor and memory put for a fee, which is connected to the motherboard: you want — to put, you do not want — do not put. Now Intel has implemented these components in Southbridge, in fact, connect this unit to the system bus and did not use a dedicated network link (as provided for standard IPMI, describing the function block IUD) for network service, and the service tunnel all network traffic to the core network adapters. Then I found out from the documentation that the program on the flash chip block Navy encrypted as they are opened for a special hardware cryptographic module, also integrated into the South Bridge. Such blocks Navy I had not come across.
Not to be unfounded, I cite an excerpt from the documentation on this South Bridge:
ARC4 processor working at 62.5 MHz speed.
Interface to both LAN ports of Intel ® 631xESB/632xESB I / O Controller Hub allowing direct connection to the net and access to all LAN registers.
Cryptographic module, supporting AES and RC4 encryption algorithms and SHA1 and MD5 authentication algorithms.
Secured mechanism for loadable Regulated FW.
The use of foreign cryptographic key length 40 bits is prohibited by law in Russia, and here — please! — Each server Intel kriptomodul with unknown keys of 256 bits. Moreover, these keys are used to encrypt the programs coded in the chip of the motherboard at the production stage.
It turns out that the Navy units in Russia on servers Intel, which are composed of chipset 5000h should be disabled. However, these units, however, are always in working order, even if the computer is disconnected (for the operation of naval duty enough voltage, that is inserted into the outlet cord server). It all seemed to me at the time of secondary importance, since it was necessary to begin to figure out which of the flash chips was a software module that works with hardware virtualization and hindering my giperdrayveru, and I began to experiment with firmware.
After reading the documentation, I tensed, and when he found that the efficiency giperdrayvera restored just after flashing the flash chip block navy, not even surprised. Understand further without special stands was impossible, as cryptography completely overlap Reversible code for the Navy. Documentation on the internal architecture of the integrated Navy I have not found in the datasheet on South Bridge Intel described only interface registers to control this unit using standard access methods, resulting in a classic "black box."
Collection of facts and raised the Troubling paranoid thoughts in the style spy detectives. These facts clearly states the following:
In the new series server boards based on Intel 5000 chipset have programs stitched in flash memory of the Navy and executed on the CPU, these programs work with hardware virtualization CPU.
Images of flash memory with Intel's website do not contain the software modules, thus preventing me software modules were illegally flashed the motherboard to the production stage.
Flash memory block contains encrypted Navy program modules that are impossible to collect and fill the flash memory without the knowledge of the encryption keys, therefore, the one who put these illegal software modules, known encryption keys, that is, in fact, had access to classified information.
I informed management "Kraftway" problem with the firmware flash memory block Navy and questionable in terms of legislation the situation with the new chipsets, Intel, and received quite the expected response in the style of "no haze, hurts business." I had to calm down, as against the employer does not trample.
His hands were tied, but "my thoughts, my horses" did not give me no rest, it is not clear why these difficulties and how it is done. If you have the opportunity to host your own software in memory of the Navy, why do you all this trouble with the CPU? Reasonable cause could be just what is required to control the problem under the context of the current computing on the CPU. Obviously, to keep track of the processed information to the main computer system, using only a peripheral low-speed processor with a frequency of 60 MHz, is impossible. Thus, it seems, the problem of this illegal system was renting the information processed in the main computer installation by means of hardware virtualization. Remotely control all the illegal system is obviously more to the CPU block navy, as it has its own independent access to the network adapters on the motherboard and its own MAC and IP-address. "How-to" have a more academic nature, as someone has managed to create a hypervisor, able to share resources with other hardware virtualization hypervisor and doing it correctly for all modes, except for the real mode of the CPU. Now these systems is no surprise, but then, five years ago, they were seen as a miracle, in addition, the emulation speed hit — a software emulated host without significant loss in performance was not possible.
To explain the need to go a little further into the theory. Architecture virtualization Intel and AMD does not constitute the platform of several hypervisors, but started the first hypervisor can emulate for hypervisors that run after, work on real hardware virtualization. In this case, all hypervisors running after the first run in an emulated environment host. This principle I call the "right of the first night." It can be easily implemented with a special handler to the root host, with the regime of the problem will not change, and secondary hosts hypervisors will be run as a root problem for the host. Emulation is not difficult to organize, but with the performance problems. Hardware virtualization works primarily with power VMCB (VMCS), the host program constantly refer to this unit, and for each such treatment requires 0.4-0.7 ms. Hide a software emulation for host virtualization Intel virtually impossible, too many teams have emulated virtualization software through outlets in the root host, rather than running them on real hardware.
Tell us a little about the differences between virtualization architecture. System hardware virtualization from Intel and AMD are quite unlike each other. The main architectural feature of these systems is in operation, the host. The system works with AMD host disabled hardware virtualization, ie, its programs are executed on a real CPU. Virtualization secondary host systems from AMD Virtualization requires only team VMRUN (we can assume that other teams do not.) Managed VMCB-block in the architecture of AMD occurs through the normal commands to access memory that allows you to control by a secondary host only execute commands VMRUN and touch up if necessary VMCB-block before the actual entrance to the mode problem. Lengthen the event loop twice is still possible, and on the platform AMD is a viable emulation. The system virtualization Intel is more complicated. To access the VMCB-unit uses special commands and VMREAD VMLOAD, must-virtualized. Usually handlers host dozens, if not hundreds of times to the fields VMCB-block, and each operation must emulate. In this case, notice that the speed drops by an order, it is very inefficient.
It became clear that to emulate the unknown colleagues used a more efficient mechanism. And clues as to which one, I found the documentation. Host at Intel itself is a virtual environment that is nothing, in fact, in this respect is no different from the environment of the task and simply controlled by another VMCB (see diagram).
In addition, the documentation describes the concept of "dual monitor" virtualization SMM-mode (system management), when in fact, the two hosts are active at once, and therefore two VMSB block, and host virtualized system management mode, controls the main host as a problem but only to the call of system management interrupt.
This body of circumstantial evidence suggests that hardware virtualization Intel, probably has a control mechanism of the secondary hosts, managed by the root host, though it is not described anywhere. Also, my system is exactly what worked, and no other explanation is almost imperceptible actions hypervisor root I still do not. It became even more interesting: it looks like someone had access to these undocumented features and use them in practice.
About six months before the end of the collaboration with "Kraftveem" I took the position of a passive observer, continuing still regularly run their system on the new series of motherboards shipments from China and new samples. Sample-all continued to work steadily. When I moved to the Chinese dress, the system appeared more and more miracles. It seemed that colleagues from abroad actively improve the operation of its root hypervisor. Last suspicious party boards behaved almost normally, with the first launch of my giperdrayvera restarts the system when the operating system starts, but all subsequent launches giperdrayvera OS and ran without a hitch. In the end, something happened, something I have long expected: received a new batch series motherboards, the use of which I do not hang giperdrayver. I was beginning to doubt his paranoid suspicions, but the new case has strengthened them.
It should be noted that the company actively develops Intel hardware virtualization. If the first revision of the equipment with which I started, was version 7, the described situation occurred on the 11 th revision, that is, approximately one year revision updated twice (audit reason have only odd numbers). So, on a revision with the number 11 in terms of output as host for hardware virtualization tasks significantly, whereby in VMCB-block even introduced a new control field. When were semplovye processors that revision hardware virtualization, I wanted to try out the new features in practice. I perfected giperdrayver with new opportunities of the 11th revision of hardware virtualization, set semplovy processor serial boards from China, in which everything has worked without any problems, and start debugging. New features of equipment does not show, and again I fell into prostration, sinning on semplovy processing and documentation. After a while the motherboard is needed for other tasks, and I, resuming the experiments, rearranged for secure processors with the 11th revision of the hardware virtualization in the Canadian sample. Imagine my surprise when this sample will work!
At first I thought that somewhere nakosyachil with serial card, as new outlets in the host does not have the motherboard well completely irrelevant, it is purely a processor function. To check, I rearranged semplovy processor serial boards, and all again stopped working. So, I do not nakosyachil, but the problem lay in the fact that the motherboard in some way influenced the new hardware virtualization features of the processor. Given my suspicion was asking only conclusion — illegal root host colleagues from abroad, stitched into the flash memory of the motherboard, did not know about the new revision of hardware virtualization. When that does not know the equipment starts to work, he stopped correctly pass exit from the state of the problem in my secondary host via its own event handler. Already knowing how to deal with this scourge, I filled in the serial board firmware to block the Navy from the site of Intel, have enabled the confidence that everything will work right away, and again precipitated as left hanging. This was something new.
According to my theory, illegal hypervisor brazen and confident of their invulnerability. Apparently, it was felt that the running phase of their child was not regulated, and mask software for fault bios no longer need. After a security enabled initialization code from being overwritten in the flash memory, the tab was almost neudalimoy.
Self-righteousness, I was not needed, control experiments. We had to invent their own methods for detection of hardware hypervisor. Later, however, it turned out that I had invented the wheel. The method allows to control the timing of system commands that require mandatory emulation host hypervisor. As I used the timer counter cyclical frames in hardware USB-controller, and a program written for real mode to minimize side and uncontrolled interrupts that masked the true time of executing system commands. I spent the first check for the net-based systems semplovyh motherboards from Canada.
Run time, the photo — is a conditional value, approximately corresponding cycles. Then I ran the same test on the series motherboard and found in his paranoid assumptions — cycle instruction execution significantly lengthened.
That is in the flash memory of the Navy server boards from China, produced under the label Intel, had set the stage production of undeclared software module that acts as a host hypervisor. It remains to convince the others. First I went to the Russian representative Intel. It was not difficult, as the staff of the Russian office frequently appeared in "Kraftway."
I'm told and showed, but was not sure that the technician understood. These so-called technical experts on the level of competence differ little from those of managers. However, he promised to report everything to management. I do not know whether he did it, but no response from Intel and not followed, all gone in the sand. Work in "Kraftway" by that time is over, and I started a new project in a company related to information security. The head of this company, with whom I shared my "discoveries", took my words seriously. In this regard, it was decided to leave the leadership of the Center to protect information and special communication FSB. This structure is composed of the FSB is engaged in providing security in the country and regulates the activity of government and commercial organizations that are related to information protection. It also regulates the measures for the protection of information for government agencies and businesses that handle sensitive and confidential information. Firm in which I was then working, maintained official contacts with the Centre to certify and license their commercial projects, so to arrange a meeting at the professional level was quite easy. It was assumed that the experts of the Center will report its opinion to the leadership, and if after this guide considers it necessary to listen to us, the next step will be a meeting at a higher level.
The meeting took place, I told and showed everything that we found out, and then demonstrated the presence of illegal software module on the board examples from Canada and China. By the way, when I first heard and professional term "bookmark" indicating such a module. When the conversation turned pro Navy, in the eyes of his colleagues from the Centre appeared misunderstanding. Had to be an educational program. In the process, it became clear that they were not even aware of the existence of a special chip in the south bridge with access to the network adapter, and the presence of the Navy in the block cryptographic module that violates Russian law. Finally, we heard a sudden, that this threat model has been studied in relation to a complex of measures to counter, and generally, we are not afraid of bookmarks, because our system does not have Internet access.
Further inquiries or to nothing, everything comes down to privacy, like, we are smart and supergramotnye, and you know about what is not allowed. However, I strongly doubt their technical competence, as they simply do not understand much of what I said and pointed. Parted on the fact that they will report to their superiors, and only it will decide on further action.
Later I learned that this "secret method" detection program host. And learned quite by accident, during the negotiations at the firm — the Center licensee authorized to check the bios in the bookmark. The technicians of the company conducting research bios, said that its software modules that use hardware virtualization, we must seek the signatures of virtualization commands. Indeed, the command processor for hardware virtualization contain three or four bytes in the code, but who said that the code they find in the clear on the flash chip? As they scan the code in memory, if the memory area reserved hardware from view?
In general, the outcome of the first meeting left a bad feeling, and I'm in the dark mood expectation. Month and a half, we were invited to the Center itself is information security and special communications that we have demonstrated that we have found a bookmark. This time to listen to us got together not ordinary employees, and managers and leading specialists (at least, so they introduced). The meeting turned into a lecture, I listened almost three hours, it was clear that the first time they hear what I tell them. I have listed a new vulnerability x86 platform, showed the tab and told her to detect and answered many questions. At the end of our thanks, said that the theme should be developed in the framework of special research, and that we parted.
Euphoria vanished when unofficially reached us information that we just did not want to believe. However, this did not dampen my desire to prove their case. As it seemed to me, the decision was on the surface: it was necessary to write himself a plug-in tab. I would not have put the tab in the flash memory of the Navy, but in the main BIOS to push it I could have. I decided to equip its security hypervisor module for masking in memory and on-chip flash, and block entry to the flash chip, which will be placed the code tab, and then remove it happens just by watering it and reprogram the BIOS on an external programmer.
It remained only to decide on the "evil" features that should perform hypervisor. I remembered the statement of one of the experts of the FSB that they are not afraid favorites because their system is disconnected from the global network. But information from the outside world must somehow get into these protected local networks, at least through one-off discs. Thus, I came to the obvious conclusion and decided to analyze incoming information tab giperdrayvera means to implement, as it were doomsday weapon that is used to kill the tab of a computer system to an external command, passing it through the inlet flow of information, steganography.
Scan information flow secretly, without losing speed, the teeth only hardware virtualization. At which point the scan is also clear: the FIFO I / O disk systems and network adapter. Scan I / O buffers — a trifling task for hardware virtualization. No sooner said than done! This giperdrayver approximately 20KB in size has been registered in the BIOS of the motherboard and is equipped with an anti-detection. He blocked efforts to get overwritten when updating the BIOS and perform a single function: Resettable flash BIOS chip for receiving a command to destroy. The team itself for ease of implementation has been sewn into a text file DOC-format tags in the setting.
When all was ready, the management of the company came out again with a proposal for the FSB to see the work of our own bookmarks and make sure that virtualization technologies are a real threat. But look at our favorites in no one wanted, with the top team arrived (I did not know whose it was an order) with us no longer communicate. The main fighters for the security did not want to listen to us. Then, have almost nothing on hoping actually to clear my conscience, we tried to convey information to the user about the problem of information security. We got in touch with "Gazprom" to inform the specialists on contemporary threats to distributed process control systems.
Managed to arrange a meeting with the guidance of the corporate protection and management of complex security systems of the corporation. For them have been prepared more visual version of bookmarks with simplified command interface. Bookmark activated after downloading to a computer text file, the contents of which consisted of two words — "Gazprom" and "stop" — arranged in an arbitrary order. The computer died, but not immediately, but after a delay of five minutes. Of course, one could make the delay and at night, but then we would have missed during the time allotted for the demonstration. The staff of "Gazprom" lamented the low level of information security, and said it was not their business, as they are guided by the requirements and standards that sets the FSB. Circle, it became clear that this monolithic system "irresponsible information" does not penetrate.
For three and a half years that have passed since then, I have never heard anyone talking about the hardware virtualization as an instrument of penetration into the target system. Paradox? I do not think so. The specificity of the topic that we learn only about the failed technology. On technologies that are not found, we do not know, and their authors, of course, silent.
Keep in mind that a reliable placement of bookmarks in the BIOS is only possible in the factory. In the conditions it will have to rely on a certain model of the motherboard, and these options are not very interesting to hackers. They need the mass, they work, they say, "on the area." However, there are those who attack sighting "is a sniper." Technology placing bookmarks in the BIOS, so even with the activation apparatus virtualization, which allows effectively hide them — this, of course, a convenient tool for these "snipers". One time they almost caught up, almost by accident. I think now it will not be able to do, and catching, as you probably understand, no one.
Link to the original I can not open infu take from here: Airbase.ru
Materials on the subject:
Microsoft programs disappear from computers
On iPhone tracking software is installed
The movement "Occupy Flash» convinces delete a multimedia plugin
Voice recognition system for Android
Gadgets Apple will recognize faces
Russian national operating system will appear in 2011