Многие читатели бложика являются практиками виртуализации, кто-то даже пытается заглянуть под капот и посмотреть как это устроено.
Сегодня я подобрал несколько актуальных ссылок на статьи, описывающие как работает на уровне процессора вложенная виртуализация, как самому написать гипервизор, зачем нужны инструкции процессора для аппаратной реализации функций виртуализации.
Virtualization Internals
- Virtualization Internals Part 1 – Intro to Virtualization
- Virtualization Internals Part 2 – VMWare and Full Virtualization using Binary Translation
- Virtualization Internals Part 3 – Xen and Paravirtualization
- Virtualization Internals Part 4 – QEMU
Как написать гипервизор самому:
- Hypervisor From Scratch – Part 1: Basic Concepts & Configure Testing Environment
- Hypervisor From Scratch – Part 2: Entering VMX Operation
- Hypervisor From Scratch – Part 3: Setting up Our First Virtual Machine
- Hypervisor From Scratch – Part 4: Address Translation Using Extended Page Table (EPT)
- Hypervisor From Scratch – Part 5: Setting up VMCS & Running Guest Code
- Hypervisor From Scratch – Part 6: Virtualizing An Already Running System
- Hypervisor From Scratch – Part 7: Using EPT & Page-Level Monitoring Features
- Hypervisor From Scratch – Part 8: How To Do Magic With Hypervisor!
Вложенная виртуализация:
- Out of Hypervisor (OoH): When Nested Virtualization Becomes Practical
- Using SMT to Accelerate Nested Virtualization
- The Turtles Project: Design and Implementation of Nested Virtualization – с чего начиналось
- Improving KVM x86 Nested Virtualization
Для понимая поддержки аппаратной виртуализации в VMware ESXi есть довольно полезная, хоть и давно не обновлявшаяся таблица ESX(i) Support of Intel VT-x and AMD-V Features. Вроде, все функции были реализованы в 6.5/6.7:
Чтобы понять какие функции используют ваши виртуальные машины, достаточно открыть vmware.log после старта и найти раздел Host VT-x Capabilities:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 |
2022-10-09T04:47:55.933Z In(05) vmx - Log for VMware ESX pid=2177955 version=7.0.3 build=build-20036589 option=Release ... 2022-10-09T04:47:55.941Z In(05) vmx - FeatureCompat: No EVC masks. 2022-10-09T04:47:55.941Z In(05) vmx - hostCPUID vendor: GenuineIntel 2022-10-09T04:47:55.941Z In(05) vmx - hostCPUID family: 0x6 model: 0x6a stepping: 0x6 2022-10-09T04:47:55.941Z In(05) vmx - hostCPUID codename: Ice Lake SP 2022-10-09T04:47:55.941Z In(05) vmx - hostCPUID name: Intel(R) Xeon(R) Gold 6354 CPU @ 3.00GHz ... virtualHW.version = "19" ... 2022-10-09T04:47:56.046Z In(05) vmx - Host VT-x Capabilities: 2022-10-09T04:47:56.046Z In(05) vmx - Basic VMX Information (0x00da050000000013) 2022-10-09T04:47:56.046Z In(05) vmx - VMCS revision ID 19 2022-10-09T04:47:56.046Z In(05) vmx - VMCS region length 1280 2022-10-09T04:47:56.046Z In(05) vmx - VMX physical-address width natural 2022-10-09T04:47:56.046Z In(05) vmx - SMM dual-monitor mode yes 2022-10-09T04:47:56.046Z In(05) vmx - VMCS memory type WB 2022-10-09T04:47:56.046Z In(05) vmx - Advanced INS/OUTS info yes 2022-10-09T04:47:56.046Z In(05) vmx - True VMX MSRs yes 2022-10-09T04:47:56.046Z In(05) vmx - Exception Injection without error code no 2022-10-09T04:47:56.046Z In(05) vmx - True Pin-Based VM-Execution Controls (0x000000ff00000016) 2022-10-09T04:47:56.046Z In(05) vmx - External-interrupt exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - NMI exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Virtual NMIs {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Activate VMX-preemption timer {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Process posted interrupts {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - True Primary Processor-Based VM-Execution Controls (0xfff9fffe04006172) 2022-10-09T04:47:56.046Z In(05) vmx - Interrupt-window exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Use TSC offsetting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - HLT exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - INVLPG exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - MWAIT exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - RDPMC exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - RDTSC exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - CR3-load exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - CR3-store exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Activate tertiary controls { 0 } 2022-10-09T04:47:56.046Z In(05) vmx - CR8-load exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - CR8-store exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Use TPR shadow {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - NMI-window exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - MOV-DR exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Unconditional I/O exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Use I/O bitmaps {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Monitor trap flag {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Use MSR bitmaps {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - MONITOR exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - PAUSE exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Activate secondary controls {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Secondary Processor-Based VM-Execution Controls (0x4bdfffff00000000) 2022-10-09T04:47:56.046Z In(05) vmx - Virtualize APIC accesses {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Enable EPT {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Descriptor-table exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Enable RDTSCP {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Virtualize x2APIC mode {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Enable VPID {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - WBINVD exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Unrestricted guest {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - APIC-register virtualization {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Virtual-interrupt delivery {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - PAUSE-loop exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - RDRAND exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Enable INVPCID {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Enable VM Functions {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Use VMCS shadowing {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - ENCLS exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - RDSEED exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Enable PML {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - EPT-violation #VE {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Conceal VMX from PT {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Enable XSAVES/XRSTORS {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - PASID translation { 0 } 2022-10-09T04:47:56.046Z In(05) vmx - Mode-based execute control for EPT {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Sub-page write permissions for EPT {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - PT uses guest physical addresses {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Use TSC scaling {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Enable UMWAIT and TPAUSE { 0 } 2022-10-09T04:47:56.046Z In(05) vmx - Enable ENCLV in VMX non-root mode { 0 } 2022-10-09T04:47:56.046Z In(05) vmx - Enable EPC Virtualization Extensions { 0 } 2022-10-09T04:47:56.046Z In(05) vmx - Bus lock exiting {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Notification VM exits { 0 } 2022-10-09T04:47:56.046Z In(05) vmx - Tertiary Processor-Based VM-Execution Controls (0x0000000000000000) 2022-10-09T04:47:56.046Z In(05) vmx - LOADIWKEY exiting no 2022-10-09T04:47:56.046Z In(05) vmx - Enable HLAT no 2022-10-09T04:47:56.046Z In(05) vmx - Enable Paging-Write no 2022-10-09T04:47:56.046Z In(05) vmx - Enable Guest Paging Verification no 2022-10-09T04:47:56.046Z In(05) vmx - Enable IPI Virtualization no 2022-10-09T04:47:56.046Z In(05) vmx - True VM-Exit Controls (0x037fffff00036dfb) 2022-10-09T04:47:56.046Z In(05) vmx - Save debug controls {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Host address-space size {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Load IA32_PERF_GLOBAL_CTRL {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Acknowledge interrupt on exit {0,1} 2022-10-09T04:47:56.046Z In(05) vmx - Save IA32_PAT {0,1} 2022-10-09T04:47:56.047Z In(05) vmx - Load IA32_PAT {0,1} 2022-10-09T04:47:56.047Z In(05) vmx - Save IA32_EFER {0,1} 2022-10-09T04:47:56.047Z In(05) vmx - Load IA32_EFER {0,1} 2022-10-09T04:47:56.047Z In(05) vmx - Save VMX-preemption timer {0,1} 2022-10-09T04:47:56.047Z In(05) vmx - Clear IA32_BNDCFGS { 0 } 2022-10-09T04:47:56.047Z In(05) vmx - Conceal VMX from processor trace {0,1} 2022-10-09T04:47:56.047Z In(05) vmx - Clear IA32_RTIT MSR {0,1} 2022-10-09T04:47:56.047Z In(05) vmx - Clear IA32_LBR_CTL MSR { 0 } 2022-10-09T04:47:56.047Z In(05) vmx - Clear user-interrupt notification vector { 0 } 2022-10-09T04:47:56.047Z In(05) vmx - Load CET state { 0 } 2022-10-09T04:47:56.047Z In(05) vmx - Load PKRS { 0 } 2022-10-09T04:47:56.047Z In(05) vmx - True VM-Entry Controls (0x0006ffff000011fb) 2022-10-09T04:47:56.047Z In(05) vmx - Load debug controls {0,1} 2022-10-09T04:47:56.047Z In(05) vmx - IA-32e mode guest {0,1} 2022-10-09T04:47:56.047Z In(05) vmx - Entry to SMM {0,1} 2022-10-09T04:47:56.047Z In(05) vmx - Deactivate dual-monitor mode {0,1} 2022-10-09T04:47:56.047Z In(05) vmx - Load IA32_PERF_GLOBAL_CTRL {0,1} 2022-10-09T04:47:56.047Z In(05) vmx - Load IA32_PAT {0,1} 2022-10-09T04:47:56.047Z In(05) vmx - Load IA32_EFER {0,1} 2022-10-09T04:47:56.047Z In(05) vmx - Load IA32_BNDCFGS { 0 } 2022-10-09T04:47:56.047Z In(05) vmx - Conceal VMX from processor trace {0,1} 2022-10-09T04:47:56.047Z In(05) vmx - Load IA32_RTIT MSR {0,1} 2022-10-09T04:47:56.047Z In(05) vmx - Load user-interrupt notification vector { 0 } 2022-10-09T04:47:56.047Z In(05) vmx - Load CET state { 0 } 2022-10-09T04:47:56.047Z In(05) vmx - Load IA32_LBR_CTL MSR { 0 } 2022-10-09T04:47:56.047Z In(05) vmx - Load PKRS { 0 } 2022-10-09T04:47:56.047Z In(05) vmx - VPID and EPT Capabilities (0x00000f01067341c1) 2022-10-09T04:47:56.047Z In(05) vmx - R=0/W=0/X=1 yes 2022-10-09T04:47:56.047Z In(05) vmx - Page-walk length 3 yes 2022-10-09T04:47:56.047Z In(05) vmx - EPT memory type WB yes 2022-10-09T04:47:56.047Z In(05) vmx - 2MB super-page yes 2022-10-09T04:47:56.047Z In(05) vmx - 1GB super-page yes 2022-10-09T04:47:56.047Z In(05) vmx - INVEPT support yes 2022-10-09T04:47:56.047Z In(05) vmx - Access & Dirty Bits yes 2022-10-09T04:47:56.047Z In(05) vmx - Advanced VM exit information for EPT violations yes 2022-10-09T04:47:56.047Z In(05) vmx - Supervisor shadow-stack control no 2022-10-09T04:47:56.047Z In(05) vmx - Type 1 INVEPT yes 2022-10-09T04:47:56.047Z In(05) vmx - Type 2 INVEPT yes 2022-10-09T04:47:56.047Z In(05) vmx - INVVPID support yes 2022-10-09T04:47:56.047Z In(05) vmx - Type 0 INVVPID yes 2022-10-09T04:47:56.047Z In(05) vmx - Type 1 INVVPID yes 2022-10-09T04:47:56.047Z In(05) vmx - Type 2 INVVPID yes 2022-10-09T04:47:56.047Z In(05) vmx - Type 3 INVVPID yes 2022-10-09T04:47:56.047Z In(05) vmx - Miscellaneous VMX Data (0x000000007004c1e7) 2022-10-09T04:47:56.047Z In(05) vmx - TSC to preemption timer ratio 7 2022-10-09T04:47:56.047Z In(05) vmx - VM-Exit saves EFER.LMA yes 2022-10-09T04:47:56.047Z In(05) vmx - Activity State HLT yes 2022-10-09T04:47:56.047Z In(05) vmx - Activity State shutdown yes 2022-10-09T04:47:56.047Z In(05) vmx - Activity State wait-for-SIPI yes 2022-10-09T04:47:56.047Z In(05) vmx - Processor trace in VMX yes 2022-10-09T04:47:56.047Z In(05) vmx - RDMSR SMBASE MSR in SMM yes 2022-10-09T04:47:56.047Z In(05) vmx - CR3 targets supported 4 2022-10-09T04:47:56.047Z In(05) vmx - Maximum MSR list size 512 2022-10-09T04:47:56.047Z In(05) vmx - VMXOFF holdoff of SMIs yes 2022-10-09T04:47:56.047Z In(05) vmx - Allow all VMWRITEs yes 2022-10-09T04:47:56.047Z In(05) vmx - Allow zero instruction length yes 2022-10-09T04:47:56.047Z In(05) vmx - MSEG revision ID 0 2022-10-09T04:47:56.047Z In(05) vmx - VMX-Fixed Bits in CR0 (0x0000000080000021/0x00000000ffffffff) 2022-10-09T04:47:56.047Z In(05) vmx - Fixed to 0 0xffffffff00000000 2022-10-09T04:47:56.047Z In(05) vmx - Fixed to 1 0x0000000080000021 2022-10-09T04:47:56.047Z In(05) vmx - Variable 0x000000007fffffde 2022-10-09T04:47:56.047Z In(05) vmx - VMX-Fixed Bits in CR4 (0x0000000000002000/0x0000000000777fff) 2022-10-09T04:47:56.047Z In(05) vmx - Fixed to 0 0xffffffffff888000 2022-10-09T04:47:56.047Z In(05) vmx - Fixed to 1 0x0000000000002000 2022-10-09T04:47:56.047Z In(05) vmx - Variable 0x0000000000775fff 2022-10-09T04:47:56.047Z In(05) vmx - VMCS Enumeration (0x000000000000002e) 2022-10-09T04:47:56.047Z In(05) vmx - Highest index 0x17 2022-10-09T04:47:56.047Z In(05) vmx - VM Functions (0x0000000000000001) 2022-10-09T04:47:56.047Z In(05) vmx - Function 0 (EPTP-switching) supported. |
Читатели шлют обратную связь:
Есть такой цикл с красивыми графиками и материалом попроще, жаль последние 2 части вряд ли уже выйдут:
Virtualization Internals Part 1 – Intro to Virtualization
https://webcache.googleusercontent.com/search?q=cache:LY7b8tSIq-UJ:https://about.saferwall.com/blog/virtualization-internals-part-1-intro-to-virtualization
Virtualization Internals Part 2 – VMware and Full Virtualization using Binary Translation
https://webcache.googleusercontent.com/search?q=cache:3nTDgz0QmeIJ:https://about.saferwall.com/blog/virtualization-internals-part-2-vmware-and-virtualization-using-binary-translation
Virtualization Internals Part 3 – Xen and Paravirtualization
https://webcache.googleusercontent.com/search?q=cache:IMTfFjuVCpwJ:https://about.saferwall.com/blog/virtualization-internals-part-3-xen-and-paravirtualization
Virtualization Internals Part 4 – KVM and Qemu
Virtualization Internals Part 5 – Virtualization with Intel VT-x
И ещё эта статья (у автора они все интересные), где всё совсем понятно:
https://habr.com/ru/company/intel/blog/196444/
Статьи ожили по другому адресу:
https://docs.saferwall.com/blog/virtualization-internals-part-1-intro-to-virtualization
Разместил в основном тексте статьи