KVM: Difference between revisions

From 太極
Jump to navigation Jump to search
 
(127 intermediate revisions by the same user not shown)
Line 2: Line 2:
* [http://www.linux-kvm.org/page/Main_Page KVM] Kernel-based Virtual Machine
* [http://www.linux-kvm.org/page/Main_Page KVM] Kernel-based Virtual Machine
* https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine
* https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine
* [https://virt-manager.org/ Virtual Machine Manager] - virt-manager. It can display VNC or SPICE, and uses libvirt to lookup the graphical connection details.
* [https://virt-manager.org/ Virtual Machine Manager] - '''virt-manager'''. Two display types: '''SPICE''' or '''VNC''', and uses libvirt to lookup the graphical connection details.
** [https://ostechnix.com/how-to-manage-kvm-virtual-machines-with-virt-manager/ How To Manage KVM Virtual Machines With Virt-Manager]
** [https://ostechnix.com/how-to-manage-kvm-virtual-machines-with-virt-manager/ How To Manage KVM Virtual Machines With Virt-Manager]
** [https://www.maketecheasier.com/access-linux-virtual-machines-over-vnc/ How to Access Linux Virtual Machines Remotely Over VNC]
** [https://www.maketecheasier.com/access-linux-virtual-machines-over-vnc/ How to Access Linux Virtual Machines Remotely Over VNC]
** [https://linuxconfig.org/how-to-install-redhat-8-on-a-virtual-machine#h3-2-how-to-connect-to-your-vm-over-vnc How to Connect to Your VM Over VNC]
** [https://linuxconfig.org/how-to-install-redhat-8-on-a-virtual-machine#h3-2-how-to-connect-to-your-vm-over-vnc How to Connect to Your VM Over VNC]
** [https://fabianlee.org/2019/02/16/kvm-virt-manager-to-connect-to-a-remote-console-using-qemussh/ Connect to a remote console using qemu+ssh]
* My notes:
* My notes:
** One advantage over VB for me: No need to go to VirtualBox website for the installation and frequent updates. I can use the command line to install it.
** One advantage over VB for me: No need to go to VirtualBox website for the installation and frequent updates. I can use the command line to install it.
** The desktop screen resolution can be automatically adjusted without a need to install ''guest addition'' or ''vmware tool''. Tested Ubuntu and Pop_OS.
** The desktop screen resolution can be automatically adjusted without a need to install ''guest addition'' or ''vmware tool''. Tested Ubuntu and Pop_OS.
** It has two windows. One is virt-manager and the other is the virtual machine.
** It has two windows. One is '''virt-manager''' and the other is the virtual machine '''qemu-system-x86_64'''.
** Use Left Control + Left Alt to escape the virtual machine.  
** Use Left Control + Left Alt to escape the virtual machine.  
* https://en.wikipedia.org/wiki/QEMU
* https://en.wikipedia.org/wiki/QEMU
Line 16: Line 17:
* [http://www.thegeekstuff.com/2014/10/linux-kvm-create-guest-vm How to Install Linux KVM and Create Guest VM with Examples]. Para-virtualization, checking the virtualization support on hardware and others. It includes the usage of '''virsh''' and '''virt-install''' commands.
* [http://www.thegeekstuff.com/2014/10/linux-kvm-create-guest-vm How to Install Linux KVM and Create Guest VM with Examples]. Para-virtualization, checking the virtualization support on hardware and others. It includes the usage of '''virsh''' and '''virt-install''' commands.
* [https://linuxconfig.org/virtualization-solutions-on-linux-systems-kvm-and-virtualbox Virtualization solutions on Linux systems - KVM and VirtualBox]
* [https://linuxconfig.org/virtualization-solutions-on-linux-systems-kvm-and-virtualbox Virtualization solutions on Linux systems - KVM and VirtualBox]
* Guest additions: it seems QEUM/KVM does not have that to download.
* [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_virtualization/getting-started-with-virtualization-in-rhel-8_configuring-and-managing-virtualization#creating-vms-and-installing-an-os-using-the-rhel-8-web-console_assembly_creating-virtual-machines CHAPTER 2. GETTING STARTED WITH VIRTUALIZATION] from redhat.com
** Share clipboard. [https://askubuntu.com/a/858650 How can I copy&paste from the host to a KVM guest?] '''sudo apt install spice-vdagent''' in the guest virtual machine. had to reboot guest to be able to make it work. It works.
** [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_tuning_and_optimization_guide/sect-virtualization_tuning_optimization_guide-monitoring_in_virt_manager 2.3. Monitoring performance in virtual machine manager]
** Display resolution. The display has a resolution 1024x768. In the virtual machine window, I do 'View' -> 'Scale to Display' where I choose 'Always' and check the checkbox 'Auto resize VM with window'. It did increase the resolution. See also the command line method: [https://superuser.com/a/169243 How to increase the visualized screen resolution on QEMU / KVM?]
** Share files
*** [https://cialu.net/qemu-kvm-on-ubuntu-and-sharing-files-between-host-and-guests/ QEMU/KVM on Ubuntu and sharing files between host and guests]. It works. Modify/create a file '''/etc/rc.local''' file to automount the folder at each login. Note the share folder still belongs to the user's group. So we can actually create a soft link for this folder on the host.
*** [https://askubuntu.com/a/851565 Is there something like “VirtualBox Guest Additions” for QEMU/KVM?]
** Mouse integration. [https://ckirbach.wordpress.com/2016/01/14/kvm-mouse-pointer-integration-with-ubuntu-guest/ virt-manager KVM mouse pointer integration (client mouse mode) with Ubuntu guest]. [https://serverfault.com/a/457610 Any way to release focus on a KVM guest in virt-manager without having to click Ctrl_L + Alt_L?]
 
* [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_tuning_and_optimization_guide/sect-virtualization_tuning_optimization_guide-monitoring_in_virt_manager Monitoring performance in virtual machine manager] from redhat.com
* [http://fossboss.com/2016/08/13/use-qemu-test-operating-systems-distributions/ How To Use QEMU To Test Operating Systems & Distributions]
* [http://fossboss.com/2016/08/13/use-qemu-test-operating-systems-distributions/ How To Use QEMU To Test Operating Systems & Distributions]
* [http://www.dedoimedo.com/computers/kvm-virtualbox.html Using KVM and VirtualBox side by side] and [http://askubuntu.com/questions/413511/virtualbox-and-kvm-beside-each-other VirtualBox and KVM beside each other]
* [http://www.dedoimedo.com/computers/kvm-virtualbox.html Using KVM and VirtualBox side by side] and [http://askubuntu.com/questions/413511/virtualbox-and-kvm-beside-each-other VirtualBox and KVM beside each other]
Line 46: Line 40:
# Or edit the XML configuration file
# Or edit the XML configuration file
virsh edit ubuntu16.04
virsh edit ubuntu16.04
</pre>
</li>
</ul>
== libvirt ==
* https://en.wikipedia.org/wiki/Libvirt. '''libvirt''' is an open-source API, daemon and management tool for managing platform virtualization.[3] It can be used to manage KVM, Xen, VMware ESXi, QEMU and other virtualization technologies.
* https://libvirt.org/
* Check libvirt version: sudo head -n2 /var/log/libvirt/qemu/ubuntu2004.log  # on Ubuntu 20.04 it shows 6.0.0
== "virt-install" to install or delete a VM ==
* [https://opensource.com/article/21/1/cloud-image-virt-install Set up a Linux cloud on bare metal]
* [https://www.golinuxcloud.com/virt-install-examples-kvm-virt-commands-linux/ virt-install examples | KVM virtualization commands cheatsheet]
<pre>
$ virt-install --help | grep graphics
$ osinfo-query os
$ sudo virt-install  -n ubuntu2004 --os-type=Linux  \
  --os-variant=ubuntu18.04  --ram=2096  --vcpus=2  \
  --disk size=20  \
  --network network=br0 \
  --graphics vnc,listen=0.0.0.0  \
  --location Downloads/ubuntu-20.04.1-desktop-amd64.iso
$ sudo virt-install --name=centos7server \
  --os-type=Linux \
  --os-variant=centos7.0 \
  --vcpu=2 --ram=2048 \
  --disk size=20 \
  --graphics vnc \
  --location=/tmp/CentOS-7-x86_64-DVD-2009.iso
# Running locally
$ sudo virt-install  -n ubuntu20043 --os-type=Linux  \
  --os-variant=ubuntu20.04  \
  --ram=4096  --vcpus=2  \
  --disk size=20  \
  --network network=default  \
  --cdrom ~/Downloads/ubuntu-20.04.3-desktop-amd64.iso
</pre>
NB
<ul>
<li>If we just run a live session and power off the VM, it will automatically reject the installation medium (iso file) and reboot. So eventually it showed a message "No bootable device." </li>
<li>Using the Virtual Machine Manager GUI I found after I tried to delete a VM it won't actually delete the image file created in '''/var/lib/libvirt/images'''. Be careful when deleting files there. See [https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/ How to delete KVM VM guest using virsh command] & [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-virsh-delete Removing and deleting a VM] from RedHat.
<pre>
$ virsh list --all
Id  Name                State
------------------------------------
39  ubuntu20043        running
$ virsh dumpxml --domain ubuntu20043 | grep 'source file'
      <source file='/var/lib/libvirt/images/ubuntu20043.qcow2' index='2'/>
$ sudo virsh destroy ubuntu20043  # SHUT DOWN
# "sudo virsh undefine ubuntu20043" WILL LEAVE qcow2
$ sudo virsh undefine ubuntu20043 --remove-all-storage
Domain 'ubuntu20043' has been undefined
Volume 'vda'(/var/lib/libvirt/images/ubuntu20043.qcow2) removed.
$ sudo ls -lt /var/lib/libvirt/images/
</pre>
</li>
</ul>
== Check if running in VM ==
[https://opensource.com/article/20/12/linux-server 9 things to do in your first 10 minutes on a Linux server]
<pre>
$ cat /sys/class/dmi/id/sys_vendor
QEMU
$ sudo dmidecode -s system-manufacturer
QEMU
</pre>
On VirtualBox + CentOS vm, it returns "innotek GmbH".
On VMware + CentOS vm, it returns "VMware, Inc".
== Guest additions ==
It seems QEUM/KVM does not have that to download.
* Share clipboard. [https://askubuntu.com/a/858650 How can I copy&paste from the host to a KVM guest?] '''sudo apt install spice-vdagent''' in the guest virtual machine. had to reboot guest to be able to make it work. It works.
** Use '''sudo dpkg-query -l | grep spice-vdagent ''' to check if the package is installed or not
* Display resolution. The display has a resolution 1024x768 (4:3). I can increase to 1600x900 (16:9) or even 1920x1080 (16:9) in the guest machine ('''scrollbar is hidden''').
** In the Virtual Machine Window, I do '''View''' -> '''Scale to Display''' where I choose '''Always''' and check the checkbox '''Auto resize VM with window'''. It did increase the resolution. See also the command line method: [https://superuser.com/a/169243 How to increase the visualized screen resolution on QEMU / KVM?]
* [https://centosfaq.org/centos/virt-manager-and-full-screen-display/ Virt-Manager And Full-screen Display]. View -> Scale Display -> Auto resize VM with window.
* Share files
** [https://cialu.net/qemu-kvm-on-ubuntu-and-sharing-files-between-host-and-guests/ QEMU/KVM on Ubuntu and sharing files between host and guests]. It works. Modify/create a file '''/etc/rc.local''' file to automount the folder at each login. Note the share folder still belongs to the user's group. So we can actually create a soft link for this folder on the host.
** [https://askubuntu.com/a/851565 Is there something like “VirtualBox Guest Additions” for QEMU/KVM?]
* Mouse integration. [https://ckirbach.wordpress.com/2016/01/14/kvm-mouse-pointer-integration-with-ubuntu-guest/ virt-manager KVM mouse pointer integration (client mouse mode) with Ubuntu guest]. [https://serverfault.com/a/457610 Any way to release focus on a KVM guest in virt-manager without having to click Ctrl_L + Alt_L?]
== QEMU vs. KVM ==
[https://www.makeuseof.com/qemu-vs-kvm-which-should-you-use/ QEMU vs. KVM: Which Should You Use for Virtualization?] QEMU uses full system emulation, allowing it to emulate a complete computer system, including the processor, memory, storage, and peripheral devices. This flexibility makes QEMU an excellent choice for cross-platform virtualization since it supports a wide range of architectures.
= Bridge network =
* [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s2-networkscripts-interfaces_network-bridge 11.2.6. NETWORK BRIDGE] from redhat.com. It seems possible to share a NIC with one or more virtual NICs.
* [https://serverfault.com/a/846873 Libvirt: one NIC, multiple VM, all with Bridged mode]
* [https://www.linux-kvm.org/page/Networking#Private_Virtual_Bridge Configuring Guest Networking]
* (Not about bridge) [https://www.cyberciti.biz/faq/linux-kvm-libvirt-dnsmasq-dhcp-static-ip-address-configuration-for-guest-os/ KVM libvirt assign static guest IP addresses using DHCP on the virtual machine]
* [https://unix.stackexchange.com/a/245911 Configure public IP addresses in KVM VMs] (using '''brctl''')
* [https://www.cyberciti.biz/faq/debian-network-interfaces-bridge-eth0-eth1-eth2/ Debian Linux: Configure Network Interfaces As A Bridge / Network Switch] (using '''brctl''')
* [https://www.cyberciti.biz/faq/ubuntu-20-04-add-network-bridge-br0-with-nmcli-command/ Ubuntu 20.04 add network bridge (br0) with nmcli command] (using '''nmcli''')
<ul>
<li>There are different ways to configure the network in Ubuntu.
<ul>
<li>If we are using a Ubuntu server, we will use the '''Netplan''' method. </li>
<li>If we are using a Ubuntu desktop, we will use the Network manager command line '''nmcli''' method or the GUI tool '''nm-connection-editor'''. </li>
</ul>
</li>
<li>Use '''nmcli''' to add a bridge network and a bridge slave interface
<pre>
sudo nmcli con add ifname br0 type bridge con-name br0
sudo nmcli con add ifname eno1 type bridge-slave master br0
</pre>
</li>
<li>Use '''nmcli''' to bring down a wired network and bring up a bridge network
<pre>
sudo nmcli con down "Wired connection 1"
sudo nmcli con up br0
</pre>
</li>
<li>Use '''nmcli''' to delete unused bridge networks
<pre>
sudo nmcli device show
sudo nmcli con show
sudo nmcli con del "br0"
</pre>
</pre>
</li>
</li>
Line 51: Line 171:


= Install =
= Install =
== Install on Ubuntu ==
== Install on Ubuntu ==
* https://help.ubuntu.com/community/KVM
* https://help.ubuntu.com/community/KVM
Line 59: Line 180:
<li> [https://mintguide.org/tools/340-kvm-virtualization-on-a-home-pc-with-linux-mint.html KVM virtualization on a home PC with Linux Mint] </li>
<li> [https://mintguide.org/tools/340-kvm-virtualization-on-a-home-pc-with-linux-mint.html KVM virtualization on a home PC with Linux Mint] </li>
<li> [https://www.linuxtechi.com/install-configure-kvm-ubuntu-18-04-server/ How to Install and Configure KVM on Ubuntu 18.04 LTS Server]. I skip step 4: Configure Network Bridge for KVM virtual Machines. </li>
<li> [https://www.linuxtechi.com/install-configure-kvm-ubuntu-18-04-server/ How to Install and Configure KVM on Ubuntu 18.04 LTS Server]. I skip step 4: Configure Network Bridge for KVM virtual Machines. </li>
<li> How to bring down br0 and delete it? After some messing up I need to delete br0.
<pre>
sudo ip link set br0 down  # ifconfig command is not found anymore
sudo brctl delbr br0
</pre></li>
<li> '''[https://www.answertopia.com/ubuntu/ubuntu-essentials/ Ubuntu 20.04 Essentials (book)]''' &rarr; [https://www.answertopia.com/ubuntu/creating-an-ubuntu-kvm-networked-bridge-interface/ Creating an Ubuntu 20.04 KVM Networked Bridge Interface]. It is based on a Ubuntu Desktop environment. The description is quite detailed. Have not tried yet! </li>
<li>[https://www.tecmint.com/install-kvm-on-ubuntu/ How to Install KVM on Ubuntu 20.04]
<li>[https://www.tecmint.com/install-kvm-on-ubuntu/ How to Install KVM on Ubuntu 20.04]
<ul>
<li>qemu: qemu </li>
<li>qemu-kvm: ovmf, qemu-block-extra, qemu-system-common, qemu-system-data, qemu-system-gui, qemu-system-x86, qemu-utils, seabios, sharutils </li>
<li>libvritd-daemon: libvirt0, libvirt-daemon, libvirt-daemon-driver-qemu, ... </li>
<li>virt-manager: qemu-block-extra, qemu-utils, virt-viewer, virtinst, ...</li>
</ul>
{{Pre}}
{{Pre}}
sudo apt install -y qemu qemu-kvm libvirt-daemon \
sudo apt install -y qemu qemu-kvm libvirt-daemon \
Line 71: Line 204:
# Click the superkey (Windows) and search/launch "Virtual Machine Manager"
# Click the superkey (Windows) and search/launch "Virtual Machine Manager"
</pre>
</pre>
Error: No active connection to install on
Click the "Create a new virtual machine" button will result in an error: No active connection to install on. Rebooting does not help. But when I use the command line to create a new VM it works (It seems '''sudo''' helps).
<pre>
<pre>
sudo virt-install --name=ubuntu2004 \
sudo virt-install --name=ubuntu2004 \
Line 78: Line 211:
   --vcpu=2 --ram=2048 \
   --vcpu=2 --ram=2048 \
   --graphics vnc \
   --graphics vnc \
   --location=ubuntu-20.04.1-desktop-amd64.iso \
   --location=ubuntu-20.04.1-desktop-amd64.iso
  --network bridge:virbr0
</pre>
Add '''--network bridge:br0''' if we have added a new bridge network (''br0'') in the host (eg using '''netplan''' or '''nmcli''' or '''nm-connection-editor''' GUI Tool in Ubuntu). See some resource below.
</li>
<li>[https://www.cyberciti.biz/faq/install-kvm-server-debian-linux-9-headless-server/ How to install KVM server on Debian 9/10 Headless Server]  </li>
<li>[https://www.cyberciti.biz/faq/how-to-install-kvm-on-ubuntu-20-04-lts-headless-server/ How to install KVM on Ubuntu 20.04 LTS Headless Server].
<ul>
<li>IP 192.168.2.25 is the new static IP for the bridge network (not virtual machine) and 192.168.2.254 is the gateway current ethernet is using. </li>
<li>Note according to [https://www.cyberciti.biz/faq/how-to-add-network-bridge-with-nmcli-networkmanager-on-linux/ How to add network bridge with nmcli (NetworkManager) on Linux], WiFi is not possible. It only works with Ethernet. Confirmed! The bridge network can be pinged from another computer. </li>
<li>
[https://unix.stackexchange.com/a/219906 How to create wireless bridge connection with nmcli]. Ans: This is not possible.
</li>
<li>I follow this [https://www.cyberciti.biz/faq/how-to-add-network-bridge-with-nmcli-networkmanager-on-linux/ How to add network bridge with nmcli (NetworkManager) on Linux] disable STP, turn off “Wired connection 1” and turn on br0. </li>
<li>Question: Do I need to turn off "Wired connection 1". It will lose the original host's static IP. The new bridge IP will become the host's IP. The guest machine still obtain a new IP via DHCP. This is not what I expect? </li>
<li>Before turn off “Wired connection 1” and turn on br0:  
<pre>
$ nmcli connection show --active
NAME                UUID  TYPE      DEVICE 
Wired connection 1  XXX    ethernet  eno1   
docker0            XXX    bridge    docker0
virbr0             XXX    bridge    virbr0
</pre>
</li>
<li>After turn off “Wired connection 1” and turn on br0:
<pre>
$ nmcli connection show --active
NAME              UUID  TYPE      DEVICE 
br0                xxx    bridge    br0   
docker0            XXX    bridge    docker0
virbr0            XXX    bridge    virbr0 
bridge-slave-eno1  XXX    ethernet  eno1   
vnet0              XXX    tun      vnet0
</pre>
</li>
<li>I am not able to use remmina to VNC to the bridge VM. Not sure if this is because my host OS is a desktop (not headless). </li>
</ul>
</li>
<li>
os-variant list in '''virt-install'''
<pre>
$ osinfo-query os
</pre>
</pre>
</li>
</li>
<li>[https://www.cyberciti.biz/faq/how-to-install-kvm-on-ubuntu-20-04-lts-headless-server/ How to install KVM on Ubuntu 20.04 LTS Headless Server]</li>
</ul>
</ul>
* Where are image files saved: '''/var/lib/libvirt/images''' (/var/lib/docker for Docker)
** [https://linuxconfig.org/how-to-move-var-directory-to-another-partition How to move /var directory to another partition]
** [https://linuxconfig.org/how-to-move-var-directory-to-another-partition How to move /var directory to another partition]
* Troubleshooting: [https://askubuntu.com/a/345263 virt-manager can't connect to libvirt]. Rebooting computer!
* Troubleshooting: [https://askubuntu.com/a/345263 virt-manager can't connect to libvirt]. Rebooting computer!
* [https://itsfoss.com/qemu-ubuntu/ Install and Use Qemu on Ubuntu] including '''spice-vdagent'''
=== Where are image files saved ===
'''/var/lib/libvirt/images''' (/var/lib/docker for Docker)
From what I can see, the directory contains many large '''qcow2''' files that should be deleted once I delete the VM.
== VNC ==
<ul>
<li>[https://blog.gtwang.org/linux/kvm-qemu-virt-install-command-tutorial/amp/ KVM/QEMU 以 virt-install 指令建立虛擬機器、VNC 顯示畫面教學]
<pre>
sudo virt-install --virt-type kvm --name ubuntu1804 --ram 2048
  --disk /home/seal/kvm/ubuntu1804.qcow2,format=qcow2
  --network network=default
  --graphics vnc,listen=0.0.0.0,password=YOUR_PASSWORD
  --noautoconsole
  --os-type=linux --os-variant=ubuntu17.10
  --cdrom=/home/seal/Downloads/ubuntu-18.10-desktop-amd64.iso
virsh vncdisplay ubuntu1804 #  127.0.0.1:1
                            # pay attention to the port number
                            # :0  --> 5900 in ssh
                            # :1  --> 5901 in ssh
sudo ufw disable
# Another machine.
# Suppose the host machine has an IP 192.168.0.3
ssh -NL 5901:localhost:5901 [email protected]
ssh -NL 5901:127.0.0.1:5901 [email protected]
vncviewer 127.0.0.1:1 # from a different machine
</pre>
NOTE: I have to use local port forwarding. If I try to directly connect to the VM in RealVNC, it will show '''The connection was refused by the computer'''. A limit by KVM?
</li>
</ul>
* [https://fabianlee.org/2019/02/16/kvm-virt-manager-to-connect-to-a-remote-console-using-qemussh/ KVM: virt-manager to connect to a remote console using qemu+ssh]
The question is how do we know the client's IP? I guess '''cockpit''' method is better.
== Install qemu-system-arm in Ubuntu ==
* sudo apt install qemu-system-arm
<ul>
<li>[https://epicchasgamer.com/2020/10/18/how-to-emulate-raspberry-pi-in-qemu/ How to emulate Raspberry Pi in QEMU] which is based on [https://github.com/dhruvvyas90/qemu-rpi-kernel qemu-rpi-kernel]-Qemu kernel for emulating Rpi on QEMU.
{{Pre}}
mkdir rpi-vm && cd rpi-vm
git clone https://github.com/dhruvvyas90/qemu-rpi-kernel.git
# Download Raspbian Buster Lite
# from https://www.raspberrypi.org/downloads/raspberry-pi-os/
dd if=/dev/zero bs=1M count=4096 >> 2020-08-20-raspios-buster-armhf-lite.img
sudo losetup -f -P --show 2020-08-20-raspios-buster-armhf-lite.img
sudo mount -o rw /dev/loopXp2 /mnt
sudo nano /mnt/etc/fstab
sudo nano /etc/ld.so.preload # Comment out the line and save the file.
# Then unmount with the following commands:
# sudo umount /mnt
# sudo losetup -d /dev/loopX
qemu-system-arm \
  -M versatilepb \
  -cpu arm1176 \
  -m 256 \
  -kernel qemu-rpi-kernel/kernel-qemu-4.19.50-buster \
  -hda 2020-08-20-raspios-buster-armhf-lite.img \
  -append "dwc_otg.lpm_enable=0 root=/dev/sda2 console=tty1 rootfstype=ext4 elevator=deadline rootwait" \
  -dtb qemu-rpi-kernel/versatile-pb-buster.dtb \
  -no-reboot \
  -serial stdio
# We can install a desktop. See
# https://www.raspberrypi.org/forums/viewtopic.php?t=133691
</pre>
</li>
</ul>


== Install on CentOS ==
== Install on CentOS ==
Line 92: Line 341:
* [https://www.cyberciti.biz/faq/how-to-install-kvm-on-centos-7-rhel-7-headless-server/ How to install KVM on CentOS 7 / RHEL 7 Headless Server]
* [https://www.cyberciti.biz/faq/how-to-install-kvm-on-centos-7-rhel-7-headless-server/ How to install KVM on CentOS 7 / RHEL 7 Headless Server]
* [https://www.tecmint.com/multiple-virtual-machine-installation-using-network-install-kvm/ How to Deploy Multiple Virtual Machines using Network Install (HTTP, FTP and NFS) under KVM Environment]
* [https://www.tecmint.com/multiple-virtual-machine-installation-using-network-install-kvm/ How to Deploy Multiple Virtual Machines using Network Install (HTTP, FTP and NFS) under KVM Environment]
== qemu-system-x86_64 ==
* The command can be used to create or rung a VM. See https://www.linux-kvm.org/page/RunningKVM
* [https://fosspost.org/use-qemu-test-operating-systems-distributions/ How To Use QEMU To Test Operating Systems & Distributions]
* '''virt-install''' vs '''qemu-system-x86_64'''
** [https://www.stratoscale.com/blog/compute/using-bare-qemu-kvm-vs-libvirt-virt-install-virt-manager/ Using bare qemu-kvm vs. libvirt/virt-install/virt-manager]
* [https://subscription.packtpub.com/book/virtualization-and-cloud/9781788294676/1/ch01lvl1sec17/starting-the-qemu-vm-with-kvm-support To start a QEMU instance with KVM support]:
:<syntaxhighlight lang='bash'>
qemu-system-x86_64 -name debian \
  -vnc 146.20.141.254:0 -m 1024 \
  -drive format=raw,index=2,file=debian.img \
  -enable-kvm -daemonize
</syntaxhighlight>
:If we use Virtual Machine Manager program, you need to have KVM ('''lsmod | grep kvm''') installed on your system for this to work.
== Install using qcow2 image ==
* https://cloud.centos.org/centos/7/images/
* [https://forums.centos.org/viewtopic.php?t=67844 Requesting for username and Password for the CentOS7 Qcow image]
* [https://fabianlee.org/2020/03/14/kvm-testing-cloud-init-locally-using-kvm-for-a-centos-cloud-image/ KVM: Testing cloud-init locally using KVM for a CentOS cloud image]*
* [https://www.cyberciti.biz/faq/how-to-add-ssh-public-key-to-qcow2-linux-cloud-images-using-virt-sysprep/ How to add ssh key to qcow2 Linux cloud images using virt-sysprep]
== Running a virtual machine from a ramdisk ==
<ul>
<li>[https://en.wikipedia.org/wiki/List_of_RAM_drive_software List of RAM drive software] </li>
<li>[https://www.cloudsavvyit.com/10254/how-to-create-a-ram-drive-in-linux/ How to Create a RAM Drive in Linux]
<pre>
if [ "$(mount | grep -o "/mnt/ram")" != "/mnt/ram" ]; then
  sudo mkdir -p /mnt/ram
  sudo mount -t ramfs -o size=1g ramfs /mnt/ram
  sudo chown -R $(whoami):$(whoami) /mnt/ram
fi
mount | grep ram
sudo umount /mnt/ram
</pre> </li>
<li>https://youtu.be/fOaWrhLoVK0  </li>
<li>[https://kaurin.github.io/2018/Libvirt-In-Memory-Storage/ Libvirt In-Memory Storage]  </li>
<li>(Windows only) [https://techgenix.com/ram-disk/ USE A RAM DISK TO GET EXTREME PERFORMANCE FROM TEMPORARY VIRTUAL MACHINES]  </li>
<li>[https://www.linuxbabe.com/command-line/create-ramdisk-linux How to Easily Create RAM Disk on Debian, Ubuntu, Linux Mint, CentOS]. How to Create a RAM Disk in Any Linux Distro.
<pre>
sudo mkdir /tmp/ramdisk
sudo chmod 777 /tmp/ramdisk
htop  # check free ram
sudo mount -t tmpfs -o size=1G myramdisk /tmp/ramdisk
            # myramdisk is the device name I gave to it
mount | tail -n 1
sudo umount /tmp/ramdisk/
</pre>
Test RAM speed
<pre>
# test write speed
sudo dd if=/dev/zero of=/tmp/ramdisk/zero bs=4k count=100000
## 409600000 bytes (410 MB, 391 MiB) copied, 0.734813 s, 557 MB/s  # SATA SSD
## 409600000 bytes (410 MB, 391 MiB) copied, 0.18296 s, 2.2 GB/s  # NVME
# test read speed
sudo dd if=/tmp/ramdisk/zero of=/dev/null bs=4k count=100000
## 409600000 bytes (410 MB, 391 MiB) copied, 0.410809 s, 997 MB/s  # SATA SSD
## 409600000 bytes (410 MB, 391 MiB) copied, 0.139487 s, 2.9 GB/s  # NVME
</pre>
Auto-mount on System Boot
<pre>
sudo nano /etc/fstab
## myramdisk  /tmp/ramdisk  tmpfs  defaults,size=1G,x-gvfs-show  0  0
## To mount it immediately without reboot, run the following command.
sudo mount -a
</pre>
How to Run VirtualBox VM on RAM Disk. You should set the machine folder to the RAM disk directory ('''/tmp/ramdisk/''').
</li>
</ul>
== kvm command ==
* [https://helpmanual.io/man1/kvm/ help kvm]. The  '''kvm'''  wrapper  script  is  used  to  provide compatibility with old '''qemu-kvm''' package which has been merged into qemu as of version 1.3. The script executes '''qemu-system-x86_64 -enable-kvm''' passing all other command-line arguments to the qemu binary.
* [https://www.makeuseof.com/how-to-install-manjaro-linux-in-a-qemukvm-virtual-machine/ How to Install Manjaro Linux in a QEMU/KVM Virtual Machine]
= snapshot =
= snapshot =
<ul>
<ul>
Line 110: Line 435:
<li>
<li>
To create a snapshot using the GUI, open virtualization manager window. In the VM properties, click on the icon (two displays overlapped) which is on the right of the menu, which says '''Manage VM snapshots'''.
To create a snapshot using the GUI, open virtualization manager window. In the VM properties, click on the icon (two displays overlapped) which is on the right of the menu, which says '''Manage VM snapshots'''.
</li>
<li>
To restore a snapshot using '''Virtual Machine Manager''', right click on a snapshot and select "Start snapshot".
</li>
</li>
</ul>
</ul>


= GUI manager =
= GUI =
 
== Virtual Machine Manager/virt-manager, Cockpit, virt-viewer, remote-viewer ==
<pre>
sudo apt install cockpit cockpit-machines
</pre>
 
* [https://www.answertopia.com/ubuntu/creating-ubuntu-kvm-virtual-machines-using-cockpit-and-virt-manager/ Creating Ubuntu 20.04 KVM Virtual Machines using Cockpit and virt-manager]*.
** It shows how to handle the error of cannot opening the ISO file.
** It shows how to use virt-viewer to connect to a VM ('''virt-viewer''')
** [http://manpages.ubuntu.com/manpages/bionic/man1/remote-viewer.1.html remote-viewer] will be used if we change the Console type from 'VNC' to 'Graphics Console in Desktop Viewer' and click the 'Launch Remote Viewer' button. A text file called 'download' will be downloaded. If we click it, it will launch '''remote-viewer'''. An interesting thing is the file will be automatically deleted once we click on it:)
* Consider running a VM on a ''remote'' machine with cockpit version 215.
** For Ubuntu 20.04.1 Desktop. If I increase the display resolution from 1024x768 (4:3) to others without keeping the same ratio, the mouse calibration has a problem:( Maybe it's better not to change the resolution. The VNC method works.
** For Windows 10, we need to unplug ethernet first in order to create a local account. I've not found a way to use virt-viewer or remote-viewer to view the desktop locally. I can still use cockpit web interface (Consoles) to use the Windows desktop. I am able to increase the display resolution to 1280x800 from 1024x768 which makes the interface easy for my eyes. '''One problem is the machine will freeze''' for some unknown reason.
<ul>
<ul>
<li>[https://ostechnix.com/manage-kvm-virtual-machines-using-cockpit-web-console/ Manage KVM Virtual Machines Using Cockpit Web Console]</li>
<li>[https://ostechnix.com/manage-kvm-virtual-machines-using-cockpit-web-console/ Manage KVM Virtual Machines Using Cockpit Web Console]</li>
<li>[https://ostechnix.com/how-to-manage-kvm-virtual-machines-with-virt-manager/ How To Manage KVM Virtual Machines With Virt-Manager]. View -> Graph ->  Memory Usage</li>
<li>[https://ostechnix.com/how-to-manage-kvm-virtual-machines-with-virt-manager/ How To Manage KVM Virtual Machines With Virt-Manager]. View -> Graph ->  Memory Usage. However, it seems the reported memory is correct for CentOS, but not Ubuntu or Windows.</li>
</ul>
</ul>
* [https://dbaxps.blogspot.com/2020/07/install-kvm-web-cockpit-console-on.html Install KVM && Web Cockpit Console on SparkyLinux 2020.6]
* Tip: it may help to use mouse to click the installation screen to refresh the screen if we don't see the progress bar to progress.
* [https://ostechnix.com/manage-kvm-virtual-machines-using-cockpit-web-console/ Manage KVM Virtual Machines Using Cockpit Web Console]
* [https://askubuntu.com/a/776107 Permission error in virtual machine manager]. Solution 1: use '''sudo sudo virt-manager'''  Solution 2: copy iso file to /tmp directory. Solution 3: change the owner of the iso file to '''libvirt-qemu:kvm'''  .
=== Network Install in virtual machine manager ===
* [https://opensource.com/article/19/6/create-centos-homelab-hour Create a CentOS homelab in an hour] (I tested it and got the same error as one reader).
* [https://www.tecmint.com/multiple-virtual-machine-installation-using-network-install-kvm/ How to Deploy Multiple Virtual Machines using Network Install (HTTP, FTP and NFS) under KVM Environment – Part 2]. This guide covers 3 approaches (ftp, nfs, http) so this is the most comprehensive!
=== Share folders ===
* [https://www.debugpoint.com/share-folder-virt-manager/ Share Folder Between Guest and Host in virt-manager (KVM/Qemu/libvirt)]
* [https://www.debugpoint.com/kvm-share-folder-windows-guest/ Share Folder Between Windows Guest and Linux Host in KVM using virtiofs]
== How to quit full screen ==
[https://www.golinuxcloud.com/exit-full-screen-mode-in-qemu-virt-manager/ How to exit full screen mode in qemu virt-manager ( RHEL / CentOS 7)]. Hint: click the first of two icons shown at the top-center screen.
== Windows VM ==
[https://www.makeuseof.com/create-windows-virtual-machine-in-linux-with-kvm/ How to Create a Windows Virtual Machine in Linux With KVM]
== Windows VM by docker ==
[https://www.youtube.com/watch?v=xhGYobuG508 Running Windows in a Docker Container!]
== GNOME Boxes ==
* https://help.gnome.org/users/gnome-boxes/stable/
* [https://www.debugpoint.com/2020/05/install-use-gnome-boxes/ How to Install and Use GNOME Boxes to Create Virtual Machines]


= Network =
= Network =
<ul>
<ul>
<li>Host only (used by VirtualBox) = '''Isolated network''' (used by KVM). Step 1: Select a VM. Click Edit, choose "Connection Details" and add a new network (Isolated mode). This ''default'' network has IPv4 192.168.122.0/24 Step 2: Click Edit, choose "Virtual machine Details" and click the Information icon (circle shape). Click "Add Hardware" and choose "Isolated network" for the Network source. Leave the MAC address and Device model as defaults. This ''network'' network has IP4 192.168.100.0/24.
<li>Host only (used by VirtualBox) = '''Isolated network''' (used by KVM). Step 1: Select a VM. Click Edit, choose "Connection Details" and add a new network (Isolated mode). This ''default'' network has IPv4 192.168.122.0/24 Step 2: Click Edit, choose "Virtual machine Details" and click the Information icon (circle shape). Click "Add Hardware" and choose "Isolated network" for the Network source. Leave the MAC address and Device model as defaults. This ''network'' network has IP4 192.168.100.0/24.
<li>[https://www.tecmint.com/create-network-bridge-in-ubuntu/ How to Configure Network Bridge in Ubuntu*]. Three methods. ''Excellent instruction!''
<ul>
<li>The new (slave-type) bridge network should tie to one of existing network adapters for this to work. </li>
<li>The new adapter will be ''br0'' only if we use the '''netplan''' method. </li>
<li>If we use the '''nmcli''' or the '''nm-connection-editor''' GUI method, we need to deactivate the Ethernet interface (if we use Ethernet). The final adapters will have ''br0'' and ''bridge-br0''. </li>
<li>To set a static IP on the host, see [[Ubuntu#Set_static_IP|Ubuntu -> Set static IP]], [https://www.tecmint.com/set-static-ip-address-in-ubuntu/ How to Configure Static IP Address on Ubuntu 20.04].
</li>
</ul>
</li>
<li>[https://www.golinuxcloud.com/configure-bridged-network-in-virt-manager/ How to configure bridged network in virt-manager (CentOS / RHEL 7)] </li>
<li>[https://www.golinuxcloud.com/configure-bridged-network-in-virt-manager/ How to configure bridged network in virt-manager (CentOS / RHEL 7)] </li>
<li>[https://linuxconfig.org/install-and-set-up-kvm-on-ubuntu-20-04-focal-fossa-linux#h2-configure-the-network-bridge Install And Set Up KVM On Ubuntu 20.04 Focal Fossa Linux] </li>
<li>[https://linuxconfig.org/install-and-set-up-kvm-on-ubuntu-20-04-focal-fossa-linux#h2-configure-the-network-bridge Install And Set Up KVM On Ubuntu 20.04 Focal Fossa Linux] </li>
<li>[https://www.tecmint.com/create-network-bridge-in-ubuntu/ How to Configure Network Bridge in Ubuntu]</li>
<li>[https://fabianlee.org/2019/04/01/kvm-creating-a-bridged-network-with-netplan-on-ubuntu-bionic/ KVM: Creating a bridged network with NetPlan on Ubuntu bionic] </li>
<li>[https://fabianlee.org/2019/04/01/kvm-creating-a-bridged-network-with-netplan-on-ubuntu-bionic/ KVM: Creating a bridged network with NetPlan on Ubuntu bionic] </li>
<li>[https://wiki.libvirt.org/page/VirtualNetworking VirtualNetworking] </li>
<li>[https://wiki.libvirt.org/page/VirtualNetworking VirtualNetworking] </li>
<li>[https://libvirt.org/formatnetwork.html Network XML format] </li>
<li>[https://libvirt.org/formatnetwork.html Network XML format] </li>
<li>[https://amoldighe.github.io/2017/12/20/kvm-networking/ KVM Networking - NAT & Host-Only] </li>
<li>[https://amoldighe.github.io/2017/12/20/kvm-networking/ KVM Networking - NAT & Host-Only] </li>
<li>[https://www.pitt-pladdy.com/blog/_20150620-151456_0100_Home_Lab_Project_Network_Bridges_for_KVM_-_NAT_Host-only_Isolated/ Home Lab Project: Network Bridges for KVM - NAT, Host-only, Isolated] </li>
<li>[https://www.pitt-pladdy.com/blog/_20150620-151456_0100_Home_Lab_Project_Network_Bridges_for_KVM_-_NAT_Host-only_Isolated/ Home Lab Project: Network Bridges for KVM - NAT, Host-only, Isolated]</li>
</ul>
</ul>
</li>
</li>


= Back and restore images =
== Find out the IP of the guest machine ==
* [https://ostechnix.com/how-to-find-the-ip-address-of-a-kvm-virtual-machine/ How To Find The IP Address Of A KVM Virtual Machine]
<pre>
# Method 1:
$ virsh list
$ virsh domifaddr centos8
 
# Method 2:
$ sudo apt install net-tools  # Debian
$ sudo dnf install net-tools  # CentOS
$ arp -n              # find the rows with virbr0 adapter
 
# Method 3:
# KVM -> Info -> NIC
</pre>
 
= Backup and restore images =
[https://techsoftcenter.com/how-to-kvm-backup-and-restore-in-linux/ How to KVM Backup and Restore in Linux]
[https://techsoftcenter.com/how-to-kvm-backup-and-restore-in-linux/ How to KVM Backup and Restore in Linux]
<pre>
<pre>
Line 145: Line 535:
sudo virsh undefine win10
sudo virsh undefine win10
ls -l /etc/libvirt/qemu
ls -l /etc/libvirt/qemu
virsh list –all
virsh list --all
sudo rm /var/lib/libvirt/images/win10.qcow2
sudo rm /var/lib/libvirt/images/win10.qcow2
sudo cp /MyBackup/win10.qcow2 /var/lib/libvirt/images/
sudo cp /MyBackup/win10.qcow2 /var/lib/libvirt/images/
Line 154: Line 544:
= Cheat sheet =
= Cheat sheet =
[https://blog.programster.org/kvm-cheatsheet KVM Cheatsheet]
[https://blog.programster.org/kvm-cheatsheet KVM Cheatsheet]
= Vagrant =
* https://www.vagrantup.com/
* [https://ostechnix.com/how-to-use-vagrant-with-libvirt-kvm-provider/ How To Use Vagrant With Libvirt KVM Provider]
* [https://www.lynda.com/Vagrant-tutorials/Learning-Vagrant/685028-2.html Learning vagrant] (Lynda.com)
* [https://leftasexercise.com/2020/05/15/managing-kvm-virtual-machines-part-i-vagrant-and-libvirt/ Managing KVM virtual machines part I – Vagrant and libvirt] (it has a nice diagram to show a big picture of things related to '''libvirt'''),
** [https://leftasexercise.com/2020/05/18/managing-kvm-virtual-machines-part-ii-the-libvirt-toolkit/ Managing KVM virtual machines part II – the libvirt toolkit],
** [https://leftasexercise.com/2020/05/22/managing-kvm-virtual-machines-part-iii-using-libvirt-with-ansible/ Managing KVM virtual machines part III – using libvirt with Ansible]
* [https://computingforgeeks.com/using-vagrant-with-libvirt-on-linux/ How To Use Vagrant with '''Libvirt''' on Linux]
= How to Use Virtualbox VMs on KVM In Linux =
[https://www.tecmint.com/migrate-virtualbox-vms-into-kvm-vms/ How to Use Virtualbox VMs on KVM In Linux]


= Run macOS on KVM / QEMU =
= Run macOS on KVM / QEMU =
[https://computingforgeeks.com/how-to-run-macos-on-kvm-qemu/ How To run macOS on KVM / QEMU]
* [https://computingforgeeks.com/how-to-run-macos-on-kvm-qemu/ How To run macOS on KVM / QEMU]
* [https://www.linuxuprising.com/2021/03/install-macos-big-sur-or-catalina-in.html Install macOS Big Sur Or Catalina In A Virtual Machine Using Docker-OSX]
* [https://www.makeuseof.com/macos-ubuntu-linux-virtual-machine/ How To Install macOS in a Virtual Machine on Ubuntu Linux]
 
= Run SteamOS on QEMU =
[https://blogs.igalia.com/berto/2022/07/05/running-the-steam-decks-os-in-a-virtual-machine-using-qemu/ Running the Steam Deck’s OS in a virtual machine using QEMU]
 
= QuickEMU =
* [https://github.com/quickemu-project/quickemu QuickEMU] - Quickly create and run optimised Windows, macOS and Linux desktop virtual machines
* [https://www.tomshardware.com/how-to/run-windows-11-and-mac-os-virtual-machines-in-linux How To Run Windows 11 and macOS Virtual Machines in Linux] (enough screenshots)
* [https://bytexd.com/use-quickemu-to-run-windows-macos-linux-virtual-machines/ Use Quickemu to Run Windows, macOS & Linux Virtual Machines]
* [https://www.linuxuprising.com/2021/12/quickly-create-and-run-optimized-linux.html Quickly Create And Run Optimized Linux, macOS And Windows Virtual Machines With Quickemu (With Auto ISO Download)]
* (video) By [https://www.youtube.com/watch?v=87MCT3Ekjlo Chris Titus Tech]
* [https://ostechnix.com/test-linux-distros-online-with-distrosea/ How To Test Linux Distros Online With DistroSea]
 
Some lessons:
* By default new folders are created when we create/download guest machine in the home folder. We can create a new folder for managing these guest machines.
* Use Ctrl+Alt+g to release the mouse. The hint is at the top of window.
* Change boot="legacy" in '''windows-11.conf''' helps to show the Windows installation process (See [https://reposhub.com/linux/shell-package-management/wimpysworld-quickemu.html Simple shell script to "manage" Qemu virtual machines.]). But it still failed ''Windows could not create a partition on disk 0. The error occurred while applying the unattend answer file's <DiskConfiguration> setting. Error code: 0x80042565.''
* When I tried to install quickemu on another freshly installed xubuntu 20.04 focal fossa (Core2 Quad CPU), the installation went through without any problem.
* The app running the guest machine is '''spicy'''.
* I can install macOS Catalina, Windows 11, etc. The trick to install macOS is '''manually reboot''' the guest machine a couple times. Also it is useful to play the "Options -> Scale display" option in "spicy" to better see the text when we install macOS. Then it works.
* I need to use '''Shift + F12''' (see the hint at the bottom of spicy window) to release mouse (or exit full display) in [https://github.com/freedesktop/spice-gtk spicy] (the GUI where the guest machine is running) on my Ubuntu. '''When installing macOS, I need to use my keyboard''' (mouse does not work) to select the hdd for booting macOS (or just wait a few seconds & the 1st drive will be used to boot).
* It helps to drag and enlarge the guest machine window (by default 'Scale display' option is checked).
* When installing macOS, the screen says "About X minutes remaining". But it seems [https://github.com/quickemu-project/quickemu/issues/228 it stuck there] (the minutes is not changed) though the CPU is busy. If I closed the window, shutdown the guest machine and reboot it, installation continues. Doing this a few times and eventually a new window ("[https://github.com/quickemu-project/quickemu/issues/278 Select Your Country or Region]") will come up. In order to monitor the download, I open another terminal and run something like '''watch du -s ~/macos-big-sur'''. This will help me to decide when to restart the VM. For big-sur, the folder size stops at 26655436 and after I reboot the VM, the folder size keep increasing and stops at 32565600. For the 3rd reboot, it still downloads something and needs 2 internal reboots. So I think the problem is the data needs to be downloaded from Apple server is TOO large.
* macos-big-sur & macos-monterey took about 32G & 34G space.
* Every time I boot macOS, I shall use the right arrow to select the 2nd icon and press the Enter button to start.
* macOS Catalina asks at least 4 characters for the password (I choose 6)
* For macOS Catalina/Monterey guest machine, it automatically allocates 2 CPUs(?), 4/8GB RAM (depending on the host total RAM?), 100GB storage by default. After installing macOS, 75GB will still be available.
* (Maybe related to Ubuntu 22.04 host using wayland). Whenever I start macOS vm, it pops up a window "Allow inhibiting shortcuts. The application spicy wants to inhibit shortcuts. You can restore shortcuts by pressing Super+Escape." I choose the default "Allow" and it seems to be OK. PS: I need to do this action quickly or the macOS window will go to black and not responds to my keyboard/mouse.
* For Windows 11 guest machine, it allocates 2 CPUs, 4/8GB RAM, 64GB storage with 43GB free.
* For Lubuntu guest machine (much better speed than macOS/Windows VM, fill full screen, Qt/KDE based), it allocates 2 CPUs, 4GB RAM, 16GB storage with 8.4GB free.
 
Build QEMU from source
* https://askubuntu.com/a/1198927
* [https://mathiashueber.com/manually-update-qemu-on-ubuntu-18-04/ How to update QEMU on Ubuntu]
 
= Nested KVM =
[https://www.howtogeek.com/devops/how-to-enable-nested-kvm-virtualization/ How to Enable Nested KVM Virtualization]

Latest revision as of 14:38, 7 March 2024

Resource

libvirt

  • https://en.wikipedia.org/wiki/Libvirt. libvirt is an open-source API, daemon and management tool for managing platform virtualization.[3] It can be used to manage KVM, Xen, VMware ESXi, QEMU and other virtualization technologies.
  • https://libvirt.org/
  • Check libvirt version: sudo head -n2 /var/log/libvirt/qemu/ubuntu2004.log # on Ubuntu 20.04 it shows 6.0.0

"virt-install" to install or delete a VM

$ virt-install --help | grep graphics
$ osinfo-query os 

$ sudo virt-install  -n ubuntu2004 --os-type=Linux  \
  --os-variant=ubuntu18.04  --ram=2096  --vcpus=2  \
  --disk size=20  \
  --network network=br0 \
  --graphics vnc,listen=0.0.0.0  \
  --location Downloads/ubuntu-20.04.1-desktop-amd64.iso

$ sudo virt-install --name=centos7server \
  --os-type=Linux \
  --os-variant=centos7.0 \
  --vcpu=2 --ram=2048 \
  --disk size=20 \
  --graphics vnc \
  --location=/tmp/CentOS-7-x86_64-DVD-2009.iso

# Running locally
$ sudo virt-install  -n ubuntu20043 --os-type=Linux  \
  --os-variant=ubuntu20.04  \
  --ram=4096  --vcpus=2  \
  --disk size=20  \
  --network network=default  \
  --cdrom ~/Downloads/ubuntu-20.04.3-desktop-amd64.iso

NB

  • If we just run a live session and power off the VM, it will automatically reject the installation medium (iso file) and reboot. So eventually it showed a message "No bootable device."
  • Using the Virtual Machine Manager GUI I found after I tried to delete a VM it won't actually delete the image file created in /var/lib/libvirt/images. Be careful when deleting files there. See How to delete KVM VM guest using virsh command & Removing and deleting a VM from RedHat.
    $ virsh list --all
     Id   Name                State
    ------------------------------------
     39   ubuntu20043         running
    
    $ virsh dumpxml --domain ubuntu20043 | grep 'source file' 
          <source file='/var/lib/libvirt/images/ubuntu20043.qcow2' index='2'/>
    
    $ sudo virsh destroy ubuntu20043  # SHUT DOWN
    
    # "sudo virsh undefine ubuntu20043" WILL LEAVE qcow2
    
    $ sudo virsh undefine ubuntu20043 --remove-all-storage 
    Domain 'ubuntu20043' has been undefined
    Volume 'vda'(/var/lib/libvirt/images/ubuntu20043.qcow2) removed.
    
    $ sudo ls -lt /var/lib/libvirt/images/
    

Check if running in VM

9 things to do in your first 10 minutes on a Linux server

$ cat /sys/class/dmi/id/sys_vendor 
QEMU

$ sudo dmidecode -s system-manufacturer 
QEMU

On VirtualBox + CentOS vm, it returns "innotek GmbH".

On VMware + CentOS vm, it returns "VMware, Inc".

Guest additions

It seems QEUM/KVM does not have that to download.

QEMU vs. KVM

QEMU vs. KVM: Which Should You Use for Virtualization? QEMU uses full system emulation, allowing it to emulate a complete computer system, including the processor, memory, storage, and peripheral devices. This flexibility makes QEMU an excellent choice for cross-platform virtualization since it supports a wide range of architectures.

Bridge network

  • There are different ways to configure the network in Ubuntu.
    • If we are using a Ubuntu server, we will use the Netplan method.
    • If we are using a Ubuntu desktop, we will use the Network manager command line nmcli method or the GUI tool nm-connection-editor.
  • Use nmcli to add a bridge network and a bridge slave interface
    sudo nmcli con add ifname br0 type bridge con-name br0
    sudo nmcli con add ifname eno1 type bridge-slave master br0
    
  • Use nmcli to bring down a wired network and bring up a bridge network
    sudo nmcli con down "Wired connection 1"
    sudo nmcli con up br0
    
  • Use nmcli to delete unused bridge networks
    sudo nmcli device show
    sudo nmcli con show
    sudo nmcli con del "br0"
    

Install

Install on Ubuntu

  • Virtualization - The Debian Administrator's Handbook
  • How to install KVM on Ubuntu 14.04 LTS Headless Server from cyberciti
  • Simple Virtualization With Ubuntu 16.04 Linux and KVM from linuxconfig
  • KVM virtualization on a home PC with Linux Mint
  • How to Install and Configure KVM on Ubuntu 18.04 LTS Server. I skip step 4: Configure Network Bridge for KVM virtual Machines.
  • How to bring down br0 and delete it? After some messing up I need to delete br0.
    sudo ip link set br0 down  # ifconfig command is not found anymore
    sudo brctl delbr br0
    
  • Ubuntu 20.04 Essentials (book)Creating an Ubuntu 20.04 KVM Networked Bridge Interface. It is based on a Ubuntu Desktop environment. The description is quite detailed. Have not tried yet!
  • How to Install KVM on Ubuntu 20.04
    • qemu: qemu
    • qemu-kvm: ovmf, qemu-block-extra, qemu-system-common, qemu-system-data, qemu-system-gui, qemu-system-x86, qemu-utils, seabios, sharutils
    • libvritd-daemon: libvirt0, libvirt-daemon, libvirt-daemon-driver-qemu, ...
    • virt-manager: qemu-block-extra, qemu-utils, virt-viewer, virtinst, ...
    sudo apt install -y qemu qemu-kvm libvirt-daemon \
             libvirt-clients bridge-utils virt-manager
    sudo systemctl status libvirtd
    # enable it to start on boot 
    sudo systemctl enable --now libvirtd
    lsmod | grep -i kvm
    ip a
    # I got two new: virbr0, virbr0-nic
    # Click the superkey (Windows) and search/launch "Virtual Machine Manager"
    

    Click the "Create a new virtual machine" button will result in an error: No active connection to install on. Rebooting does not help. But when I use the command line to create a new VM it works (It seems sudo helps).

    sudo virt-install --name=ubuntu2004 \
      --os-type=Linux \
      --os-variant=ubuntu20.04 \
      --vcpu=2 --ram=2048 \
      --graphics vnc \
      --location=ubuntu-20.04.1-desktop-amd64.iso
    

    Add --network bridge:br0 if we have added a new bridge network (br0) in the host (eg using netplan or nmcli or nm-connection-editor GUI Tool in Ubuntu). See some resource below.

  • How to install KVM server on Debian 9/10 Headless Server
  • How to install KVM on Ubuntu 20.04 LTS Headless Server.
    • IP 192.168.2.25 is the new static IP for the bridge network (not virtual machine) and 192.168.2.254 is the gateway current ethernet is using.
    • Note according to How to add network bridge with nmcli (NetworkManager) on Linux, WiFi is not possible. It only works with Ethernet. Confirmed! The bridge network can be pinged from another computer.
    • How to create wireless bridge connection with nmcli. Ans: This is not possible.
    • I follow this How to add network bridge with nmcli (NetworkManager) on Linux disable STP, turn off “Wired connection 1” and turn on br0.
    • Question: Do I need to turn off "Wired connection 1". It will lose the original host's static IP. The new bridge IP will become the host's IP. The guest machine still obtain a new IP via DHCP. This is not what I expect?
    • Before turn off “Wired connection 1” and turn on br0:
      $ nmcli connection show --active
      NAME                UUID   TYPE      DEVICE  
      Wired connection 1  XXX    ethernet  eno1    
      docker0             XXX    bridge    docker0 
      virbr0              XXX    bridge    virbr0 
      
    • After turn off “Wired connection 1” and turn on br0:
      $ nmcli connection show --active
      NAME               UUID   TYPE      DEVICE  
      br0                xxx    bridge    br0     
      docker0            XXX    bridge    docker0 
      virbr0             XXX    bridge    virbr0  
      bridge-slave-eno1  XXX    ethernet  eno1    
      vnet0              XXX    tun       vnet0 
      
    • I am not able to use remmina to VNC to the bridge VM. Not sure if this is because my host OS is a desktop (not headless).
  • os-variant list in virt-install
    $ osinfo-query os 
    


Where are image files saved

/var/lib/libvirt/images (/var/lib/docker for Docker)

From what I can see, the directory contains many large qcow2 files that should be deleted once I delete the VM.

VNC

  • KVM/QEMU 以 virt-install 指令建立虛擬機器、VNC 顯示畫面教學
    sudo virt-install --virt-type kvm --name ubuntu1804 --ram 2048 
      --disk /home/seal/kvm/ubuntu1804.qcow2,format=qcow2 
      --network network=default 
      --graphics vnc,listen=0.0.0.0,password=YOUR_PASSWORD 
      --noautoconsole 
      --os-type=linux --os-variant=ubuntu17.10 
      --cdrom=/home/seal/Downloads/ubuntu-18.10-desktop-amd64.iso
    
    virsh vncdisplay ubuntu1804 #  127.0.0.1:1
                                # pay attention to the port number
                                # :0  --> 5900 in ssh
                                # :1  --> 5901 in ssh
    sudo ufw disable
    
    # Another machine. 
    # Suppose the host machine has an IP 192.168.0.3
    ssh -NL 5901:localhost:5901 [email protected]
    ssh -NL 5901:127.0.0.1:5901 [email protected]
    
    vncviewer 127.0.0.1:1 # from a different machine
    

    NOTE: I have to use local port forwarding. If I try to directly connect to the VM in RealVNC, it will show The connection was refused by the computer. A limit by KVM?

The question is how do we know the client's IP? I guess cockpit method is better.

Install qemu-system-arm in Ubuntu

  • sudo apt install qemu-system-arm
  • How to emulate Raspberry Pi in QEMU which is based on qemu-rpi-kernel-Qemu kernel for emulating Rpi on QEMU.
    mkdir rpi-vm && cd rpi-vm
    git clone https://github.com/dhruvvyas90/qemu-rpi-kernel.git
    
    # Download Raspbian Buster Lite
    # from https://www.raspberrypi.org/downloads/raspberry-pi-os/
    dd if=/dev/zero bs=1M count=4096 >> 2020-08-20-raspios-buster-armhf-lite.img
    
    sudo losetup -f -P --show 2020-08-20-raspios-buster-armhf-lite.img
    
    sudo mount -o rw /dev/loopXp2 /mnt
    
    sudo nano /mnt/etc/fstab
    
    sudo nano /etc/ld.so.preload # Comment out the line and save the file. 
    # Then unmount with the following commands:
    # sudo umount /mnt
    # sudo losetup -d /dev/loopX
    
    qemu-system-arm \
      -M versatilepb \
      -cpu arm1176 \
      -m 256 \
      -kernel qemu-rpi-kernel/kernel-qemu-4.19.50-buster \
      -hda 2020-08-20-raspios-buster-armhf-lite.img \
      -append "dwc_otg.lpm_enable=0 root=/dev/sda2 console=tty1 rootfstype=ext4 elevator=deadline rootwait" \
      -dtb qemu-rpi-kernel/versatile-pb-buster.dtb \
      -no-reboot \
      -serial stdio
    
    # We can install a desktop. See 
    # https://www.raspberrypi.org/forums/viewtopic.php?t=133691
    

Install on CentOS

qemu-system-x86_64

qemu-system-x86_64 -name debian \
  -vnc 146.20.141.254:0 -m 1024 \
  -drive format=raw,index=2,file=debian.img \
  -enable-kvm -daemonize
If we use Virtual Machine Manager program, you need to have KVM (lsmod | grep kvm) installed on your system for this to work.

Install using qcow2 image

Running a virtual machine from a ramdisk

  • List of RAM drive software
  • How to Create a RAM Drive in Linux
    if [ "$(mount | grep -o "/mnt/ram")" != "/mnt/ram" ]; then
      sudo mkdir -p /mnt/ram
      sudo mount -t ramfs -o size=1g ramfs /mnt/ram
      sudo chown -R $(whoami):$(whoami) /mnt/ram
    fi
    mount | grep ram
    
    sudo umount /mnt/ram
    
  • https://youtu.be/fOaWrhLoVK0
  • Libvirt In-Memory Storage
  • (Windows only) USE A RAM DISK TO GET EXTREME PERFORMANCE FROM TEMPORARY VIRTUAL MACHINES
  • How to Easily Create RAM Disk on Debian, Ubuntu, Linux Mint, CentOS. How to Create a RAM Disk in Any Linux Distro.
    sudo mkdir /tmp/ramdisk
    sudo chmod 777 /tmp/ramdisk
    htop  # check free ram
    sudo mount -t tmpfs -o size=1G myramdisk /tmp/ramdisk
                # myramdisk is the device name I gave to it
    mount | tail -n 1
    sudo umount /tmp/ramdisk/
    

    Test RAM speed

    # test write speed
    sudo dd if=/dev/zero of=/tmp/ramdisk/zero bs=4k count=100000
    ## 409600000 bytes (410 MB, 391 MiB) copied, 0.734813 s, 557 MB/s  # SATA SSD
    ## 409600000 bytes (410 MB, 391 MiB) copied, 0.18296 s, 2.2 GB/s   # NVME
    
    # test read speed
    sudo dd if=/tmp/ramdisk/zero of=/dev/null bs=4k count=100000
    ## 409600000 bytes (410 MB, 391 MiB) copied, 0.410809 s, 997 MB/s  # SATA SSD
    ## 409600000 bytes (410 MB, 391 MiB) copied, 0.139487 s, 2.9 GB/s  # NVME
    

    Auto-mount on System Boot

    sudo nano /etc/fstab
    ## myramdisk  /tmp/ramdisk  tmpfs  defaults,size=1G,x-gvfs-show  0  0
    
    ## To mount it immediately without reboot, run the following command.
    sudo mount -a 
    

    How to Run VirtualBox VM on RAM Disk. You should set the machine folder to the RAM disk directory (/tmp/ramdisk/).

kvm command

snapshot

  • How to Create, Revert and Delete KVM Virtual machine (domain) snapshot with virsh command, How to create snapshot in Linux KVM VM/Domain
    $ sudo su
    # virsh list --all
    
    # virsh snapshot-create-as --domain win10 --name win10_snap --description "Windows 10 snapshot"    # Create a snapshot
    # virsh snapshot-list win10
    # virsh snapshot-info --domain win10 --snapshotname win10_snap
    
    # virsh snapshot-revert win10 win10_snap    # Restore
    
    # virsh snapshot-delete --domain win10 --snapshotname win10_snap
    
  • To create a snapshot using the GUI, open virtualization manager window. In the VM properties, click on the icon (two displays overlapped) which is on the right of the menu, which says Manage VM snapshots.
  • To restore a snapshot using Virtual Machine Manager, right click on a snapshot and select "Start snapshot".

GUI

Virtual Machine Manager/virt-manager, Cockpit, virt-viewer, remote-viewer

sudo apt install cockpit cockpit-machines
  • Creating Ubuntu 20.04 KVM Virtual Machines using Cockpit and virt-manager*.
    • It shows how to handle the error of cannot opening the ISO file.
    • It shows how to use virt-viewer to connect to a VM (virt-viewer)
    • remote-viewer will be used if we change the Console type from 'VNC' to 'Graphics Console in Desktop Viewer' and click the 'Launch Remote Viewer' button. A text file called 'download' will be downloaded. If we click it, it will launch remote-viewer. An interesting thing is the file will be automatically deleted once we click on it:)
  • Consider running a VM on a remote machine with cockpit version 215.
    • For Ubuntu 20.04.1 Desktop. If I increase the display resolution from 1024x768 (4:3) to others without keeping the same ratio, the mouse calibration has a problem:( Maybe it's better not to change the resolution. The VNC method works.
    • For Windows 10, we need to unplug ethernet first in order to create a local account. I've not found a way to use virt-viewer or remote-viewer to view the desktop locally. I can still use cockpit web interface (Consoles) to use the Windows desktop. I am able to increase the display resolution to 1280x800 from 1024x768 which makes the interface easy for my eyes. One problem is the machine will freeze for some unknown reason.

Network Install in virtual machine manager

Share folders

How to quit full screen

How to exit full screen mode in qemu virt-manager ( RHEL / CentOS 7). Hint: click the first of two icons shown at the top-center screen.

Windows VM

How to Create a Windows Virtual Machine in Linux With KVM

Windows VM by docker

Running Windows in a Docker Container!

GNOME Boxes

Network

Find out the IP of the guest machine

# Method 1:
$ virsh list
$ virsh domifaddr centos8

# Method 2:
$ sudo apt install net-tools   # Debian
$ sudo dnf install net-tools   # CentOS
$ arp -n               # find the rows with virbr0 adapter

# Method 3:
# KVM -> Info -> NIC

Backup and restore images

How to KVM Backup and Restore in Linux

# Backup 
virsh list --all
virsh dumpxml win10 > /MyBackup/win10.xml
sudo cp /var/lib/libvirt/images/win10.qcow2 /MyBackup/win10.qcow2
sudo chown $USER:$USER /MyBackup/*.qcow2

# Restore Making
sudo virsh undefine win10
ls -l /etc/libvirt/qemu
virsh list --all
sudo rm /var/lib/libvirt/images/win10.qcow2
sudo cp /MyBackup/win10.qcow2 /var/lib/libvirt/images/
sudo virsh define --file /MyBackup/win10.xml
# virsh start win10

Cheat sheet

KVM Cheatsheet

Vagrant

How to Use Virtualbox VMs on KVM In Linux

How to Use Virtualbox VMs on KVM In Linux

Run macOS on KVM / QEMU

Run SteamOS on QEMU

Running the Steam Deck’s OS in a virtual machine using QEMU

QuickEMU

Some lessons:

  • By default new folders are created when we create/download guest machine in the home folder. We can create a new folder for managing these guest machines.
  • Use Ctrl+Alt+g to release the mouse. The hint is at the top of window.
  • Change boot="legacy" in windows-11.conf helps to show the Windows installation process (See Simple shell script to "manage" Qemu virtual machines.). But it still failed Windows could not create a partition on disk 0. The error occurred while applying the unattend answer file's <DiskConfiguration> setting. Error code: 0x80042565.
  • When I tried to install quickemu on another freshly installed xubuntu 20.04 focal fossa (Core2 Quad CPU), the installation went through without any problem.
  • The app running the guest machine is spicy.
  • I can install macOS Catalina, Windows 11, etc. The trick to install macOS is manually reboot the guest machine a couple times. Also it is useful to play the "Options -> Scale display" option in "spicy" to better see the text when we install macOS. Then it works.
  • I need to use Shift + F12 (see the hint at the bottom of spicy window) to release mouse (or exit full display) in spicy (the GUI where the guest machine is running) on my Ubuntu. When installing macOS, I need to use my keyboard (mouse does not work) to select the hdd for booting macOS (or just wait a few seconds & the 1st drive will be used to boot).
  • It helps to drag and enlarge the guest machine window (by default 'Scale display' option is checked).
  • When installing macOS, the screen says "About X minutes remaining". But it seems it stuck there (the minutes is not changed) though the CPU is busy. If I closed the window, shutdown the guest machine and reboot it, installation continues. Doing this a few times and eventually a new window ("Select Your Country or Region") will come up. In order to monitor the download, I open another terminal and run something like watch du -s ~/macos-big-sur. This will help me to decide when to restart the VM. For big-sur, the folder size stops at 26655436 and after I reboot the VM, the folder size keep increasing and stops at 32565600. For the 3rd reboot, it still downloads something and needs 2 internal reboots. So I think the problem is the data needs to be downloaded from Apple server is TOO large.
  • macos-big-sur & macos-monterey took about 32G & 34G space.
  • Every time I boot macOS, I shall use the right arrow to select the 2nd icon and press the Enter button to start.
  • macOS Catalina asks at least 4 characters for the password (I choose 6)
  • For macOS Catalina/Monterey guest machine, it automatically allocates 2 CPUs(?), 4/8GB RAM (depending on the host total RAM?), 100GB storage by default. After installing macOS, 75GB will still be available.
  • (Maybe related to Ubuntu 22.04 host using wayland). Whenever I start macOS vm, it pops up a window "Allow inhibiting shortcuts. The application spicy wants to inhibit shortcuts. You can restore shortcuts by pressing Super+Escape." I choose the default "Allow" and it seems to be OK. PS: I need to do this action quickly or the macOS window will go to black and not responds to my keyboard/mouse.
  • For Windows 11 guest machine, it allocates 2 CPUs, 4/8GB RAM, 64GB storage with 43GB free.
  • For Lubuntu guest machine (much better speed than macOS/Windows VM, fill full screen, Qt/KDE based), it allocates 2 CPUs, 4GB RAM, 16GB storage with 8.4GB free.

Build QEMU from source

Nested KVM

How to Enable Nested KVM Virtualization