Self hosting

Resource

• https://forum.proxmox.com/
• Helpful Youtube Channels for Self-Hosting
• Awesome-Selfhosted in Github

Proxmox Virtual Environment

• Bare metal hypervisor
• Discovered it from the post
• Learning Proxmox VE (book)
• Mastering Promox (book)
• Iso image location in proxmox: /var/lib/vz/template/iso
• Proxmox as a home virtualization solution
• Proxmox – a great Hypervisor for running servers
• What are Proxmox advantages vs VMware?
• Comparison of Server Virtualization
• Videos
  • Proxmox VE 6.0 Beginner Tutorial - Installing Proxmox & Creating a virtual machine. Need 3 drives; one for proxmox, one for ISOs and one for VMs.
  • Launching a Windows VM in Proxmox 9/28/2021
  • $149 Low Cost Virtual Machine Server & installation process
• ProxMox; A VM Server For Your Home
• Docker
  • Running Docker on Proxmox
  • Running Docker on Proxmox
  • (TUTORIAL) Proxmox with Docker (and Portainer)
• Proxmox vs Docker for a Home server
  • Using a vm give you snapshots, easy backups, live migration and easy swapping between server
  • If you have any "security" concerns, there's less chance of a breakout from a VM than from docker. It is also much easier to move between networks / vlans as a VM than a container.
• Install with single SSD? from Reddit
• Before I do anything on Proxmox, I do this first...
• How to upgrade Use apt dist-upgrade or the GUI, NOT apt upgrade!

Set up

• FQDN
  • Change FQDN pmgchanged.proxmox.com
  • Checking Hostname (FQDN): hostname -f
  • Hostname (FQDN).......huh?
  • How to install the Proxmox Virtual Environment virtualization management solution pve.example.invalid
  • pve.pi5. It means the hostname is "pve". Installing Proxmox 8.1 on Raspberry Pi 5
• My NVMe SSD was not detected. It turns out I need to choose ACHI instead of RAID in SATA Operation under System Configuration in BIOS. See AHCI VS RAID: Which One Should I Choose for Better Performance?.
• Proxmox VE 7 Initial Installation Checklist. 2021
  • To access it, use https://IP:8006
  • Username is 'root'.
  • Change to pve-no-subscription Repo. We need to modify package sources before we run apt update and apt dist-upgrade
• My initial setup (disk storage is 240GB). However in web page, the 'Disks' tab will show 240GB for /dev/sda and /dev/sda3 (LVM) shows 93% of usage.

  root@pve:~# df -h
  Filesystem            Size  Used Avail Use% Mounted on
  udev                  7.7G     0  7.7G   0% /dev
  tmpfs                 1.6G  1.3M  1.6G   1% /run
  /dev/mapper/pve-root   55G  3.1G   49G   6% /
  tmpfs                 7.8G   37M  7.7G   1% /dev/shm
  tmpfs                 5.0M     0  5.0M   0% /run/lock
  /dev/sda2             511M  328K  511M   1% /boot/efi
  tmpfs                 1.6G     0  1.6G   0% /run/user/0
  /dev/fuse             128M   16K  128M   1% /etc/pve
  

  I don't have a separate disk for storing ISOs. I use the local disk. The iso files are saved in /var/lib/vz/template/iso directory. The bridge network works so I can ssh to the new virtual machine (after install openssh-server).

• Proxmox Beginner Tutorial: How to set up your first virtual machine on a secondary hard disk
  • You need at least 2 physical drives; one for proxmox and the other for virtual machines.
• Trying out Proxmox VE, an open-source virtualization platform 2021
• How to build a Self-hosted environment on Linux with Proxmox

Linux VM

• Make sure the storage is LVM so we can extend it later
• Install and start spice and qemu agents.

sudo apt install spice-vdagent # may be installed already
sudo systemctl start spice-vdagent # needed
sudo apt install qemu-guest-agent
sudo systemctl start qemu-guest-agent

Windows VM

• How to Unlock a Proxmox VM. Then reboot pve.
• For Windows installation, if we are dropped at the EFI shell, we can use reset -s to shut down the system.
• Windows VirtIO Drivers & Creating Windows virtual machines using virtIO drivers from fedoraproject.org
• (Videos) Launching a Windows VM in Proxmox (Win10), Virtualize Windows 10 with Proxmox VE (Win10)
• In OS tab, be sure to choose the right Guest OS Version; if we choose (10/2016/2019), we will select 2019 in the driver folder locations OR if choose (11/2022), we will select the corresponding driver folders. This affects the later steps when we select the drivers to install.
• There are 3 drivers we shall install.
  • vioscsi\win10\amd64\vioscsi.inf (we need it in order to see the virtual disk)
  • NetKVM\w10\amd64\netkvm.inf (in order to get network working, this can be installed later from Windows Device Manager)
  • Balloon\win10\amd64\balloon.inf (memory balloon driver, this can be installed later)
  • Guest agent (Qemu-guest-agent). Make sure we have installed all missing drivers from Device Manager. We can go to virtual machine Summary tab to check if the IPs shows the Windows IP.
• Balloon driver
  • If balloon driver is not installed on Windows Virtual Machines then memory metrics will not be collected for windows virtual machine. Steps to Install balloon driver on Windows Virtual Machines
  • A balloon driver is a component of the VMware Tools package of drivers and utilities that help virtual machines (VMs) run better. The purpose of the balloon driver is to take physical RAM from a VM and release it back to the VMkernel. This reclaim usually happens when the ESXi server is short on RAM and this specific VM is the loser in the competition for physical RAM. How the balloon driver helps VM performance
• Change network to use a static IP.
• Enable RDP.
• (Related to Audio device). Change Hardware - Display - SPICE (instead of default). W/o doing that, we'll get an error "Can't start vms with audio device(SPICE) in pve 7.3 "
  • Might be better to disable audio so we don't need to use SPICE for Windows
  • SPICE driver - Proxmox wiki.
  • Windows 7 mouse cursor alignment in VNC console when SPICE is enabled
  • Spice with Windows 10 - Mouse Does Not Work. Download spice tool does get rid of the 2nd cursor. But the response seems to be getting slower and alignment is not quite correct. So in the end I still disable audio device and switch from SPICE to default.
  • Windows 10 guest best practices
• For Audio, we need to add it to the hardware list (driver=spice is OK). PS: audio works when I tested using the Microsoft Remote Desktop app on mac. It does not work when I use Remmina app from Ubuntu initially but if I change the sound setting (from "Off" to "Local" in the "Advanced" tab) it works.
• Windows 11 (10/21/2021). We can still use 10/2016/2019 for the Version selection. When I check the Task Manager (Windows 11 Pro, Version 21H2, OS build 22000.675), it shows 1.8/4GB was used and 101 out of 127GB is free.
  • OS: version 10/2016/2019
  • System: BIOS: OVMF (UEFI). Machine - q35. Add TPM. Qemu Agent. SCSI: VirtIO SCCI
  • HD: Bus-VirtIO Block. Disk size >=32
  • CPU: Cores >=2. Type - Host
  • Memory
  • Network: VirtIO
  • After finish, adding CD/DVD - VirtIO iso
  • Install: Browse VirtIO disk. AMD - win10. Next, browse NetKVM - win10.
  • After reboot. Continue to answer questions. Reboot
  • File manager. VirtIO disk. Run virtio-win-gt-x64.exe.
  • Right click on Start. Device Manager. All clean.
• Windows 11 VM / Office 2019 install error 0-2054
• Windows 11 virtual machine on Proxmox 8/19/2022
• AMD/NVIDIA GPU Passthrough in Window 11 - Proxmox Guide 7/22/2022
• Windows 11 23H2 VM in Proxmox Tutorial - 2024. It works.
  • ISO: Win11_23H2_English_x64v2.iso. Guest OS: leave it to Linux.
  • System: UEFI. Select UEFI storage. Check TPM. Select TPM storage.
  • The key in all the steps is selecting SATA in Disks. Set disk size to 64.
  • CPU: 2 cores. Memory: 8192.
  • Shift + F10. Type OOBE\PASSNRO . See Set up Windows 11 without internet.
  • For internet connection, attach Virtio ISO (I'm using the version 0.1.240) and run "virtio-win-gt-x64". Go to Device Manager and check the ethernet problem is gone.
  • It shows 48.6GB free of 63.1GB.

Mac VM

• Installing macOS 12 “Monterey” on Proxmox 7
• Running a MacOS Monterey VM in ProxMox VE
• Installing macOS 13 Ventura on Proxmox 7.2. It works (tested 2/11/2023).
  • I also got an error "macOS Update Assistant”. After I clicked "x" to dismiss it, I can continue the installation.
  • MacOS requires at least 4 characters for the password (something like 'asdf' will suffice).
  • I followed the instruction up to 'Sleep management'.
  • It took about 1 hour.
  • The default resolution is 1920x1080. So I need to use noVNC and choose 'Fullscreen' (ESC to return back) from the control bar on LHS.
  • On the new Ventura system, the memory usage is 1.7GB. So the default 4GB is still enough.
  • How to turn on SSH server on Mac in macOS Ventura 13 or newer.
  • The current IP address can be found by ifconfig command or ipconfig getifaddr en0 or System preferences-Network.
• OSX-PROXMOX - Run macOS on ANY Computer - AMD & Intel
• Hackintosh Install Script For Proxmox

Upgrade

• Upgrade from 7 to 8
• How-to: Proxmox VE 7.4 to 8.0 Upgrade. I got an email notification with the title "apt-listchanges: news for pve" for the first time. root@EMAILAddress to root@EMAILAddress.
  • https://github.com/tteck/Proxmox
• Not So Simple To Upgrade Proxmox 8.1 with SDN (software defined network)

Install on Debian

Installing Proxmox VE 7.x on Debian Bullseye for custom partition layout (video)

Cheat sheet

https://github.com/vzamora/Proxmox-Cheatsheet

SSD/HDD choices

• First time installation of PVE - SSD/HDD choices
• Try these commands to avoid heavy wear on SSD's (only if not clustered) Do I need a SSD reserved for Proxmox?

Home Server

My Proxmox Home Server Walk-Through

• Part 1 (TrueNAS, Portainer, Portainer, Wireguard),
• Part 2: Remote Gaming and Streaming

ZFS and RAID

• ZFS (Zettabyte File System): A file system developed by Sun Microsystems for use in their Solaris operating system. It is now available on many other operating systems.
• Yes, ZFS can be used without LVM. Even on a workstation, you could use ZFS to pool your disks into a single large pool of storage rather than keep them separate or rely on LVM. How to Install and Use ZFS on Ubuntu (and Why You’d Want To)

• How to configure Proxmox storage (ZFS + RAID10) from the video 'Before I do anything on Proxmox, I do this first...'.

• ZFS vs RAID-0
  • ZFS is not like RAID-0. RAID-0 is a type of RAID that stripes data across multiple disks without any redundancy. If one disk fails, all data is lost. ZFS, on the other hand, provides data redundancy and checksumming to avoid silent data corruption.

• ZFS cons
  • XFS vs ZFS vs Linux Raid. ZFS is a very CPU-intensive filesystem. This can lead to slower performance on systems with limited CPU resources.

• Proxmox installation.
  • The default file system is ext4. ZFS is an alternative to ext4.
  • ZFS on linux
  • The Proxmox VE installer, which partitions the local disk(s) with ext4, XFS, BTRFS (technology preview), or ZFS and installs the operating system.
  • The main advantage of ZFS over EXT4 is guaranteed data integrity . ZFS protects your data by enabling volume management on filesystem level. EXT4 does not offer volume management on the filesystem level
  • (Video) Setting Up NAS Server On Proxmox

• ZFS vs EXT4: Best File System for Linux and Other Operating Systems
• RAM
  • How *I* Reduced Proxmox RAM Usage with NFS Instead of ZFS
  • The memory usage in Proxmox is higher than expected

Users, groups

Create Users, Groups and Assign Permissions in Proxmox VE

Live session

• To create a VM that is not meant to be installed to a disk, just make sure no disks have been added.
• Tested on Ubuntu 24.04 desktop.
• For the RAM,
  • If I use 6144 as minimum and 8192 as max. "df -h" shows 3.7G as "/" and 3.7G as "/tmp".
  • If I use 8192 as minimum and 8192 as max. "df -h" shows 3.9G as "/" and 3.9G as "/tmp".
  • If I use 5120 as minimum and 6144 as max. "df -h" shows 2.4G as "/" and 2.5G as "/tmp".
• It took about 1 minute to show up the desktop no matter what RAM allocation is used. I can press "x" to close a window that asked about a few questions. I allocate 4 VCPU. The CPU on the host is i5-8500T @2.1GHz.

After installation

Don’t run Proxmox without these settings!

Change subscription repository

• Package Repositories. Comment out the line in /etc/apt/sources.list.d/pve-enterprise.list and add a line deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription to "/etc/apt/sources.list" file.
• See Proxmox Update No Subscription Repository Configuration for Proxmox 8.

Removing ProxMox Subscription Notice

• Removing ProxMox Subscription Notice.
• How to: Remove “You do not have a valid subscription for this server….” from Proxmox Virtual Environment/Proxmox VE (PVE 6.1 to 7.1 and up) works. I find I need to use Ctrl + F5 to force a page to reload, ignoring the cache files for that page. We may need to restart the browser too. On macOS, we can use Shift + Cmd + R. No reboot is needed!

.bashrc and ls colors

• nano .bashrc and uncomment 5 lines so "ls" output has colors.
• nano .bashrc and add the following. It will change the "ls" directory color to CYAN.

export LS_COLORS="di=1;36"

Login timeout

Proxmox GUI Session Timeout. The login ticket is valid for 2 hours and gets refreshed every 15 minutes.

Login in browser

• Import certificate in browser
• User Management. Authentication Realms
  • Proxmox VE Authentication Server
  • Linux PAM Standard Authentication. What is my username on my RStudio Workbench / RStudio Server installation?.
• How to save PVE Web Loginpassword on Firefox/Chrome?
• Mobile browser

SPICE

Display, copy-and-paste

• noVNC is still a type of VNC.
  • noVNC is an open source VNC client that runs well in any modern browser including mobile browsers (iOS and Android). It is both a VNC client JavaScript library as well as an application built on top of that library. noVNC follows the standard VNC protocol, but unlike other VNC clients it does require WebSockets support.
  • Kali In The Browser (noVNC)
• VNC Client Access
• By default, Proxmox assigns a standard VGA device for BIOS-based virtual machines and a QXL device for UEFI-based virtual machines.
• For Windows OS, we can use the default (noVNC)
  • QXL: How to Install virtio Drivers on KVM-QEMU Windows Virtual Machines
• For Linux OS, SPICE is better (clipboard in noVNC does not work). Screen can be scaled as we want. Copy and paste still does not work after I installed spice-vdagent. virt-manager Spice copy paste doesn't work. But the copy-paste function works in Debian VM launched by Virtual Machine Manager (actually the menu bar are different. One is called 'Remote Viewer'/remote-viewer and the other embedded viewer from /usr/bin/qemu-system-x86_64 is called 'QEMU/KVM').
  • (Debian11 and antiX VMs) When I use "ps -ef | grep spice", I got /usr/sbin/spice-vdagentd & /usr/bin/spice-vdagent as shown in the screenshot here for the VM launched by QEMU/KVM. But I did not see /usr/bin/spice-vdagent in the VM launched by Proxmox.
  • B/C a hint from above, I found a solution here. After I run /usr/bin/spice-vdagent , copy-and-paste works!
  • (Fedora 35). Copy-and-paste works out of box (vdagentd & vdagent are automatically running in the background). Maybe it's because Fedora is a Red Hat based Linux OS.
  • How to add spice-vdagent to VirtIO VM?
• Summary,

  sudo apt install spice-vdagent
  sudo systemctl start spice-vdagent

• Comparison

  Tool	Pros	Cons
  noVNC	It is a lighter approach, as it has less services required (less overhead), which allows for a quick “one off connection” solution. It is an open source VNC client JavaScript library as well as an application built on top of that library. It runs well in any modern browser including mobile browsers (iOS and Android).	The clipboard does not work. Audio device?
  Spice	presents the guest windowing system with an X driver that captures X protocol operations directly. This means that Spice can provide better performance than VNC.	It requires more services than noVNC.

Sound/audio

• https://en.wikipedia.org/wiki/Simple_Protocol_for_Independent_Computing_Environments
• SPICE (Simple Protocol for Independent Computing Environments) is a communication protocol used for virtual environments. It provides a remote display system, allowing users to view a virtualized desktop on their local machine and interact with it using keyboard and mouse input.
  • SPICE is often used in conjunction with virtualization platforms such as QEMU/KVM, and is widely used in enterprise and cloud computing environments.
• SPICE (Simple Protocol for Independent Computing Environments) does not have any direct alternatives as it is a specific communication protocol used for remote display in virtualized environments. However, there are other remote display protocols such as RDP (Remote Desktop Protocol), VNC (Virtual Network Computing), and NX (NoMachine's Remote X protocol) that can be used as alternatives to SPICE in certain situations. Nonetheless, the most suitable protocol for a specific use case depends on various factors such as the nature of the application, the network bandwidth available, the desired level of graphics performance, and more.
• https://pve.proxmox.com/wiki/SPICE
  • Add sound hardware to VM
  • Change Display from default to Spice
  • (For Lubuntu) sudo apt install spice-vdagent spice-webdavd
• Choose SPICE when launching the VM, it will download a vv file.
• In ubuntu, "remote-viewer" will be used to open the vv file when we double clicked the downloaded file. Proxmox SPICE console apt install virt-viewer

• In macOS, see Proxmox - SPICE client setup for MacOS. In fact, the resolution of VM on macbook pro is poor. Using noVNC seems to be good enough.

brew tap jeffreywildman/homebrew-virt-manager
brew install virt-viewer
remote-viewer pve-spice.vv

Share a folder

• To share a folder, you should install spice-webdavd daemon inside the Proxmox VM. Also make sure the spice-vdagent service is installed and running in the VM in order to fix display resolution issues.
• https://pve.proxmox.com/wiki/SPICE. HINT: only the Linux version of Virt-viewer (remote-viewer) supports this.
• Share Files Between Guest and Host OS in GNOME Boxes

Remote Desktop through browser

• How to: Create a Remote Desktop Gateway server on Proxmox VE (PVE)

Guest agent

This affects whether we can see IP in the Summary option.

• Qemu-guest-agent
• Windows 10 guest best practices including nested virtualization for installing WSL. Just make sure to select host as processor.
• RAM usage
• RAM always like this since install, any idea why?

• For Ubuntu VM,
  • Proxmox -> VM -> Options -> QEMU Guest Agent. Check both options: Use QEMU Guest Agent & Run guest-trim after a disk move or VM migration.
  • On Ubuntu VM,

  sudo apt-get install qemu-guest-agent
  sudo systemctl start qemu-guest-agent
  

• For Windows VM,
  • Proxmox -> VM -> Options -> QEMU Guest Agent. Check the 1st option is enough.

Improve performance

• Allocate Sufficient Resources
• Use VirtIO Drivers
• Install QEMU Guest Agent:
• Enable GPU Passthrough (Optional)

can't shutdown a VM

Use the command qm unlock XXX

qm stop XXX
# can't lock file '/var/lock/qemu-server/lock-996.conf' - got timeout

qm unlock XXX
qm stop XXX

Now we can go back to proxmox GUI to remove the vm.

But if the "qm unlock" does not work, we can use the kill command. See Proxmox can’t stop VM – How we fix it!

ps aux | grep "/usr/bin/kvm -id VMID"
kill -9 PID  # VM will stop

Errors and solutions

• Error 0 occurred while receiving the document.: it's a browser problem. Change from Chrome to FF fixed it.

Read-only system

• fsck -f -c -y /dev/mapper/pve-root and on the vm drive fsck -f -c -y /dev/nvme0n1p1
• Read-only file system on proxmox server. The filesystem will usually go into read-only while the system is running if there is a filesystem consistency issue. This is specified in fstab as errors=remount-ro and will occur when a FS access fails.
• journalctl -b showed EXT4-fs error.

Memory usage

• PVE showing high memory usage but VM is not. It's just the cache. Please look at the yellow bar in htop.
• Reduce the minimum memory in Hardware settings.

Network

• https://pve.proxmox.com/wiki/Network_Configuration
• What is IP Masquerading?
  • IP Masquerading, helps machines with non-routable IP addresses to access the Internet via the machine that is masquerading. It is a form of NAT (Network Address Translation).
  • What is IP masquerading and when is it of use?
  • Beginners Guide to Masquerading and Port Forwarding in CentOS/RHEL 7
• Virtualizing An Internal Network With pfSense In ProxMox (video)
• VLAN:
  • Configuring VLANs on Proxmox- An Introductory Guide
  • Before I do anything on Proxmox, I do this first... (video)
• How to Assign a Second IP Address to Proxmox VE Hosts
• NAT (default network in Virtualbox)
  • Create Private Network Bridge on Proxmox VE with NAT
  • *Proxmox and using NAT with a Virtual Machine 2012
  • Proxmox networking setup in a Hetzner data center
  • Enable NAT Networking Mode from How to Install Proxmox VE
• DHCP
  • Using Dnsmasq DHCP Server in Proxmox for VMs IP

Ethernet port

How many Ethernet ports do I need on my Proxmox?

Linux bridge commands

An introduction to Linux bridging commands and features

LXC

• LXC vs Docker: Which Container Platform Is Right for You?
• Linux Container and pveam tool

  # pveam update
  # pveam available
  # pveam available --section system
  # pveam download local debian-10.0-standard_10.0-1_amd64.tar.gz
  # pveam list local
  

• Download http://download.proxmox.com/images/system/
• Debian Appliance Builder, VM Templates and Clones
• By default static IP was chosen when we created the container. If we don't specify anything then the container has no internet. So we need to change to DHCP in network setting in order to get internet connection (tested on Debian 10 template). ubuntu 20.04 LXC + sshd uses only 37MB memory. It runs 22 processes. Booting is very fast! (cf ubuntu 20.04 server + sshd uses 145MB and runs 99 processes)
• Password needs at least 5 characters.
• To allow ssh log in (it's OK even firewall checked), we need to modify /etc/ssh/sshd_config file (so we can use the root account to log in). See How to SSH to LXC containers?

  LoginGraceTime 2m
  PermitRootLogin yes   # prohibit-password
  StrictModes yes
  

• This LXC looks very much like a server VM not Docker (we can also install desktop environment in an LXC)
• Convert Docker image to Proxmox lxc
• Using a Desktop with a LXC Proxmox 7, Is there a GUI for Ubuntu Desktop in a container?
• New users (eg brian)

  adduser brian
  usermod -aG sudo brian
  mkdir -p /home/brian/.ssh
  # if we want to copy the ssh key from root to the new user
  cp ./.ssh/* /home/brian/.ssh
  chown -R brian:brian /home/brian/.ssh/
  

Proxmox help scripts

• Proxmox Help Scripts
• Proxmox Automation with Proxmox Helper Scripts!
• 40+ Scripts To Streamline Your Proxmox Homelab

LXC images

• https://images.linuxcontainers.org/images/. See Installing Proxmox 8.1 on Raspberry Pi 5
• Download templates LXC containers. Rename rootfs.tar.xz to be the name you want for the template ie debian_bookworm_20230714.tar.xz. You can then use it as a container template as normal.
• Variants, cloud vs default images.
• Desktop images for LXD VM
• Desktop with LXC:
  • installing a desktop environment in an Ubuntu LXC container can work, but there are some considerations to keep in mind. LXC containers are designed to be lightweight and do not include all the components of a full virtual machine, which can affect how a desktop environment operates within them.
  • you may encounter issues with services that expect to interact with hardware directly, as containers are more restricted compared to virtual machines.
• To use images from https://images.linuxcontainers.org/
  • Install LXC/LXD
  • Initialize LXD: lxd init
  • Use the lxc command to download the desired desktop image; e.g., lxc launch images:ubuntu/20.04/desktop --vm
  • Launch the container
  • Access the container
  • Install a desktop environment
  • Set up remote access
  • Connect to the desktop environment

Pi hole

Installing Pi-Hole inside a Proxmox LXC Container. 2GB disk, 1 CPU core, and 256MB of memory. The memory usage is pretty flat around 53MB according to Proxmox gui). I am using Debian 11 template.

apt update
apt upgrade
nano /etc/sysctl.conf # disable IPv6 
reboot
apt install curl
curl -sSL https://install.pi-hole.net | bash
pihole -a -p # change to a simpler password

QCOW2

• How To Import QCOW2 Image Into Proxmox
• qcow
• The “disks.qcow2” file is a disk image format used by QEMU virtualization software, and it’s not typically used directly for LXC containers in Proxmox. A QCOW2 file is a disk image saved in the second version of the QEMU Copy On Write (QCOW2) format, which is used by QEMU virtualization software.
• DietPi. See Unraid case for the kernel panic issue.

SMART and wearout

I saw the wearout is 99% on my host disk (240GB Kingston SSD). I cannot delete a VM. If I use the command "qm destroy XXX", it shows "Unable to create output file '/var/log/pve/tasks/1/UPID:pvv....:qmdestroy:108:root@pam:' - Read-only file system". The host disk uses only 37% of storage on root partition. The solution: reboot Proxmox.

• Proxmox on SSD wearout
• Proxmox VMs on SSD - high wearout
• SSD Wear Out Calc?

Storage Drive

• Format a disk Prepare the drive

  fdisk /dev/nvme0n1
   : p
   : d
   :   ENTER
   : p
   : w
   : n ENTER ENTER ENTER
   : p
   : w
   : q 
  

  Now to go GUI, pve -> Disks -> Directory -> Create Dir.
• lsblk, df -h and more

  # lsblk
  NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
  sda            8:0    0 223.6G  0 disk 
  ├─sda1         8:1    0  1007K  0 part 
  ├─sda2         8:2    0   512M  0 part /boot/efi
  └─sda3         8:3    0 223.1G  0 part 
    ├─pve-swap 253:0    0     8G  0 lvm  [SWAP]
    └─pve-root 253:1    0 215.1G  0 lvm  /
  nvme0n1      259:0    0 465.8G  0 disk 
  └─nvme0n1p1  259:1    0 465.8G  0 part /mnt/pve/vm1
  
  # df -h
  Filesystem            Size  Used Avail Use% Mounted on
  udev                  7.7G     0  7.7G   0% /dev
  tmpfs                 1.6G  1.3M  1.6G   1% /run
  /dev/mapper/pve-root  214G   30G  176G  15% /
  tmpfs                 7.8G   40M  7.7G   1% /dev/shm
  tmpfs                 5.0M     0  5.0M   0% /run/lock
  /dev/nvme0n1p1        458G  2.0G  433G   1% /mnt/pve/vm1
  /dev/sda2             511M  328K  511M   1% /boot/efi
  /dev/fuse             128M   16K  128M   1% /etc/pve
  tmpfs                 1.6G     0  1.6G   0% /run/user/0
  
  # fdisk -l
  Disk /dev/nvme0n1: 465.76 GiB, 500107862016 bytes, 976773168 sectors
  Disk model: WDC WDS500G2B0C                         
  ...
  Device         Start       End   Sectors   Size Type
  /dev/nvme0n1p1  2048 976773134 976771087 465.8G Linux filesystem
  
  Disk /dev/sda: 223.57 GiB, 240057409536 bytes, 468862128 sectors
  Disk model: KINGSTON SA400S3
  ...
  Device       Start       End   Sectors   Size Type
  /dev/sda1       34      2047      2014  1007K BIOS boot
  /dev/sda2     2048   1050623   1048576   512M EFI System
  /dev/sda3  1050624 468862094 467811471 223.1G Linux LVM
  
  Disk /dev/mapper/pve-swap: 8 GiB, 8589934592 bytes, 16777216 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  
  Disk /dev/mapper/pve-root: 215.07 GiB, 230925795328 bytes, 451026944 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  
  # pvdisplay
    --- Physical volume ---
    PV Name               /dev/sda3
    VG Name               pve
    PV Size               <223.07 GiB / not usable <3.57 MiB
    Allocatable           yes (but full)
    PE Size               4.00 MiB
    Total PE              57105
    Free PE               0
    Allocated PE          57105
    PV UUID               4taiYE-DDJa-4UdU-v3QY-kd2s-7r5i-shhJ7Q
  

• Benchmark HD speed

  hdparm -t --direct /dev/nvme0n1p1
  hdparm -t --direct /dev/sda3
  

• lvresize vs lvextend. lvextend can only increase the size of a volume, whereas lvresize can increase or reduce it. Increase the size of an LVM logical volume.
• How to Manage and Use LVM (Logical Volume Management) in Ubuntu

Storage type

https://pve.proxmox.com/wiki/Storage

local vs local-lvm

• What is the difference between “local” and “local-lvm” on Proxmox VE (PVE)? Which to use? Why use local/local-lvm?
• local-lvm is actually a lvm-thin volume .
• local: The path is /var/lib/vz and vz is a folder.

  root@pve:~# tree -d /var/lib/vz/
  /var/lib/vz/
  ├── dump
  ├── images
  └── template
      ├── cache
      └── iso
  
  6 directories
  

• local-lvm: This is not a directory. In a simple way, lvm-thin storage - like a physical hard drive (image), /dev/pve/vm-100-disk-1 like a partition on that drive. local-lvm path?

  root@udoo:~# ls -l /dev/pve
  total 0
  lrwxrwxrwx 1 root root 7 Jul 18 17:40 root -> ../dm-1
  lrwxrwxrwx 1 root root 7 Jul 18 17:40 swap -> ../dm-0
  lrwxrwxrwx 1 root root 7 Jul 18 18:36 vm-100-disk-0 -> ../dm-6
  lrwxrwxrwx 1 root root 7 Jul 18 18:41 vm-101-disk-0 -> ../dm-7
  lrwxrwxrwx 1 root root 7 Jul 18 18:20 vm-102-disk-0 -> ../dm-8
  lrwxrwxrwx 1 root root 7 Jul 18 18:54 vm-103-disk-0 -> ../dm-9
  lrwxrwxrwx 1 root root 8 Jul 19 11:16 vm-104-disk-0 -> ../dm-10
  

lvm-thin

• Storage: LVM Thin. LVM-thin, or thin provisioning, is a feature of LVM that allows you to create logical volumes with a virtual size that can be larger than the available storage. Blocks in a standard LVM logical volume are allocated when the volume is created, but blocks in a thin-provisioned volume are allocated as they are written. This means that you can create a thin-provisioned volume with a large virtual size, but it will only consume physical storage space as data is written to it1. This can be useful for managing storage more efficiently and cost-effectively.

Single drive

If we have only one drive, we may like to delete local-lvm and then increase the space in local. See How to install Proxmox VE 7.0.

• DataCenter -> Storage -> local-lvm -> Remove.
• Go to Shell

  lvremove /dev/pve/data -y
  lvresize -l +100%FREE /dev/pve/root
  resize2fs /dev/mapper/pve-root
  

• Go to DataCenter -> Node -> Summary -> / HD space to verify the size.

Add a new storage

• /etc/fstab & mount. The key is to mount the drive first through a terminal. New folders will be created based on the "Content" we choose. The existing files on the drive won't be erased if we don't ask to do that.
  • Add additional drive in proxmox | How to?
  • How to add storage to Proxmox /etc/fstab & mount
  • How To Add External USB Storage To Proxmox /etc/fstab & mount
  • easiest way to mount USB drive Proxmox 7? /etc/fstab & mount & UUID
  • Add Storage Drive To Proxmox 7 /etc/fstab & mount
• Add a new physical hard drive to Proxmox VE 4x 5x LVM
• Storage: LVM Thin
  • How to: Delete/Remove local-lvm from Proxmox VE (PVE) (How to Expand/Extend/Grow LVM/Ext3/Ext4/XFS) (and Some LVM basics, commands)
  • How to ERASE LVM-THIN and create Legacy LVM
  • Need to delete local-lvm and reuse the size

    # delete local-lvm storage in gui; pve -> Disks -> LVM-Thin -> More -> Restroy
    lvdisplay # /dev/pve/swap, /dev/pve/root, /dev/pve/data
    
    lvremove /dev/pve/data
    lvresize -l +100%FREE /dev/pve/root
    resize2fs /dev/mapper/pve-root
    
    vgdisplay pve | grep Free # should be 0 now
    

• (2023/7/16). Added a USB disk. Go to pve -> Disks -> Directory -> "Create: Directory". Choose Disk, Filesystem (ext4) and Name (usb). The new disk will become "/mnt/pve/usb" in Proxmox. Now when I go back to pve -> Disks, I can see it'll be one of devices (/dev/sdb). I can use it for backup (Datacenter -> Backup). For some reason, Proxmox web interface did not work after I plugged in my USB disk but ssh still worked. Reboot the server solved the problem.
• If I remove the usb disk (Datacenter -> Storage -> Remove) and put the usb disk in a Linux OS, I see it has several directories: dump, images, lost+found, private, snippets, and template. To add the disk back to the node, use (Datacenter -> Storage -> Add -> Directory). ID=usb, Directory=/mnt/pve/usb, Content: all.
• (2023/7/29). Suppose I have an existing formatted USB disk. I plugged it into the machine. I first manually go to the console to create a new directory /mnt/usb and run chown root:root -R /mnt/usb; chmod 755 -R /mnt/usb. Now in the PVE GUI, I can go to the DataCenter -> Storage -> Add -> Directory. Choose ID=usb, Directory=/mnt/usb, Content: anything I want. Now if I run "ls /mnt/usb", I'll see directories "dump images private template". My original files on the disk are intact. I can use the disk as I like.

Passthrough a HDD

5 Things I Would Do On Fresh Install Of ProxMox. Change to "No subscription", IOMMU, VM Template, and HDD passthrough.

Passthrough a USB to LXC

• This is different from adding a USB storage to "Proxmox"
• USB Passthrough to an LXC (Proxmox)
• USB Printer Passthrough to LXC in Proxmox (video)
• NOTE: OpenMediaVault is installed through iso so it is not belonged to LXC.

Passthrough a USB or a physical drive to VM

• Concept: if we passthrough a usb disk to a vm from Proxmox, does it mean Proxmox won't see the usb disk and only the VM can see and use the usb disk?
  • Answer: Yes, that’s correct. When you pass through a USB disk to a virtual machine (VM) from Proxmox, the USB disk becomes directly accessible to the VM and not to the Proxmox host (including fdisk or lsblk). This is because the USB device is being assigned directly to the VM, making it appear as if the device is connected to the VM rather than the host.
• Proxmox documentation
  • USB Devices in Virtual Machines
    • Reassign to Host.
      1. Identify the VM and USB device,
      2. Remove the USB device from the VM's configuration (If using the Proxmox web interface, navigate to the VM’s hardware tab, select the USB device, and click on "Remove", If using the command-line interface, you can edit the VM’s configuration file directly. The configuration files are located in /etc/pve/qemu-server/ and are named after the VM’s ID (e.g., 100.conf for a VM with ID 100). In the configuration file, remove the line that corresponds to the USB device),
      3. Restart the VM.
  • Passthrough Physical Disk to Virtual Machine (VM)

• GUI How to: Passthrough USB devices in Proxmox VE (PVE) 6.2 (Easiest and quick).
  • My example: Datacenter -> node name -> VM -> hardware -> Add -> USB Device -> Select the correct USB device to passthrough (lsusb command shows my USB storage is Bus 001 Device 002: ID 152d:0576 JMicron Technology Corp. so I choose the Use USB Vendor/Device ID option).
  • Now when I go to the OpenMediaVault - Storage - Disks, I would be able to see the USB disk (Done). This is very easy compared to LXC case. To remove the USB drive, we just need to remove the USB device from the VM.
  • No qm set command is needed.

• How to Pass Through USB Devices in Proxmox
  • The screenshot of the output of lsusb is similar to what I saw on my PVE.

• Command Line Passthrough Physical Disk or USB to VM on Proxmox VE
• Command Line Passing USB Storage Drive to Proxmox LXC
• Passing a Physical Drive through to a VM in ProxMox (video)
• Open-Media-Vault (OMV) VM Running in Proxmox | Proxmox Home Server | Home Lab (video).
  • Use qm set command.
  • 8GB is enough for OMV OS.
  • Remember to edit the attached hard disk and uncheck the backup option
  • Now we can boot OMV. It will ask your hostname first and then your domain name.
  • Default port is 80 to access OMV.
  • username/password: admin/openmediavault (which is different from the OS root/PASSWORD)
  • Create a shared folder, SMB
  • Users. A new user belongs to the group users.
  • Go go another machine and enter smb://IP

How to safely remove a USB HDD

• How to 'safely' remove a USB HDD (spindown). OMV case.
• lsblk has a column MOUNTPOINTS showing if a disk is mounted or not.
• Assume I insert a second USB drive and the drive has not been used. If I just remove the 2nd USB drive, the 1st USB drive will be affected and not seen by lsusb command by PVE. So in order to safely remove the 2nd USB drive, I need to use the eject /dev/sdb command where "/dev/sdb" is determined by fdisk -l command.

Upgrade storage

• https://pve.proxmox.com/wiki/Storage
• Upgrade data disk
  1. stop all guests
  2. Backup all guests to NAS, USB Disk using Vzdump or send it to a PBS
  3. remove the Storage at "Datacenter -> Storage"
  4. shutdown server and replace disk
  5. wipe new disk (can be done using webUI since PVE 7.X, otherwise do it manually using CLI) at "YourNode -> Disks -> select new disk -> wipe"
  6. use the webUI to create a new VM/LXC storage (LVM-Thin, ZFS or whatever you like) at "YourNode -> Disks -> LVM-Thin/ZFS -> Create: Thinpool/ZFSpool"
  7. restore backups o new VM/LXC storage

• How to Add Extra Storage to Proxmox
• Resizing after copy to bigger SSD my experience

Shared storage

• Proxmox VE Full Course: Class 14 - Shared Storage. TrueNAS was used.
• How To Setup NFS Shared Storage In Proxmox

SAMBA/CIF

• Adding a Samba share to Proxmox as Storage
  • Directly work on Proxmox interface will show a message: create storage failed: storage 'xxx' is not online (500).
  • This method works.
  • When modifying the file /etc/fstab, //[ip of server]/[name of share] /media/share cifs credentials=/root/.smb,users,rw,iocharset=utf8, pay attention that the name of share is not a directory name. If we mess up the setting, we will get an error can not use mount.cifs: mount error(2): No such file or directory when we call mount -a. No need to use the "vers" option in my situation.
• I learned that the samba shared directory won't be mounted automatically on boot. The solution samba network share fails to mount at boot time or Fstab - Use SystemD automount works. One long line below.

  //<ip_of_server>/<name_of_share> /media/share cifs credentials=/root/.smb,users,rw,iocharset=utf8,noauto,x-systemd.automount,x-systemd.device-timeout=10 0 0
  

  • noauto: This option means that the device will not be mounted automatically during boot or with the mount -a command. It needs to be mounted explicitly.
  • x-systemd.automount: When this option is used, systemd will enable an “automount unit”, also known as an automount trap, or a mount point (path) where a file system may later be mounted. The file system itself is a separate unit (a “mount unit”) and will only be mounted if there is a subsequent demand to use that path. Attempts to alter the above behavior by setting either “auto” or “noauto” will have no effect.
  • The last option x-systemd.device-timeout=10 sets the timeout for the device to 10 seconds. If the device is not available within this time when you or a process attempt to mount it manually, systemd will stop trying to mount it.
  • The 0 0 at the end of the line are two different options: The first 0 refers to dump, a backup utility. By setting it to 0, you’re telling dump to ignore this file system. The second 0 is for fsck, the file system check utility. This 0 tells fsck not to check this file system at boot time.
• If I just run mount -a, it does not show any errors. But the network drive is still not available. If I just run the "mount -t cifs" command in the shell, I got the following message.

  mount: (hint) your fstab has been modified, but systemd still uses
         the old version; use 'systemctl daemon-reload' to reload.
  

• After the Samba network share is available in PVE, we can add it to the web interface
  • Datacenter (not hostname)
  • Storage -> Add Directory. The ID will be shown on the PVE LHS panel. The "Directory" refers to the directory mounted on PVE, e.g., /media/share.

Mount a Network Share in a Linux Container

• How to mount a Network Share in a Linux Container under Proxmox

Ceph stroage

• What is CEPH??? Ceph is a storage CLUSTERING solution.
  • You can add any number of disks on any number of machines into one big storage cluster. Then you set up the configuration for ceph, most notably the number of copies of a file.
  • If you set this to 2 for instance, it means that the cluster will always have 3 copies of all the objects this setting applies to. Ceph is also self-managing, meaning that it will automatically try to distribute these copies over 3 physical machines (if possible), onto 3 separate disks.
  • When any disk or machine dies, ceph will immediately use the 2 remaining copies of the affected objects and create a 3rd copy in the cluster.
  • What this does is eliminate the requirement to manually restock your spare disk in a conventional RAID setup as long as you have enough total storage to fit all the objects 3 times.
• Deploy Hyper-Converged Ceph Cluster

Plex

Proxmox LXC Intel Quick Sync Transcode for Plex

Proxmox increase vm disk size

• Important: Make sure LVM was selected when we installed Linux. Otherwise, we need to boot from a LIVE CD.
  • Shut down the VM from Proxmox.
  • In Proxmox, add a CD-ROM drive to the VM and attach a Debian or other Linux live ISO.
  • Start the VM and boot from the live CD.
  • Once booted, open a terminal

  sudo parted /dev/sda
  print
  # This will show you the current partition layout
  
  resizepart 2 100%
  # This resizes partition 2 to use all available space
  
  print
  # Verify the new size
  quit
  
  # After exiteing parted, run
  sudo partprobe /dev/sda
  

• https://pve.proxmox.com/wiki/Resize_disks
• How to Increase VM Disk Size in Proxmox
• Proxmox Resize Disk VM / Extend Disk VM LVM
• Expand logical volume - Ubuntu on Proxmox. It works when I increase my Ubuntu22.04 from 16GB to 32GB.
  • Step 1: Work in PVE. This can be done through Proxmox UI (Hardware -> Disk Action -> Resize disk -> change 0 to 16 for example if we want to increase size by 16GB).

  # qm resize <vmid> <disk> <size> 
  # qm resize 102 scsi0 +16G

  • Step 2: Work in the VM. PS: for some reason, running lvextend does not show the filesystem has been extend. After I called gparted to extend the partition, then lvextend will show the filesystem has a new blocks length.

  $ lsblk  #  or df -h. We can see the size of sda is larger than the root partition.
  
  # Make LVM aware of any changes in the size of the underlying partition 
  # (in this case, /dev/sda3) that contains the physical volume.
  # LVM scans the specified device (/dev/sda3) to determine its current size.
  # If the size of the partition has changed (usually increased), LVM updates its metadata to reflect the new size of the physical volume.
  $ sudo pvresize /dev/sda3
    Physical volume "/dev/sda3" changed
    1 physical volume(s) resized or updated / 0 physical volume(s) not resized
  
  $ df -h       # /dev/mapper/ubuntu--vg-ubuntu--lv is around 16GB, no changed yet
  Filesystem                         Size  Used Avail Use% Mounted on
  /dev/mapper/ubuntu--vg-ubuntu--lv   15G   12G  2.4G  84% /
  
  # Extend LV to use up all space from VG
  $ sudo lvextend -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv
    Size of logical volume ubuntu-vg/ubuntu-lv changed from <15.00 GiB (3839 extents) to <30.00 GiB (7679 extents).
    Logical volume ubuntu-vg/ubuntu-lv successfully resized.
  
  # resize file system
  $ sudo resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv
  resize2fs 1.46.5 (30-Dec-2021)
  Filesystem at /dev/mapper/ubuntu--vg-ubuntu--lv is mounted on /; on-line resizing required
  old_desc_blocks = 2, new_desc_blocks = 4
  The filesystem on /dev/mapper/ubuntu--vg-ubuntu--lv is now 7863296 (4k) blocks long.
  
  $ df -h
  Filesystem                         Size  Used Avail Use% Mounted on
  /dev/mapper/ubuntu--vg-ubuntu--lv   30G   12G   17G  42% /

Clone (full clone vs convert to template)

• Clone a VM in Proxmox – Here’s how to safely duplicate your VM.
• A clone VM works like a regular VM
• Note if a VM is converted to a template, it can't be started (like a regular VM).

Backup and restore VM

• Backup and Restore from proxmox wiki. How to Backup Proxmox? Proxmox Backup and Recovery Methods.
  • vzdump
  • pct and qmrestore
• How to backup and transfer a Proxmox VM to another Proxmox Node
  • Backup file has a timestamp on the filename and it is saved to /var/lib/vz/dump if it is saved in local (pve) or /mnt/pve/vm1/dump if it is saved on my vm1 storage.
  • The backup file can be seen on the GUI under $STORAGE (pve) -> Backups where $STORAGE is the storage name (e.g. local). From there, it has a 'Restore' button where we can restore it with an option to enter a new VM ID.
  • If the backup file is saved in local (pve), the backup file can also be seen under the VM|Backup menu.
  • After restoring, the new VM has a new ID but the VM name is still the same as the original one (so we can only use the ID to distinguish the VMs). Therefore, if we use the static IP in a VM, it is better to shut down the old one before we Start the new VM.
  • If we remove/delete the restored VM, the backup file is not affected (not deleted).
  • It seems backup + restore = clone.
  • The backup speed is quick. Using the "top" command it shows it is the command zstd running for the backup.
• Restore Proxmox VM from backup – Here are the steps to recover your VM
• VM ID:
  • One biggest problem is I cannot tell what the backup file is from the file name after I back up the files to another location. The backup file however contains VM ID on its filename. That is the only clue we can use to find out what the VM is on the original Proxmox.
  • The Backup Notes is useful actually. In the backup folders on Proxmox, it also create *.notes files if we create it on the GUI.
  • Maybe it's useful for me to create a text file along with the backup files to show what the files represents.
• Question: why sometimes my backup files are not shown on GUI.
  Ans: the default backup storage is "local". We need to toggle that from the GUI. PS: the storage we changed cannot be memorized.
• Question: backup status mailings for "backup success" despite all jobs being set to "On Failure only" 2024/1/19
• Schedule backup: Backup in Proxmox VE with screenshots.
• Proxmox VE Full Course: Class 10 - Backups and Snapshots
  • Snapshots (for testing someting). PS: No need to stop the VM. Taking snapshots and rollback is fast. It always saves the snapshots on the same (?) disk.
  • Backup. PS: No need to stop the VM. It will let you choose where to back up and other options. Mode: Snapshot, Suspend, Stop.
  • Automatic backup. Datacenter -> Backup -> Add (Create Backup Job).
• lxc container backup suspend mode exit code 23. LXC backup failed - code 23. Choose Backup Mode from snapshot to stop. The LXC will auto-restart after backup is finished.

The current guest configuration does not support taking new snapshots

• How to Fix (Workaround) Proxmox VE (PVE) with TPM 2.0 can’t have snapshots “The current guest configuration does not support taking new snapshots”
• Snapshots in Proxmox VE

Change backup file names

• How do you set the vzdump output file name to include guestname in the name?
• The backup VM has a filename vzdump-qemu-$ID-$Date-$Time.vma.zst (compression by default). If we select the file, we can click the 'Restore' button to restore the VM.
• If we back up a container, the backup file name has a format vzdump-lxc-$ID-$Date-$Time.tar.zst.

Restore error: data corruption

• An example of log from a failed restoration.

  restore vma archive: zstd -q -d -c /media/wd2t/dump/vzdump-qemu-201-2024_06_29-08_59_07.vma.zst | vma extract -v -r /var/tmp/vzdumptmp552388.fifo - /var/tmp/vzdumptmp552388
  ...
  progress 1% (read 343605248 bytes, duration 2 sec)
  progress 2% (read 687210496 bytes, duration 3 sec)
  progress 3% (read 1030815744 bytes, duration 4 sec)
  _29-08_59_07.vma.zst : Decoding error (36) : Data corruption detected 
  vma: restore failed - short vma extent (3282432 < 3797504)
  

• Run zstd command

  # zstd -q -d -c /media/wd2t/dump/vzdump-qemu-201-2024_06_29-08_59_07.vma.zst > /var/tmp/vzdump-qemu-201-2024_06_29-08_59_07.vma
  _29-08_59_07.vma.zst : Decoding error (36) : Data corruption detected 
  

• How does zstd detected data corruption
  • File Format Verification: Zstandard begins by checking the magic number of the file, which identifies it as a Zstandard compressed file. If this initial identifier is missing or incorrect, Zstandard will immediately flag the file as corrupted.
  • Frame Header Check: The decompression process involves reading the frame headers, which contain metadata about the compressed data, such as block sizes and checksums. If these headers are malformed or inconsistent, Zstandard will detect corruption.
  • Checksum Verification: Zstandard uses checksums to verify the integrity of each block of data. When a file is compressed, Zstandard computes and stores a checksum for each block. During decompression, it recomputes the checksum for each decompressed block and compares it to the stored value. If they don't match, it indicates data corruption.
  • Block Integrity: Each block of compressed data is decompressed independently. If any block is incomplete, truncated, or contains unexpected data patterns that do not conform to the expected compression format, Zstandard will detect this as corruption.
  • End of Stream Marker: Zstandard expects a specific marker at the end of the stream to signify the end of the compressed data. If this marker is missing or incorrect, it indicates that the file may be incomplete or corrupted.
• Example workflow to verify integrity

  sha256sum original_file.vma
  zstd original_file.vma
  zstd -t original_file.vma.zst  # 't'est integrity
  # If there are no errors, the file should be intact.
  

VM locked after I stopped the backup

vm locked after failed backup, can't unlock

qm unlock <vmid>
qm start <vmid>

I did not try above commands. However, after I did a reboot and the locking disappeared.

rclone

SYNC Proxmox backups to BACKBLAZE using RCLONE | OFF-SITE Backups | Proxmox Home Server Series

Setup a MediaWiki Server

How to Setup a MediaWiki 1.31 Server on a Debian 10 Proxmox container

Multiple node cluster

• Cluster Manager
• How to Configure a Proxmox VE 4 Multiple Node Cluster
• Join Cluster Failed: "* this host already contains virtual guests". The solution is delete all VM. It works.

Remove a node

• How to Remove a Node From a Cluster in Proxmox. But it does not work when I checked the web interface.

# shell in the node we want to keep
pvecm nodes
pvecm delnode [NODE_NAME]
pvecm nodes

• The instruction asks to power off the node we want to remove before calling "pvecm delnode". If I follow it, I got an error cluster not ready - no quorum?. The solution at here works (without to reboot the main node). However, the 2nd node still showed the 1st node:(

pvecm nodes
pvecm expected 1 # assume my cluster expected 1 node after I removed extras
pvecm delnode udoo
# Could not kill node (error = CS_ERR_NOT_EXIST)
# Killing node 2
pvecm nodes
# Now only 1 node is left

Migration VM

• Proxmox VE Full Course: Class 15 - Clustering
• How to Move virtual machine (VM) between different Proxmox VE (PVE) hosts or clusters (General ideas/Methods) 2022/4
• Migrate VM to different host
• Migration of servers to Proxmox VE
• My example:
  • Suppose two Proxmox servers have the same samba storage (look at the cylinder icons for storages) and the 1st Proxmox server has backed up some VM/CT to the SMB storage.
  • Go to the SMB storage on the 2nd sever. Click on Backup -> Restore. Select the file we want to restore. Enter the new storage and VM ID. Restore. Done.
  • Google docs (Proxmox mediawiki)

High Availability

Proxmox VE Full Course: Class 16 - High Availability

NVIDIA GPU drivers

How to Install the Official NVIDIA GPU Drivers on Proxmox VE 8

USB passthrough

• USB Devices in Virtual Machines
• How to Passthrough USB Disk to a Virtual Machine in Proxmox 6.2, Adding USB Devices to Proxmox VM (video). This assumes the USB device is on the proxmox host.
• Share ext SSD for samba and VMs
• (video) Adding USB Redirection to the Raspberry Pi + Proxmox Thin Client. The USB device is on the client.
  • Converting Any USB Device to A Wireless USB, VirtualHere ($49, locked to one device). The server (USB devices are plugged in) can be any where.
• My example
  • Plugin a USB device. the "lsusb" command in Proxmox should show the new device with an ID in the format xxxx/xxxx representing Vendor/Device. The ID of the new device can vary if we use a game controller that supports different modes. For example, on the BETOP controller, each of Xbox360/regular PC/Android modes gives a different ID.
  • It seems using Proxmox GUI is easier to manage a new USB device. Click "Hardware" and add "USB device". Select the 3rd one "Usb USB vendor/Device ID" and choose the one you have just plugged in.
  • If the USB is a flash drive, it should show under My PC.
  • If the USB is a game controller, in the search box in Taskbar, enter "game controller" and find the one "Set up USB game controllers" in Control panel. it should show the name of the controller. Open edge and go to https://gamepadtester.net to test each buttons of the controller.
  • If we are done with the USB device, we shut down the VM first. Go to the Hardware in Proxmox and remove the USB item.

• Unraid.
  • Virtual Machines Settings
  • Howto Virtualize Unraid on a Proxmox host. It basically passes through the USB drive to the Proxmox VM. However, it includes a solution for resolving the kernel panic error.

Thin client

Raspberry Pi THIN CLIENT for Proxmox VMs

Android app

Proxmox Virtual Environment

Nested virtualization

3 reasons why you should set up nested virtualization on your home lab

Android emulator and nested virtualization

Is there a guide to getting Android x86 installed on Proxmox?

Security

Am I compromised? If you need true remote access, set up a VPN that you connect to on your router.

OpenWRT router

• See Netboot.
• Table of Hardware: Ideal for OpenWrt + Wifi6 (802.11ax) supported
• GL.iNet GL-MT3000
• OpenWrt One WiFi 6 router samples are now available for $89
  • Banana Pi OpenWrt One Router page

Error 401: no ticket

Empty browser's cache. It works.

Warning email about NVME temperature

• SMART error (Health) detected on host: XXXX. I received an email about the temperature warning from the sender root email created when we set up Proxmox.
• There is no /var/log/syslog file. To check the log, use journalctl command.

SMB server

• Setting Up NAS Server On Proxmox
• SMB Server In Docker with ZFS! Simple, Cheap, and Efficient!

Check SMB Share availability before starting a VM

add a startup delay to your VMs and CTs if this is the only time you have this issue. Then set up a script in crontab to run @reboot that just waits x seconds, then remounts the share.

Cloud image, Cloud-init

• https://help.ubuntu.com/community/CloudInit
• What are Ubuntu cloud images?
• Using Ubuntu cloud-images without a cloud
• KVM: Testing cloud-init locally using KVM for an Ubuntu cloud image
• Launching Ubuntu Cloud Images with QEMU
• Proxmox
  • How to create a Proxmox Ubuntu cloud-init image
  • Creating Ubuntu and Debian OS Template on Proxmox VE
  • Perfect Proxmox Template with Cloud Image and Cloud Init (video)

Proxmox vs. ESXi

• https://www.reddit.com/r/homelab/comments/5vit59/proxmox_vs_esxi/
• Comparison of Server Virtualization

VMWare ESXi

• vSphere Hypervisor
• VMware ESXi Free vs Paid: A Look at License Limitations

Proxmox Backup Server/PBS

• How To Install Proxmox Backup Server Step by Step
• Install PBS in VM (video)
• Getting Started With Proxmox Backup Server
• Can PBS be used as standalone backup server with no integration to proxmox ve?

proxmox-backup-client

• Backup Client Usage

Remote machine management

Self-Hosted Remote Desktop Connection Alternatives

Remotely

• https://github.com/immense/Remotely
• Remotely, a free open source Remote Support Alternative to Anydesk GTA LogmeIn TeamViewer and more and notes
  • Server + client
  • Server is operating on the browser to access the client's computer
  • Client (Windows or Linux only, macOS is not supported) needs to install a program
• Remotely Update - an Update on an open source, free alternative to TeamViewer, Anydesk, GoToAssist & notes

RPort

• https://rport.io/, doc
• RPort - an open source, self hosted Remote Machine Management System running on Linux (video) , notes