Vagrant

Official website https://www.vagrantup.com/

Resources

What is vagrant

• Vagrant is an open source tool used for creating a portable virtual environment.
• Using Vagrant, developers and sysadmins can create any virtual environment instantly.

Download

Download link for the latest version or use sudo apt install virtualbox vagrant.

Use vagrant -v to see the vagrant version currently installed in your machine.

Documentation, medias

• https://docs.vagrantup.com/v2/
• Vagrant Simplified from LinuxJournal.

Books

• Vagrant Virtual Development Environment Cookbook (2015)
• Creating Development Environments with Vagrant (2015, 2nd Ed)
• Pro Vagrant (2015)
• Vagrant: Up and Running (2013)

Windows

• http://www.sitepoint.com/getting-started-vagrant-windows/
• http://www.seascapewebdesign.com/blog/part-1-getting-started-vagrant-windows-7-and-8

On Windows OS w/o admin privilege, I need to use Admin/PM to open CMD. Otherwise, 'vagrant up' will give a permission error.

Also, since Windows does not have the ssh command, it is better to borrow the command from the Git package. After installing it, open Git Bash from the start menu (Use Admin/PM). Then cd to the vagrant project directory. Continue to run vagrant up or vagrant ssh as you want.

Too bad. On Windows 10, we will get an error with the current version of vagrant.

URL: ["https://atlas.hashicorp.com/ubuntu/precise32"]
Error:

Need to downgrade vagrant to 1.7.4.

Some commands

vagrant -v # get the vagrant version

vagrant up
vagrant halt # gracefully shut down
vagrant reload # restarts vagrant machine, loads new Vagrantfile configuration
vagrant global-status  # output status of all vagrant machines (not boxes)
                       # If a machine is up before, it is shown as 'running' in the output
                       # EVEN IT IS NOT RUNNING now. For example, the following machine
                       # is currently off now.
# id       name    provider   state   directory                           
# ------------------------------------------------------------------------
# a48f6f3  default virtualbox running /mnt/bigdisk/vagrant/trusty64-gui 
# BUT THE VIRTUALBOX shows it is off now.

vagrant suspend 
vagrant status  # displays the currently initialized vagrant environments
vagrant resume

vagrant destroy # stops the running machine Vagrant is managing and destroys all resources
vagrant destroy a48f6f3  # Use id

vagrant box SUBCOMMANDS
vagrant box remove ubuntu/trusty32

vagrant ssh
vagrant ssh-config  # You can set any of the values in the Vagrantfile if you need to modify it

vagrant package # This packages a currently running VirtualBox environment into a re-usable box.

Examples

Create Ubuntu machines

# Method 1: 14.04 (Trusty Tahr)
mkdir -p ~/Vagrant/trusty64
cd ~/Vagrant/trusty64
## create a Vagrantfile by entering its content from https://app.vagrantup.com/ubuntu/boxes/trusty64
cat > Vagrantfile <<EOF
Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/trusty64"
end
EOF
vagrant up # a hidden directory .vagrant will be created.
vagrant ssh # based on the hidden environment to log in
vagrant halt # shutdown
vagrant destroy # delete

# Method 2: 16.04 (Xenial Xerus)
mkdir ~/Vagrant/xenial64
vagrant box add ubuntu/xenial64
vagrant init ubuntu/xenial64 # Create <Vagrantfile> under the current directory
vagrant up
...

# Method 3: 18.04 (Bionic Beaver)
mkdir ~/Vagrant/bionic64
                             # Skip the "box" command
vagrant init ubuntu/bionic64 # Create <Vagrantfile> under the current directory
vagrant up
...

R box

Not available!

A simple example

$ mkdir precise32
$ cd precise32
$ vagrant init hashcorp/precise32
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
$ cat Vagrant
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "hashcorp/precise32"
end
$ vagrant up
vm
* The box 'hashcorp/precise32' could not be found.

The solution from stackoverflow works.

If we open VirtualBox GUI, we will see a new guest machine called precise32_default_XXXXXXXX is created and running though we do not see Ubuntu precise 32 desktop application in a new window.

A web server

VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.define "web", primary: true do |web|
    web.vm.box = "ubuntu/trusty64"
    web.vm.network "forwarded_port", guest:80, host:8888
    web.vm.provision "shell", inline: "apt-get install -y nginx"
  end
end

Start the machine with the vagrant up command. Open a web browser on your host machine at http://localhost:8888. This will present the nginx Welcome page.

See my gist for the output.

A GUI example

Go to

1. https://atlas.hashicorp.com/boxes/search. Type 'GUI' to search
2. https://atlas.hashicorp.com/chad-thompson/boxes/ubuntu-trusty64-gui

Run vagrant init chad-thompson/ubuntu-trusty64-gui. Edit Vagrantfile.

Vagrant.configure(2) do |config|
  config.vm.box = "chad-thompson/ubuntu-trusty64-gui"
  config.vm.provider "virtualbox" do |vbox|
    vbox.gui = true
  end
end

Then run vagrant up (1.5GB download).

Note that when I run df -h in the VM, it shows

Filesystem  Size Used Avail  Mounted on
/dev/sda1   21G  3.1G  17G   /
/vagrant    1.8T 445G 1.4T   /vagrant

The /vagrant partition contains the Vagrantfile.

We can also make the Vagrantfile a bit more general by including multiple providers in the same Vagrantfile.

Vagrant.configure(2) do |config|
  config.vm.box = "chad-thompson/ubuntu-trusty64-gui"
  config.vm.provider "virtualbox" do |vbox|
    vbox.gui = true
  end
  config.vm.provider "vmware_fusion" do |fusion|
    fusion.gui = true
  end
end

Share folders/Synced folders

Default implementation from providers

Suppose we want to set up a web server to develop html files. We share a source folder that contains html documentation with the guest.

1. In the working directory, create a new dir called vagrantsite.
2. In the vagrantsite dir, create a file named index.html.
3. Create a new Vagrantfile in the working directory.
4. Add a synced_folder directive to the web server configuration.
5. Create an additional command to the web.vm.provision line to create a symbolic link from this directory to a directory in the nginx default web directory.

VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.define "web", primary: true do |web|
    web.vm.box = "ubuntu/trusty64"
    web.vm.network "forwarded_port", guest:80, host:8888
    web.vm.synced_folder "vagrantsite/", "/opt/vagrantsite"
    web.vm.provision "shell", inline, "apt-get install -y nginx; ln -s /opt/vagrantsite /usr/share/nginx/html/vagrantsite"
  end
end 

Now executing vagrant up command and open the page http://localhost:8888/vagrantsite/ in a web browser.

Note

• Vagrant shares the working directory of the Vagrantfile in the root vagrant directory by default. Executing ls /vagrant command should return this information.
• Sharing folders will require proper tools to be installed in the host machine. For VirtualBox, this means having the guest additions installed. For VMware, this means VMware tools installed.

Shared folders can also be implemented in different ways

• Using networked file systems such as NFS (Linux, OS X) and SMB (Windows). These options might offer significantly better file-sharing performance than shared folder functionality of the hypervisor; i.e. NFS is better than using the guest additions method in VirtualBox. See http://mitchellh.com/comparing-filesystem-performance-in-virtual-machines.
• A Vagrant machine executing on a remote hypervisor (eg a cloud service) will usually not have shared filesystems available. Vagrant provides methods to copy files from host ot guest using the rsync protocol.

Share Network File Systems

Sharing directories from the host machine will require the host system to export a directory for use by the guest machine. When using NFS, this means that Vagrant will add an entry to the native /etc/exports file to define the rule to export the specified directory to the guest box.

Prior to starting, make sure that Network File Share Daemon (nfsd) is installed on your host machine.

Notice the type option in the synced_folder line.

VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.define "web", primary: true do |web|
    web.vm.box = "ubuntu/trusty64"
    web.vm.network "forwarded_port", guest:80, host:8888
    web.vm.synced_folder "vagrantsite/", "/opt/vagrantsite", type: "nfs"
    web.vm.provision "shell", inline, "apt-get install -y nginx; ln -s /opt/vagrantsite /usr/share/nginx/html/vagrantsite"
  end
end 

You can verify the shared folder is from NFS by typing vagrant ssh and executing the mount command. The mount command should contain an entry that shows type nfs for one partition.

For Windows Server Message Block (SMB) protocol, modify the synced_folder to

    web.vm.synced_folder "vagrantsite/", "/opt/vagrantsite", type:" smb"

rsync

'rsync method is useful when

• NFS or VirtualBox shared folders aren't available in the guest machine.
• Processes that generate significant disk activity (I/O) on shared folders
• Vagrant can be used to control vm in remote locations (even in remote data centers over the public internet)

Pay attention to the synced_folder line.

VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.define "web", primary: true do |web|
    web.vm.box = "ubuntu/trusty64"
    web.vm.network "forwarded_port", guest:80, host:8888
    web.vm.synced_folder "vagrantsite/", "/opt/vagrantsite", type: "rsync"
    web.vm.provision "shell", inline, "apt-get install -y nginx; ln -s /opt/vagrantsite /usr/share/nginx/html/vagrantsite"
  end
end 

When the files in the host machine are changed, we need to run vagrant rsync command on the host to synchronize the /opt/vagrantsite directory.

Customizing virtual machine settings in VirtualBox

The primary example is changing the settings of the virtual machine to use more (or less) system memory (2GB in the following example) and virtual processors (2 cpus).

Vagrant.configure(2) do |config|
  config.vm.define "web", primary: true do |web|
    web.vm.box = "ubuntu/trusty64"
    web.vm.network "forwarded_port", guest:80, host:8888
    web.vm.provision "shell", inline: "apt-get install -y nginx"
    web.vm.provider "virtualbox" do |vbox|
      vbox.memory = 2048
      vbox.cpus = 2
    end
  end
end

Start the vm with the vagrant up command. Verify the system has the amount of memory allocated by issuing the command head /proc/meminfo.

If we like to start a GUI on the vm, just add vbox.gui = true in the vbox block.

    web.vm.provider "virtualbox" do |vbox|
      vbox.memory = 2048
      vbox.cpus = 2
      vbox.gui = true
      vbox.customize ["modifyvm", :id, "--bioslogofadein", "off"]
    end

The last line use the VBoxManage command to fade the VirtualBox startup logo immediately.

A full listing of available options to modify the runtime is available in the VirtualBox documentation http://www.virtualbox.org/manual/ch08.html.

Where is vagrant saving boxes files

http://stackoverflow.com/questions/10155708/where-does-vagrant-download-its-box-files-to

• Windows: C:/Users/USERNAME/.vagrant.d/boxes
• Linux and Mac: ~/.vagrant.d/boxes/

We can change the default directory by modifying the VAGRANT_HOME variable. See https://docs.vagrantup.com/v2/other/environmental-variables.html.

Linux

One time use.

export VAGRANT_HOME=my/new/path/goes/here/

Windows

• https://harvsworld.com/2014/change-vagrant_home-directory-windows/

set VAGRANT_HOME=X:\PATH\TO\VAGRANT # one time only
setx VAGRANT_HOME "X:/your/path"    # permanent

Double check

set VAGRANT_HOME

Vagrantfile

Vagrantfile is just Ruby.

Vagrantfile template

If we run vagrant init, we will get the following vagrantfile.

VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  # ...
end

Precise32

VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "hashcorp/precise32"
end

config.vm

Config namespace: config.vm

The settings within config.vm modify the configuration of the machine that Vagrant manages.

• config.vm.boot_timeout
• config.vm.box
• config.vm.box_url
• config.vm.communicator
• config.vm.hostname
• config.vm.provider
• config.vm.synced_folder

Boxes

https://app.vagrantup.com/boxes/search

Download a box w/ initializing an env

vagrant box add ubuntu/trusty64

You can also quickly initialize a Vagrant environment with vagrant init ubuntu/trusty64.

We can also specify an URL to add/download a box.

vagrant box add http://servername/boxes/environment.box

Remove a box

Something like

vagrant box remove ubuntu/trusty32

List downloaded boxes

vagrant box list

List running VirtualBox machines

VBoxManage list runningvms

Create a new vagrant box/package from a VirtualBox machine

The final note is saved under Google Drive!!

The instruction at http://aruizca.com/steps-to-create-a-vagrant-base-box-with-ubuntu-14-04-desktop-gui-and-virtualbox/ works for me. Lots of information is given there like install vagrant public key, compact space, install OpenSSH server. Two things that are missing are

• rm ~/.ssh/known_hosts
• cat /dev/null > ~/.bash_history && history -c && exit.

(Naive approach. See the above link at aruizca.com for more accurate steps) Suppose we use vagrant to create a new box based on ubuntu/trusty64. We ssh to the box and install some software (eg r-base-core). We use vagrant halt to shut down the box. Now we can package this new box.

$ VBoxManage list vms # find out the vm name (vm is created by Vagrant)
$ vagrant package --base=trusty64_default_1451665747284_29528 --output=mytest.box

To test the new box, we can copy the box to your local Vagrant cache.

$ vagrant box add mytest.box --name=mytestbox
$ vagrant box list
mytestbox       (virtualbox, 0)
ubuntu/trusty64 (virtualbox, 20151217.0.0)

Create a new directory so we can test the new box.

cd ..
mkdir mytestbox
cd mytestbox
vagrant init mytestbox  # create Vagrantfile

Note that we can not use vagrant up to start the box now. If we try to run it, we will get the following error about the SSH authentication.

    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Authentication failure. Retrying...

I have to use Ctrl+C to stop the process and then use vagrant destroy to destroy the (unsuccessfully) running virtual machine.

Go ahead to modify the Vagrantfile created by vagrant init mytestbox command to use the correct SSH username/password. By default, Vagrant relies on a common public key that is used by most box publishers that allows access to an account called vagrant. After the first login, Vagrant will place a key in the appropriate account; so, if desired, the password can be removed from the Vagrantfile after the first boot. (verified!)

Vagrant.configure(2) do |config|
  config.vm.box = "mytestbox"
  config.ssh.username="ubuntu"
  config.ssh.password="ubuntu"
end

Strangely, though I can run vagrant up, I cannot use the password I created when I try the vagrant ssh command. I found the solution is to make sure the username and password used in the Vagrantfile the same as the virtual machine; in this case since I am using the ubuntu/trusty64 box, I should use the vagrant/vagrant as the username/password in the Vagrantfile. That is the correct Vagrantfile should be

Vagrant.configure(2) do |config|
  config.vm.box = "mytestbox"
  config.ssh.username="vagrant"
  config.ssh.password="vagrant"
end

Now we can run vagrant up and vagrant ssh commands. We could check R should be available in the virtual machine.

Create a template for the virtual machine images: Packer

• https://www.packer.io/
• https://youtu.be/6-7WjA-hHvg?t=10m44s
• Rolling Your Own Images With Packer
• Orchestrating a community-developed computational workshop and accompanying training materials by Sean Davis 2018
• Examples:
  • Bento project
  • https://seandavi.github.io/post/2018/07/infrastructure-as-code-building-the-bioconductor-conference-ami-with-packer/

Provisioning

Running basic shell commands

Pay attention to the line containing config.vm.provision.

VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "ubuntu/trusty64"
  config.vm.provision "shell", 
    inline: "echo 'Knock Knock!'  > /etc/motd"
end

Once the machine is booted, access the machine with vagrant ssh command. The login shell will display the message created with the inline shell script.

Executing shell scripts in a Vagrantfile

VAGRANTFILE_API_VERSION = "2"

$nginx_install = <<SCRIPT
   if [ ! -x /usr/sbin/nginx ]; then
      apt-get install -y nginx;
   fi

   # Default NGINX directory: /usr/share/nginx/html
   # Replace this with symbolic link to vagrant directory.
   if [ ! -L /usr/share/nginx/html ]; then
      rm -rf /usr/share/nginx/html
      ln -s /vagrant/html /usr/share/nginx/html
   fi 
SCRIPT

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "puppetlabs/ubuntu-14.04-32-nocm"
  config.vm.provision "shell", inline: $nginx_install
  config.vm.network "forwarded_port", guest:80, host:8080
end

External shell scripts

The working directory should contain both Vagrantfile and the shell script file.

Pay attention to the line containing config.vm.provision.

VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "ubuntu/trusty64"
  config.vm.provision "shell", path: "nginx-install.sh"
  config.vm.network "forwarded_port", guest:80, host:8080
end

The script file <nginx-install.sh> looks like

#!/bin/bash

   if [ ! -x /usr/sbin/nginx ]; then
      apt-get install -y nginx;
   fi

   # Default NGINX directory: /usr/share/nginx/html
   # Replace this with symbolic link to vagrant directory.
   if [ ! -L /usr/share/nginx/html ]; then
      rm -rf /usr/share/nginx/html
      ln -s /vagrant/html /usr/share/nginx/html
   fi

Execute the vagrant up command and open the browser at http://localhost:8080 in the local machine.

As a test of idempotency, execute the vagrant provision command. You shall see this command will exit quickly as all conditions within the script should be satisfied.

Using management tools/languages: Puppet

• https://docs.vagrantup.com/v2/provisioning/puppet_apply.html
• https://docs.vagrantup.com/v2/provisioning/puppet_agent.html

Vagrant machines use the Puppet agent infrastructure to perform provisioning operations on a machine. Puppet agents can function in one of two ways.

• By connecting to a Puppymaster to retrieve configuration information. A Puppetmaster is a server that is a centroalized location for systems to retrieve system configurations.
• By executing a puppy apply command to interpret and apply configurations locally. This is often referred to as the masterless Puppet approach.

Install Puppet

• Puppet Server
• Install Puppet Agent in Linux
• How to install Puppet master and client in Ubuntu 14.04 https://www.howtoforge.com/puppet-ubuntu-14.04
• http://www.bogotobogo.com/DevOps/Puppet/puppet_install_on_ubuntu_14_04_trusty.php
• https://help.ubuntu.com/lts/serverguide/puppet.html
• https://www.digitalocean.com/community/tutorials/how-to-install-puppet-to-manage-your-server-infrastructure

Masterless approach

We apply Puppet configurations locally with some reusable code (a Puppet module) that is obtained from the Puppet Forget repository.

• Vagrant box: it is necessary to use a Vagrant box that either has the Puppet agent installed or creates a bootstrapping script that configures packages repositories and installs the Puppet agent in the virtual machine. For simplicity, we can use a box provided by Puppet Labs.
• Host machine: the host machine also needs to install the Puppet agent. Having the agent installed on the host will allow for management of Puppet modules and resources by the host for Vagrant guests to be provisioned. A common example is to use the puppet module utility to manage and use modules.

The working directory has the following file structure

|-- Vagrantfile
|-- puppet
     |-- manifests
     |-- modules

The Vagrantfile looks like (pay attention to the line containing config.vm.provision. The Puppet provisioner requires parameters to be set that define paths to our manifest and modules directories, as well as a manifest filename <site.pp> in this case).

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  # A Vagrant box with puppet pre-installed.
  config.vm.define "web", :primary => true do |web|
     web.vm.box = "ubuntu/trusty64"
     web.vm.hostname = "web"
     web.vm.network "forwarded_port", guest: 80, host: 8080

     web.vm.provision "puppet" do |puppet|
       puppet.manifests_path = "puppet/manifests"
       puppet.manifest_file  = "site.pp"
       puppet.module_path = "puppet/modules"
     end
  end
end

With the Puppet provisioner configured, we can write the Puppet code.

Configure Puppet We use code downloaded from the Puppet Forget to install the Apache web server

1. Open Puppet Forget (http://forge.puppetlabs.com)
2. Search for 'apache'
3. Select the puppetlabs/apache module. There are 2 ways you can do this:
   • If you have puppet installed on your host, install the module with the puppet module command: puppet module install --modulepath=puppet/modules puppetlabs-apache. This approach will resolve dependencies and download them as well as the specified module itself.
   • Download the tar.gz file and extract it into the modules directory. As this method does not resolve dependencies, this is something that need to be done manually. For this apache example, as we can see from the metadata.json file, the module dependencies are concat and stdlib.
4. With the module dependencies installed, we need to create a manifest file <site.pp> that will govern how resources and modules are used.

node web {
   class{"apache":
     default_vhost => false,
   }
   
   apache::vhost{"default-host":
      docroot => "/var/www/html",
      docroot_owner => 'www-data',
      docroot_group => 'www-data',
      default_vhost => true,
      logroot => '/var/log/apache2',
      port => 80,
   }
}

This manifest file does 3 things: specifies an action for the web node, calls the apache class to install the Apache web server, and defines an apache::vhost type that will create a default virtual host for our web server.

The working directory now has a structure

|-- Vagrantfile
|-- puppet
     |-- manifests -- site.pp
     |-- modules
           |-- apache -- lots of files and subdirectories
           |-- concat -- lots of files and subdirectories
           |-- stdlib -- lots of files and subdirectories

Run the vagrant up command in the working directory. Open http://localhost:8080 in a web browser.

Providers

Docker

Vagrant vs Docker

Others

Resize a hard disk

• https://gist.github.com/christopher-hopper/9755310
• http://askubuntu.com/questions/317338/how-can-i-increase-disk-size-on-a-vagrant-vm