Windows, virtually (pt4)
or, can you repeat that?
Scripting for fun and profit
Everything is looking pretty neat at this point, but it is time to start automating things around. I could easily start using libvirt now to set up the VM and that would definitely help in creating new ones in the future if/when we add more GPUs, but lets be honest... that is definitely not how I roll!
For our initial attempt we're going to be using my basic tool bag language, Python, some bash scripting and just code up a few helpers. Some prep work is needed on every boot up, so lets script that first:
#!/bin/bash
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbindNext up is the actual qemu command needed to run each VM. Basing this on the final qemu command line from last entry:
sudo qemu-system-x86_64 \
> -name winvm1,process=winvm1 \
> -machine type=q35,accel=kvm \
> -cpu EPYC,kvm=off \
> -smp 4,sockets=1,cores=2,threads=2 \
> -m 8G \
> -rtc clock=host,base=localtime \
> -vga none \
> -nographic \
> -serial none \
> -parallel none \
> -usb \
> -device usb-host,vendorid=0x1bcf,productid=0x0005 \
> -device usb-host,vendorid=0x04d9,productid=0x1702 \
> -device vfio-pci,host=0a:00.0,multifunction=on,romfile=/home/ce/ZotacGTX1080TImini.rom \
> -device vfio-pci,host=0a:00.1 \
> -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
> -drive if=pflash,format=raw,file=/tmp/my_vars.fd \
> -boot order=dc \
> -drive id=disk0,if=virtio,cache=none,format=raw,file=/dev/disk/by-id/wwn-0x500a0751f002394a \
> -drive file=/home/ce/win10.iso,index=1,media=cdrom \
> -drive file=/home/ce/virtio-win-0.1.171.iso,index=2,media=cdrom \
> -netdev type=tap,id=net0,ifname=vmtap0,vhost=on \
> -device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:01We can thus create a template and just fill in the parts that are VM specific. To deal with variable substitution we'll be using Python's string format operation, which will not be terribly generic, as the substitutions will need to already be properly formatted for their position in the command line, but that is plenty good for now.
cp /usr/share/OVMF/OVMF_VARS.fd /tmp/ovmf_vars-{vmid:03d}.fd
qemu-system-x86_64 \
-name {vmname},process={vmname} \
-machine type=q35,accel=kvm \
-cpu EPYC,kvm=off \
-smp {cpuunits},sockets={cpusockets},cores={cpucores},threads={cputhreads} \
-m {ram} \
-rtc clock=host,base=localtime \
-vga none \
-nographic \
-serial none \
-parallel none \
-usb \{usb}{vfio}
-drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
-drive if=pflash,format=raw,file=/tmp/ovmf_vars-{vmid:03d}.fd \
-boot order={predrives}c \{drives}
-netdev type=tap,id=net0,ifname=vmtap{vmid:03d},vhost=on \
-device virtio-net-pci,netdev=net0,mac={mac}A bunch of these variables will need some processing and, as mentioned above, proper formatting, but we save the text for the template in a file called kvm.template and lets prepare the vm parser/formatter next:
#!/usr/bin/env python3
import json
import os
import sys
PWD = os.path.abspath(os.path.dirname(__file__))
DEFAULTS = {
'vmid': 0,
'cpusockets': 1,
'cpucores': 1,
'cputhreads': 1,
'ram': '2G',
'usb': [],
'vfio': [],
'predrives': '',
'drives': {
'raw': [],
'iso': [],
}
}
def readjson(filename):
config = json.load(open(filename))
if not 'vmid' in config:
# try to infer vmid from filename
vmid = os.path.splitext(os.path.basename(filename))[0]
if vmid.isdigit():
config['vmid'] = int(vmid)
return config
def format(config):
actualconfig = {}
actualconfig.update(DEFAULTS)
actualconfig.update(config)
if not 'vmname' in actualconfig:
actualconfig['vmname'] = 'vm{}'.format(actualconfig['vmid'])
if not 'mac' in actualconfig:
actualconfig['mac'] = '00:16:3e:00:01:{:02x}'.format(actualconfig['vmid'])
tmp = []
for e in actualconfig['usb']:
tmp.append('\n-device usb-host,vendorid=0x{:04x},productid=0x{:04x} \\'.format(*e))
actualconfig['usb'] = ''.join(tmp)
tmp = []
for e in actualconfig['vfio']:
tmp.append('\n-device vfio-pci,host={} \\'.format(e))
actualconfig['vfio'] = ''.join(tmp)
tmp = []
for e in actualconfig['drives'].get('raw', []):
tmp.append('\n-drive id=disk{},if=virtio,cache=none,format=raw,file={} \\'.format(
len(tmp), e))
for e in actualconfig['drives'].get('iso', []):
tmp.append('\n > -drive index={},file={},media=cdrom \\'.format(
len(tmp), e))
actualconfig['drives'] = ''.join(tmp)
actualconfig['cpuunits'] = actualconfig['cpusockets'] * actualconfig['cpucores'] * actualconfig['cputhreads']
template = open(os.path.join(PWD, 'kvm.template')).read()
return template.format(**actualconfig)
if __name__ == '__main__':
if len(sys.argv) > 1:
config = readjson(sys.argv[1])
else:
config = {}
print(format(config))And that allows for the original command line to be reproduced based on a configuration, which we'll now create in json so that we can later tweak easily and also add new VM instances. The configuration is just the overriding of the DEFAULTS from formatter.py with the appropriate entries, and we use the filename to set the vmid:
{
"cpucores": 2,
"cputhreads": 2,
"ram": "16G",
"usb": [
[7119, 5],
[1241, 4890]
],
"vfio": [
"0a:00.0,multifunction=on,romfile=/home/ce/ZotacGTX1080TImini.rom",
"0a:00.1"
],
"drives": {
"raw": [
"/dev/disk/by-id/wwn-0x500a0751f002394a"
]
}
}Lets make the scripts executable just to save some typing and try a test run:
ce@bear:~/vms$ chmod +x formatter.py
ce@bear:~/vms$ chmod +x prepare.sh
ce@bear:~/vms$ # Lets try the default formatter entries first
ce@bear:~/vms$ ./formatter.py
cp /usr/share/OVMF/OVMF_VARS.fd /tmp/ovmf_vars-000.fd
qemu-system-x86_64 \
-name vm0,process=vm0 \
-machine type=q35,accel=kvm \
-cpu EPYC,kvm=off \
-smp 1,sockets=1,cores=1,threads=1 \
-m 2G \
-rtc clock=host,base=localtime \
-vga none \
-nographic \
-serial none \
-parallel none \
-usb \
-drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
-drive if=pflash,format=raw,file=/tmp/ovmf_vars-000.fd \
-boot order=c \
-netdev type=tap,id=net0,ifname=vmtap000,vhost=on \
-device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:00
ce@bear:~/vms$ # And now the configuration matching our target VM
ce@bear:~/vms$ ./formatter.py conf/001.json
cp /usr/share/OVMF/OVMF_VARS.fd /tmp/ovmf_vars-001.fd
qemu-system-x86_64 \
-name vm1,process=vm1 \
-machine type=q35,accel=kvm \
-cpu EPYC,kvm=off \
-smp 4,sockets=1,cores=2,threads=2 \
-m 16G \
-rtc clock=host,base=localtime \
-vga none \
-nographic \
-serial none \
-parallel none \
-usb \
-device usb-host,vendorid=0x1bcf,productid=0x0005 \
-device usb-host,vendorid=0x04d9,productid=0x131a \
-device vfio-pci,host=0a:00.0,multifunction=on,romfile=/home/ce/ZotacGTX1080TImini.rom \
-device vfio-pci,host=0a:00.1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
-drive if=pflash,format=raw,file=/tmp/ovmf_vars-001.fd \
-boot order=c \
-drive id=disk0,if=virtio,cache=none,format=raw,file=/dev/disk/by-id/wwn-0x500a0751f002394a \
-netdev type=tap,id=net0,ifname=vmtap001,vhost=on \
-device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:0Learning to listen
At this point, I can start the VM on the shell without much work, I could add another GPU, keyboard and mouse (in theory, at least) and build a separate, independent virtual machine.
USB Keypad
All that is nice and good, but I need to ssh into the machine to start, restart or stop anything, as the physical linux console is not available. For this reason I plan to use yet another input method to "manage" the machines, simulating the power on and reset buttons of a physical machine. I had planned originally to buy myself an elgato Stream Deck to use for all the control and monitoring needs of this machine, but it is a bit expensive and does not support linux natively (though that seems to be less of a problem now).
Still, scavenging for parts I already have that may prove to be of use here, I came up with a USB keypad that I can just take control over, so lets use that for now.
We can then thus state the target solution as:
- Runs on boot
- Makes the needed preparations and checks for VM spinning
- Allows for control over ssh
- Allows for control over the USB keypad
- Can start, reset, clean shutdown and force power off each existing VM independently
To make things simpler, I'm going to target running all of this off the root user, though with a little work it is certainly possible to do this through a normal user. Perhaps at a later date we can investigate that option. First things first, running on boot is potentially a simple matter of having a single startup script that gets run from crontab or similar, and that script is our first stop.
To accomplish the control over ssh part I could run a controller service in the background and communicate with it using unix sockets or similar, but a simpler approach is to use something console driven in the foreground and a terminal multiplexer like screen or, my favourite, tmux. So, lets create a vmlauncher.cron shell script to be run on boot, do the prep work and run a tmux session, on which it then launches a python script that will handle all the orchestration of the VMs, including listening to the keypad as well as the stdin for VM handling commands.
#!/bin/bash
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
tmux new-session -d -s vmhandler
tmux send-keys -t vmhandler "login -f root" C-m
tmux send-keys -t vmhandler "cd /home/ce/vms/" C-m
tmux send-keys -t vmhandler "./launcher.py" C-mNow lets connect the keypad and find out how to take control over it:
ce@bear:~$ lsusb
<snip>
Bus 001 Device 007: ID 04d9:a02a Holtek Semiconductor, Inc.
Bus 001 Device 006: ID 05e3:0608 Genesys Logic, Inc. Hub
<snip>
ce@bear:~$ # Is it handled by linux's generic input events?
ce@bear:~$ ls /dev/input/by-id/
usb-04d9_a02a-event-kbd usb-_USB_Keyboard-event-if01
usb-1bcf_USB_Optical_Mouse-event-mouse usb-_USB_Keyboard-event-kbd
usb-1bcf_USB_Optical_Mouse-mouse
ce@bear:~$ # Apparently so, lets try and use it from python then
ce@bear:~$ sudo apt install python3-pip
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
dh-python libexpat1-dev libpython3-dev libpython3.6-dev python-pip-whl python3-crypto
python3-dev python3-keyring python3-keyrings.alt python3-secretstorage python3-setuptools
python3-wheel python3-xdg python3.6-dev
Suggested packages:
python-crypto-doc gnome-keyring libkf5wallet-bin gir1.2-gnomekeyring-1.0
python-secretstorage-doc python-setuptools-doc
The following NEW packages will be installed:
dh-python libexpat1-dev libpython3-dev libpython3.6-dev python-pip-whl python3-crypto
python3-dev python3-keyring python3-keyrings.alt python3-pip python3-secretstorage
python3-setuptools python3-wheel python3-xdg python3.6-dev
0 upgraded, 15 newly installed, 0 to remove and 0 not upgraded.
Need to get 47.9 MB of archives.
After this operation, 84.1 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
<snip>
ce@bear:~$ sudo pip3 install evdev
Collecting evdev
Installing collected packages: evdev
Successfully installed evdev-1.2.0
ce@bear:~$ sudo python3 # We need sudo because our user is not in the input group
Python 3.6.9 (default, Nov 7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import evdev
>>> evdev.list_devices()
['/dev/input/event12', '/dev/input/event11', '/dev/input/event10', '/dev/input/event9', '/dev/input/event8',
'/dev/input/event7', '/dev/input/event6', '/dev/input/event5', '/dev/input/event4', '/dev/input/event3',
'/dev/input/event2', '/dev/input/event1', '/dev/input/event0']
>>> print (evdev.InputDevice('/dev/input/by-id/usb-04d9_a02a-event-kbd'))
device /dev/input/by-id/usb-04d9_a02a-event-kbd, name "HID 04d9:a02a", phys "usb-0000:01:00.0-10.2/input0"
>>> ctrlkp = evdev.InputDevice('/dev/input/by-id/usb-04d9_a02a-event-kbd')
>>> # While not pressing any keypad keys
...
>>> ctrlkp.active_keys()
[]
>>> # While pressing the keypad '1' key
...
>>> ctrlkp.active_keys()
[79]
>>> # While pressing the keypad '1', '2' and '3' keys simultaneously
...
>>> ctrlkp.active_keys()
[79, 80, 81]
>>> evdev.ecodes.KEY[79]
'KEY_KP1'
>>> evdev.ecodes.KEY[80]
'KEY_KP2'
>>> evdev.ecodes.KEY[81]
'KEY_KP3'
>>> # While pressing the keypad '1', '2' and '3' keys simultaneously
...
>>> ctrlkp.active_keys(verbose=True)
[('KEY_KP1', 79), ('KEY_KP2', 80), ('KEY_KP3', 81)]This should be all we need to manage the keypad, except for exclusivity, so we make sure the keys we press aren't bleeding out into the physical console that happens to be open on the machine. For that we'll use the grab() method as described in the evdev api documentation. As for the method for monitoring the keypad, we are spoiled for choice; there's a simple manual loop, optionally on a thread, that just reads from the inputs of interest, there's select, selectors and asyncio for python 3.6+ available event loop handling libraries and since I have never dealt with selectors, although it seems to be a simple wrapper around select, that's the one I'll be trying first.
We'll need to handle both the keypad and stdin, so a simple test shows that we can, indeed, handle both easily with selectors:
ce@bear:~/vms$ sudo python3
Python 3.6.9 (default, Nov 7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import evdev
>>> import selectors
>>> import sys
>>>
>>> sel = selectors.DefaultSelector()
>>> sel.register(sys.stdin, selectors.EVENT_READ)
SelectorKey(fileobj=<_io.TextIOWrapper name='<stdin>' mode='r' encoding='UTF-8'>, fd=0, events=1,
data=None)
>>>
>>> ctrlkp = evdev.InputDevice('/dev/input/by-id/usb-04d9_a02a-event-kbd')
>>> sel.register(ctrlkp, selectors.EVENT_READ)
SelectorKey(fileobj=InputDevice('/dev/input/by-id/usb-04d9_a02a-event-kbd'), fd=4, events=1, data=None)
>>>
>>> while True:
... for key, mask in sel.select():
... device = key.fileobj
... if isinstance(device, evdev.InputDevice):
... for event in device.read():
... event = evdev.util.categorize(event)
... if isinstance(event, evdev.events.KeyEvent):
... print(event)
... else:
... print (device.readlines(1))
...
I just typed this on the console
['I just typed this on the console\n']
Now I'll press keys on the keypad:
["Now I'll press keys on the keypad:\n"]
key event at 1575834985.711380, 69 (KEY_NUMLOCK), down
key event at 1575834985.719336, 69 (KEY_NUMLOCK), up
key event at 1575834985.727370, 79 (KEY_KP1), down
key event at 1575834985.791376, 79 (KEY_KP1), up
key event at 1575834985.799377, 69 (KEY_NUMLOCK), down
key event at 1575834985.807346, 69 (KEY_NUMLOCK), up
key event at 1575834988.279669, 69 (KEY_NUMLOCK), down
key event at 1575834988.287637, 69 (KEY_NUMLOCK), up
key event at 1575834988.295671, 80 (KEY_KP2), down
key event at 1575834988.367678, 80 (KEY_KP2), up
key event at 1575834988.375679, 69 (KEY_NUMLOCK), down
key event at 1575834988.383648, 69 (KEY_NUMLOCK), up
key event at 1575834989.735840, 69 (KEY_NUMLOCK), down
key event at 1575834989.743807, 69 (KEY_NUMLOCK), up
key event at 1575834989.751845, 80 (KEY_KP2), down
key event at 1575834989.751845, 81 (KEY_KP3), down
key event at 1575834989.927863, 80 (KEY_KP2), up
key event at 1575834989.927863, 81 (KEY_KP3), up
key event at 1575834989.935861, 69 (KEY_NUMLOCK), down
key event at 1575834989.943831, 69 (KEY_NUMLOCK), upThe final two pieces for this puzzle are the enumerating of available VMs by reading the contents of the conf directory, and controlling the qemu instances by feeding commands to its stdin, as I'll just ignore the stdout for now. For the latter I'll be using python's subprocess which always does the trick:
ce@bear:~$ sudo python3
Python 3.6.9 (default, Nov 7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> q = subprocess.Popen(('bash','/tmp/001'), stdin=subprocess.PIPE)
>>> QEMU 4.1.0 monitor - type 'help' for more information
(qemu) # Wait for windows to spin up and login
...
>>> q.stdin.write(b'system_powerdown\n')
17
>>> q.stdin.flush()
>>> system_powerdown
(qemu) q.wait(60)
0Adding everything together we can finally cook up the launcher.py file that will keep tabs on everything:
#!/usr/bin/env python3
import evdev
import formatter
import os
import selectors
import subprocess
import sys
PWD = os.path.abspath(os.path.dirname(__file__))
handlers = {}
def prepare_configs ():
confdir = os.path.join(PWD, 'conf')
for conf in os.listdir(confdir):
conf = formatter.readjson(os.path.join(confdir, conf))
vmid = str(conf.get('vmid', 0))
runner = '/tmp/{}.run'.format(vmid)
handlers[vmid] = {'runner': runner, 'proc': None}
frunner = open(runner, 'w')
frunner.write(formatter.format(conf))
frunner.close()
def handle_stdin (line):
line = tuple(filter(None, line.strip().split(' ')))
vm = None
if len(line) > 0:
if len(line) > 1:
if line[1] in handlers:
vm = handlers[line[1]]
if line[0] == 'start' and vm is not None:
if vm['proc'] is not None:
print("VM {} already started".format(line[1]))
else:
print("VM {} is starting".format(line[1]))
vm['proc'] = subprocess.Popen(('bash', vm['runner']), stdin=subprocess.PIPE)
elif line[0] == 'stop' and vm is not None:
if vm['proc'] is None:
print("VM {} not running".format(line[1]))
else:
vm['proc'].stdin.write(b'system_powerdown\n')
vm['proc'].stdin.flush()
vm['proc'].wait(60)
vm['proc'] = None
print("VM {} has stopped".format(line[1]))
elif line[0] == 'reset' and vm is not None:
if vm['proc'] is None:
print("VM {} not running".format(line[1]))
else:
print("sendinf VM {} system_reset".format(line[1]))
vm['proc'].stdin.write(b'system_reset\n')
vm['proc'].stdin.flush()
elif line[0] == 'quit' and vm is not None:
if vm['proc'] is None:
print("VM {} not running".format(line[1]))
else:
vm['proc'].stdin.write(b'quit\n')
vm['proc'].stdin.flush()
vm['proc'].wait(60)
vm['proc'] = None
print("VM {} has quit".format(line[1]))
KEYPAD_MAP = {'KEY_KP1': 'start 1',
'KEY_KP4': 'reset 1',
'KEY_KP7': 'stop 1',
'KEY_TAB': 'quit 1',}
def handle_keypad(event):
if event.keystate == event.key_up and event.keycode != 'KEY_NUMLOCK':
if event.keycode in KEYPAD_MAP:
handle_stdin(KEYPAD_MAP[event.keycode])
else:
print(event, "unhandled")
def handle_inputs ():
sel = selectors.DefaultSelector()
try:
ctrlkp = evdev.InputDevice('/dev/input/by-id/usb-04d9_a02a-event-kbd')
sel.register(ctrlkp, selectors.EVENT_READ)
except:
print("No Keypad found!")
sel.register(sys.stdin, selectors.EVENT_READ)
while True:
for key, mask in sel.select():
device = key.fileobj
if isinstance(device, evdev.InputDevice):
for event in device.read():
event = evdev.util.categorize(event)
if isinstance(event, evdev.events.KeyEvent):
handle_keypad(event)
else:
for l in device.readlines(1):
handle_stdin(l)
if __name__ == '__main__':
prepare_configs()
vms = list(handlers.keys())
vms.sort()
print("Found {} VM configs:".format(len(vms)))
for vm in vms:
print("\t{}\t{}".format(vm, handlers[vm]['runner']))
handle_inputs()I made launcher.py executable and added the entry @reboot /home/ce/vms/vmlauncher.cron to root's crontab, and after making it executable and owned by root (just to be on the safe side) it is time to give this a spin! Spoiler alert, it works great... Restarted the machine, logged in as ce, ran sudo tmux a to attach myself to the running tmux session for root and;
root@bear:~# cd /home/ce/vms/
root@bear:/home/ce/vms# ./launcher.py
Found 1 VM configs:
1 /tmp/1.run
VM 1 is starting
QEMU 4.1.0 monitor - type 'help' for more information
(qemu) stop
stop 1
system_powerdown
(qemu) VM 1 has stoppedThat is going to be it for this post, you can grab all the files as I have them at this point in the archive Windows_virtually_pt4-src.tar.bz2
Note
This entry's image was created using The Virtual Keypunch by mass:werk, which beautifully replicates a card punched by the once common IBM 29 Card Punch machine.