SkiaLab

Overview

Skia's buildbots are hosted in three places:

• Google Compute Engine. This is the preferred location for bots which don‘t need to run on physical hardware, ie. anything that doesn’t require a GPU, stable performance numbers, or a specific hardware configuration. Most of our compile bots live here, along with some non-GPU test bots on Linux and Windows.
• Chrome Golo. This is the preferred location for bots which require specific hardware or OS configurations that are not supported by GCE. We have several Mac, Linux, and Windows bots in the Golo.
• The local SkiaLab in Chapel Hill. Anything we can't get in GCE or the Golo lives here. This includes newer or uncommon GPUs and all Android, ChromeOS, and iOS devices.

This page covers the local SkiaLab in Chapel Hill.

Layout

The SkiaLab consists of three wireframe racks which hold machines connected to two KVM switches. Each KVM switch has a monitor, mouse, and keyboard and is the primary mode of access to the lab machines. In general, the machines are on the same rack as the KVM switch used to access them. The switch nearest the door (labeled “DOOR”), is connected to machines on its own rack as well as a smaller rack closer to the door.

Each machine is labeled with its hostname and the number or letter used to access it on the KVM switch. Android devices are located on the rack nearest the interior of the office (the KVM switch is labeled “OFFICE”). They are labeled with their serial number and the name of the buildslave they are associated with. Each device connects to a host machine, either directly or by way of a powered USB hub.

Disclaimer: Please ONLY make changes on a lab machine as a last resort, as it is disruptive to the running bots and can leave the machines in a dirty state. If you must make changes, such as cloning a copy of Skia to run tests and debug failures, be sure to clean up after yourself. If a permanent change needs to be made on the machine (such as a driver update), please contact an infra team member.

Common Tasks

Locating the host machine for a failing bot

Sometimes failures can only be reproduced on a particular hardware configuration. In these cases, it is sometimes necessary to log into the host machine where a failing bot is running in order to debug the failure.

From the Status page:

1. Click on the box associated with a failed build.
2. A popup will appear with some information about the build, including the builder and buildslave. Click the “Lookup” link next to “Host machine”. This will bring you to the SkiaLab Hosts page, which contains information about the machines in the lab, pre-filtered to select the machine which runs the buildslave in question.
3. The information box will display the hostname of the machine as well as the KVM switch and number used to access the machine, if the machine is in the SkiaLab.
4. Walk over to the lab. While standing at the KVM switch indicated by the host information page, double tap <ctrl> and then press the number or letter from the information page. It may be necessary to move or click the mouse to wake the machine up.
5. Log in to the machine if necessary. The password is stored in Valentine as “Chapel Hill buildbot slave password”.

Rebooting a problematic Android device

Follow the same process as above, with some slight changes:

1. On the Status page, click the box for the failed build.

2. Click the “Lookup” link for the host machine. Remember the name of the buildslave which ran the build.

3. The hosts page will display the information used to access the host machine for the device as well as the serial number for the device next to the name of its buildsave.

4. Walk over to the lab and find the Android device with the serial number from the hosts page. Hold the power and volume-up buttons until the device reboots.

5. Access the host machine for the device, per the above instructions. Use the which_devices.py script to verify that the device has re-attached. From the home directory:

    $ python buildbot/scripts/which_devices.py
   

Maintenance Tasks

Bringing up a new buildbot host machine

This assumes that we‘re just adding a host machine for a new buildbot slave, and doesn’t cover how to make changes to the buildbot code to change the behavior of the builder itself.

1. Obtain the machine itself and place it on the racks in the lab. Connect power, ethernet, and KVM cables.
2. If we already have a disk image appropriate for this machine, follow the instructions for flashing a disk image to a machine below. Otherwise, follow the instructions for bringing up a new machine from scratch.
3. Power on the machine. Be sure to kill any buildbot processes that start up, eg. killall python on Linux and Mac, and just close any cmd instances which pop up on Windows.
4. Set the hostname for the machine.
5. Ensure that the machine is labeled with its hostname and KVM number.
6. Add the new slave to the slaves.cfg file on the appropriate master, eg. https://chromium.googlesource.com/chromium/tools/build/+/master/masters/master.client.skia/slaves.cfg, and upload the change for code review.
7. Add an entry for the new host machine to the slave_hosts_cfg.py file in the Skia infra repo: https://skia.googlesource.com/buildbot/+/master/site_config/slave_hosts_cfg.py, and upload it for review.
8. Commit the change to add the slave to the master. Once it lands, commit the slave_hosts_cfg.py change immediately afterward.
9. Restart the build master. Either ask borenet@ to do this or file a ticket for a trooper to do it.
10. Reboot the machine and monitor the build master to ensure that it connects. This can take some time, since the bot needs to sync Chrome.

Bringing up a new Android bot

1. Locate or add a host machine. We generally want to keep the number of devices attached to each host below 5 or so. If a new host machine is required, follow the above instructions for bringing up a new buildbot host machine, with the exception that the slave corresponds to the Android device, not the host machine itself.

2. Ensure that the buildslave is not yet running:

    $ killall python
   

3. Disable MTP and PTP on the device. Some devices require one or the other to be enabled; in that case, select PTP and choose to ‘do nothing’ when attaching to the host machine.

4. Connect the device to the host machine, either through a powered USB hub or directly to the machine.

5. Make sure that the device is in developer mode and that USB debugging is enabled.

6. Authorize the device for USB debugging on the host machine by checking the “always allow” box on dialog box which appears on the Android device after plugging it into the host.

7. Ensure that the device appears as “connected” when you run the which_devices.py script:

    $ python buildbot/scripts/which_devices.py
   

8. Reboot the machine to start the buildslave.

Bringing up a new machine from scratch

TODO(borenet): Migrate from Google Docs.

OS-specific instructions are available in a Google Doc

Flashing a disk image to a machine

1. Find the USB key labeled, “Clonezilla” in the SkiaLab and insert it into the machine.
2. Turn on the machine and load the boot menu. For Shuttle machines, press <del> or <esc>. Mac machines require that you plug in the Mac keyboard and press the <option> key at boot. Boot from the USB key. It's typically UEFI and named something like “FlashBlu” or “Kanguru”.
3. At the Clonezilla menu, choose the “to RAM” option.
4. Choose your preferred language.
5. “Don't touch keymap”.
6. “Start Clonezilla”.
7. “device-image”.
8. “local_dev”.
9. Unplug the flash drive and plug in the external hard drive labeled, “Disk images.” Wait for the “Attached Enclosure device” message to appear, then hit <enter>.
10. Select the external drive to use for /home/partimag, something like, “1000GB_ntfs_My_Passport”.
11. Select the bot_img directory.
12. Hit <enter> to continue.
13. “Beginner”
14. “restoredisk”
15. Select the image to use. Make sure that it's compatible with this machine.
16. Choose the hard drive in the machine. It should be the only option.
17. “y” and “y”
18. Choose “reboot” after flashing the image to the machine.
19. Set the hostname of the machine so that it doesn't conflict with any existing machines.

Capturing a disk image

1. Make sure that the machine is in a clean state: no pre-existing buildslave checkouts, extra software, etc.
2. Find the USB key labeled, “Clonezilla” in the SkiaLab and insert it into the machine.
3. Turn on the machine and load the boot menu. For Shuttle machines, press <del> or <esc>. Mac machines require that you plug in the Mac keyboard and press the <option> key at boot. Boot from the USB key. It's typically UEFI and named something like “FlashBlu” or “Kanguru”.
4. At the Clonezilla menu, choose the “to RAM” option.
5. Choose your preferred language.
6. “Don't touch keymap”.
7. “Start Clonezilla”.
8. “device-image”.
9. “local_dev”
10. Unplug the flash drive and plug in the external hard drive labeled, “Disk images.” Wait for the “Attached Enclosure device” message to appear, then hit <enter>.
11. Select the external drive to use for /home/partimag, something like, “1000GB_ntfs_My_Passport”.
12. Select the bot_img directory.
13. “Beginner”
14. “savedisk”
15. Choose a name for the disk image. The convention is: skiabot-<hardware type>-<OS>-<disk image revision #>
16. Choose the hard drive in the machine. It should be the only option.
17. “y”
18. Choose “reboot” or “shut down” when finished.