Test Flow Description

This page will give a description in prose of every job queued up for the test flow as well as a description of each test that actually runs on the hardware.

Jobs

• clone_repo

  Tests to see if the given commit url + commit hash can be cloned, may return a git error message.

• build_firmware_image

  Attempts to build the docker image used to build the team’s firmware, may return output log from docker build.

• generate_secrets

  Build’s the team’s design docker image and then uses it to generate secrets for 32 randomly generated group ids. May return the following log outputs, output log from docker build of design image, and output from invoking the design container to generate secrets. When successful this job return The generated secrets and the 32 randomly generated group ids.

• build_firmware

  Generates 2 hsm binaries using the previously build firmware docker image. Both binaries have the same randomly generated pin. Binary A has permission string 'G0=---:G1=R--:G2=-W-:G3=RW-:G4=--C:G5=R-C:G6=-WC:G7=RWC' where instead of G0-G7 it uses the first 8 group ids generated from generate_secrets. Binary B has permission string 'G2=R-C:G3=R-C:G4=RWC' where instead of G2-G4 we make the same replacement as Binary A. This job will produce: A & B Binaries, pin used for both, and A & B permission strings.

• prot_firmware

  Encrypts the two binaries from the previous build step. Will produce protected versions of A & B firmwares (can only be flashed on attack boards).

• run_tests

  This job runs the first set of functional tests. This test set focuses on capturing edge cases. Full details of this test below in Tests from the run_tests job. Will produce a log.html which will give the details on which tests succeeded and failed.

• create_attack_package_phase_1

  This job runs the first phase of building the attack package. Generates new secrets, Builds firmware for Attacker, Engineer, and Litho. This job does not does not rebuild docker images it reuses previously built images for firmware and design. Will produce new plaintext binaries for Attacker, Engineer, and Litho. Also produces a new global.secrets file.

• create_attack_package_phase_2

  This job runs the second phase of building the attack package. Flashes all three binaries onto boards and fills slot 0 with file needed for the scenario. Then it reads out the full binary and encrypts the new binary now packaged with a starting file. Will produce the three encrypted packed binaries: Attacker, Engineer, and Litho. Also produces scenario_info.yaml, which has all the extra details for the attack phase. First in the yaml are the group ids for design, update, telemetry, & calibration group. Also in the yaml are the file UUIDs for the files loaded on each device. Then in the yaml are the pins for each device. The yaml also has the Device UUID for each device. Lastly in the yaml are the permission string for each device.

• run_scenario_tests

  Runs a set of tests which guarantee that the minimal set of needed operations work for the actual firmwares used in the attack packages. also tests digests Full details of this test below in Tests from the run_scenario_tests job. Will produce a log.html which will give the details on which tests succeeded and failed.

Warning

The exact tests run are subject to change during the design phase! Please be aware that the organizers WILL add more tests before we move into the attack phase.

Tests from the run_tests job

• flash

  Flashes hsm A and B.

• list_empty

  Lists on hsm A and B.

• write_1

  Writes a file from A on slot 0 and then lists on A and reads the file on slot 0.

• interrogate_1

  B interrogates A.

• receive_1

  B receives the file on slot 0 on A

• write_max

  Writes the same example file to every empty slot on A. Then lists on A and reads every file on A. Finally B interrogates A

• overwrite

  A file is overwritten on slot 7 of A. Then list and read file on A. Finally B interrogates and receives the file on A and then list and read the file on B.

• pass_file_back_and_forth

  Write file on slot 0 of B. A interrogates and receives from B and list on A. B interrogates and receives same file from A but in a new slot. Finally list on B and read both original file as well as file received from A.

• write_max_file_name

  Write a file to A on slot 0 with max file name list and then read file on A.

• write_max_file_size

  Write a file to A of the max file size of 8192 bytes to slot 0. List and read the file on slot 0 on A. Finally write that file again to slots 1-7 on A.

• receive_max_file

  B interrogates and receives on of the max size files from A. B then lists and reads this file.

• write_all_ascii

  Write a file on A containing all ascii characters from 0-127 then list and read the file.

• bad_pin

  Perform a list on A using the wrong pin, no specific necessary output but it shouldn’t crash. Then list and read a file using the correct pin.

• read_without_perms

  Write a file to A for which A does NOT have the read permission. Attempt to read the file, no specific necessary output but it shouldn’t crash. Finally list on A.

• write_without_perms

  Write a file to A for which A foes NOT have the write permission, no specific necessary behavior but shouldn’t crash. Write a file to A on the same slot where A does have write permissions then list on A.

• receive_without_perms

  Write a file to A where A has read and write permissions, but not receive permissions, on slot 3. B interrogates and receives file on slot 3 from A. Overwrite file on slot 3 on A. A interrogates and receives file from B, no specific necessary behavior but shouldn’t crash. Write again to slot 3 on A and then list on A.

• write_0_byte_file

  Write a file to A with no contents.

• read_0_byte_file

  List and read the 0 byte file on A.

• receive_0_byte_file

  B interrogates and receives the 0 byte file from A. List and read the file on B.

Tests from the run_scenario_tests job

• flash_engineer_litho_pair

  Flash engineer and Litho HSM.

• engineer_digest

  Get the digest from design_file1 on engineer HSM.

• litho_digest

  Get the digest from design_file2 on litho HSM.

• engineer_list

  List on engineer HSM.

• litho_list

  List on litho HSM.

• engineer_read

  Read design_file1 on engineer HSM.

• litho_read

  Read design_file2 on litho HSM.

• engineer_litho_swap

  Litho interrogates and receives design_file1 from engineer. Engineer then interrogates and receives design_file2 from Litho HSM. List on engineer and litho HSM. Read design_file2 on engineer HSM and read read design_file1 on litho HSM.

• create_file_engineer

  Write a design file on engineer HSM on slot 2. List and read design file on slot 2. Then Litho interrogates and receives file on slot 2 from engineer HSM. Finally list and read file on slot 2 on litho HSM.

• triple_digest_transfer

  Write a file on slot 0 to engineer HSM then get the digest of slot 0 on engineer HSM. Litho interrogates and receives file on slot 0 from engineer then get digest of that file on litho HSM. Flash an alternate version of engineer HSM with different device id. Alternate engineer HSM interrogates and receives file on slot 0 from litho then get digest of that file on alternate engineer HSM.

• flash_attacker_litho_pair

  Flash attack HSM and flash litho HSM.

• attacker_digest

  Get the digest from update_file on litho HSM.

• attacker_list

  List on attacker HSM.

• attacker_litho_swap_update

  Litho interrogates and receives update_file from attacker. List on litho HSM then read update_file on litho HSM.

• attacker_write_calibration

  Attacker writes calibration file to slot 1. List on attacker HSM then read slot 1 on attacker HSM.

• attacker_litho_swap_calibration

  Litho interrogates and receives calibration file from attacker. List on litho HSM then read calibration file on litho HSM. Attacker interrogates and receives calibration file from litho on a new slot. List on attacker HSM then read new calibration file on attacker HSM.

• litho_write_telemetry

  Write telemetry file on litho HSM then list on litho HSM.

• attacker_litho_swap_telemetry

  Attacker interrogates and receives telemetry file from litho HSM. List on attacker HSM then read telemetry file on attacker HSM.