Automated Testing Service

The eCTF team creates an Automated Testing Service to allow teams to verify that their design meets the Functional Requirements.

The testing service can be interacted with through Slack.

Submitting a Test

Tests are submitted through the eCTF Slackbot. To submit a test, use the /testdesign command from within your team’s private Slack channel.

In order to submit, you will need the URL of your team’s git repository, and the tag/branch of your design intended for testing. If testing your team’s design, you will need to add the ectf account on GitHub, GitLab, or BitBucket to your design.

Tests may take some time to finish. The testing infrastructure will take your design, build it, and test it on physical hardware. You are sharing the system with other teams, so be courteous of the quantity of tests that you submit!

Your team’s design must pass testing to continue through the Handoff process.

You can check the status of your test with the /queuecheck Slack command.

Testing Flag

To receive the testing flag, simply submit a test with the organizer provided example design!

Testing Results

After your test has been executed, you will receive a log of how your design performed against the functional requirement tests. An example of a passing test is shown below.

[10:41:47] SUCCESS  [Logger] PASS Substep: Remove old repo
[10:41:52] SUCCESS  [Logger] PASS Substep: git clone
[10:41:52] SUCCESS  [Logger] PASS Step: CloneDesign
[10:41:52] SUCCESS  [Logger] PASS Substep: Remove .git folder
[10:41:52] SUCCESS  [Logger] PASS Substep: Remove __pycache__ folder
[10:41:52] SUCCESS  [Logger] PASS Step: PruneGitRepo
[10:42:00] SUCCESS  [Logger] PASS Substep: Build Deployment
[10:42:00] SUCCESS  [Logger] PASS Substep: Hash Deployment
[10:42:04] SUCCESS  [Logger] PASS Substep: Build Comp 0
[10:42:08] SUCCESS  [Logger] PASS Substep: Build Comp 1
[10:42:11] SUCCESS  [Logger] PASS Substep: Build Comp 2
[10:42:15] SUCCESS  [Logger] PASS Substep: Build Comp 3
[10:42:19] SUCCESS  [Logger] PASS Substep: Build AP
[10:42:19] SUCCESS  [Logger] PASS Substep: Hash Deployment
[10:42:19] SUCCESS  [Logger] PASS Substep: Compare Hashes
[10:42:19] SUCCESS  [Logger] PASS Step: BuildTestImages
[10:42:19] SUCCESS  [Logger] PASS Substep: Protect AP
[10:42:20] SUCCESS  [Logger] PASS Substep: Protect C0
[10:42:20] SUCCESS  [Logger] PASS Substep: Protect C1
[10:42:20] SUCCESS  [Logger] PASS Substep: Protect C2
[10:42:20] SUCCESS  [Logger] PASS Substep: Protect C3
[10:42:20] SUCCESS  [Logger] PASS Step: ProtectTestImages
[10:43:26] SUCCESS  [Logger] PASS Substep: Flash AP
[10:44:31] SUCCESS  [Logger] PASS Substep: Flash C0
[10:45:35] SUCCESS  [Logger] PASS Substep: Flash C1
[10:45:42] SUCCESS  [Logger] PASS Substep: Reset boards
[10:45:42] SUCCESS  [Logger] List
[10:45:42] SUCCESS  [Logger] PASS Substep: List
[10:45:42] SUCCESS  [Logger] Found expected result
[10:45:42] SUCCESS  [Logger] PASS Substep: Validate List Output
[10:45:42] SUCCESS  [Logger] Attest
[10:45:42] SUCCESS  [Logger] PASS Substep: Attest c0
[10:45:42] SUCCESS  [Logger] Found expected result
[10:45:42] SUCCESS  [Logger] PASS Substep: Validate Attest
[10:45:42] SUCCESS  [Logger] Attest
[10:45:42] SUCCESS  [Logger] PASS Substep: Attest c1
[10:45:42] SUCCESS  [Logger] Found expected result
[10:45:42] SUCCESS  [Logger] PASS Substep: Validate Attest
[10:45:42] SUCCESS  [Logger] Boot
[10:45:42] SUCCESS  [Logger] PASS Substep: Boot
[10:45:42] SUCCESS  [Logger] Found expected result
[10:45:42] SUCCESS  [Logger] PASS Substep: Validate Boot Output
[10:45:48] SUCCESS  [Logger] PASS Substep: Reset boards
[10:45:48] SUCCESS  [Logger] Replace
[10:45:48] SUCCESS  [Logger] PASS Substep: Replace c0
[10:45:48] SUCCESS  [Logger] Found expected result
[10:45:48] SUCCESS  [Logger] PASS Substep: Validate replace
[10:45:48] SUCCESS  [Logger] List
[10:45:48] SUCCESS  [Logger] PASS Substep: List
[10:45:48] SUCCESS  [Logger] Found expected result
[10:45:48] SUCCESS  [Logger] PASS Substep: Validate List Output
[10:45:54] SUCCESS  [Logger] PASS Substep: Reset boards
[10:45:54] SUCCESS  [Logger] List
[10:45:54] SUCCESS  [Logger] PASS Substep: List
[10:45:54] SUCCESS  [Logger] Found expected result
[10:45:54] SUCCESS  [Logger] PASS Substep: Validate List Output
[10:47:00] SUCCESS  [Logger] PASS Substep: Flash C2
[10:47:06] SUCCESS  [Logger] PASS Substep: Reset boards
[10:47:06] SUCCESS  [Logger] List
[10:47:06] SUCCESS  [Logger] PASS Substep: List
[10:47:06] SUCCESS  [Logger] Found expected result
[10:47:06] SUCCESS  [Logger] PASS Substep: Validate List Output
[10:47:06] SUCCESS  [Logger] Boot
[10:47:06] SUCCESS  [Logger] PASS Substep: Boot
[10:47:06] SUCCESS  [Logger] Found expected result
[10:47:06] SUCCESS  [Logger] PASS Substep: Validate Boot Output
[10:47:08] SUCCESS  [Logger]
[10:47:09] SUCCESS  [Logger] PASS Substep: Post Boot:
[10:47:09] SUCCESS  [Logger] PASS Step: RunTests
[10:47:09] SUCCESS  [Logger] Verify Package Contents Flow Passed