Clone specific software-layer-commit and implement CI to check merged status

[casparvl]

This PR is an initial step in creating a workflow where we can use PRs to software-layer-scripts directly, and then once they are merged, just update the SHA checksum to a (Github-signed) merge commit, rebuild, and be done :)

Edit: better description...

This PR contains three changes:

1. bot/build.sh now clones a particular commit, which is specified in bot/commit_sha
2. CI job Verify software-layer-scripts / check_bot_build_checksum verifies the checksum of bot/build.sh against a reference hardcoded in the workflow file. This way, a malicious contributor would have to modify both bot/build.sh and the workflow file, which would (hopefully) stand out to a reviewer.
3. CI job Verify software-layer-scripts / check_software_layer_scripts_commit which check that bot/commit_sha is a commit that is part of the upstream https://github.com/EESSI/software-layer-scripts , is on the main branch (i.e. has been merged), is a merge commit, and is signed with the public Github GPG for the web interface.

We might need to update the commit_sha already (not sure if there have been more merges to software-layer-scripts since I started this) before we merge this to make sure this PR doesn't actually revert us to using an older version.

[casparvl]

Let's do a small test build to see if the new build.sh works, i.e. if it correctly clones the software-layer-scripts repo from a given commit.

[casparvl]

Perfect. CI run on 2cd6082 shows

Commit c0a3ff09a3a38737af5a922fdf581aa7b2dd6c88 is NOT merged into origin/main.
Error: Process completed with exit code 1.

as expected, since this commit is on a feature branch but is not merged. Then, using a merge commit as in 6d954c4 the CI now passes.

[casparvl]

TODO: I guess in this same PR we should still add a check that verifies that the SHA-checksum of bot/build.sh itself has remained unchanged (there should be no reason to change it, since the sha-checksum is external to this file).

[casparvl]

Perfect. As expected, after changing the bot/build.sh in f1fdcca and fixing a typo in the workflow in c4b1f9a I get:

Computed checksum: bb805939ae22f3ca2e6fc85d13613aeb9b3fc81974a2e1ef3bfc85a7f3ae8a0f
Reference checksum: 9d33368cac2e38e10147eeb0aafc321651ebaa5912387ecef97683570906773a
ERROR: Checksum mismatch! The file bot/build.sh has been modified.

Changing the bot/build.sh back to it's original version in 0494884 and having the CI run on a subsequent merge commit 72fbb29 I now get

Computed checksum: 9d33368cac2e38e10147eeb0aafc321651ebaa5912387ecef97683570906773a
Reference checksum: 9d33368cac2e38e10147eeb0aafc321651ebaa5912387ecef97683570906773a
Checksum for bot/build.sh matches the reference value

[casparvl]

Ok, as expected, both Verify software-layer-scripts / check-bot_build_checksum and Verify software-layer-scripts / check_software_layer_scripts_commit pass on bce9bbc

Let's test again by changing the sha checksum...

[casparvl]

Ok, on bee1d29 we again have the expected failure:

Commit c0a3ff09a3a38737af5a922fdf581aa7b2dd6c88 is NOT merged into origin/main.
Error: Process completed with exit code 1.

Let's change the sha checksum back, and now change something in bot/build.sh to test the other CI job.

[casparvl]

Again, we get the expected failure:

Computed checksum: 93705d4ae3517d9dfcac79d4e7a113e62977d9187b1af9bd4797c03367a9cdfb
Reference checksum: 9d33368cac2e38e10147eeb0aafc321651ebaa5912387ecef97683570906773a
ERROR: Checksum mismatch! The file bot/build.sh has been modified.
Error: Process completed with exit code 1.

[bedroge]

I would make it clear that this sha belongs to a software-layer-scripts commit, so maybe rename it to software-layer-scripts.commit_sha or something?

[casparvl]

Discussed in support meeting. One issue came up - though it doesn't affect this specific PR, nor the CI checks in it.

The issue is related to how we might use the commit_sha introduced in this PR to make the bot refuse to deploy. The rationale is: if a bot has tarballs with a commit_sha that do not match the bot/commit_sha, those are invalid and should not be deployed. However, what may happen is this:

• software-layer-scripts PR 1 introduces a change (e.g. to eb_hooks), commit X
• software-layer PR 1 uses it to build some software, and proves that the new eb_hooks solves some issue
• software-layer-scripts PR 1 is merged, creating a merge-commit Y
• software-layer PR 1 updates bot/commit_sha to Y and rebuilds all tarballs
• software-layer-scripts PR 2 introduces a different change, which gets accepted and merged with merge commit Z
• software-layer PR 2 uses merge commit Z to build all tarballs
• software-layer PR 2 gets deployed and merged
• software-layer PR 1 gets synced with main. This means bot/commit_sha should be updated to Z (as Z happened after Y, and we don't want a merge of software-layer PR 1 to 'downgrade' the bot/commit_sha to an older version than the one on main). This now invalidates all builds done with bot/commit_sha=Y - even though those are perfectly valid builds!

I'm not yet sure how to avoid this. It is not possible to retain Y in SW PR1, as a merge of PR1 would then cause main to contain bot/commit_sha=Y, which we don't want if Z is newer than Y. So we have to come up with a way that if we then accept Z in bot/commit_sha in the PR 1, that builds with Y are still considered 'valid'.

Note that I don't think we should hold off the current PR: it's a first step that'll provide us with the ability to build in software-layer from PRs to software-layer-scripts, which makes this workflow smoother already. And it guards at least against merging this PR if the commit_sha doesn't belong to a signed merge commit. Those two aspects in themselves add value - even without any modifications to the bot-deploy procedure.