| 1 | Soil
|
| 2 | ====
|
| 3 |
|
| 4 | Continuous testing on many platforms.
|
| 5 |
|
| 6 | ## Server Setup
|
| 7 |
|
| 8 | soil/web-init.sh deploy # deploy static assets and tools
|
| 9 |
|
| 10 | ## Data
|
| 11 |
|
| 12 | https://test.oils-for-unix.org/ # not Soil, Coil could be a better name
|
| 13 |
|
| 14 | static-api/ # used by the maybe-merge task
|
| 15 | github/
|
| 16 | 1234/ GITHUB_RUN_ID (or NUMBER)
|
| 17 | dev-minimal # file with status 0
|
| 18 | cpp-small
|
| 19 |
|
| 20 | github-jobs/
|
| 21 | index.html # pretty list of runs, links to commits/ and foo.wwz/index
|
| 22 | raw.html # flat
|
| 23 | 1234/ # GITHUB_RUN_NUMBER
|
| 24 | dev-minimal.wwz/
|
| 25 | index.html # worker runs format-wwz-index for TSV -> HTML
|
| 26 | dev-minimal.{tsv,json}
|
| 27 | cpp-small.wwz/
|
| 28 | index.html
|
| 29 | cpp-small.{tsv,json}
|
| 30 | commits/
|
| 31 |
|
| 32 | srht-jobs/
|
| 33 | index.html
|
| 34 | raw.html
|
| 35 | 345/ # JOB_ID
|
| 36 | dev-minimal.wwz/
|
| 37 | index.html
|
| 38 | dev-minimal.{tsv,json}
|
| 39 | 344/
|
| 40 | cpp-small.wwz/
|
| 41 | index.html
|
| 42 | cpp-small.{tsv,json}
|
| 43 | commits/ # index grouped by commit?
|
| 44 | 09ab09ab.html # links to ../345/dev-minimal.wwz/
|
| 45 | 1010abab.html
|
| 46 |
|
| 47 | ## Tokens / Authentication
|
| 48 |
|
| 49 | - `SOIL_GITHUB_API_TOKEN` - used by `maybe-merge` task, to use Github API to fast forward
|
| 50 | - appears in `.github/workflows/all-builds.yml` for **only** the `maybe-merge` task
|
| 51 | - `OILS_GITHUB_KEY` - used by all tasks to publish HTML
|
| 52 | - - should really be called `OILS_SSH_FROM_GITHUB_ACTIONS`
|
| 53 |
|
| 54 | ## Code
|
| 55 |
|
| 56 | Running a job starts at either:
|
| 57 |
|
| 58 | - `.github/workflows/all-builds.yml` for Github Actions
|
| 59 | - `.builds/worker{1,2,3,4}.yml` for sourcehut
|
| 60 |
|
| 61 | The YAML files either:
|
| 62 |
|
| 63 | - Directly invoke `soil/worker.sh` for a raw VM job.
|
| 64 | - Invoke wrappers `soil/github-actions.sh` / `soil/sourcehut.sh`, which in turn
|
| 65 | uses `soil/host-shim.sh` to run a job in a container.
|
| 66 | - They wrappers also publish via SSH with `soil/web-worker.sh`
|
| 67 |
|
| 68 | `soil/host-shim.sh` pulls and start an OCI container each node, and then runs
|
| 69 | `soil/worker.sh` inside the container.
|
| 70 |
|
| 71 | `soil/worker.sh` runs the job, and publishes 3 files to `travis-ci.oilshell.org`:
|
| 72 |
|
| 73 | 1. JSON metadata about the commit and build environment
|
| 74 | 1. TSV metadata for each "toil" step
|
| 75 | 1. A `.wwz` file (servable zip file) of logs
|
| 76 |
|
| 77 | `soil/web.sh` is a wrapper around `soil/web.py`, and it runs on the SERVER
|
| 78 | (`travis-ci.oilshell.org`). It reads and joins metadata from every job to
|
| 79 | construct `index.html` and `$RUN_NUMBER/index.html`.
|
| 80 |
|
| 81 | The server-side components are deployed by `soil/web-init.sh`.
|
| 82 |
|
| 83 | ## Terminology
|
| 84 |
|
| 85 | Three level hierarchy of executables:
|
| 86 |
|
| 87 | - Run - A group of jobs run per commit, usually on different machines.
|
| 88 | Github has `GITHUB_RUN_{NUMBER,ID}`, but sourcehut lacks this concept.
|
| 89 | - Job - A group of tasks run on the same machine, in the same container, or on a raw VM.
|
| 90 | - Task - A shell command that's run within the Oil repo. Tasks are currently
|
| 91 | run sequentially.
|
| 92 |
|
| 93 | TODO:
|
| 94 |
|
| 95 | - Tasks need explicit dependencies
|
| 96 | - Dynamically schedule tasks, and remove the idea of a job. It should just be
|
| 97 | a flat list of tasks run per commit, on various machines, with
|
| 98 | dynamically-scheduled and resolved dependencies.
|
| 99 |
|
| 100 | ## Events
|
| 101 |
|
| 102 | - End Job.
|
| 103 | - First upload wwz, tsv, and JSON last
|
| 104 | - web.py can rewrites the following files in one pass, based on list-json
|
| 105 | state
|
| 106 | - github-jobs/tmp-$$.{index,raw}.html - shell script does mv
|
| 107 | - github-jobs/commits/tmp-$$.$HASH.html - shell script does mv
|
| 108 | - this is based on github-jobs/$RUN/*.tsv -- similar to format-wwz-index
|
| 109 | - or srht-jobs/*/*.tsv and filtered by commit
|
| 110 | - github-jobs/tmp-$$.remove.txt - shell script does rm
|
| 111 | - status-api/github-jobs/$RUN/$job -- PUT this
|
| 112 |
|
| 113 | - Start job. TODO: This doesn't exist.
|
| 114 | - github-jobs/index.html should show Started / Pass / Fail
|
| 115 |
|
| 116 | ## Web Interface / Security
|
| 117 |
|
| 118 | This one is a pure uploader, which you can borrow from picdir.
|
| 119 |
|
| 120 | POST https://test.oils-for-unix.org/results
|
| 121 | ?jobName=dev-minimal
|
| 122 | &workerHost=github
|
| 123 |
|
| 124 | It does multi-part file upload of wwz, TSV, JSON, and saves the files. Start
|
| 125 | with basic auth over HTTPS? See picdir.
|
| 126 |
|
| 127 |
|
| 128 | POST https://test.oils-for-unix.org/event
|
| 129 |
|
| 130 | ?eventName=start
|
| 131 | &workerHost=github
|
| 132 | &runId=1234
|
| 133 | &jobName=dev-minimal
|
| 134 |
|
| 135 | ?eventName=start
|
| 136 | &workerHost=sourcehut
|
| 137 | &jobId=345
|
| 138 | &jobName=dev-minimal
|
| 139 |
|
| 140 | # whether to update the status-api
|
| 141 | ?eventName=done
|
| 142 | &workerHost=github
|
| 143 | &runId=1234
|
| 144 | &jobName=dev-minimal
|
| 145 | &status=124
|
| 146 | &updateStatusApi=1
|
| 147 |
|
| 148 | This PHP script can just run a shell script synchronously? Events start and
|
| 149 | done are meant to be quick.
|
| 150 |
|
| 151 | Other events:
|
| 152 |
|
| 153 | - begin pulling container
|
| 154 | - done pulling container
|
| 155 | - done with task, although we want to get rid of this abstraction
|
| 156 |
|
| 157 |
|
| 158 | ## Notes
|
| 159 |
|
| 160 | ### Idea For Tuple Space?
|
| 161 |
|
| 162 | Listening to `oilshell/oil`:
|
| 163 |
|
| 164 | - `dev-minimal`
|
| 165 | - Can a contributor quickly get started with the Oil repo?
|
| 166 | - They just want to run bin/osh without installing much
|
| 167 | - `ovm-tarball`
|
| 168 | - `cpp`
|
| 169 | - `other-tests`
|
| 170 |
|
| 171 | Listening to `oilshell/tarballs`:
|
| 172 |
|
| 173 | - Travis
|
| 174 | - OS X (bin-darwin)
|
| 175 | - Sourcehut
|
| 176 | - Alpine (bin-alpine)
|
| 177 | - OpenBSD (bin-openbsd)
|
| 178 | - Alternate architectures like ARM
|
| 179 |
|
| 180 | All of these build both oil.ovm and oil-native. Need maintainers. We build
|
| 181 | them as a "start".
|
| 182 |
|
| 183 | It would nice to implement this as SQL.
|
| 184 |
|
| 185 |
|
| 186 |
|