OILS / soil / README.md View on Github | oilshell.org

186 lines, 134 significant
1Soil
2====
3
4Continuous testing on many platforms.
5
6## Server Setup
7
8 soil/web-init.sh deploy # deploy static assets and tools
9
10## Data
11
12 https://test.oils-for-unix.org/ # not Soil, Coil could be a better name
13
14 static-api/ # used by the maybe-merge task
15 github/
16 1234/ GITHUB_RUN_ID (or NUMBER)
17 dev-minimal # file with status 0
18 cpp-small
19
20 github-jobs/
21 index.html # pretty list of runs, links to commits/ and foo.wwz/index
22 raw.html # flat
23 1234/ # GITHUB_RUN_NUMBER
24 dev-minimal.wwz/
25 index.html # worker runs format-wwz-index for TSV -> HTML
26 dev-minimal.{tsv,json}
27 cpp-small.wwz/
28 index.html
29 cpp-small.{tsv,json}
30 commits/
31
32 srht-jobs/
33 index.html
34 raw.html
35 345/ # JOB_ID
36 dev-minimal.wwz/
37 index.html
38 dev-minimal.{tsv,json}
39 344/
40 cpp-small.wwz/
41 index.html
42 cpp-small.{tsv,json}
43 commits/ # index grouped by commit?
44 09ab09ab.html # links to ../345/dev-minimal.wwz/
45 1010abab.html
46
47## Tokens / Authentication
48
49- `SOIL_GITHUB_API_TOKEN` - used by `maybe-merge` task, to use Github API to fast forward
50 - appears in `.github/workflows/all-builds.yml` for **only** the `maybe-merge` task
51- `OILS_GITHUB_KEY` - used by all tasks to publish HTML
52 - - should really be called `OILS_SSH_FROM_GITHUB_ACTIONS`
53
54## Code
55
56Running a job starts at either:
57
58- `.github/workflows/all-builds.yml` for Github Actions
59- `.builds/worker{1,2,3,4}.yml` for sourcehut
60
61The YAML files either:
62
63- Directly invoke `soil/worker.sh` for a raw VM job.
64- Invoke wrappers `soil/github-actions.sh` / `soil/sourcehut.sh`, which in turn
65 uses `soil/host-shim.sh` to run a job in a container.
66 - They wrappers also publish via SSH with `soil/web-worker.sh`
67
68`soil/host-shim.sh` pulls and start an OCI container each node, and then runs
69`soil/worker.sh` inside the container.
70
71`soil/worker.sh` runs the job, and publishes 3 files to `travis-ci.oilshell.org`:
72
731. JSON metadata about the commit and build environment
741. TSV metadata for each "toil" step
751. A `.wwz` file (servable zip file) of logs
76
77`soil/web.sh` is a wrapper around `soil/web.py`, and it runs on the SERVER
78(`travis-ci.oilshell.org`). It reads and joins metadata from every job to
79construct `index.html` and `$RUN_NUMBER/index.html`.
80
81The server-side components are deployed by `soil/web-init.sh`.
82
83## Terminology
84
85Three level hierarchy of executables:
86
87- Run - A group of jobs run per commit, usually on different machines.
88 Github has `GITHUB_RUN_{NUMBER,ID}`, but sourcehut lacks this concept.
89- Job - A group of tasks run on the same machine, in the same container, or on a raw VM.
90- Task - A shell command that's run within the Oil repo. Tasks are currently
91 run sequentially.
92
93TODO:
94
95- Tasks need explicit dependencies
96- Dynamically schedule tasks, and remove the idea of a job. It should just be
97 a flat list of tasks run per commit, on various machines, with
98 dynamically-scheduled and resolved dependencies.
99
100## Events
101
102- End Job.
103 - First upload wwz, tsv, and JSON last
104 - web.py can rewrites the following files in one pass, based on list-json
105 state
106 - github-jobs/tmp-$$.{index,raw}.html - shell script does mv
107 - github-jobs/commits/tmp-$$.$HASH.html - shell script does mv
108 - this is based on github-jobs/$RUN/*.tsv -- similar to format-wwz-index
109 - or srht-jobs/*/*.tsv and filtered by commit
110 - github-jobs/tmp-$$.remove.txt - shell script does rm
111 - status-api/github-jobs/$RUN/$job -- PUT this
112
113- Start job. TODO: This doesn't exist.
114 - github-jobs/index.html should show Started / Pass / Fail
115
116## Web Interface / Security
117
118This one is a pure uploader, which you can borrow from picdir.
119
120 POST https://test.oils-for-unix.org/results
121 ?jobName=dev-minimal
122 &workerHost=github
123
124It does multi-part file upload of wwz, TSV, JSON, and saves the files. Start
125with basic auth over HTTPS? See picdir.
126
127
128 POST https://test.oils-for-unix.org/event
129
130 ?eventName=start
131 &workerHost=github
132 &runId=1234
133 &jobName=dev-minimal
134
135 ?eventName=start
136 &workerHost=sourcehut
137 &jobId=345
138 &jobName=dev-minimal
139
140 # whether to update the status-api
141 ?eventName=done
142 &workerHost=github
143 &runId=1234
144 &jobName=dev-minimal
145 &status=124
146 &updateStatusApi=1
147
148This PHP script can just run a shell script synchronously? Events start and
149done are meant to be quick.
150
151Other events:
152
153- begin pulling container
154- done pulling container
155- done with task, although we want to get rid of this abstraction
156
157
158## Notes
159
160### Idea For Tuple Space?
161
162Listening to `oilshell/oil`:
163
164- `dev-minimal`
165 - Can a contributor quickly get started with the Oil repo?
166 - They just want to run bin/osh without installing much
167- `ovm-tarball`
168- `cpp`
169- `other-tests`
170
171Listening to `oilshell/tarballs`:
172
173- Travis
174 - OS X (bin-darwin)
175- Sourcehut
176 - Alpine (bin-alpine)
177 - OpenBSD (bin-openbsd)
178 - Alternate architectures like ARM
179
180All of these build both oil.ovm and oil-native. Need maintainers. We build
181them as a "start".
182
183It would nice to implement this as SQL.
184
185
186