| 1 | `deps/` Directory
|
| 2 | =================
|
| 3 |
|
| 4 | Scripts to build various dependencies. They can used on the host machine, or
|
| 5 | in a Dockerfile to build a container. The container is used in our CI build,
|
| 6 | and can also be used locally.
|
| 7 |
|
| 8 | ## Docker Files
|
| 9 |
|
| 10 | Our images:
|
| 11 |
|
| 12 | - <https://hub.docker.com/u/oilshell>
|
| 13 |
|
| 14 | Base image maintained by:
|
| 15 |
|
| 16 | - <https://hub.docker.com/_/debian>
|
| 17 | - <https://github.com/debuerreotype/debuerreotype>
|
| 18 |
|
| 19 | Note: we could use names like `buster-20220912-slim` instead of `buster-slim`.
|
| 20 |
|
| 21 | - How can we get notified when the image is updated?
|
| 22 | - <https://crazymax.dev/diun/> has many notification mechanisms
|
| 23 |
|
| 24 |
|
| 25 | ## Container Dir Structure
|
| 26 |
|
| 27 | The file system in our images looks like this:
|
| 28 |
|
| 29 | /home/uke/
|
| 30 | tmp/ # Dockerfiles copy build scripts here
|
| 31 | oil/ # soil/host-shim.sh mounts the repo here
|
| 32 | _tmp/ # published as .wwz files to CI
|
| 33 | oil_DEPS/ # Built into containers
|
| 34 | cpython-full/ # build of Python
|
| 35 | py3/ # like cpython-full
|
| 36 |
|
| 37 | mycpp-venv/ # installed by python3 -m pip
|
| 38 | mypy/ # git repo
|
| 39 |
|
| 40 | spec-bin/
|
| 41 | wild/ # Pure data
|
| 42 |
|
| 43 | cmark/
|
| 44 |
|
| 45 | libcmark.so # shared library
|
| 46 | python3 # Plain executable
|
| 47 |
|
| 48 | Note that `build/dev-shell.sh` puts some executables in `$PATH`.
|
| 49 |
|
| 50 | TODO:
|
| 51 |
|
| 52 | - We should `--mount type=bind` "wedges" into say `/wedge/foo`
|
| 53 | - These can be either data or executable tools
|
| 54 | - I suppose `/home/uke/{oil,oil_DEPS}` are also wedges? They are meant not to conflict
|
| 55 | - But we want wedges to work OUTSIDE containers too.
|
| 56 | - We should also be able to COPY wedges into an image, for easier deployment on
|
| 57 | cloud services like Github Actions and fly.io
|
| 58 | - Then run an arbitrary shell command that uses the code and data
|
| 59 | - Outputs can go back into `/wedge`
|
| 60 | - Problems: absolute paths, root, `make install`, etc.
|
| 61 |
|
| 62 | Dir structure:
|
| 63 |
|
| 64 | /wedge/ # or you can build your own /home/andy/wedge
|
| 65 | oilshell.org/ # built for this distro, e.g. ubuntu-18.04
|
| 66 | DEPS/
|
| 67 | Python-2.7.13/
|
| 68 | Python-3.10.4/
|
| 69 | bash-4.4/
|
| 70 | bash-5.1/
|
| 71 |
|
| 72 | And then we'll need some symlinks like `python3`, etc.
|
| 73 |
|
| 74 | ## Kinds of Tarballs / "Dir Values"
|
| 75 |
|
| 76 | - Source vs. Intermediate - what repo stores it?
|
| 77 | - Absolute vs. Relative - where is it mounted?
|
| 78 | - Layer vs. Wedge - how is it mounted?
|
| 79 |
|
| 80 | ## Soil C++ Tarball Notes
|
| 81 |
|
| 82 | cpp-tarball is consumed by these jobs now:
|
| 83 |
|
| 84 | - raw-vm
|
| 85 | - wild
|
| 86 | - app-tests for ble.sh
|
| 87 |
|
| 88 | If we use it in more tasks, we could try to remove MyPy/Python 3 from Docker
|
| 89 | images. It should especially speed up the pull on sourcehut -- 30-40 seconds
|
| 90 | for pea/other-tests, vs. 2:13 for benchmarks2
|
| 91 |
|
| 92 | So these can also consume the tarball:
|
| 93 |
|
| 94 | - benchmarks
|
| 95 | - benchmarks2
|
| 96 | - interactive uses the 'benchmarks' image for some reason
|
| 97 | - cpp-spec
|
| 98 | - build ASAN binary from tarball
|
| 99 | - needs both osh and ysh
|
| 100 | - cpp-coverage -- would need to include C++ unit tests and Ninja in tarball,
|
| 101 | which is not a bad idea
|
| 102 |
|
| 103 | More:
|
| 104 |
|
| 105 | - Alpine testing
|
| 106 | - Smoosh testing with OCaml, similar to spec tests
|
| 107 |
|