Why Sponsor Oils? | source | all docs for version 0.22.0 | all versions | oilshell.org
This doc describes the YSH language from clean slate perspective. We don't assume you know Unix shell, or the compatible OSH. But shell users will see the similarity, with simplifications and upgrades.
Remember, YSH is for Python and JavaScript users who avoid shell! See the project FAQ for more color on that.
This document is long because it demonstrates nearly every feature of the language. You may want to read it in multiple sittings, or read The Simplest Explanation of Oil first. (Until 2023, YSH was called the "Oil language".)
Here's a summary of what follows:
cd /tmp, and builtin functions like
join().Keep these points in mind as you read the details below.
Start YSH just like you start bash or Python:
bash$ ysh # assuming it's installed
ysh$ echo 'hello world' # command typed into YSH
hello world
In the sections below, we'll save space by showing output in comments, with
=>:
echo 'hello world' # => hello world
Multi-line output is shown like this:
echo one
echo two
# =>
# one
# two
You can also type commands into a file like hello.ysh. This is a complete
YSH program, which is identical to a shell program:
echo 'hello world' # => hello world
Unlike shell, YSH has var and const keywords:
const name = 'world' # const is rarer, used the top-level
echo "hello $name" # => hello world
They take rich Python-like expressions on the right:
var x = 42 # an integer, not a string
setvar x = x * 2 + 1 # mutate with the 'setvar' keyword
setvar x += 5 # Increment by 5
echo $x # => 6
var mylist = [x, 7] # two integers [6, 7]
Expressions are often surrounded by ():
if (x > 0) {
echo 'positive'
} # => positive
for i, item in (mylist) { # 'mylist' is a variable, not a string
echo "[$i] item $item"
}
# =>
# [0] item 6
# [1] item 7
YSH has Ruby-like blocks:
cd /tmp {
echo hi > greeting.txt # file created inside /tmp
echo $PWD # => /tmp
}
echo $PWD # prints the original directory
And utilities to read and write JSON:
var person = {name: 'bob', age: 42}
json write (person)
# =>
# {
# "name": "bob",
# "age": 42,
# }
echo '["str", 42]' | json read # sets '_reply' variable by default
The = keyword evaluates and prints an expression:
= _reply
# => (List) ["str", 42]
(Think of it like var x = _reply, without the var.)
Let's describe the word language first, and then talk about commands and expressions. Words are a rich language because strings are a central concept in shell.
You can choose the quoting style that's most convenient to write a given string.
Double-quoted strings allow interpolation with $:
var person = 'alice'
echo "hi $person, $(echo bye)" # => hi alice, bye
Write operators by escaping them with \:
echo "\$ \" \\ " # => $ " \
In single-quoted strings, all characters are literal (except ', which
can't be expressed):
echo 'c:\Program Files\' # => c:\Program Files\
If you want C-style backslash character escapes, use a J8 string, which is like JSON, but with single quotes::
echo u' A is \u{41} \n line two, with backslash \\'
# =>
# A is A
# line two, with backslash \
The u'' strings are guaranteed to be valid Unicode (unlike JSON), but you can
also use b'' strings:
echo b'byte \yff' # byte that's not valid unicode, like \xff in other languages
# do not confuse with \u{ff}
Multi-line strings are surrounded with triple quotes. They come in the same three varieties, and leading whitespace is stripped in a convenient way.
sort <<< """
var sub: $x
command sub: $(echo hi)
expression sub: $[x + 3]
"""
# =>
# command sub: hi
# expression sub: 9
# var sub: 6
sort <<< '''
$2.00 # literal $, no interpolation
$1.99
'''
# =>
# $1.99
# $2.00
sort <<< u'''
C\tD
A\tB
''' # b''' strings also supported
# =>
# A B
# C D
(Use multiline strings instead of shell's here docs.)
YSH has syntax for 3 types of substitution, all of which start with $. These
things can all be converted to a string:
The syntax $a or ${a} converts a variable to a string:
var a = 'ale'
echo $a # => ale
echo _${a}_ # => _ale_
echo "_ $a _" # => _ ale _
The shell operator :- is occasionally useful in YSH:
echo ${not_defined:-'default'} # => default
The $(echo hi) syntax runs a command and captures its stdout:
echo $(hostname) # => example.com
echo "_ $(hostname) _" # => _ example.com _
The $[myexpr] syntax evaluates an expression and converts it to a string:
echo $[a] # => ale
echo $[1 + 2 * 3] # => 7
echo "_ $[1 + 2 * 3] _" # => _ 7 _
There are four constructs that evaluate to an list of strings, rather than a single string.
Globs like *.py evaluate to a list of files.
touch foo.py bar.py # create the files
write *.py
# =>
# foo.py
# bar.py
If no files match, it evaluates to an empty list ([]).
The brace expansion mini-language lets you write strings without duplication:
write {alice,bob}@example.com
# =>
# alice@example.com
# bob@example.com
The @ operator splices an array into a command:
var myarray = :| ale bean |
write S @myarray E
# =>
# S
# ale
# bean
# E
You also have @[] to splice an expression that evaluates to a list:
write -- @[split('ale bean')]
# =>
# ale
# bean
Each item will be converted to a string.
There's also a variant of command sub that splits first:
write @(seq 3) # write gets 3 arguments
# =>
# 1
# 2
# 3
A simple command is a space-separated list of words, which are often unquoted.
YSH looks up the first word to determine if it's a proc or shell builtin.
echo 'hello world' # The shell builtin 'echo'
proc greet (name) { # A proc is like a procedure or process
echo "hello $name"
}
# Now the first word will resolve to the proc
greet alice # => hello alice
If it's neither, then it's assumed to be an external command:
ls -l /tmp # The external 'ls' command
Commands accept traditional string arguments, as well as typed arguments in parentheses:
# 'write' is a string arg; 'x' is a typed expression arg
json write (x)
You can redirect stdin and stdout of simple commands:
echo hi > tmp.txt # write to a file
sort < tmp.txt
Idioms for using stderr (identical to shell):
ls /tmp 2>errors.txt
echo 'fatal error' 1>&2
"Simple" commands in YSH can also have typed () and block {} args, which
we'll see in the section on "procs".
Pipelines are a powerful method manipulating data streams:
ls | wc -l # count files in this directory
find /bin -type f | xargs wc -l # count files in a subtree
The stream may contain (lines of) text, binary data, JSON, TSV, and more. Details below.
The YSH ... prefix lets you write long commands, pipelines, and && chains
without \ line continuations.
... find /bin # traverse this directory and
-type f -a -executable # print executable files
| sort -r # reverse sort
| head -n 30 # limit to 30 files
;
When this mode is active:
;
terminator.var, setvar, const to Declare and MutateConstants can't be modified:
const myconst = 'mystr'
# setvar myconst = 'foo' would be an error
Modify variables with the setvar keyword:
var num_beans = 12
setvar num_beans = 13
A more complex example:
var d = {name: 'bob', age: 42} # dict literal
setvar d.name = 'alice' # d.name is a synonym for d['name']
echo $[d.name] # => alice
That's most of what you need to know about assignments. Advanced users may
want to use setglobal or call myplace->setValue(42) in certain situations.
More details: Variable Declaration and Mutation.
for LoopShell-style for loops iterate over words:
for word in 'oils' $num_beans {pea,coco}nut {
echo $word
}
# =>
# oils
# 13
# peanut
# coconut
You can also request the loop index:
for i, word in README.md *.py {
echo "$i - $word"
}
# =>
# 0 - README.md
# 1 - __init__.py
To iterate over lines of stdin, use:
for line in (io.stdin) {
echo $line
}
# lines are buffered, so it's much faster than `while read --rawline`
Ask for the loop index:
for i, line in (io.stdin) {
echo "$i $line"
}
To iterate over a typed data, use parentheses around an expression. The
expression should evaluate to an integer Range, List, or Dict:
for i in (3 .. 5) { # range operator ..
echo "i = $i"
}
# =>
# i = 3
# i = 4
List:
var foods = ['ale', 'bean']
for item in (foods) {
echo $item
}
# =>
# ale
# bean
Again, you can request the index with for i, item in ....
Here's the most general form of the loop over Dict:
var mydict = {pea: 42, nut: 10}
for i, k, v in (mydict) {
echo "$i - $k - $v"
}
# =>
# 0 - pea - 42
# 1 - nut - 10
There are two simpler forms:
for k in (mydict)for k, v in (mydict)(One way to think of it: for loops in YSH have the functionality Python's
enumerate(), items(), keys(), and values().)
while LoopWhile loops can use a command as the termination condition:
while test --file lock {
sleep 1
}
Or an expression, which is surrounded in ():
var i = 3
while (i < 6) {
echo "i = $i"
setvar i += 1
}
# =>
# i = 3
# i = 4
# i = 5
if elif ConditionalIf statements test the exit code of a command, and have optional elif and
else clauses:
if test --file foo {
echo 'foo is a file'
rm --verbose foo # delete it
} elif test --dir foo {
echo 'foo is a directory'
} else {
echo 'neither'
}
Invert the exit code with !:
if ! grep alice /etc/passwd {
echo 'alice is not a user'
}
As with while loops, the condition can also be an expression wrapped in
():
if (num_beans > 0) {
echo 'so many beans'
}
var done = false
if (not done) { # negate with 'not' operator (contrast with !)
echo "we aren't done"
}
case ConditionalThe case statement is a series of conditionals and executable blocks. The
condition can be either an unquoted glob pattern like *.py, an eggex pattern
like /d+/, or a typed expression like (42):
var s = 'README.md'
case (s) {
*.py { echo 'Python' }
*.cc | *.h { echo 'C++' }
* { echo 'Other' }
}
# => Other
case (s) {
/ dot* '.md' / { echo 'Markdown' }
(30 + 12) { echo 'the integer 42' }
(else) { echo 'neither' }
}
# => Markdown
(Shell style like if foo; then ... fi and case $x in ... esac is also legal,
but discouraged in YSH code.)
If statements are also used for error handling. Builtins and external commands use this style:
if ! test -d /bin {
echo 'not a directory'
}
if ! cp foo /tmp {
echo 'error copying' # any non-zero status
}
Procs use this style (because of shell's disabled errexit quirk):
try {
myproc
}
if failed {
echo 'failed'
}
For a complete list of examples, see YSH Error Handling. For design goals and a reference, see YSH Fixes Shell's Error Handling.
break, continue, return, exitThe exit keyword exits a process (it's not a shell builtin.) The other 3
control flow keywords behave like they do in Python and JavaScript.
Here's a builtin command that takes a literal block argument:
shopt --unset errexit { # ignore errors
cp ale /tmp
cp bean /bin
}
Blocks are a special kind of typed argument passed to commands like shopt.
Their type is value.Command.
procYou can define units of code with the proc keyword.
proc mycopy (src, dest) {
### Copy verbosely
mkdir -p $dest
cp --verbose $src $dest
}
The ### line is a "doc comment", and can be retrieved with pp proc. Simple
procs like this are invoked like a shell command:
touch log.txt
mycopy log.txt /tmp # first word 'mycopy' is a proc
Procs have more features, including four kinds of arguments:
{ }.At the call site, they can look like any of these forms:
cd /tmp # word arg
json write (d) # word arg, then positional arg
# error 'failed' (status=9) # word arg, then named arg
cd /tmp { echo $PWD } # word arg, then block arg
var mycmd = ^(echo hi) # expression for a value.Command
eval (mycmd) # positional arg
At the definition site, the kinds of parameters are separated with ;, similar
to the Julia language:
proc p2 (word1, word2; pos1, pos2, ...rest_pos) {
echo "$word1 $word2 $[pos1 + pos2]"
json write (rest_pos)
}
proc p3 (w ; ; named1, named2, ...rest_named; block) {
echo "$w $[named1 + named2]"
eval (block)
json write (rest_named)
}
proc p4 (; ; ; block) {
eval (block)
}
YSH also has Python-like functions defined with func. These are part of the
expression language, which we'll see later.
For more info, see the Informal Guide to Procs and Funcs (under construction).
Shell builtins like cd and read are the "standard library" of the
command language. Each one takes various flags:
cd -L . # follow symlinks
echo foo | read --all # read all of stdin
Here are some categories of builtin:
echo write readcd testfork wait forkwait execshopt shvarcommand builtin runproc type evalYSH expressions look and behave more like Python or JavaScript than shell. For
example, we write if (x < y) instead of if [ $x -lt $y ]. Expressions are
usually surrounded by ( ).
At runtime, variables like x and y are bounded to typed data, like
integers, floats, strings, lists, and dicts.
funcAt the end of the Command Language, we saw that procs are shell-like units of
code. Now let's talk about Python-like functions in YSH, which are
different than procs:
func keyword.Here's a function that mutates its argument:
func popTwice(mylist) {
call mylist->pop()
call mylist->pop()
}
var mylist = [3, 4]
# The call keyword is an "adapter" between commands and expressions,
# like the = keyword.
call popTwice(mylist)
Here's a pure function:
func myRepeat(s, n; special=false) { # positional; named params
var parts = []
for i in (0 .. n) {
append $s (parts)
}
var result = join(parts)
if (special) {
return ("$result !!") # parens required for typed return
} else {
return (result)
}
}
echo $[myRepeat('z', 3)] # => zzz
echo $[myRepeat('z', 3, special=true)] # => zzz !!
Funcs are named using camelCase, while procs use kebab-case. See the
Style Guide for more conventions.
In addition, to builtin commands, YSH has Python-like builtin functions. These are like the "standard library" for the expression language. Examples:
len() type()bool() int() float() str() list() ...split() join() glob() maybe()Int, Str, List, Dict, ...YSH has data types, each with an expression syntax and associated methods.
Mutating methods are looked up with a thin arrow ->:
var foods = ['ale', 'bean']
var last = foods->pop() # bean
write @foods # => ale
You can ignore the return value with the call keyword:
call foods->pop()
Transforming methods use a fat arrow =>:
var line = ' ale bean '
var trimmed = line => trim() => upper() # 'ALE BEAN'
If the => operator doesn't find a method with the given name in the object's
type, it looks for free functions:
# list() is a free function taking one arg
# join() is a free function taking two args
var x = {k1: 42, k2: 43} => list() => join('/') # 'K1/K2'
This allows a left-to-right "method chaining" style.
Now let's go through the data types in YSH. We'll show the syntax for literals, and what methods they have.
YSH uses JavaScript-like spellings these three "atoms":
var x = null
var b1, b2 = true, false
if (b1) {
echo 'yes'
} # => yes
There are many ways to write integers:
var small, big = 42, 65_536
echo "$small $big" # => 42 65536
var hex, octal, binary = 0x0001_0000, 0o755, 0b0001_0101
echo "$hex $octal $binary" # => 65536 493 21
Floats are written like you'd expect:
var small = 1.5e-10
var big = 3.14
See the section above called Three Kinds of String Literals. It described
'single quoted', "double ${quoted}", and u'J8-style\n' strings; as well
as their multiline variants.
Strings are UTF-8 encoded in memory, like strings in the Go language. There isn't a separate string and unicode type, as in Python.
Strings are immutable, as in Python and JavaScript. This means they only have transforming methods:
var x = s => trim()
Other methods:
trimLeft() trimRight()trimPrefix() trimSuffix()upper() lower() (not implemented)All lists can be expressed with Python-like literals:
var foods = ['ale', 'bean', 'corn']
var recursive = [1, [2, 3]]
As a special case, list of strings are called arrays. It's often more convenient to write them with shell-like literals:
# No quotes or commas
var foods = :| ale bean corn |
# You can use the word language here
var other = :| foo $s *.py {alice,bob}@example.com |
Lists are mutable, as in Python and JavaScript. So they mainly have mutating methods:
call foods->reverse()
write -- @foods
# =>
# corn
# bean
# ale
Dicts use syntax that's more like JavaScript than Python. Here's a dict literal:
var d = {
name: 'bob', # unquoted keys are allowed
age: 42,
'key with spaces': 'val'
}
There are two syntaxes for key lookup. If the key doesn't exist, it's a fatal error.
var v1 = d['name']
var v2 = d.name # shorthand for the above
var v3 = d['key with spaces'] # no shorthand for this
Keys names can be computed with expressions in []:
var key = 'alice'
var d2 = {[key ++ '_z']: 'ZZZ'} # Computed key name
echo $[d2.alice_z] # => ZZZ # Reminder: expression sub
Omitting the value causes it to be taken from a variable of the same name:
var d3 = {key} # value is taken from the environment
echo "name is $[d3.key]" # => name is alice
More:
var empty = {}
echo $[len(empty)] # => 0
Dicts are mutable, as in Python and JavaScript. But the keys() and values()
methods return new List objects:
var keys = d2 => keys() # => alice_z
# var vals = d3 => values() # => alice
Place type / "out params"The read builtin can either set an implicit variable _reply:
whoami | read --all # sets _reply
Or you can pass a value.Place, created with &
var x # implicitly initialized to null
whoami | read --all (&x) # mutate this "place"
echo who=$x # => who=andy
These types are for reflection on YSH code. Most YSH programs won't use them directly.
Command: an unevaluated code block.
^(ls | wc -l)Expr: an unevaluated expression.
^[42 + a[i]]Operators are generally the same as in Python:
if (10 <= num_beans and num_beans < 20) {
echo 'enough'
} # => enough
YSH has a few operators that aren't in Python. Equality can be approximate or exact:
var n = ' 42 '
if (n ~== 42) {
echo 'equal after stripping whitespace and type conversion'
} # => equal after stripping whitespace type conversion
if (n === 42) {
echo "not reached because strings and ints aren't equal"
}
Pattern matching can be done with globs (~~ and !~~)
const filename = 'foo.py'
if (filename ~~ '*.py') {
echo 'Python'
} # => Python
if (filename !~~ '*.sh') {
echo 'not shell'
} # => not shell
or regular expressions (~ and !~). See the Eggex section below for an
example of the latter.
Concatenation is ++ rather than + because it avoids confusion in the
presence of type conversion:
var n = 42 + 1 # string plus int does implicit conversion
echo $n # => 43
var y = 'ale ' ++ "bean $n" # concatenation
echo $y # => ale bean 43
An Eggex is a type of YSH expression that denote regular expressions. They
translate to POSIX ERE syntax, for use with tools like egrep, awk, and sed --regexp-extended (GNU only).
They're designed to be readable and composable. Example:
var D = / digit{1,3} /
var ip_pattern = / D '.' D '.' D '.' D'.' /
var z = '192.168.0.1'
if (z ~ ip_pattern) { # Use the ~ operator to match
echo "$z looks like an IP address"
} # => 192.168.0.1 looks like an IP address
if (z !~ / '.255' %end /) {
echo "doesn't end with .255"
} # => doesn't end with .255"
See the Egg Expressions doc for details.
Let's review what we've seen before moving onto other YSH features.
Here are the languages we saw in the last 3 sections:
'mystr'${x} and $(hostname)*.shreadif, forproc['ale', 'bean'] or :| ale bean |{name: 'bob', age: 42}split('ale bean') and join(['pea', 'nut'])Here are two examples:
(1) In this this command, there are four words. The fourth word is an
expression sub $[].
write hello $name $[d['age'] + 1]
# =>
# hello
# world
# 43
(2) In this assignment, the expression on the right hand side of =
concatenates two strings. The first string is a literal, and the second is a
command sub.
var food = 'ale ' ++ $(echo bean | tr a-z A-Z)
write $food # => ale BEAN
So words, commands, and expressions are mutually recursive. If you're a conceptual person, skimming Syntactic Concepts may help you understand this on a deeper level.
In addition to languages for code, YSH also deals with languages for data. JSON is a prominent example of the latter.
UTF-8 is the foundation of our textual data languages.
Traditional Unix tools like grep and awk operate on streams of lines. YSH
supports this style, just like any other shell.
But YSH also has J8 Notation, a data format based on JSON.
It lets you encode arbitrary byte strings into a single (readable) line, including those with newlines and terminal escape sequences.
Example:
# A line with a tab char in the middle
var mystr = u'pea\t' ++ u'42\n'
# Print it as JSON
write $[toJson(mystr)] # => "pea\t42\n"
# JSON8 is the same, but it's not lossy for binary data
write $[toJson8(mystr)] # => "pea\t42\n"
You can write and read tree-shaped data as JSON:
var d = {key: 'value'}
json write (d) # dump variable d as JSON
# =>
# {
# "key": "value"
# }
echo '["ale", 42]' > example.json
json read (&d2) < example.json # parse JSON into var d2
pp (d2) # pretty print it
# => (List) ['ale', 42]
JSON will lose information when strings have binary data, but the slight JSON8 upgrade won't:
var b = {binary: $'\xff'}
json8 write (b)
# =>
# {
# "binary": b'\yff'
# }
Table-shaped data can be read and written as TSV8. (TODO: not yet implemented.)
Although we describe OSH and YSH as different languages, they use the same
interpreter under the hood. This interpreter has various shopt flags that
are flipped for different behavior, e.g. with shopt --set ysh:all.
Understanding this interpreter and its interface to the Unix kernel will help you understand both languages!
The Interpreter State doc is under construction. It will cover:
{name -> cell} mapping.Bool, Int, Str, etc.readonly, export, and nameref flags.shopt: parse_paren, simple_word_eval, etc.shvar: IFS, PATH$? and _error$! for the last PID_this_dir_replyThe Process Model doc is under construction. It will cover:
execfork, forkwaitYSH is a large language that evolved from Unix shell. It has shell-like commands, Python-like expressions on typed data, and Ruby-like command blocks.
Even though it's large, you can "forget" the bad parts of shell like [ $x -lt $y ].
These concepts are central to YSH:
These shell features are part of YSH, but aren't shown for brevity.
fork and forkwait builtins, for concurrent execution and subshells.diff <(sort left.txt) <(sort right.txt)The shared interpreter supports many shell constructs that are deprecated:
|| and && in limited circumstances, since errexit
is on by default.local and declare. Use YSH keywords.[[ x =~ $pat ]]. Use YSH expressions.$(( x + 1 )) and (( y = x )). Use YSH expressions.until loop can always be replaced with a while loop${} can be written in other ways. For example
${s#/tmp} could be s => removePrefix('/tmp') (TODO).This document mentions a few constructs that aren't yet implemented. Here's a summary:
# Unimplemented syntax:
echo ${x|html} # formatters
echo ${x %.2f} # statically-parsed printf
var x = j"line\n"
echo j"line\n" # JSON-style string literal
var x = "<p>$x</p>"html
echo "<p>$x</p>"html # tagged string
var x = 15 Mi # units suffix
Important builtins that aren't implemented:
describe for testingparseArgs() to parse flagsYSH can be used to write simple "shell scripts" or longer programs. It has procs and modules to help with the latter.
A module is just a file, like this:
#!/usr/bin/env ysh
### Deploy script
source-guard main || return 0 # declaration, "include guard"
source $_this_dir/lib/util.ysh # defines 'log' helper
const DEST = '/tmp/ysh-tour'
proc my-sync(...files) {
### Sync files and show which ones
cp --verbose @files $DEST
}
proc main {
mkdir -p $DEST
touch {foo,bar}.py {build,test}.sh
log "Copying source files"
my-sync *.py *.sh
if test --dir /tmp/logs {
cd /tmp/logs
log "Copying logs"
my-sync *.log
}
}
if is-main { # The only top-level statement
main @ARGV
}
You wouldn't bother with the boilerplate for something this small. But this
example illustrates the idea, which is that the top level often contains these
words: proc, const, module, source, and use.