OILS / doc / idioms.md View on Github | oilshell.org

949 lines, 602 significant
1---
2default_highlighter: oils-sh
3---
4
5YSH vs. Shell Idioms
6====================
7
8This is an informal, lightly-organized list of recommended idioms for the
9[YSH]($xref) language. Each section has snippets labeled *No* and *Yes*.
10
11- Use the *Yes* style when you want to write in YSH, and don't care about
12 compatibility with other shells.
13- The *No* style is discouraged in new code, but YSH will run it. The [OSH
14 language]($xref:osh-language) is compatible with
15 [POSIX]($xref:posix-shell-spec) and [bash]($xref).
16
17[J8 Notation]: j8-notation.html
18
19<!-- cmark.py expands this -->
20<div id="toc">
21</div>
22
23## Use [Simple Word Evaluation](simple-word-eval.html) to Avoid "Quoting Hell"
24
25### Substitute Variables
26
27No:
28
29 local x='my song.mp3'
30 ls "$x" # quotes required to avoid mangling
31
32Yes:
33
34 var x = 'my song.mp3'
35 ls $x # no quotes needed
36
37### Splice Arrays
38
39No:
40
41 local myflags=( --all --long )
42 ls "${myflags[@]}" "$@"
43
44Yes:
45
46 var myflags = :| --all --long |
47 ls @myflags @ARGV
48
49### Explicitly Split, Glob, and Omit Empty Args
50
51YSH doesn't split arguments after variable expansion.
52
53No:
54
55 local packages='python-dev gawk'
56 apt install $packages
57
58Yes:
59
60 var packages = 'python-dev gawk'
61 apt install @[split(packages)]
62
63Even better:
64
65 var packages = :| python-dev gawk | # array literal
66 apt install @packages # splice array
67
68---
69
70YSH doesn't glob after variable expansion.
71
72No:
73
74 local pat='*.py'
75 echo $pat
76
77
78Yes:
79
80 var pat = '*.py'
81 echo @[glob(pat)] # explicit call
82
83---
84
85YSH doesn't omit unquoted words that evaluate to the empty string.
86
87No:
88
89 local e=''
90 cp $e other $dest # cp gets 2 args, not 3, in sh
91
92Yes:
93
94 var e = ''
95 cp @[maybe(e)] other $dest # explicit call
96
97### Iterate a Number of Times (Split Command Sub)
98
99No:
100
101 local n=3
102 for x in $(seq $n); do # No implicit splitting of unquoted words in YSH
103 echo $x
104 done
105
106OK:
107
108 var n = 3
109 for x in @(seq $n) { # Explicit splitting
110 echo $x
111 }
112
113Better;
114
115 var n = 3
116 for x in (1 .. n+1) { # Range, avoids external program
117 echo $x
118 }
119
120Note that `{1..3}` works in bash and YSH, but the numbers must be constant.
121
122## Avoid Ad Hoc Parsing and Splitting
123
124In other words, avoid *groveling through backslashes and spaces* in shell.
125
126Instead, emit and consume [J8 Notation]($xref:j8-notation):
127
128- J8 strings are [JSON]($xref) strings, with an upgrade for byte string
129 literals
130- [JSON8]($xref) is [JSON]($xref), with this same upgrade
131- [TSV8]($xref) is TSV with this upgrade (not yet implemented)
132
133Custom parsing and serializing should be limited to "the edges" of your YSH
134programs.
135
136### More Strategies For Structured Data
137
138- **Wrap** and Adapt External Tools. Parse their output, and emit [J8 Notation][].
139 - These can be one-off, "bespoke" wrappers in your program, or maintained
140 programs. Use the `proc` construct and `flagspec`!
141 - Example: [uxy](https://github.com/sustrik/uxy) wrappers.
142 - TODO: Examples written in YSH and in other languages.
143- **Patch** Existing Tools.
144 - Enhance GNU grep, etc. to emit [J8 Notation][]. Add a
145 `--j8` flag.
146- **Write Your Own** Structured Versions.
147 - For example, you can write a structured subset of `ls` in Python with
148 little effort.
149
150<!--
151 ls -q and -Q already exist, but --j8 or --tsv8 is probably fine
152-->
153
154## The `write` Builtin Is Simpler Than `printf` and `echo`
155
156### Write an Arbitrary Line
157
158No:
159
160 printf '%s\n' "$mystr"
161
162Yes:
163
164 write -- $mystr
165
166The `write` builtin accepts `--` so it doesn't confuse flags and args.
167
168### Write Without a Newline
169
170No:
171
172 echo -n "$mystr" # breaks if mystr is -e
173
174Yes:
175
176 write --end '' -- $mystr
177 write -n -- $mystr # -n is an alias for --end ''
178
179### Write an Array of Lines
180
181 var myarray = :| one two three |
182 write -- @myarray
183
184## New Long Flags on the `read` builtin
185
186### Read a Line
187
188No:
189
190 read line # Bad because it mangles your backslashes!
191
192For now, please use this bash idiom to read a single line:
193
194 read -r line # Easy to forget -r for "raw"
195
196YSH used to have `read --line`, but there was a design problem: reading
197buffered lines doesn't mix well with reading directly from file descriptors,
198and shell does the latter.
199
200That is, `read -r` is suboptimal because it makes many syscalls, but it's
201already established in shell.
202
203### Read a Whole File
204
205No:
206
207 read -d '' # harder to read, easy to forget -r
208
209Yes:
210
211 read --all # sets $_reply
212 read --all (&myvar) # sets $myvar
213
214### Read Until `\0` (consume `find -print0`)
215
216No:
217
218 # Obscure syntax that bash accepts, but not other shells
219 read -r -d '' myvar
220
221Yes:
222
223 read -0 (&myvar)
224
225## YSH Enhancements to Builtins
226
227### Use `shopt` Instead of `set`
228
229Using a single builtin for all options makes scripts easier to read:
230
231Discouraged:
232
233 set -o errexit
234 shopt -s dotglob
235
236Idiomatic:
237
238 shopt --set errexit
239 shopt --set dotglob
240
241(As always, `set` can be used when you care about compatibility with other
242shells.)
243
244### Use `:` When Mentioning Variable Names
245
246YSH accepts this optional "pseudo-sigil" to make code more explicit.
247
248No:
249
250 read -0 record < file.bin
251 echo $record
252
253Yes:
254
255 read -0 (&myvar) < file.bin
256 echo $record
257
258
259### Consider Using `--long-flags`
260
261Easier to write:
262
263 test -d /tmp
264 test -d / && test -f /vmlinuz
265
266 shopt -u extglob
267
268Easier to read:
269
270 test --dir /tmp
271 test --dir / && test --file /vmlinuz
272
273 shopt --unset extglob
274
275## Use Blocks to Save and Restore Context
276
277### Do Something In Another Directory
278
279No:
280
281 ( cd /tmp; echo $PWD ) # subshell is unnecessary (and limited)
282
283No:
284
285 pushd /tmp
286 echo $PWD
287 popd
288
289Yes:
290
291 cd /tmp {
292 echo $PWD
293 }
294
295### Batch I/O
296
297No:
298
299 echo 1 > out.txt
300 echo 2 >> out.txt # appending is less efficient
301 # because open() and close()
302
303No:
304
305 { echo 1
306 echo 2
307 } > out.txt
308
309Yes:
310
311 fopen > out.txt {
312 echo 1
313 echo 2
314 }
315
316The `fopen` builtin is syntactic sugar -- it lets you see redirects before the
317code that uses them.
318
319### Temporarily Set Shell Options
320
321No:
322
323 set +o errexit
324 myfunc # without error checking
325 set -o errexit
326
327Yes:
328
329 shopt --unset errexit {
330 myfunc
331 }
332
333### Use the `forkwait` builtin for Subshells, not `()`
334
335No:
336
337 ( cd /tmp; rm *.sh )
338
339Yes:
340
341 forkwait {
342 cd /tmp
343 rm *.sh
344 }
345
346Better:
347
348 cd /tmp { # no process created
349 rm *.sh
350 }
351
352### Use the `fork` builtin for async, not `&`
353
354No:
355
356 myfunc &
357
358 { sleep 1; echo one; sleep 2; } &
359
360Yes:
361
362 fork { myfunc }
363
364 fork { sleep 1; echo one; sleep 2 }
365
366## Use Procs (Better Shell Functions)
367
368### Use Named Parameters Instead of `$1`, `$2`, ...
369
370No:
371
372 f() {
373 local src=$1
374 local dest=${2:-/tmp}
375
376 cp "$src" "$dest"
377 }
378
379Yes:
380
381 proc f(src, dest='/tmp') { # Python-like default values
382 cp $src $dest
383 }
384
385### Use Named Varargs Instead of `"$@"`
386
387No:
388
389 f() {
390 local first=$1
391 shift
392
393 echo $first
394 echo "$@"
395 }
396
397Yes:
398
399 proc f(first, @rest) { # @ means "the rest of the arguments"
400 write -- $first
401 write -- @rest # @ means "splice this array"
402 }
403
404You can also use the implicit `ARGV` variable:
405
406 proc p {
407 cp -- @ARGV /tmp
408 }
409
410### Use "Out Params" instead of `declare -n`
411
412Out params are one way to "return" values from a `proc`.
413
414No:
415
416 f() {
417 local in=$1
418 local -n out=$2
419
420 out=PREFIX-$in
421 }
422
423 myvar='init'
424 f zzz myvar # assigns myvar to 'PREFIX-zzz'
425
426
427Yes:
428
429 proc f(in, :out) { # : is an out param, i.e. a string "reference"
430 setref out = "PREFIX-$in"
431 }
432
433 var myvar = 'init'
434 f zzz :myvar # assigns myvar to 'PREFIX-zzz'.
435 # colon is required
436
437### Note: Procs Don't Mess With Their Callers
438
439That is, [dynamic scope]($xref:dynamic-scope) is turned off when procs are
440invoked.
441
442Here's an example of shell functions reading variables in their caller:
443
444 bar() {
445 echo $foo_var # looks up the stack
446 }
447
448 foo() {
449 foo_var=x
450 bar
451 }
452
453 foo
454
455In YSH, you have to pass params explicitly:
456
457 proc bar {
458 echo $foo_var # error, not defined
459 }
460
461Shell functions can also **mutate** variables in their caller! But procs can't
462do this, which makes code easier to reason about.
463
464## Use Modules
465
466YSH has a few lightweight features that make it easier to organize code into
467files. It doesn't have "namespaces".
468
469### Relative Imports
470
471Suppose we are running `bin/mytool`, and we want `BASE_DIR` to be the root of
472the repository so we can do a relative import of `lib/foo.sh`.
473
474No:
475
476 # All of these are common idioms, with caveats
477 BASE_DIR=$(dirname $0)/..
478
479 BASE_DIR=$(dirname ${BASH_SOURCE[0]})/..
480
481 BASE_DIR=$(cd $($dirname $0)/.. && pwd)
482
483 BASE_DIR=$(dirname (dirname $(readlink -f $0)))
484
485 source $BASE_DIR/lib/foo.sh
486
487Yes:
488
489 const BASE_DIR = "$this_dir/.."
490
491 source $BASE_DIR/lib/foo.sh
492
493 # Or simply:
494 source $_this_dir/../lib/foo.sh
495
496The value of `_this_dir` is the directory that contains the currently executing
497file.
498
499### Include Guards
500
501No:
502
503 # libfoo.sh
504 if test -z "$__LIBFOO_SH"; then
505 return
506 fi
507 __LIBFOO_SH=1
508
509Yes:
510
511 # libfoo.sh
512 module libfoo.sh || return 0
513
514### Taskfile Pattern
515
516No:
517
518 deploy() {
519 echo ...
520 }
521 "$@"
522
523Yes
524
525 proc deploy() {
526 echo ...
527 }
528 runproc @ARGV # gives better error messages
529
530## Error Handling
531
532[YSH Fixes Shell's Error Handling (`errexit`)](error-handling.html) once and
533for all! Here's a comprehensive list of error handling idioms.
534
535### Don't Use `&&` Outside of `if` / `while`
536
537It's implicit because `errexit` is on in YSH.
538
539No:
540
541 mkdir /tmp/dest && cp foo /tmp/dest
542
543Yes:
544
545 mkdir /tmp/dest
546 cp foo /tmp/dest
547
548It also avoids the *Trailing `&&` Pitfall* mentioned at the end of the [error
549handling](error-handling.html) doc.
550
551### Ignore an Error
552
553No:
554
555 ls /bad || true # OK because ls is external
556 myfunc || true # suffers from the "Disabled errexit Quirk"
557
558Yes:
559
560 try ls /bad
561 try myfunc
562
563### Retrieve A Command's Status When `errexit` is On
564
565No:
566
567 # set -e is enabled earlier
568
569 set +e
570 mycommand # this ignores errors when mycommand is a function
571 status=$? # save it before it changes
572 set -e
573
574 echo $status
575
576Yes:
577
578 try mycommand
579 echo $_status
580
581### Does a Builtin Or External Command Succeed?
582
583These idioms are OK in both shell and YSH:
584
585 if ! cp foo /tmp {
586 echo 'error copying' # any non-zero status
587 }
588
589 if ! test -d /bin {
590 echo 'not a directory'
591 }
592
593To be consistent with the idioms below, you can also write them like this:
594
595 try cp foo /tmp
596 if (_status !== 0) {
597 echo 'error copying'
598 }
599
600### Does a Function Succeed?
601
602When the command is a shell function, you shouldn't use `if myfunc` directly.
603This is because shell has the *Disabled `errexit` Quirk*, which is detected by
604YSH `strict_errexit`.
605
606**No**:
607
608 if myfunc; then # errors not checked in body of myfunc
609 echo 'success'
610 fi
611
612**Yes**. The *`$0` Dispatch Pattern* is a workaround that works in all shells.
613
614 if $0 myfunc; then # invoke a new shell
615 echo 'success'
616 fi
617
618 "$@" # Run the function $1 with args $2, $3, ...
619
620**Yes**. The YSH `try` builtin sets the special `_status` variable and returns
621`0`.
622
623 try myfunc # doesn't abort
624 if (_status === 0) {
625 echo 'success'
626 fi
627
628### `try` Also Takes a Block
629
630A block arg is useful for multiple commands:
631
632 try { # stops at the first error
633 chmod +x myfile
634 cp myfile /bin
635 }
636 if (_status !== 0) {
637 echo 'error'
638 }
639
640
641### Does a Pipeline Succeed?
642
643No:
644
645 if ps | grep python; then
646 echo 'found'
647 fi
648
649This is technically correct when `pipefail` is on, but it's impossible for
650YSH `strict_errexit` to distinguish it from `if myfunc | grep python` ahead
651of time (the ["meta" pitfall](error-handling.html#the-meta-pitfall)). If you
652know what you're doing, you can disable `strict_errexit`.
653
654Yes:
655
656 try {
657 ps | grep python
658 }
659 if (_status === 0) {
660 echo 'found'
661 }
662
663 # You can also examine the status of each part of the pipeline
664 if (_pipeline_status[0] !== 0) {
665 echo 'ps failed'
666 }
667
668### Does a Command With Process Subs Succeed?
669
670Similar to the pipeline example above:
671
672No:
673
674 if ! comm <(sort left.txt) <(sort right.txt); then
675 echo 'error'
676 fi
677
678Yes:
679
680 try {
681 comm <(sort left.txt) <(sort right.txt)
682 }
683 if (_status !== 0) {
684 echo 'error'
685 }
686
687 # You can also examine the status of each process sub
688 if (_process_sub_status[0] !== 0) {
689 echo 'first process sub failed'
690 }
691
692(I used `comm` in this example because it doesn't have a true / false / error
693status like `diff`.)
694
695### Handle Errors in YSH Expressions
696
697 try {
698 var x = 42 / 0
699 echo "result is $[42 / 0]"
700 }
701 if (_status !== 0) {
702 echo 'divide by zero'
703 }
704
705### Test Boolean Statuses, like `grep`, `diff`, `test`
706
707The YSH `boolstatus` builtin distinguishes **error** from **false**.
708
709**No**, this is subtly wrong. `grep` has 3 different return values.
710
711 if grep 'class' *.py {
712 echo 'found' # status 0 means found
713 } else {
714 echo 'not found OR ERROR' # any non-zero status
715 }
716
717**Yes**. `boolstatus` aborts the program if `egrep` doesn't return 0 or 1.
718
719 if boolstatus grep 'class' *.py { # may abort
720 echo 'found' # status 0 means found
721 } else {
722 echo 'not found' # status 1 means not found
723 }
724
725More flexible style:
726
727 try grep 'class' *.py
728 case $_status {
729 (0) echo 'found'
730 ;;
731 (1) echo 'not found'
732 ;;
733 (*) echo 'fatal'
734 exit $_status
735 ;;
736 }
737
738## Use YSH Expressions, Initializations, and Assignments (var, setvar)
739
740### Initialize and Assign Strings and Integers
741
742No:
743
744 local mystr=foo
745 mystr='new value'
746
747 local myint=42 # still a string in shell
748
749Yes:
750
751 var mystr = 'foo'
752 setvar mystr = 'new value'
753
754 var myint = 42 # a real integer
755
756### Expressions on Integers
757
758No:
759
760 x=$(( 1 + 2*3 ))
761 (( x = 1 + 2*3 ))
762
763Yes:
764
765 setvar x = 1 + 2*3
766
767### Mutate Integers
768
769No:
770
771 (( i++ )) # interacts poorly with errexit
772 i=$(( i+1 ))
773
774Yes:
775
776 setvar i += 1 # like Python, with a keyword
777
778### Initialize and Assign Arrays
779
780Arrays in YSH look like `:| my array |` and `['my', 'array']`.
781
782No:
783
784 local -a myarray=(one two three)
785 myarray[3]='THREE'
786
787Yes:
788
789 var myarray = :| one two three |
790 setvar myarray[3] = 'THREE'
791
792 var same = ['one', 'two', 'three']
793 var typed = [1, 2, true, false, null]
794
795
796### Initialize and Assign Dicts
797
798Dicts in YSH look like `{key: 'value'}`.
799
800No:
801
802 local -A myassoc=(['key']=value ['k2']=v2)
803 myassoc['key']=V
804
805
806Yes:
807
808 # keys don't need to be quoted
809 var myassoc = {key: 'value', k2: 'v2'}
810 setvar myassoc['key'] = 'V'
811
812### Get Values From Arrays and Dicts
813
814No:
815
816 local x=${a[i-1]}
817 x=${a[i]}
818
819 local y=${A['key']}
820
821Yes:
822
823 var x = a[i-1]
824 setvar x = a[i]
825
826 var y = A['key']
827
828### Conditions and Comparisons
829
830No:
831
832 if (( x > 0 )); then
833 echo 'positive'
834 fi
835
836Yes:
837
838 if (x > 0) {
839 echo 'positive'
840 }
841
842### Substituting Expressions in Words
843
844No:
845
846 echo flag=$((1 + a[i] * 3)) # C-like arithmetic
847
848Yes:
849
850 echo flag=$[1 + a[i] * 3] # Arbitrary YSH expressions
851
852 # Possible, but a local var might be more readable
853 echo flag=$['1' if x else '0']
854
855
856## Use [Egg Expressions](eggex.html) instead of Regexes
857
858### Test for a Match
859
860No:
861
862 local pat='[[:digit:]]+'
863 if [[ $x =~ $pat ]]; then
864 echo 'number'
865 fi
866
867Yes:
868
869 if (x ~ /digit+/) {
870 echo 'number'
871 }
872
873Or extract the pattern:
874
875 var pat = / digit+ /
876 if (x ~ pat) {
877 echo 'number'
878 }
879
880### Extract Submatches
881
882No:
883
884 if [[ $x =~ foo-([[:digit:]]+) ]] {
885 echo "${BASH_REMATCH[1]}" # first submatch
886 }
887
888Yes:
889
890 if (x ~ / 'foo-' <capture d+> /) { # <> is capture
891 echo $[_group(1)] # first submatch
892 }
893
894## Glob Matching
895
896No:
897
898 if [[ $x == *.py ]]; then
899 echo 'Python'
900 fi
901
902Yes:
903
904 if (x ~~ '*.py') {
905 echo 'Python'
906 }
907
908
909No:
910
911 case $x in
912 *.py)
913 echo Python
914 ;;
915 *.sh)
916 echo Shell
917 ;;
918 esac
919
920Yes (purely a style preference):
921
922 case $x { # curly braces
923 (*.py) # balanced parens
924 echo 'Python'
925 ;;
926 (*.sh)
927 echo 'Shell'
928 ;;
929 }
930
931## TODO
932
933### Distinguish Between Variables and Functions
934
935- `$RANDOM` vs. `random()`
936- `LANG=C` vs. `shopt --setattr LANG=C`
937
938## Related Documents
939
940- [Shell Language Idioms](shell-idioms.html). This advice applies to shells
941 other than YSH.
942- [What Breaks When You Upgrade to YSH](upgrade-breakage.html). Shell constructs that YSH
943 users should avoid.
944- [YSH Fixes Shell's Error Handling (`errexit`)](error-handling.html). YSH fixes the
945 flaky error handling in POSIX shell and bash.
946- TODO: Go through more of the [Pure Bash
947 Bible](https://github.com/dylanaraps/pure-bash-bible). YSH provides
948 alternatives for such quirky syntax.
949