How to iterate over lines in a variable in Bash?

Question

How does one properly iterate over lines in bash either in a variable, or from the output of a command?

Simply setting the IFS variable to a new line works for the output of a command but not when processing a variable that contains new lines.

For example

#!/bin/bash
list="One\ntwo\nthree\nfour"
#Print the list with echo
echo -e "echo: \n$list"
#Set the field separator to new line
IFS=$'\n'
#Try to iterate over each line
echo "For loop:"
for item in $list
do
        echo "Item: $item"
done
#Output the variable to a file
echo -e $list > list.txt
#Try to iterate over each line from the cat command
echo "For loop over command output:"
for item in cat list.txt
do
        echo "Item: $item"
done

This gives the output:

echo: 
One
two
three
four
For loop:
Item: One\ntwo\nthree\nfour
For loop over command output:
Item: One
Item: two
Item: three
Item: four

As you can see, echoing the variable or iterating over the cat command prints each of the lines one by one correctly. However, the first for loop prints all the items on a single line. Any ideas?

Answer 1

With bash, if you want to embed newlines in a string, enclose the string with $'':

$ list="One\ntwo\nthree\nfour"
$ echo "$list"
One\ntwo\nthree\nfour
$ list=$'One\ntwo\nthree\nfour'
$ echo "$list"
One
two
three
four

And if you have such a string already in a variable, you can read it line-by-line with:

while IFS= read -r line; do
    echo "... $line ..."
done <<< "$list"

---

@wheeler makes a good point about <<< adding a trailing newline.

Suppose the variable ends with a newline

list=$'One\ntwo\nthree\nfour\n'

Then the while loop outputs

while IFS= read -r line; do
    echo "... $line ..."
done <<< "$list"

... One ...
... two ...
... three ...
... four ...
...  ...

To get around that, use a redirection from a process substitution instead of a here-string

while IFS= read -r line; do
    echo "... $line ..."
done < <(printf '%s' "$list")

... One ...
... two ...
... three ...
... four ...

But now this "fails" for strings without a trailing newline

list2=$'foo\nbar\nbaz'
while IFS= read -r line; do
    echo "... $line ..."
done < <(printf '%s' "$list2")

... foo ...
... bar ...

The read documentation says

The exit status is zero, unless end-of-file is encountered

and because the input does not end with a newline, EOF is encountered before read can get a whole line. read exits non-zero and the while loop completes.

The characters are consumed into the variable though.

So, the absolutely proper way to loop over the lines of a string is:

while IFS= read -r line || [[ -n $line ]]; do
    echo "... $line ..."
done < <(printf '%s' "$list2")

This outputs the expected for both $list and $list2

Answer 2

You can use while + read:

some_command | while read line ; do
   echo === $line ===
done

Btw. the -e option to echo is non-standard. Use printf instead, if you want portability.

Answer 3

#!/bin/sh

items="
one two three four
hello world
this should work just fine
"

IFS='
'
count=0
for item in $items
do
  count=$((count+1))
  echo $count $item
done

Answer 4

Here's a funny way of doing your for loop:

for item in ${list//\\n/
}
do
   echo "Item: $item"
done

A little more sensible/readable would be:

cr='
'
for item in ${list//\\n/$cr}
do
   echo "Item: $item"
done

But that's all too complex, you only need a space in there:

for item in ${list//\\n/ }
do
   echo "Item: $item"
done

You $line variable doesn't contain newlines. It contains instances of \ followed by n. You can see that clearly with:

$ cat t.sh
#! /bin/bash
list="One\ntwo\nthree\nfour"
echo $list | hexdump -C

$ ./t.sh
00000000  4f 6e 65 5c 6e 74 77 6f  5c 6e 74 68 72 65 65 5c  |One\ntwo\nthree\|
00000010  6e 66 6f 75 72 0a                                 |nfour.|
00000016

The substitution is replacing those with spaces, which is enough for it to work in for loops:

$ cat t.sh
#! /bin/bash
list="One\ntwo\nthree\nfour"
echo ${list//\\n/ } | hexdump -C

$ ./t.sh 
00000000  4f 6e 65 20 74 77 6f 20  74 68 72 65 65 20 66 6f  |One two three fo|
00000010  75 72 0a                                          |ur.|
00000013

---

Demo:

$ cat t.sh
#! /bin/bash
list="One\ntwo\nthree\nfour"
echo ${list//\\n/ } | hexdump -C
for item in ${list//\\n/ } ; do
    echo $item
done

$ ./t.sh 
00000000  4f 6e 65 20 74 77 6f 20  74 68 72 65 65 20 66 6f  |One two three fo|
00000010  75 72 0a                                          |ur.|
00000013
One
two
three
four

Answer 5

You can also first convert the variable into an array, then iterate over this.

lines="abc
def
ghi"

declare -a theArray

while read -r line
do
    theArray+=("$line")            
done <<< "$lines"

for line in "${theArray[@]}"
do
    echo "$line"
    #Do something complex here that would break your read loop
done

This is only usefull if you do not want to mess with the IFS and also have issues with the read command, as it can happen, if you call another script inside the loop that that script can empty your read buffer before returning, as it happened to me.

Answer 6

list="One\ntwo\nthree\nfour"

for line in $(echo ${list} | xargs -L1 echo)
do
  echo ${line}
done

Answer 7

Using a string var with carefully crafted newline-delimited substrings:

tmp1=' M file*4.bin'$'\n''AM file5\0345.txt'$'\n''M  file6a$\t.rtf'

Note the concatenation via $'\n', in order to actually insert newline control chars, instead of just '\n' substrings and single-quoted substrings, to preserve special chars/avoid escaping.

The code below should work with arrays as well:

tmp2=(' M file*1.bin' 'AM file2\0345.txt' 'M  file3a$\t.rtf')

Iterating the string in a for loop now can be achieved via unquoted expansion to a single string ([*]) and IFS restriction to newline (the printf variant can give more control, but invokes a subshell:

_IFS="$IFS"; IFS=$'\n'
#for e in $(printf '%s\n' "${tmp1[*]}"); do 
for e in ${tmp1[*]}; do
  printf 'ELEM: [%s]\n' "$e"
done
IFS="$_IFS"

Note the missing double quotes around ${tmp1[*]}, in order to expand and split at the newline control chars, instead of expanding to just a single string here.

The surrounding IFS modification and reset should prevent accidential splitting at spaces or tabs, but might also affect commands inside the loop.

Also consider the hint on requiring to disable globbing from 'dave_thompson_085' below, in case the input strings may contain escaped or special chars (not single-quoted).

A while loop over a quoted expanded Here String may be the better choice, depending on the input. Narrowing the IFS to only newlines for just the loop is possible, but shouldn't be necessary:

while IFS=$'\n' read -r; do 
  printf 'ELEM: [%s]\n' "$REPLY"
done <<<"$tmp1"

When we have an array, it can be simply iterated with a for loop, from a quoted array Parameter Expansion ([@]) - note the double-quotes here:

for e in "${tmp2[@]}"; do 
  printf 'ELEM: [%s]\n' "$e"
done

or with while loop:

_IFS="$IFS"; IFS=$'\n'
while read -r; do 
  printf 'ELEM: [%s]\n' "$REPLY"
done <<<"${tmp2[*]}"
# done < <(printf '%s\n' "${tmp2[*]}")
IFS="$_IFS"
# done <<<"${tmp2[@]@E}"

The usage of @E as operator for the array Parameter Expansion ([@]), is required to expand the newline control chars, but this will also expand other special chars end escape sequences, which may be undesired.

To avoid this, restricting the IFS to newline (and restoring it) and expanding the array to a single string ([*]) is used instead.

Alternatively, the array splitting can be done via Command Substitution and printf.

This code should work for newline-delimited strings, as well as for arrays.