prepend heredoc to a file, in bash

Question

I was using:

cat <<<"${MSG}" > outfile

to begin with writing a message to outfile, then further processing goes on, which appends to that outfile from my awk script.

But now logic has changed in my program, so I'll have to first populate outfile by appending lines from my awk program (externally called from my bash script), and then as a final step prepend that ${MSG} heredoc to the head of my outfile..

How could I do that from within my bash script, not awk script?

EDIT

this is MSG heredoc:

read -r -d '' MSG << EOF
-----------------------------------------------
--   results of processing - $CLIST
--   used THRESHOLD ($THRESHOLD)
-----------------------------------------------
l
EOF
# trick to pertain newline at the end of a message
# see here: http://unix.stackexchange.com/a/20042
MSG=${MSG%l}

NB: cat with a herestring? The portable way to do that would have been `printf '%s\n' "$MSG" > outfile`. Avoid bashisms where you can. — Jens, Jul 08 '14 at 14:15
FYI, what you're doing here is necessarily inefficient -- "necessarily" meaning that POSIX doesn't provide any way (in *any* programming language) to prepend content to a file without rewriting that whole file. If it's a large file, then, you should try to avoid any process which would have you doing this frequently. — Charles Duffy, Jul 08 '14 at 15:00
@CharlesDuffy yes you are right, in my script this occures only once. I tried with that MSG writing being handled with `awk`, but, as that `awk` script gets called more than once from within my bash script, putting that MSG into BEGIN block wouldn't help, because I would have many instances, and I want only one at the top. — branquito, Jul 08 '14 at 15:32
As an aside: you can make do without `l` trick for preserving trailing newlines if you instead use the following: `IFS= read -r -d '' MSG << EOF ...`. — mklement0, Jul 08 '14 at 15:48
@mklement0 Would I have to revert it back to old `IFS` value, or it doesn't matter, because it would go with the script being finished? — branquito, Jul 08 '14 at 15:51
No, there is NO need to restore the value, because prepending a variable assignment directly to a command that way _localizes_ it to that command - that is (loosely speaking), it automatically reverts to the previous value when the command finishes. — mklement0, Jul 08 '14 at 15:52
Oh yes, I forgot that, silly me. Thanks, your tip saves me two lines of code :) — branquito, Jul 08 '14 at 15:54

chepner · Answer 1 · 2014-07-08T15:14:54.023

5

Use a command group:

{
    echo "$MSG"
    awk '...'
} > outfile

If outfile already exists, you have no choice but to use a temporary file and copy it over the original. This is due to how files are implemented by all(?) file systems; you cannot prepend to a stream.

{
     # You may need to rearrange, depending on how the original
     # outfile is used.
     cat outfile
     echo "$MSG"
     awk '...'
} > outfile.new && mv outfile.new outfile

Another non-POSIX feature you could use with cat is process substitution, which makes the output of an arbitrary command look like a file to cat:

cat <(echo $MSG) outfile <(awk '...') > outfile.new && mv outfile.new outfile

edited Jul 08 '14 at 15:14

answered Jul 08 '14 at 14:10

chepner

497,756
71
530
681

I cannot do that, because this happens in a moment when I already have outfile generated. I have a case checking if `outfile` exists, if it does, than I need to prepend my $MSG – branquito Jul 08 '14 at 14:15
I did exactly that: `cat <<< $MSG outfile > outfile.tmp && mv outfile.tmp outfile`, but ended up with `outfile` being empty, and still having that `outfile.tmp` in my directory? That was the reason I posted question here ;) – branquito Jul 08 '14 at 14:35
my bad.. now I saw, entered `>` between `mv outfile.tmp` and `outfile`, overseeing that, led me to post here.. sorry for taking your time – branquito Jul 08 '14 at 14:43
don't know why, but `cat <<< $MSG outfile > outfile.tmp && mv outfile.tmp outfile` doesn't prepend $MSG, `outfile` is there, but no $MSG prepended? – branquito Jul 08 '14 at 15:09
2

`<<<` provides standard input, and `cat` ignores standard input when it receives one or more filename arguments. – chepner Jul 08 '14 at 15:13
Oh, that was the case. Thanks. – branquito Jul 08 '14 at 15:19

score 5 · Accepted Answer · edited Jan 17 '20 at 17:54

5

You can use awk to insert a multiline string at beginning of a file:

awk '1' <(echo "$MSG") file

Or even this echo should work:

echo "${MSG}$(<file)" > file

edited Jan 17 '20 at 17:54

cautionbug

435
5
18

answered Jul 08 '14 at 14:12

anubhava

761,203
64
569
643

$MSG contains more than one line, it's a `heredoc` banner-like message – branquito Jul 08 '14 at 14:14
can you explain a bit, will this work only for that number of lines, so it will break if I change my `$MSG` to have more lines? – branquito Jul 08 '14 at 14:46
1

`MSG` variable can have any number of lines actually. awk's `substr` function has been used to strip off `$'` from front and `\n'` from end (those are added by `printf "%q"`). I have tested it on my OSX before posting the answer. – anubhava Jul 08 '14 at 14:47
I lost my newline at the end of $MSG. Why is that, and how to keep it? – branquito Jul 08 '14 at 15:26
Sure just change `substr` to `substr(m,3,length(m)-3)` – anubhava Jul 08 '14 at 15:39
I'm a little worried that this relies a bit much on implementation-defined behavior -- `printf '%q'` promises only to create a string that bash can parse, not to do it in any specific way. – Charles Duffy Jul 08 '14 at 15:51
Actually I couldn't think of any other way of passing a multiline string to `awk` earlier. But I have edited it now to remove `printf "%q"` – anubhava Jul 08 '14 at 15:57
1

+1 for simple solutions, but you should note that (a) `awk` is not the right tool for this job _in general_ - it makes sense here _specifically_, because of the OP's specific requirements, and (b) the `echo` solution should only be used for _small_ files, given that the entire string is built up _in memory_ (right?). Also, it sounds like `$MSG` is already `\n`-terminated, so you should use `<(printf '%s' "$MSG")`. – mklement0 Jul 09 '14 at 03:01
1

@CharlesDuffy: To finish the discussion, even though `printf %q` is no longer in the mix: I share your concern; for instance, if the string happens to contain no characters that need escaping (though it will in this case), the result of `printf %q` is _not_ `$'...'`-enclosed. Note that GNU `awk` and `mawk` actually DO accept strings with embedded newlines on the command line - FreeBSD/OSX `awk` doesn't (which is actually the POSIX-compliant behavior). The `awk`-POSIX-compliant solution is to replace newlines with `'\n'` strings: `-v MSG="${MSG//$'\n'/\\n}". See http://goo.gl/uT1oh9 for more. – mklement0 Jul 09 '14 at 03:13
1

Highly underrated answer. Thank you! For the sake of completion, the `echo` version just goes to stdout - it doesn't rewrite the original file (per the question). The adjustment is simple: `echo "${MSG}$( file` – cautionbug Jan 16 '20 at 18:09

Charles Duffy · Answer 3 · 2014-07-08T15:41:39.357

5

Use - as a placeholder on the cat command line for the point where you want new content inserted:

{ cat - old-file >new-file && mv new-file old-file; } <<EOF
header
EOF

edited Jul 08 '14 at 15:41

answered Jul 08 '14 at 15:34

Charles Duffy

280,126
43
390
441

Can that `< – branquito Jul 08 '14 at 15:39
2

+1 for the prefix form, but it's simpler to use `cat <>old-file` for the suffix form. – mklement0 Jul 08 '14 at 15:40
@mklement0, indeed, that was silly. I've removed it. – Charles Duffy Jul 08 '14 at 15:41
1

@branquito, indeed, using a herestring there will be fine. – Charles Duffy Jul 08 '14 at 15:42
I like your answer the best, but not shure If would be now ok to accept it as I already accepted another one earlier, before you posted. – branquito Jul 08 '14 at 15:44
1

@branquito, there's nothing wrong with the current answer, even if this one is a little bit cleaner, so perfectly fine to keep it. – Charles Duffy Jul 08 '14 at 15:47
Yes, that's why I said, your solution looks neat. Anyway I learned a few new and important things from all of you. – branquito Jul 08 '14 at 15:49

prepend **heredoc** to a file, in bash

3 Answers3

prepend heredoc to a file, in bash