DEV Community

CKalt
CKalt

Posted on

ChatGPT Help: Bash heredoc usage for CMD strings- bonus how to use arrays and not need eval or heredoc!

I've used Bash for I don't know--- 20 years or more--- and I've never quite understood why I can sometimes use heredocs and sometimes I cannot. The gritty details of how variables are expanded and how bash treats variable assignments has always been a bit too dirty for me to become an expert in. So below ChatGPT teaches me this cool trick (I call it a bonus) how to use arrays to avoid using an eval. This is at the end of this post, but in my opinion. It's the best part!

Happy geeking/ err reading....

Me:

On linux or macOS running bash how would I assign a string to variable called CMD that I wish to run later, using heredoc syntax, something like below? (which I know is bad syntax but in communicates my goal)

CMD=

curl --no-progress-meter -X 'POST' \
  "https://hosttarget.com/placetarget/${method}" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d "DeviceSerial=${DEVICE_SERIAL}&DateStartSecUtc=${DATE_START_SEC_UTC}&DateNumSteps=${STEPS}&UserKey=${USER_KEY}"
Enter fullscreen mode Exit fullscreen mode

I want to be able to both run the above $CMD and echo it to stdout. But I wish to use heredoc syntax for assigning the CMD var for readability.

ChatGPT:

You can use a heredoc in Bash to assign a multi-line string to a variable like so:


read -r -d '' CMD << 'EOF'
curl --no-progress-meter -X 'POST' \
  "https://hosttarget/placetarget/${method}" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d "DeviceSerial=${DEVICE_SERIAL}&DateStartSecUtc=${DATE_START_SEC_UTC}&DateNumSteps=${STEPS}&UserKey=${USER_KEY}"
EOF
Enter fullscreen mode Exit fullscreen mode

Here's what the various parts mean:

read -r -d '': This uses the read built-in to read into a variable. The -r flag prevents backslashes from being treated as escape characters. The -d '' sets the delimiter to an empty string, which allows you to capture the entire heredoc, including newlines.
CMD: This is the variable you're reading into.
EOF: This marks the start and end of the heredoc. You can use any word you want, as long as it appears by itself on a line. The quotes around 'EOF' prevent parameter substitution within the heredoc.
Enter fullscreen mode Exit fullscreen mode

After you run this, the CMD variable will contain your curl command. You can then echo it or evaluate it:


echo "$CMD"

# Evaluate it. Be careful when doing this; make sure CMD only contains what you expect.
eval "$CMD"
Enter fullscreen mode Exit fullscreen mode

Note that the eval command should be used carefully, as it will execute the contents of the variable as a shell command. Make sure you trust the content that's going into CMD.

Me:

Why do we need to use read why cannot we simply do:

CMD=<<EOF
curl --no-progress-meter -X 'POST' \
  "https://hosttarget.com/placetarget/${method}" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d "DeviceSerial=${DEVICE_SERIAL}&DateStartSecUtc=${DATE_START_SEC_UTC}&DateNumSteps=${STEPS}&UserKey=${USER_KEY}"
EOF
Enter fullscreen mode Exit fullscreen mode

ChatGPT:

The heredoc (<<EOF ... EOF) syntax in Bash is often used for redirecting multiple lines of text to a command's standard input or into a file, rather than directly assigning it to a variable. Unlike some other scripting languages where multi-line string assignment is straightforward, Bash doesn't allow the heredoc syntax to be used directly for variable assignment in the way you've shown (CMD=<<EOF ...).

The read -r -d '' trick is a commonly used idiom to work around this limitation. It reads multiple lines into a variable, effectively capturing the heredoc content into that variable. Using read ensures that the variable gets populated with exactly the content of the heredoc, including any newlines and other formatting.

That's why you can't just do CMD=<<EOF ... EOF, but you have to use a command like read to capture the heredoc's content into a variable.

Me: What happens if you try to use heredoc for variable assignment?

ChatGPT:

Attempting to use heredoc syntax for direct variable assignment in Bash, like this:

CMD=<<EOF
some text here
EOF
Enter fullscreen mode Exit fullscreen mode

will result in a syntax error. You'll likely see an error message similar to the following:


bash: syntax error near unexpected token `<<'
Enter fullscreen mode Exit fullscreen mode

This is because the <<EOF ... EOF heredoc syntax is designed for input redirection, not for direct assignment to variables. In Bash, variable assignment using = expects a value immediately following the =, without any spaces. The heredoc syntax does not satisfy this requirement.

As mentioned earlier, you can use the read command as a workaround to capture the content of a heredoc into a variable. Another common workaround is to use command substitution to assign a variable using $(...), but that typically works better for capturing the output of a command rather than for storing multi-line text.

Me:

Why not use:

CMD=$(cat <<'EOF' ... EOF)
Enter fullscreen mode Exit fullscreen mode

instead of read?

ChatGPT:

Using CMD=$(cat <<'EOF' ... EOF) is another valid way to assign multi-line strings to a variable using heredoc in Bash. Both methods have their own characteristics, and either could be more suitable depending on what you want to do

Me:

Why if I run this script it works fine:

#!/bin/bash
method=DeviceData
#DEVICE_SERIAL="00B811B8"
DEVICE_SERIAL="B1400166"
DATE_START_SEC_UTC=1682380800     # Pacific/US: 2023-04-24 17:00:00
STEPS=24                          # Each Step 60min. So 24=1day
USER_KEY="xxxxx""

CMD=$(cat <<EOF
curl --no-progress-meter -X 'POST' \
  "https://hosttarget.com/placetarget/${method}" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d "DeviceSerial=${DEVICE_SERIAL}&DateStartSecUtc=${DATE_START_SEC_UTC}&DateNumSteps=${STEPS}&UserKey=${USER_KEY}"
EOF
)

echo $CMD
eval $CMD
Enter fullscreen mode Exit fullscreen mode

however if I remove the eval command and try to run the script this is what I get:

curl --no-progress-meter -X 'POST' "https://hosttarget.com/placetarget/DeviceData" -H 'accept: application/json' -H 'Content-Type: application/x-www-form-urlencoded' -d "DeviceSerial=B1400166&DateStartSecUtc=1682380800&DateNumSteps=24&UserKey=xxxxx"
curl: (3) URL rejected: Port number was not a decimal number between 0 and 65535
curl: (6) Could not resolve host: application
curl: (6) Could not resolve host: application
Enter fullscreen mode Exit fullscreen mode

ChatGPT:

The issue is related to the way shell expansion and command execution work. When you echo $CMD, you see the command string as you expect it, but there are hidden characters (like line breaks and extra spaces) in it.

In your script, CMD is a variable containing a string with embedded newlines, multiple spaces, and even tabs if they exist in the <<EOF ... EOF block.
When you eval $CMD, it evaluates that string as if you typed it directly into the shell, so it works as expected.
When you just copy-paste the output of echo $CMD into the shell, the newlines and extra spaces aren't properly handled, which leads to the errors you're seeing.
Enter fullscreen mode Exit fullscreen mode

Here's what you could do to make it more straightforward:

Get rid of the CMD variable and the eval. Just use the curl command directly in the script.
Enter fullscreen mode Exit fullscreen mode 
#!/bin/bash
method=DeviceData
DEVICE_SERIAL="B1400166"
DATE_START_SEC_UTC=1682380800     # Pacific/US: 2023-04-24 17:00:00
STEPS=24                          # Each Step 60min. So 24=1day
USER_KEY="xxxxx"

curl --no-progress-meter -X 'POST' \
  "https://hosttarget.com/placetarget/${method}" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d "DeviceSerial=${DEVICE_SERIAL}&DateStartSecUtc=${DATE_START_SEC_UTC}&DateNumSteps=${STEPS}&UserKey=${USER_KEY}"
Enter fullscreen mode Exit fullscreen mode

Alternatively, if you want to keep using a variable for the command:

Use array variables to keep each argument separate, which helps in preserving spaces and special characters as they are.
Enter fullscreen mode Exit fullscreen mode 
#!/bin/bash
method=DeviceData
DEVICE_SERIAL="B1400166"
DATE_START_SEC_UTC=1682380800     # Pacific/US: 2023-04-24 17:00:00
STEPS=24                          # Each Step 60min. So 24=1day
USER_KEY="xxxxx"

CMD=("curl" "--no-progress-meter" "-X" "POST"
  "https://hosttarget.com/placetarget/${method}"
  "-H" "accept: application/json"
  "-H" "Content-Type: application/x-www-form-urlencoded"
  "-d" "DeviceSerial=${DEVICE_SERIAL}&DateStartSecUtc=${DATE_START_SEC_UTC}&DateNumSteps=${STEPS}&UserKey=${USER_KEY}")

"${CMD[@]}"
Enter fullscreen mode Exit fullscreen mode

Either way should work without requiring eval.

Me:

If I used your array variable approach, how would I echo the command to standard output that would be copied and pased and run in another bash session?

ChatGPT:

If you want to echo the command in a way that it could be copied and pasted into another bash session, you could iterate over the array and print each element, taking care to quote any elements that contain special characters.

Here's a sample script that would echo the command before running it:

#!/bin/bash

method=DeviceData
DEVICE_SERIAL="B1400166"
DATE_START_SEC_UTC=1682380800
STEPS=24
USER_KEY="xxxxx"

CMD=("curl" "--no-progress-meter" "-X" "POST"
  "https://hosttarget.com/placetarget/${method}"
  "-H" "accept: application/json"
  "-H" "Content-Type: application/x-www-form-urlencoded"
  "-d" "DeviceSerial=${DEVICE_SERIAL}&DateStartSecUtc=${DATE_START_SEC_UTC}&DateNumSteps=${STEPS}&UserKey=${USER_KEY}")

# Echo the command in a way that it can be copy-pasted
printf "%q " "${CMD[@]}"
echo

# Run the command
"${CMD[@]}"
Enter fullscreen mode Exit fullscreen mode

The %q format specification in the printf command will print the string in a way that it can be reused as shell input. This means special characters will be escaped appropriately.

Top comments (0)