DevOps - Bash Scripting
0x04. Loops, conditions and parsing - shellcheck
SSH (Secure Shell) keys
SSH (Secure Shell) keys are a pair of cryptographic keys used for secure authentication and communication between two computers over a network. The pair consists of a private key and a public key. The private key is kept on the client-side and should be kept secret, while the public key is placed on the server-side to allow secure access.
When you connect to a remote server using SSH, the server checks if your client has the corresponding private key. If the private key matches the public key stored on the server, the authentication is successful, and you are granted access.
Here's how you can create SSH keys on various operating systems:
If you have Git Bash installed, you can use the same command as above in the Git Bash terminal:
```bash
ssh-keygen -t rsa
```
Again, it will prompt you for a save location and passphrase.
Once you have generated your SSH keys, you can use the public key by copying its content and adding it to the `~/.ssh/authorized_keys` file on the server you want to connect to. The private key should be kept secure and not shared with others.
If you already have an existing id_rsa (private key) and id_rsa.pub (public key) and you need to retrieve them, you can find them in the default SSH directory of your user account. Here's how to locate them:
Linux/MacOS: Open a terminal and navigate to the
~/.ssh/directory, which is the default location for SSH keys. You can use the following command:
cd ~/.ssh/Once you are in the ~/.ssh/ directory, you can see the files using the ls command:
lsThis will list all the files in the directory, including your
id_rsa(private key) andid_rsa.pub(public key).
Remember that SSH keys provide a secure way of authentication, but it's essential to protect your private key with a strong passphrase to ensure an additional layer of security.
What is the advantage of using #!/usr/bin/env bash over #!/bin/bash
The difference between using `#!/usr/bin/env bash` and `#!/bin/bash` lies in how the operating system locates the desired interpreter (in this case, the Bash shell).
#!/usr/bin/env bash
Using this shebang instructs the system to look for the "bash" executable in the directories listed in the user's PATH environment variable. When the script is executed, the system will search for "bash" using the PATH, and it will use the first occurrence of "bash" found in the directories listed in PATH. This approach allows for more flexibility since it does not require the exact path to the interpreter. It makes the script more portable since it is not tied to a specific location of the "bash" executable.
Advantages:
-Greater portability: The script will work on different systems as long as Bash is available in the user's PATH.
- No hard-coded path: The script does not rely on a specific path to the Bash interpreter, making it easier to move the script between different systems without modifications.
2. `#!/bin/bash`:
This shebang explicitly specifies the path to the Bash interpreter. It points directly to the location of the Bash executable, which is usually in the `/bin/` directory.
Advantages:
- Explicitness: It leaves no ambiguity about which interpreter will be used to run the script. It ensures that the script will always be interpreted by Bash, even if there are other shells installed on the system.
- Specificity: If you have multiple versions of Bash installed on your system (e.g., different locations or custom builds), using the absolute path ensures you're running the desired version.
In summary, the advantage of using `#!/usr/bin/env bash` is its portability and flexibility, while `#!/bin/bash` provides explicitness and ensures a specific interpreter is used, using `#!/usr/bin/env bash` is a recommended practice as it offers better portability.
How to use while, until and for loops
While Loop
The `while` loop executes a block of code as long as a given condition is true.
```bash
while [ condition ]; do
# Code to execute while the condition is true
# The loop will continue until the condition becomes false
done
```
Example: Print numbers from 1 to 5 using a while loop.
```bash
#!/bin/bash
counter=1
while [ $counter -le 5 ]; do
echo $counter
((counter++))
done
```
Until Loop
The `until` loop is similar to the `while` loop but runs until a condition becomes true.
```bash
until [ condition ]; do
# Code to execute until the condition becomes true
done
```
Example: Print numbers from 1 to 5 using an until loop.
```bash
#!/bin/bash
counter=1
until [ $counter -gt 5 ]; do
echo $counter
((counter++))
done
```
For Loop
The `for` loop iterates over a list of items and executes a block of code for each item in the list.
```bash
for variable in list; do
# Code to execute for each item in the list
done
```
Example: Print numbers from 1 to 5 using a for loop.
```bash
#!/bin/bash
for (( counter=1; counter<=5; counter++ )); do
echo $counter
done
```
The `while` and `until` loops are useful when you want to repeat a block of code until a particular condition is met or no longer met, respectively. The `for` loop is ideal when you want to iterate over a predefined sequence of items, such as numbers, filenames, or array elements.
Nested Loops are also in Bash:
You can use nested while loops in Bash to achieve more complex iterations. Here's an example of a Bash script with nested while loops:
To loop through files in the current directory, you can use the
forloop with the*wildcard, which represents all files and directories in the current directory. Here's how you can do it:
file will hold the name of the current file or directory in each iteration.How to use if, else, elif and case condition statements
1. How to use while, until, and for loops:
**While Loop:**
The `while` loop executes a block of code as long as a given condition is true.
```bash
while [ condition ]; do
# Code to execute while the condition is true
# The loop will continue until the condition becomes false
done
```
Example: Print numbers from 1 to 5 using a while loop.
```bash
#!/bin/bash
counter=1
while [ $counter -le 5 ]; do
echo $counter
((counter++))
done
```
**Until Loop:**
The `until` loop is similar to the `while` loop but runs until a condition becomes true.
```bash
until [ condition ]; do
# Code to execute until the condition becomes true
done
```
Example: Print numbers from 1 to 5 using an until loop.
```bash
#!/bin/bash
counter=1
until [ $counter -gt 5 ]; do
echo $counter
((counter++))
done
```
**For Loop:**
The `for` loop iterates over a list of items and executes a block of code for each item in the list.
```bash
for variable in list; do
# Code to execute for each item in the list
done
```
Example: Print numbers from 1 to 5 using a for loop.
```bash
#!/bin/bash
for (( counter=1; counter<=5; counter++ )); do
echo $counter
done
```
2. How to use if, else, elif, and case condition statements:
**If-Else Statement:**
The `if` statement allows you to conditionally execute a block of code based on a condition. Optionally, you can use `else` to define an alternative block of code to execute if the condition is not true.
```bash
if [ condition ]; then
# Code to execute if the condition is true
else
# Code to execute if the condition is false (optional)
fi
```
**Elif (Else If) Statement:**
The `elif` statement is used when you have multiple conditions to check. It provides an alternative condition to check if the initial `if` condition is false.
```bash
if [ condition1 ]; then
# Code to execute if condition1 is true
elif [ condition2 ]; then
# Code to execute if condition2 is true
else
# Code to execute if both condition1 and condition2 are false (optional)
fi
```
Example: Check if a number is positive, negative, or zero.
```bash
#!/bin/bash
read -p "Enter a number: " number
if [ $number -gt 0 ]; then
echo "Positive"
elif [ $number -lt 0 ]; then
echo "Negative"
else
echo "Zero"
fi
```
Case Statement
The `case` statement allows you to match a variable against multiple patterns and execute code accordingly.
while loop in combination with a case statement to create powerful and flexible scripts that handle different cases or options. The case statement allows you to perform pattern matching against a variable and execute different blocks of code based on the matched pattern.3. How to use the `cut` command:
The `cut` command is used to extract specific columns or fields from a text file or a stream. It is especially useful when dealing with delimited files (e.g., CSV files) where columns are separated by a delimiter like tabs or commas.
```bash
cut OPTIONS FILE
```
Common options:
- `-d`: Specify the delimiter used in the input file. By default, it's a tab character.
- `-f`: Specify the field/column number(s) to extract. You can provide multiple field numbers separated by commas.
- `--complement`: Complement the selection (extract all fields except the specified ones).
Example: Suppose we have a file named `data.csv` with the following content:
```
Name,Age,Gender
John,30,Male
Jane,25,Female
Tom,40,Male
```
To extract the "Name" and "Gender" columns:
```bash
cut -d ',' -f 1,3 data.csv
```
Output:
```
Name,Gender
John,Male
Jane,Female
Tom,Male
```
In this example, we specified a comma (`,`) as the delimiter using `-d ','`, and then we used `-f 1,3` to extract the first and third fields (columns).
Remember to adjust the delimiter and field numbers according to your specific use case.
How to use the cut command
The `cut` command is used to extract specific columns or fields from a text file or a stream. It is especially useful when dealing with delimited files (e.g., CSV files) where columns are separated by a delimiter like tabs or commas.
The basic syntax of the `cut` command is as follows:
```bash
cut OPTIONS FILE
```
Here are some of the common options you can use with the `cut` command:
- `-d`: Specify the delimiter used in the input file. By default, it's a tab character.
- `-f`: Specify the field/column number(s) to extract. You can provide multiple field numbers separated by commas.
- `--complement`: Complement the selection (extract all fields except the specified ones).
Now, let's see some examples to better understand how to use the `cut` command:
Suppose we have a file named `data.csv` with the following content:
```
Name,Age,Gender
John,30,Male
Jane,25,Female
Tom,40,Male
```
1. To extract the "Name" column:
```bash
cut -d ',' -f 1 data.csv
```
Output:
```
Name
John
Jane
Tom
```
In this example, we specified a comma (`,`) as the delimiter using `-d ','`, and then we used `-f 1` to extract the first field (column).
2. To extract the "Name" and "Gender" columns:
```bash
cut -d ',' -f 1,3 data.csv
```
Output:
```
Name,Gender
John,Male
Jane,Female
Tom,Male
```
In this example, we used `-f 1,3` to extract the first and third fields (columns).
3. To extract all fields except the "Age" column:
```bash
cut -d ',' --complement -f 2 data.csv
```
Output:
```
Name,Gender
John,Male
Jane,Female
Tom,Male
```
In this example, we used `--complement` to extract all fields except the one specified with `-f 2`.
The `cut` command is a powerful tool for extracting specific data from text files based on column positions or delimiters.
Business Case:
Calculate the total salary of all employees in the Finance department
Create a simple employee data file and perform some data processing tasks using loops, conditional statements, and the cut command.
Let's assume we have a file named employees.txt with the following data:
ShellCheck
ShellCheck is a tool used for static analysis and linting of shell scripts. It helps identify potential issues and improvements in your shell scripts. While it is primarily designed for Unix-like systems, you can install and use ShellCheck on Windows using various methods. Here are a couple of ways to install ShellCheck on Windows:
1. Install ShellCheck using WSL (Windows Subsystem for Linux):
- If you have WSL (Windows Subsystem for Linux) installed on your Windows machine, you can use it to install ShellCheck.
- Open a WSL terminal (e.g., Ubuntu, Debian, or any other distribution you have installed).
- Update the package list and install ShellCheck:
```bash
sudo apt update
sudo apt install shellcheck
```
- After installation, you can use ShellCheck within the WSL terminal to check your shell scripts.
2. Install ShellCheck using Chocolatey:
- If you have Chocolatey package manager installed on your Windows machine, you can use it to install ShellCheck.
- Open a Command Prompt or PowerShell window with administrator privileges.
- Install ShellCheck using Chocolatey:
```bash
choco install shellcheck
```
- After installation, you can use ShellCheck in your Command Prompt, PowerShell, or other terminals to check your shell scripts.
3. Install ShellCheck using Git Bash:
- If you have Git Bash installed on your Windows machine, you can use it to install ShellCheck.
- Open Git Bash, which provides a Unix-like terminal environment.
- Install ShellCheck using the package manager appropriate for your Git Bash distribution. For example, if you are using Git for Windows, you can install ShellCheck with the following command:
```bash
pacman -S shellcheck
```
- After installation, you can use ShellCheck within the Git Bash terminal to check your shell scripts.
Please note that when using ShellCheck on Windows, you may need to provide the correct paths to your shell scripts if they are located outside the WSL, Chocolatey, or Git Bash environments.
Choose the method that best fits your setup, and once installed
you can run ShellCheck on your shell scripts by passing the script file as an argument:
```bash
shellcheck your_script.sh
```
ShellCheck will analyze your script and provide feedback on any potential issues or improvements.
Hints:
The
<<<symbol
Known as a Here String in Bash. It is used to redirect a string as input to a command, allowing you to pass the contents of a variable or a literal string to a command that expects input from standard input (stdin).
command <<< "string"When you use <<<, the string following it is treated as input to the command on its left side. It's similar to providing input via a pipe (|), but it's more convenient when you want to pass a single string or a variable's content.
checks if the current item referenced by
$fileis a regular file (not a directory or a special file by:if [ -f "$file" ]To check if a file name contains a certain delimiter ex. comma delimiter in Bash, you can use the
casestatement or other string manipulation techniques like parameter expansion. Here's an example using thecasestatement:
casestatement checks if the variable$file(representing each file name) contains a comma (*,*).
4. how to check if If the file exists, if the file is empty if the file is a regular file ?
if [ -e "$file" ]; # Check if the file exists
if [ -s "$file" ]; # Check if the file is empty
if [ -f "$file" ]; # Check if the file is a regular file