Streamlining DevOps Workflow: Mastering Advanced Linux Commands and Scripts

Introduction:

Welcome back to my Hashnode blog! In this second article, I'm thrilled to share a treasure trove of advanced Linux commands and scripts that are tailor-made for DevOps engineers. As we navigate the complexities of modern-day development and operations, these powerful tools will become your trusty companions, simplifying tasks and boosting productivity.

To search for Text or Pattern matching.

In Unix-like operating systems like Linux, the grep command is a strong and popular text-searching tool. Its main use is to look for particular regular expressions or patterns in text files or other command output. The acronym "grep" stands for "Global Regular Expression Print."

ubuntu@ip-172-xx-xx-xxx:~$ grep -ir "hello Harshit" example.txt

• The "grep" command searches line by line across the provided file or input stream for instances of the pattern. When a match is made, it prints the complete matching line to the standard output. Grep is useful for simultaneously scanning several files since it displays the filename where the match occurred if multiple files are searched.

• -i: stands for "ignore case." When this option is combined with "grep," the search is no longer case-sensitive and will match patterns in both capital and lowercase letters. For instance, if you search for "example" with the suffix "-i," it will match "example," "Example," "EXAMPLE," and so on.

• -r: option stands for "recursive." When combined with "grep," this option makes the search recursive, allowing it to look through all of the files in the specified directory and its subdirectories in addition to the specified file(s). When you need to search for a pattern over a whole directory tree, this is quite helpful.

To change the group ownership of Files and Directories.

Linux and other Unix-based operating systems allow users to modify the group ownership of files and directories using the "chgrp" command. Its name, which means "change group," perfectly summed up its main purpose.

ubuntu@ip-172-xx-xx-xxx:~$ chgrp tester file(s)

• tester: This is the name of the new group you want to assign to the specified files or directories. It can be either the group name or the group's numeric group ID (GID).

• file(s): These are the files or directories whose group ownership you want to modify. You can specify multiple files and directories as arguments to the command.

• The "chgrp" command allows you to change the group ownership of one or more files or directories to a specified group. Only the owner of the file or the superuser (root) has permission to change the group ownership of a file.

To add a new group.

The "groupadd" command creates a new group with the specified name and assigns a unique group ID (GID) to it. Groups are used to organize and manage users with similar access rights and permissions. Files and directories can be associated with specific groups, allowing group members to access and modify them based on the permissions set for the group.

ubuntu@ip-172-xx-xx-xxx:~$ sudo groupadd developer

• developer: This is the name of the new group we want to create.

• After running this command, the "developer" group will be created with a unique group ID (GID). You can confirm the creation of the group by checking the "/etc/group" file, where group information is stored.

• The "groupadd" command typically requires administrative privileges, commonly granted through the "sudo" command. Also, it is essential to ensure that the group name you choose does not conflict with any existing group names on the system. Group names are case-sensitive and must follow certain naming conventions specific to the operating system.

To add users to a specific group.

When a user is added to a group, group membership is granted using the "gpasswd -m" command. This enables the user to access any resources, files, or directories that have been assigned the group's particular permissions. The user inherits the group's access rights and privileges by joining the group.

ubuntu@ip-172-xx-xx-xxx:~$ gpasswd -m Paliwal,Choubisa Harshit

• If you use the gpasswd command without the -M option and simply provide the username and group name, it will add the specified user to the group. The user will become a regular member of the group, without any administrative privileges. Regular members can access resources associated with the group and have the permissions set for the group, but they cannot manage the group's membership.

• Paliwal,Choubisa: This is a comma-separated list of usernames that you want to set as the members of the group. The users specified in the list will become the only members of the group, replacing any existing members.

• Harshit: This is the name of the group for which you want to set the members.

• The gpasswd command without the -M option and providing a list of users. It will replace the existing group members entirely with the new list you specify. Always ensure that you have the correct list of users before executing the command to avoid unintended changes to group membership.

To Filter Structured or Unstructured data.

For text processing, Linux and other Unix-based operating systems use the awk command. Data extraction, manipulation, and reporting are its main uses. The word "awk" is derived from the initials of the program's creators, Alfred Aho, Peter Weinberger, and Brian Kernighan.

ubuntu@ip-172-xx-xx-xxx:~$ awk 'pattern { action }' input_file

• pattern: This is a condition or a regular expression to match against each input line. If the pattern is not specified, the action will be applied to all lines.

• { action }: This is the block of code that gets executed for lines that match the specified pattern. It can include commands for data processing, calculations, or printing the output.

• input_file: This is the name of the file from which the input data is read. If not provided, awk reads input from the standard input (e.g., text piped from another command).

ubuntu@ip-172-xx-xx-xxx:~$ awk ' NR>=1 && NR<=10 /docker/ { print NR,$1,$2,$5 } ' input_file

In this example, we are using awk to process the "input_file" and extract specific fields for lines containing the word "docker." Here's a short explanation of the command:

• awk: Invokes the awk command for text processing.

• ' NR>=1 && NR<=10 /docker/ { print NR,$1,$2,$5 } ': This is the awk script, specifying the pattern and action to be performed.

  • NR>=1 && NR<=10: The condition checks if the line number (NR) is between 1 and 10, limiting the processing to the first ten lines of the file.

  • /docker/: The pattern searches for lines that contain the word "docker."

  • { print NR,$1,$2,$5 }: The action specifies what to print for lines that match the pattern. It prints the line number (NR), the first field ($1), the second field ($2), and the fifth field ($5) of the matching lines.

• input_file: The name of the file to be processed by awk.

In summary, awk is a versatile tool for text processing, and while it shines when dealing with structured data, it can still be very useful for working with unstructured data using the power of regular expressions. With a little creativity, awk can efficiently handle a wide range of text-processing tasks in the Linux command-line environment.

Essential Scripting Rules.

Before embarking on the exciting journey of scripting with Bash, it's crucial to understand the fundamental rules that shape well-structured and efficient scripts. We'll explore the key rules that every aspiring Bash scriptwriter should know.

• Script File Extension:

  To identify a file as a Bash script, it's essential to use the correct file extension. By convention, Bash scripts end with the ".sh" extension, signifying that they are executable shell scripts.

• Shebang (Shbang):

  The shebang, represented by "#!/bin/bash" at the beginning of a script, is a vital rule. It serves as a directive to the system, indicating the interpreter (in this case, "/bin/bash") to use when executing the script. The shebang ensures that your script runs in the correct environment, regardless of where it is executed.

• Setting Script Permissions:

  Before executing a Bash script, it needs to be granted executable permissions. To achieve this, you can use the "chmod" command with the value "700" (read, write, and execute permissions for the script owner) followed by the filename.

ubuntu@ip-172-xx-xx-xxx:~$ chmod 700 filename.sh

Executing the Script: Once the script's executable permissions have been set, you can run it directly using "./filename.sh" without explicitly invoking the "bash" command. The "./" indicates that the script is in the current working directory.

Taking Input in scripts.

To take input in Bash scripts we can use the read command. It allows you to prompt the user for input and store their response in a variable.

#!/bin/bash

# Prompt user for their name
echo "Please enter your name:"
read name

# Display a greeting message
echo "Hello, $name! Welcome to our script."

In this example, the read command is used to ask the user to enter their name. The input provided by the user is stored in the name variable. Then, the script displays a personalized greeting message using the entered name.

To take arguments for Script.

In Bash scripts, we can take command-line arguments by accessing special variables that hold the arguments provided when the script is executed. These variables are denoted by positional parameters, such as "$1," "$2," and so on.

Example: Let's create a simple script named "greet.sh" that takes two command-line arguments and uses them to display a personalized greeting message:

#!/bin/bash

# Check if two arguments are provided
if [ $# -ne 2 ]; then
  echo "Usage: $0 <first_name> <last_name>"
  exit 1
fi

# Access the command-line arguments
first_name=$1
last_name=$2

# Display a personalized greeting
echo "Hello, $first_name $last_name! Welcome:)"

Usage:

To use this script, you can execute it from the terminal with two arguments as follows.

ubuntu@ip-172-xx-xx-xxx:~$ ./greet.sh Tushar Yadav

Output:

Hello, Tushar Yadav! Welcome :)

This script will also make sure that the user gives the required number of input while executing the script, so let's now understand the conditional statement for it.

• $#: This special variable holds the number of command-line arguments passed to the script. For example, if you execute the script with bashgreet.sh arg1 arg2, then $# will be equal to 2.

• -ne: This is a comparison operator that stands for "not equal." It is used to compare the value on the left side with the value on the right side. In this case, it checks if the number of command-line arguments ($#) is not equal to 2.

• then: This keyword is used to indicate the start of the code block to be executed if the condition in the if statement is true.

• echo "Usage: $0 <first_name> <last_name>": If the number of command-line arguments is not equal to 2, the script displays a usage message to guide the user on how to use the script. The special variable $0 holds the name of the script itself.

• exit 1: This command exits the script with an exit status of 1. An exit status of 0 usually indicates successful execution, while a non-zero exit status typically indicates an error or failure. By convention, a non-zero exit status signifies that something went wrong during script execution.

• fi: This keyword marks the end of the if statement block.

Using For loop to Iterate over Files.

In linux, we can use 'For' loop to Iterate over files in a directory so let's understand It with the help of an example.

Example:

my_files/
  ├── file1.txt
  ├── file2.txt
  └── file3.txt

#!/bin/bash

# Directory path
directory=/home/Ubuntu/*

# Iterate over files in the directory
for file in $directory 
do
    echo $file
done

In this example, the for loop iterates over all the files in the directory. The variable file takes the value of each file in the loop, and the echo command displays the name of each processed file.

Understanding "If" statement.

The if statement is a key conditional construct used in scripting that enables you to run particular commands or blocks of code in response to certain circumstances.

#!/bin/bash

# Read a number from user input
echo "Enter a number:"
read num

# Check if the number is positive or not
if [ "$num" -gt 0 ]
then
    echo "The number is positive."
else
    echo "The number is not positive."
fi

In this example, the script first prompts the user to enter a number. The "greater than" sign (-gt) is used in the if statement to determine whether the entered number (num) is greater than zero. If the condition is satisfied—that is, if the number is positive—the script will say "The number is positive." The script displays "The number is not positive" if the condition is false (that is, the number is zero or negative).

Script to Add User

#!/bin/bash

# Prompt for the new username
echo "Enter the new username:"
read username

# Check if the username is empty
if [ -z "$username" ]; then
  echo "Error: Username cannot be empty."
  exit 1
fi

# Prompt for the user's full name
echo "Enter the full name of the user:"
read full_name

# Create the new user
useradd -m -c "$full_name" "$username"

# Check if the user creation was successful
if [ $? -eq 0 ]; then
  echo "User '$username' has been added successfully."
else
  echo "Error: Failed to add user '$username'."
fi

1. #!/bin/bash: The shebang line specifies the interpreter to be used, which is /bin/bash in this case.

2. echo "Enter the new username:": The script prompts the user to enter a new username.

3. read username: The read command reads the user's input and assigns it to the username variable.

4. if [ -z "$username" ]; then: The script checks if the username variable is empty using the -z test. If it's empty, it means the user didn't provide a username, and the script displays an error message and exits with status 1.

5. echo "Enter the full name of the user:": The script prompts the user to enter the full name of the new user.

6. read full_name: The read command reads the user's input and assigns it to the full_name variable.

7. useradd -m -c "$full_name" "$username": The script uses the useradd command to create the new user. The -m option creates the user's home directory, and the -c option sets the user's comment (full name).

8. if [ $? -eq 0 ]; then: After creating the user, the script checks the exit status of the useradd command. If it's equal to 0 (indicating success), it displays a success message. Otherwise, it displays an error message.

Script to take Backup

#!/bin/bash

src=/home/Ubuntu/bash/scripts
trg=/home/Ubuntu/backup/

filename=$(date + '%d-%m-%y').tar.gz

echo "Backup Started"
tar -cvf $trg/$filename $src

1. #!/bin/bash: The shebang line specifies the interpreter to be used, which is /bin/bash in this case.

2. src="/home/Ubuntu/bash/scripts": This line sets the source directory that contains the files you want to back up. Make sure to replace this path with the actual directory you want to back up.

3. trg="/home/Ubuntu/backup/": This line sets the target directory where the backup will be saved. Ensure that the specified directory exists and has appropriate permissions.

4. filename=$(date +'%d-%m-%y').tar.gz: This line uses the date command to generate a filename based on the current date in the format "dd-mm-yy". The ".tar.gz" extension indicates that the backup will be in tarball format and compressed using gzip.

5. echo "Backup Started": This line displays a message indicating that the backup process has started.

6. tar -cvf "$trg/$filename" "$src": The tar command is used to create the backup. The -c option indicates that we are creating an archive, -v enables verbose mode to show the progress, and -f specifies the filename of the resulting tarball. The $trg/$filename variable represents the full path to the backup file, and $src is the path to the source directory that contains the files to be backed up.

Scheduling script using Crontab

CronTab, a time-based job scheduler in Unix-like operating systems, empowers users to automate repetitive tasks effortlessly.

1. Understanding CronTab: CronTab is a built-in utility that manages the scheduling of tasks (cron jobs) on a system. It operates with a time-based syntax, allowing users to specify when and how often a script or command should be executed.

2. Scheduling a Script: To schedule a script using CronTab, open your system's crontab editor with the command "crontab -e." Use the syntax to define the schedule in the following format:

* * * * * /path/to/your/script.sh

The five asterisks represent the time intervals for minute, hour, day of the month, month, and day of the week, respectively. To execute your script every day at 2:30 PM, the schedule would be:

30 14 * * * /path/to/your/script.sh

An effective tool for automating script execution is CronTab, which makes it simple to schedule effective tasks. You may use CronTab to optimize your productivity and streamline your workflow while lowering manual intervention and raising job accuracy by following the instructions.

Conclusion

In conclusion, the world of advanced Linux commands and scripts opens doors to enhanced productivity and efficiency for DevOps engineers. Armed with powerful tools like grep, user management commands, and the art of scripting, you now have the ability to automate tasks, extract valuable insights, and optimize workflows. With the prowess of CronTab, you can schedule tasks to run precisely when needed, freeing up time for more strategic endeavors. Embrace these skills, and unlock the true potential of your Linux journey. Happy scripting!