AWS Project: A Guide to Shell Scripting

Shell scripting involves writing a series of commands for the Unix shell (like Bash) to automate tasks. It allows you to combine and execute commands (in a .sh file), use control structures, and manage variables to streamline operations.

In DevOps and AWS shell scripting is used for :

Automation: Streamlines deployments, config management, and monitoring, minimizing errors and saving time.
Infrastructure Management: Efficiently handles cloud resources, interacts with APIs (e.g., AWS CLI), and manages configurations.
Integration: Seamlessly integrates with CI/CD pipelines to deploy and orchestrate applications across environments.
Flexibility: Customizes and extends functionalities to fit project needs and workflows.
Operational Efficiency: Ensures consistency, quick troubleshooting, and error handling, optimizing operational workflows.

In this project, we will be using bash scripting

Aim:

Develop a script to generate a comprehensive report on AWS resource usage including EC2 instances, S3 buckets, IAM users, and Lambda functions, enhancing visibility and management of cloud resources.

Implementation:

Find the code at https://github.com/roishub/aws-bash-script-projects/blob/main/Advanced/aws-resource-monitoring-advanced.sh

Explanation:

#!/bin/bash

Shebang: Indicates the script should be run using the Bash shell.

################################
# Author: Rois 
# Date: 12th-june-24
# Version: v2
# This script will report the AWS resource usage 
# We will monitor EC2, S3, IAM, Lambda
######################################################

Comment Block: Provides metadata about the script including author, date, version, and purpose.

# Define AWS region and profile (if any)
REGION="us-east-1"
PROFILE="default"

Variables: Sets default AWS region and profile.

# Enhanced output formatting
echo "========================================="
echo "AWS Resource Usage Report"
echo "Date: $(date)"
echo "Region: $REGION"
echo "========================================="

Output Formatting: Prints a formatted header for the report including date and region.

# Error handling function
error_handler() {
    echo "Error occurred in script at line: $1"
    exit 1
}

Error Handler: Defines a function to handle errors, printing the line number where the error occurred.

trap 'error_handler $LINENO' ERR

Trap Errors: Uses the error handler function to catch any errors and execute it.

# List S3 buckets with sizes
echo "Print list of S3 buckets:"
aws s3 ls --profile $PROFILE --region $REGION
for bucket in $(aws s3api list-buckets --query "Buckets[].Name" --output text --profile $PROFILE --region $REGION); do
    echo "Bucket: $bucket"
    aws s3 ls s3://$bucket --recursive --summarize --human-readable --profile $PROFILE --region $REGION | grep "Total Size"
done

S3 Buckets: Lists all S3 buckets and prints their total sizes.

# List EC2 instances with status
echo "Print list of EC2 instances:"
aws ec2 describe-instances --profile $PROFILE --region $REGION --query "Reservations[].Instances[].{ID:InstanceId,Type:InstanceType,State:State.Name,Tags:Tags}" --output table

EC2 Instances: Lists all EC2 instances with their ID, type, state, and tags.

# List AWS Lambda functions with memory size
echo "Print list of Lambda functions:"
aws lambda list-functions --profile $PROFILE --region $REGION --query "Functions[].{Name:FunctionName,MemorySize:MemorySize}" --output table

Lambda Functions: Lists all Lambda functions with their names and memory sizes.

# List IAM users
echo "Print list of IAM users:"
aws iam list-users --profile $PROFILE --query "Users[].{UserName:UserName,CreateDate:CreateDate}" --output table

IAM Users: Lists all IAM users with their usernames and creation dates.

# List IAM roles
echo "Print list of IAM roles:"
aws iam list-roles --profile $PROFILE --query "Roles[].{RoleName:RoleName,CreateDate:CreateDate}" --output table

IAM Roles: Lists all IAM roles with their names and creation dates.

# Send email notification (requires configured AWS SES)
# echo "Sending email notification..."
# aws ses send-email --from "you@example.com" --destination "ToAddresses=recipient@example.com" --message "Subject={Data=AWS Resource Usage Report,Charset=utf8},Body={Text={Data=$(cat /path/to/report.log),Charset=utf8}}" --profile $PROFILE --region $REGION

Email Notification: (Commented out) Example command to send an email notification using AWS SES.

# Redirect output to a log file with timestamp
LOG_FILE="aws_resource_report_$(date +'%Y%m%d_%H%M%S').log"
exec > >(tee -a $LOG_FILE) 2>&1

Logging: Redirects output to a log file with a timestamp.

# CSV Export (example for EC2 instances)
echo "Exporting EC2 instance list to CSV..."
aws ec2 describe-instances --profile $PROFILE --region $REGION --query "Reservations[].Instances[].{ID:InstanceId,Type:InstanceType,State:State.Name}" --output text | tr -s '\t' ',' > ec2_instances.csv

CSV Export: Exports the list of EC2 instances to a CSV file.

echo "AWS Resource Usage Report Completed"

Conclusion:

By automating AWS resource monitoring with this script, we achieve efficient, accurate, and timely visibility into our cloud infrastructure, enhancing management capabilities and ensuring optimal resource utilization.