Skip to content
MechCrate

Shell Scripting Guide: DevOps Excellence

Version: 1.0 Last Updated: 2026-01-08 Purpose: Comprehensive guide to production-grade shell scripting for DevOps, emphasizing security, modularity, cross-platform compatibility, and maintainability

Table of Contents

  1. Core Principles
  2. Cross-Platform Compatibility
  3. Security Best Practices
  4. Script Architecture & Modularity
  5. Error Handling & Validation
  6. Testing Strategies
  7. Documentation Standards
  8. DevOps Integration Patterns
  9. Quality Gates & Validation
  10. Common Patterns & Anti-Patterns

1. Core Principles

1.1 Framework Alignment

Shell scripts in our ecosystem follow the same rigorous standards as application code:

Primary Directive: “Security > reliability > maintainability > convenience”

Key Principles:

  • Evidence > Assumptions: Validate all inputs, environment state, and dependencies
  • Fail Fast, Fail Explicitly: Detect errors immediately with actionable context
  • Modularity: Single Responsibility Principle applies to scripts and functions
  • Composability: Scripts should be pipeline-friendly and reusable
  • Observability: Comprehensive logging with structured output
  • Idempotency: Scripts should be safely re-runnable

1.2 Script Classification

operational_scripts:
  purpose: "Production operations, deployments, automation"
  requirements: "Maximum reliability, comprehensive error handling"
  examples: ["deploy.sh", "backup.sh", "health-check.sh"]


development_scripts:
  purpose: "Local development, testing, tooling"
  requirements: "Developer experience, clear feedback"
  examples: ["dev-setup.sh", "test-runner.sh", "db-seed.sh"]


infrastructure_scripts:
  purpose: "System provisioning, configuration management"
  requirements: "Idempotency, auditability, cross-platform"
  examples: ["setup-node.sh", "configure-firewall.sh"]


utility_scripts:
  purpose: "Helper functions, shared libraries"
  requirements: "Pure functions, composability, zero side effects"
  examples: ["lib/logger.sh", "lib/validators.sh"]

1.3 Quality Standards

All production scripts must meet:

  • Shellcheck: Zero warnings at default strictness
  • POSIX Compliance: Or explicit bash requirement documentation
  • Error Handling: Comprehensive with exit codes
  • Documentation: Inline comments + usage help
  • Testing: Critical paths covered with bats or shunit2
  • Security Review: Pass security checklist (see §3)

2. Cross-Platform Compatibility

2.1 Platform Detection

Reliable platform detection pattern:

#!/usr/bin/env bash
# detect-platform.sh - Portable platform detection


# ✅ Robust platform detection
detect_platform() {
    local platform
    case "$(uname -s)" in
        Linux*)     platform="linux";;
        Darwin*)    platform="macos";;
        CYGWIN*|MINGW*|MSYS*) platform="windows";;
        FreeBSD*)   platform="freebsd";;
        OpenBSD*)   platform="openbsd";;
        *)          platform="unknown";;
    esac
    echo "${platform}"
}


# ✅ Architecture detection
detect_arch() {
    local arch
    case "$(uname -m)" in
        x86_64|amd64)   arch="amd64";;
        aarch64|arm64)  arch="arm64";;
        armv7l)         arch="armv7";;
        i386|i686)      arch="386";;
        *)              arch="unknown";;
    esac
    echo "${arch}"
}


# Usage
PLATFORM=$(detect_platform)
ARCH=$(detect_arch)


echo "Running on: ${PLATFORM}/${ARCH}"

2.2 POSIX vs Bash

Decision Matrix:

Use POSIX (#!/bin/sh)Use Bash (#!/usr/bin/env bash)
Maximum portability neededAdvanced features required
Alpine/BusyBox environmentsArrays, associative arrays needed
Minimal dependenciesExtended pattern matching
CI/CD containersLocal development scripts
System init scriptsComplex data processing

POSIX-Compatible Example:

#!/bin/sh
# posix-example.sh - Maximum portability


set -eu  # POSIX supports these


# ✅ POSIX-compatible string operations
string_contains() {
    case "$1" in
        *"$2"*) return 0;;
        *) return 1;;
    esac
}


# ✅ POSIX-compatible loops
for item in "$@"; do
    if string_contains "$item" "test"; then
        echo "Found test item: $item"
    fi
done

Bash-Specific Example:

#!/usr/bin/env bash
# bash-example.sh - Requires bash 4.0+


set -euo pipefail


# ✅ Bash arrays
declare -a services=("api" "worker" "scheduler")


# ✅ Bash associative arrays
declare -A config=(
    [env]="production"
    [region]="us-east-1"
    [replicas]="3"
)


# ✅ Bash parameter expansion
for service in "${services[@]}"; do
    echo "Deploying ${service} to ${config[region]}"
done


# ✅ Bash extended globbing
shopt -s extglob nullglob
for file in !(*.md|*.txt); do
    echo "Processing: $file"
done

2.3 Command Portability

Platform-Specific Command Handling:

portable-commands.sh
#!/usr/bin/env bash
# ✅ Handle GNU vs BSD command differences
portable_sed() {
    if sed --version >/dev/null 2>&1; then
        # GNU sed
        sed -i "$@"
    else
        # BSD sed (macOS)
        sed -i '' "$@"
    fi
}


# ✅ Portable readlink (for getting absolute paths)
portable_realpath() {
    local path="$1"


    if command -v realpath >/dev/null 2>&1; then
        # Linux: use realpath
        realpath "$path"
    elif command -v greadlink >/dev/null 2>&1; then
        # macOS with coreutils: use greadlink
        greadlink -f "$path"
    else
        # Fallback: Python
        python3 -c "import os; print(os.path.realpath('$path'))"
    fi
}


# ✅ Portable date handling
portable_date() {
    if date --version >/dev/null 2>&1; then
        # GNU date
        date -d "$1" "$2"
    else
        # BSD date (macOS)
        date -j -f "%Y-%m-%d" "$1" "$2"
    fi
}


# ✅ Cross-platform clipboard
copy_to_clipboard() {
    local content="$1"


    if command -v pbcopy >/dev/null 2>&1; then
        # macOS
        echo "$content" | pbcopy
    elif command -v xclip >/dev/null 2>&1; then
        # Linux with X11
        echo "$content" | xclip -selection clipboard
    elif command -v wl-copy >/dev/null 2>&1; then
        # Linux with Wayland
        echo "$content" | wl-copy
    else
        echo "Error: No clipboard utility found" >&2
        return 1
    fi
}

2.4 Dependency Management

Graceful dependency handling:

check-dependencies.sh
#!/usr/bin/env bash
# ✅ Required dependencies (fail if missing)
declare -a REQUIRED_DEPS=(
    "git"
    "docker"
    "jq"
)


# ✅ Optional dependencies (warn if missing)
declare -A OPTIONAL_DEPS=(
    [gh]="GitHub CLI - required for PR automation"
    [terraform]="Infrastructure provisioning"
    [kubectl]="Kubernetes deployments"
)


check_required_dependencies() {
    local missing_deps=()


    for cmd in "${REQUIRED_DEPS[@]}"; do
        if ! command -v "$cmd" >/dev/null 2>&1; then
            missing_deps+=("$cmd")
        fi
    done


    if [ ${#missing_deps[@]} -gt 0 ]; then
        echo "Error: Missing required dependencies:" >&2
        printf '  - %s\n' "${missing_deps[@]}" >&2
        return 1
    fi
}


check_optional_dependencies() {
    for cmd in "${!OPTIONAL_DEPS[@]}"; do
        if ! command -v "$cmd" >/dev/null 2>&1; then
            echo "Warning: Optional dependency '$cmd' not found" >&2
            echo "  Purpose: ${OPTIONAL_DEPS[$cmd]}" >&2
        fi
    done
}


# Run checks
check_required_dependencies || exit 1
check_optional_dependencies

3. Security Best Practices

3.1 Security Principles

Zero Trust Model for Scripts:

  • Never trust input: Validate everything
  • Never trust environment: Verify state before operations
  • Principle of Least Privilege: Request minimum necessary permissions
  • Defense in Depth: Multiple validation layers
  • Fail Securely: Default to safe state on errors

3.2 Input Validation & Sanitization

input-validation.sh
#!/usr/bin/env bash
# ❌ DANGEROUS: Command injection vulnerability
dangerous_example() {
    local user_input="$1"
    eval "echo $user_input"  # NEVER DO THIS
    sh -c "ls $user_input"    # NEVER DO THIS
}


# ✅ SAFE: Proper input validation
validate_alphanumeric() {
    local input="$1"
    if [[ ! "$input" =~ ^[a-zA-Z0-9_-]+$ ]]; then
        echo "Error: Invalid input. Only alphanumeric, dash, underscore allowed." >&2
        return 1
    fi
}


# ✅ SAFE: Whitelist validation
validate_environment() {
    local env="$1"
    case "$env" in
        dev|staging|production) return 0;;
        *)
            echo "Error: Invalid environment. Must be: dev, staging, production" >&2
            return 1
            ;;
    esac
}


# ✅ SAFE: Path sanitization
sanitize_path() {
    local path="$1"


    # Remove leading/trailing whitespace
    path=$(echo "$path" | xargs)


    # Prevent directory traversal
    if [[ "$path" == *".."* ]]; then
        echo "Error: Path traversal detected" >&2
        return 1
    fi


    # Ensure path is within allowed directory
    local base_dir="/var/app"
    local full_path="${base_dir}/${path}"
    local canonical_path
    canonical_path=$(portable_realpath "$full_path")


    if [[ ! "$canonical_path" == "$base_dir"* ]]; then
        echo "Error: Path outside allowed directory" >&2
        return 1
    fi


    echo "$canonical_path"
}


# ✅ SAFE: URL validation
validate_url() {
    local url="$1"


    # Basic URL pattern validation
    if [[ ! "$url" =~ ^https?://[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}(/.*)?$ ]]; then
        echo "Error: Invalid URL format" >&2
        return 1
    fi


    # Additional security: block private IP ranges
    if [[ "$url" =~ (localhost|127\.0\.0\.1|10\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.) ]]; then
        echo "Error: Private IP ranges not allowed" >&2
        return 1
    fi
}

3.3 Secrets Management

secrets-management.sh
#!/usr/bin/env bash
# ❌ NEVER: Hardcode secrets
DANGEROUS_API_KEY="sk-1234567890abcdef"  # NEVER DO THIS


# ❌ NEVER: Log secrets
echo "API Key: $API_KEY"  # NEVER DO THIS


# ✅ SAFE: Load from secure environment
load_secrets() {
    if [ -z "$API_KEY" ]; then
        echo "Error: API_KEY environment variable not set" >&2
        echo "Load from: vault, AWS Secrets Manager, or secure env file" >&2
        return 1
    fi
}


# ✅ SAFE: Use environment files securely
load_env_file() {
    local env_file="${1:-.env}"


    if [ ! -f "$env_file" ]; then
        echo "Error: Environment file not found: $env_file" >&2
        return 1
    fi


    # Check file permissions (should be 600 or 400)
    local perms
    perms=$(stat -c %a "$env_file" 2>/dev/null || stat -f %A "$env_file")
    if [[ ! "$perms" =~ ^[46]00$ ]]; then
        echo "Warning: Insecure permissions on $env_file (found: $perms, expected: 600)" >&2
    fi


    # Load without exposing in process list
    set -a
    # shellcheck source=/dev/null
    source "$env_file"
    set +a
}


# ✅ SAFE: Mask secrets in logs
mask_secret() {
    local secret="$1"
    local length=${#secret}


    if [ "$length" -le 4 ]; then
        echo "****"
    else
        local prefix="${secret:0:2}"
        local suffix="${secret: -2}"
        echo "${prefix}****${suffix}"
    fi
}


# ✅ SAFE: Clean up secrets on exit
cleanup_secrets() {
    unset API_KEY
    unset DATABASE_PASSWORD
    unset JWT_SECRET
}
trap cleanup_secrets EXIT

3.4 File Operations Security

secure-file-ops.sh
#!/usr/bin/env bash
# ✅ Create temporary files securely
create_temp_file() {
    local temp_file
    temp_file=$(mktemp) || {
        echo "Error: Failed to create temporary file" >&2
        return 1
    }


    # Set restrictive permissions immediately
    chmod 600 "$temp_file"


    echo "$temp_file"
}


# ✅ Secure temporary directory
create_temp_dir() {
    local temp_dir
    temp_dir=$(mktemp -d) || {
        echo "Error: Failed to create temporary directory" >&2
        return 1
    }


    chmod 700 "$temp_dir"
    echo "$temp_dir"
}


# ✅ Atomic file operations
atomic_write() {
    local target_file="$1"
    local content="$2"
    local temp_file


    temp_file=$(create_temp_file) || return 1


    # Write to temp file
    echo "$content" > "$temp_file" || {
        rm -f "$temp_file"
        return 1
    }


    # Atomic move
    mv -f "$temp_file" "$target_file" || {
        rm -f "$temp_file"
        return 1
    }
}


# ✅ Safe file deletion
secure_delete() {
    local file="$1"


    if [ ! -f "$file" ]; then
        echo "Warning: File not found: $file" >&2
        return 1
    fi


    # Overwrite before deletion (for sensitive data)
    if [ -w "$file" ]; then
        dd if=/dev/urandom of="$file" bs=1k count=$(du -k "$file" | cut -f1) 2>/dev/null
    fi


    rm -f "$file"
}

3.5 Process Security

process-security.sh
#!/usr/bin/env bash
# ✅ Safe command execution (avoid shell injection)
safe_execute() {
    local -a cmd=("$@")


    # Use array to prevent word splitting and globbing
    "${cmd[@]}" || {
        echo "Error: Command failed with exit code $?" >&2
        return 1
    }
}


# ✅ Execute with timeout
execute_with_timeout() {
    local timeout="$1"
    shift
    local -a cmd=("$@")


    if command -v timeout >/dev/null 2>&1; then
        # GNU timeout
        timeout "$timeout" "${cmd[@]}"
    else
        # Fallback for systems without timeout
        "${cmd[@]}" &
        local pid=$!


        ( sleep "$timeout"; kill -TERM "$pid" 2>/dev/null ) &
        local killer=$!


        wait "$pid"
        local exit_code=$?
        kill "$killer" 2>/dev/null


        return $exit_code
    fi
}


# ✅ Safely handle background processes
run_with_cleanup() {
    local -a cmd=("$@")
    local pid


    # Start background process
    "${cmd[@]}" &
    pid=$!


    # Ensure cleanup on exit
    trap "kill $pid 2>/dev/null; wait $pid 2>/dev/null" EXIT INT TERM


    # Wait for completion
    wait "$pid"
}


# ✅ Drop privileges when possible
drop_privileges() {
    local target_user="$1"


    if [ "$(id -u)" -eq 0 ]; then
        if command -v sudo >/dev/null 2>&1; then
            exec sudo -u "$target_user" "$0" "$@"
        elif command -v su >/dev/null 2>&1; then
            exec su -s /bin/bash "$target_user" -c "$0 $*"
        else
            echo "Error: Cannot drop privileges - no sudo or su available" >&2
            return 1
        fi
    fi
}

4. Script Architecture & Modularity

4.1 Single Responsibility Principle

Script Organization:

# ❌ BAD: Monolithic script doing everything
#!/usr/bin/env bash
# deploy-everything.sh - 2000 lines of mixed concerns


# ✅ GOOD: Modular architecture
project-root/
├── scripts/
│   ├── deploy.sh              # Main orchestration script
│   ├── lib/                   # Shared libraries
│   │   ├── logger.sh          # Logging utilities
│   │   ├── validators.sh      # Input validation
│   │   ├── aws.sh             # AWS operations
│   │   └── notifications.sh   # Slack/email alerts
│   ├── deploy/                # Deployment modules
│   │   ├── build.sh           # Build application
│   │   ├── test.sh            # Run tests
│   │   ├── backup.sh          # Backup before deploy
│   │   └── rollback.sh        # Rollback procedures
│   └── utils/                 # Utility scripts
│       ├── health-check.sh    # Service health checks
│       └── cleanup.sh         # Cleanup resources

4.2 Module Pattern

Creating reusable library modules:

# lib/logger.sh - Structured logging module


#!/usr/bin/env bash
# Prevent double-sourcing
[ -n "${LOGGER_SH_LOADED:-}" ] && return 0
readonly LOGGER_SH_LOADED=1


# Configuration
LOG_LEVEL="${LOG_LEVEL:-INFO}"
LOG_FILE="${LOG_FILE:-}"
LOG_JSON="${LOG_JSON:-false}"


# Log levels
readonly LOG_LEVEL_DEBUG=0
readonly LOG_LEVEL_INFO=1
readonly LOG_LEVEL_WARN=2
readonly LOG_LEVEL_ERROR=3
readonly LOG_LEVEL_FATAL=4


# Convert log level name to number
_log_level_to_number() {
    case "${1^^}" in
        DEBUG) echo $LOG_LEVEL_DEBUG;;
        INFO)  echo $LOG_LEVEL_INFO;;
        WARN)  echo $LOG_LEVEL_WARN;;
        ERROR) echo $LOG_LEVEL_ERROR;;
        FATAL) echo $LOG_LEVEL_FATAL;;
        *)     echo $LOG_LEVEL_INFO;;
    esac
}


# Core logging function
_log() {
    local level="$1"
    local message="$2"
    shift 2
    local context=("$@")


    local level_num
    local config_level_num
    level_num=$(_log_level_to_number "$level")
    config_level_num=$(_log_level_to_number "$LOG_LEVEL")


    # Skip if below configured level
    [ "$level_num" -lt "$config_level_num" ] && return 0


    local timestamp
    timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")


    if [ "$LOG_JSON" = "true" ]; then
        # JSON structured logging
        printf '{"timestamp":"%s","level":"%s","message":"%s"' \
            "$timestamp" "$level" "$message"


        # Add context fields
        for ctx in "${context[@]}"; do
            local key="${ctx%%=*}"
            local value="${ctx#*=}"
            printf ',"%s":"%s"' "$key" "$value"
        done


        printf '}\n'
    else
        # Human-readable logging
        local color_reset="\033[0m"
        local color
        case "$level" in
            DEBUG) color="\033[0;36m";;  # Cyan
            INFO)  color="\033[0;32m";;  # Green
            WARN)  color="\033[0;33m";;  # Yellow
            ERROR) color="\033[0;31m";;  # Red
            FATAL) color="\033[1;31m";;  # Bold Red
        esac


        printf "%b[%s] %-5s %s%b" \
            "$color" "$timestamp" "$level" "$message" "$color_reset"


        # Add context
        if [ ${#context[@]} -gt 0 ]; then
            printf " |"
            for ctx in "${context[@]}"; do
                printf " %s" "$ctx"
            done
        fi


        printf "\n"
    fi | if [ -n "$LOG_FILE" ]; then
        tee -a "$LOG_FILE"
    else
        cat
    fi
}


# Public API
log_debug() { _log "DEBUG" "$@"; }
log_info()  { _log "INFO" "$@"; }
log_warn()  { _log "WARN" "$@"; }
log_error() { _log "ERROR" "$@"; }
log_fatal() { _log "FATAL" "$@"; exit 1; }


# Export functions
export -f log_debug log_info log_warn log_error log_fatal

Using the logger module:

#!/usr/bin/env bash
# deploy.sh - Main deployment script


set -euo pipefail


# Get script directory
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"


# Source libraries
# shellcheck source=lib/logger.sh
source "${SCRIPT_DIR}/lib/logger.sh"


# Configure logging
export LOG_LEVEL="INFO"
export LOG_FILE="/var/log/deploy.log"
export LOG_JSON="false"


main() {
    log_info "Starting deployment" "env=production" "version=1.2.3"


    # Deployment logic
    if ! perform_deployment; then
        log_error "Deployment failed" "stage=deploy"
        return 1
    fi


    log_info "Deployment completed successfully"
}


main "$@"

4.3 Configuration Management

# lib/config.sh - Configuration module


#!/usr/bin/env bash
[ -n "${CONFIG_SH_LOADED:-}" ] && return 0
readonly CONFIG_SH_LOADED=1


# Configuration hierarchy (later sources override earlier)
declare -a CONFIG_SOURCES=(
    "/etc/app/config.env"           # System-wide defaults
    "${HOME}/.config/app/config.env" # User defaults
    "${PWD}/.env"                    # Project-specific
    "${ENV_FILE:-}"                  # Explicit override
)


# Load configuration with validation
load_config() {
    local -A loaded_configs=()


    for config_file in "${CONFIG_SOURCES[@]}"; do
        if [ -n "$config_file" ] && [ -f "$config_file" ]; then
            log_debug "Loading config from: $config_file"


            # Validate before loading
            validate_config_file "$config_file" || {
                log_error "Invalid config file: $config_file"
                return 1
            }


            # Load safely
            set -a
            # shellcheck source=/dev/null
            source "$config_file"
            set +a


            loaded_configs["$config_file"]=1
        fi
    done


    # Log which configs were loaded
    if [ ${#loaded_configs[@]} -gt 0 ]; then
        log_info "Loaded ${#loaded_configs[@]} config files"
    else
        log_warn "No config files loaded"
    fi
}


# Validate config file format
validate_config_file() {
    local file="$1"


    # Check syntax (no execution)
    bash -n "$file" 2>/dev/null || {
        log_error "Syntax error in config file: $file"
        return 1
    }


    # Check for dangerous patterns
    if grep -qE '(rm -rf|eval |exec |>\s*/dev/)' "$file"; then
        log_error "Dangerous commands detected in config: $file"
        return 1
    fi


    return 0
}


# Get required config value
config_require() {
    local var_name="$1"
    local var_value="${!var_name:-}"


    if [ -z "$var_value" ]; then
        log_error "Required configuration missing: $var_name"
        return 1
    fi


    echo "$var_value"
}


# Get config value with default
config_get() {
    local var_name="$1"
    local default="${2:-}"
    local var_value="${!var_name:-$default}"


    echo "$var_value"
}

4.4 Function Design

Pure functions and side effect isolation:

function-design.sh
#!/usr/bin/env bash
# ✅ GOOD: Pure function - no side effects
calculate_sha256() {
    local file="$1"
    sha256sum "$file" | awk '{print $1}'
}


# ✅ GOOD: Clear input/output contract
validate_semver() {
    local version="$1"


    if [[ "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?(\+[a-zA-Z0-9.]+)?$ ]]; then
        return 0
    else
        return 1
    fi
}


# ✅ GOOD: Side effects isolated and documented
# Side effects: Creates directory, writes file, modifies filesystem
initialize_project() {
    local project_dir="$1"


    mkdir -p "$project_dir" || return 1
    touch "$project_dir/README.md" || return 1


    log_info "Initialized project" "dir=$project_dir"
}


# ✅ GOOD: Function composition
process_deployment() {
    local version="$1"


    validate_semver "$version" || return 1


    local artifact_path
    artifact_path=$(build_artifact "$version") || return 1


    local checksum
    checksum=$(calculate_sha256 "$artifact_path") || return 1


    upload_artifact "$artifact_path" "$checksum" || return 1
}


# ✅ GOOD: Named parameters pattern
deploy_service() {
    local service=""
    local environment=""
    local version=""
    local dry_run="false"


    # Parse named parameters
    while [ $# -gt 0 ]; do
        case "$1" in
            --service)    service="$2"; shift 2;;
            --env)        environment="$2"; shift 2;;
            --version)    version="$2"; shift 2;;
            --dry-run)    dry_run="true"; shift;;
            *)
                echo "Unknown parameter: $1" >&2
                return 1
                ;;
        esac
    done


    # Validate required parameters
    [ -z "$service" ] && { echo "Error: --service required" >&2; return 1; }
    [ -z "$environment" ] && { echo "Error: --env required" >&2; return 1; }
    [ -z "$version" ] && { echo "Error: --version required" >&2; return 1; }


    # Execute with parameters
    log_info "Deploying service" \
        "service=$service" \
        "env=$environment" \
        "version=$version" \
        "dry_run=$dry_run"
}

4.5 Script Template

Production-ready script template:

#!/usr/bin/env bash
#
# script-name.sh - Brief description of what this script does
#
# Usage: script-name.sh [options] <arguments>
#
# Options:
#   -h, --help         Show this help message
#   -v, --verbose      Enable verbose output
#   -d, --dry-run      Run without making changes
#   -e, --env ENV      Environment (dev|staging|production)
#
# Examples:
#   script-name.sh --env production
#   script-name.sh --dry-run --verbose
#
# Exit codes:
#   0 - Success
#   1 - General error
#   2 - Invalid arguments
#   3 - Dependency missing
#


set -euo pipefail
IFS=$'\n\t'


# Script metadata
readonly SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly SCRIPT_VERSION="1.0.0"


# Configuration defaults
VERBOSE="${VERBOSE:-false}"
DRY_RUN="${DRY_RUN:-false}"
ENVIRONMENT="${ENVIRONMENT:-dev}"


# Load libraries
# shellcheck source=lib/logger.sh
source "${SCRIPT_DIR}/lib/logger.sh"


#######################################
# Display usage information
# Globals: None
# Arguments: None
# Outputs: Usage text to stdout
#######################################
usage() {
    cat << EOF
Usage: ${SCRIPT_NAME} [options] <arguments>


Options:
    -h, --help         Show this help message
    -v, --verbose      Enable verbose output
    -d, --dry-run      Run without making changes
    -e, --env ENV      Environment (dev|staging|production)


Examples:
    ${SCRIPT_NAME} --env production
    ${SCRIPT_NAME} --dry-run --verbose


Exit codes:
    0 - Success
    1 - General error
    2 - Invalid arguments
    3 - Dependency missing
EOF
}


#######################################
# Check required dependencies
# Globals: None
# Arguments: None
# Returns: 0 if all dependencies present, 3 otherwise
#######################################
check_dependencies() {
    local -a required_commands=("git" "jq" "curl")
    local missing_commands=()


    for cmd in "${required_commands[@]}"; do
        if ! command -v "$cmd" >/dev/null 2>&1; then
            missing_commands+=("$cmd")
        fi
    done


    if [ ${#missing_commands[@]} -gt 0 ]; then
        log_error "Missing required dependencies: ${missing_commands[*]}"
        return 3
    fi
}


#######################################
# Parse command line arguments
# Globals: VERBOSE, DRY_RUN, ENVIRONMENT
# Arguments: All script arguments ($@)
# Returns: 0 on success, 2 on invalid arguments
#######################################
parse_arguments() {
    while [ $# -gt 0 ]; do
        case "$1" in
            -h|--help)
                usage
                exit 0
                ;;
            -v|--verbose)
                VERBOSE="true"
                export LOG_LEVEL="DEBUG"
                shift
                ;;
            -d|--dry-run)
                DRY_RUN="true"
                shift
                ;;
            -e|--env)
                ENVIRONMENT="$2"
                shift 2
                ;;
            -*)
                log_error "Unknown option: $1"
                usage >&2
                return 2
                ;;
            *)
                break
                ;;
        esac
    done


    # Validate environment
    case "$ENVIRONMENT" in
        dev|staging|production) ;;
        *)
            log_error "Invalid environment: $ENVIRONMENT"
            return 2
            ;;
    esac
}


#######################################
# Cleanup function (called on EXIT)
# Globals: None
# Arguments: None
#######################################
cleanup() {
    local exit_code=$?


    # Cleanup temporary files
    if [ -n "${TEMP_DIR:-}" ] && [ -d "$TEMP_DIR" ]; then
        rm -rf "$TEMP_DIR"
    fi


    # Log exit
    if [ $exit_code -eq 0 ]; then
        log_info "Script completed successfully"
    else
        log_error "Script failed with exit code: $exit_code"
    fi


    exit $exit_code
}


#######################################
# Main script logic
# Globals: All configuration variables
# Arguments: Remaining positional arguments
# Returns: 0 on success, 1 on error
#######################################
main() {
    log_info "Starting ${SCRIPT_NAME}" \
        "version=${SCRIPT_VERSION}" \
        "env=${ENVIRONMENT}"


    # Your script logic here
    if [ "$DRY_RUN" = "true" ]; then
        log_info "DRY RUN: Would execute deployment"
        return 0
    fi


    # Actual operations
    log_info "Executing deployment..."


    return 0
}


# Trap cleanup on exit
trap cleanup EXIT


# Execute
check_dependencies || exit $?
parse_arguments "$@" || exit $?
main "$@"

5. Error Handling & Validation

5.1 Robust Error Handling

error-handling.sh
#!/usr/bin/env bash
# ✅ Set strict error handling
set -euo pipefail
IFS=$'\n\t'


# Explanation:
# -e: Exit on any command failure
# -u: Exit on undefined variable usage
# -o pipefail: Pipelines fail if any command fails
# IFS: Prevent word splitting issues


# ✅ Better error handling with custom trap
error_handler() {
    local line_no="$1"
    local bash_lineno="$2"
    local command="$3"
    local exit_code="$4"


    log_error "Command failed" \
        "line=$line_no" \
        "command=$command" \
        "exit_code=$exit_code"


    # Optional: Send alert
    send_error_notification "$command" "$exit_code"
}


# Set error trap
trap 'error_handler ${LINENO} ${BASH_LINENO} "$BASH_COMMAND" $?' ERR


# ✅ Defensive programming patterns
safe_function() {
    # Validate preconditions
    [ $# -eq 0 ] && { log_error "No arguments provided"; return 1; }


    local input="$1"
    [ -z "$input" ] && { log_error "Empty input"; return 1; }


    # Main logic with error checking
    if ! some_command "$input"; then
        log_error "Command failed for input: $input"
        return 1
    fi


    # Validate postconditions
    if [ ! -f "expected-output.txt" ]; then
        log_error "Expected output file not created"
        return 1
    fi


    return 0
}


# ✅ Retry logic with exponential backoff
retry_with_backoff() {
    local max_attempts="$1"
    local base_delay="$2"
    shift 2
    local -a command=("$@")


    local attempt=1
    local delay="$base_delay"


    while [ $attempt -le "$max_attempts" ]; do
        log_debug "Attempt $attempt of $max_attempts"


        if "${command[@]}"; then
            return 0
        fi


        if [ $attempt -lt "$max_attempts" ]; then
            log_warn "Command failed, retrying in ${delay}s..." \
                "attempt=$attempt"
            sleep "$delay"


            # Exponential backoff
            delay=$((delay * 2))
        fi


        attempt=$((attempt + 1))
    done


    log_error "Command failed after $max_attempts attempts"
    return 1
}


# ✅ Circuit breaker pattern
declare -A CIRCUIT_BREAKERS=()
readonly CIRCUIT_BREAKER_THRESHOLD=5
readonly CIRCUIT_BREAKER_TIMEOUT=60


circuit_breaker_execute() {
    local circuit_name="$1"
    shift
    local -a command=("$@")


    # Check if circuit is open
    local failure_count="${CIRCUIT_BREAKERS[$circuit_name]:-0}"
    local last_failure="${CIRCUIT_BREAKERS[${circuit_name}_last]:-0}"
    local current_time
    current_time=$(date +%s)


    if [ "$failure_count" -ge "$CIRCUIT_BREAKER_THRESHOLD" ]; then
        local time_since_last=$((current_time - last_failure))


        if [ "$time_since_last" -lt "$CIRCUIT_BREAKER_TIMEOUT" ]; then
            log_error "Circuit breaker open for: $circuit_name" \
                "failures=$failure_count" \
                "retry_in=$((CIRCUIT_BREAKER_TIMEOUT - time_since_last))s"
            return 1
        else
            # Reset circuit breaker
            log_info "Circuit breaker reset: $circuit_name"
            CIRCUIT_BREAKERS[$circuit_name]=0
        fi
    fi


    # Execute command
    if "${command[@]}"; then
        # Success - reset failure count
        CIRCUIT_BREAKERS[$circuit_name]=0
        return 0
    else
        # Failure - increment counter
        CIRCUIT_BREAKERS[$circuit_name]=$((failure_count + 1))
        CIRCUIT_BREAKERS[${circuit_name}_last]=$current_time


        log_warn "Circuit breaker failure recorded" \
            "circuit=$circuit_name" \
            "count=${CIRCUIT_BREAKERS[$circuit_name]}"


        return 1
    fi
}

5.2 Validation Patterns

validation-patterns.sh
#!/usr/bin/env bash
# ✅ Pre-execution validation checklist
validate_environment() {
    log_info "Validating environment..."


    # Check required environment variables
    local -a required_vars=(
        "AWS_REGION"
        "ENVIRONMENT"
        "APP_VERSION"
    )


    for var in "${required_vars[@]}"; do
        if [ -z "${!var:-}" ]; then
            log_error "Required environment variable not set: $var"
            return 1
        fi
    done


    # Check file system
    if [ ! -w "/var/app" ]; then
        log_error "Application directory not writable: /var/app"
        return 1
    fi


    # Check disk space
    local available_space
    available_space=$(df -k /var/app | awk 'NR==2 {print $4}')
    local required_space=$((1024 * 1024))  # 1GB in KB


    if [ "$available_space" -lt "$required_space" ]; then
        log_error "Insufficient disk space" \
            "available=${available_space}KB" \
            "required=${required_space}KB"
        return 1
    fi


    # Check network connectivity
    if ! ping -c 1 -W 2 8.8.8.8 >/dev/null 2>&1; then
        log_error "No network connectivity"
        return 1
    fi


    log_info "Environment validation passed"
}


# ✅ State validation
validate_deployment_state() {
    local service="$1"
    local expected_version="$2"


    log_info "Validating deployment state" \
        "service=$service" \
        "version=$expected_version"


    # Check service is running
    if ! systemctl is-active --quiet "$service"; then
        log_error "Service not running: $service"
        return 1
    fi


    # Check version matches
    local actual_version
    actual_version=$(get_deployed_version "$service")


    if [ "$actual_version" != "$expected_version" ]; then
        log_error "Version mismatch" \
            "expected=$expected_version" \
            "actual=$actual_version"
        return 1
    fi


    # Check health endpoint
    local health_url="http://localhost:8080/health"
    local response
    response=$(curl -sf "$health_url" || echo "FAILED")


    if [ "$response" = "FAILED" ]; then
        log_error "Health check failed: $health_url"
        return 1
    fi


    log_info "Deployment state validation passed"
}


# ✅ Data integrity validation
validate_backup() {
    local backup_file="$1"
    local checksum_file="${backup_file}.sha256"


    # Verify backup exists
    if [ ! -f "$backup_file" ]; then
        log_error "Backup file not found: $backup_file"
        return 1
    fi


    # Verify checksum file exists
    if [ ! -f "$checksum_file" ]; then
        log_error "Checksum file not found: $checksum_file"
        return 1
    fi


    # Verify checksum matches
    local expected_checksum
    expected_checksum=$(cat "$checksum_file")


    local actual_checksum
    actual_checksum=$(sha256sum "$backup_file" | awk '{print $1}')


    if [ "$expected_checksum" != "$actual_checksum" ]; then
        log_error "Backup checksum mismatch" \
            "file=$backup_file" \
            "expected=$expected_checksum" \
            "actual=$actual_checksum"
        return 1
    fi


    # Verify backup is not empty
    local file_size
    file_size=$(stat -c%s "$backup_file" 2>/dev/null || stat -f%z "$backup_file")


    if [ "$file_size" -eq 0 ]; then
        log_error "Backup file is empty: $backup_file"
        return 1
    fi


    log_info "Backup validation passed" "file=$backup_file" "size=${file_size}B"
}

6. Testing Strategies

6.1 Testing Framework Setup

Using BATS (Bash Automated Testing System):

Terminal window
# Install BATS
# macOS: brew install bats-core
# Linux: git clone https://github.com/bats-core/bats-core.git && cd bats-core && ./install.sh /usr/local


# tests/test-helper.bash - Shared test utilities


#!/usr/bin/env bash


# Setup function (runs before each test)
setup() {
    # Create temporary directory for test
    export TEST_TEMP_DIR="$(mktemp -d)"


    # Load the script under test
    load "../lib/validators.sh"
}


# Teardown function (runs after each test)
teardown() {
    # Cleanup
    if [ -n "${TEST_TEMP_DIR:-}" ] && [ -d "$TEST_TEMP_DIR" ]; then
        rm -rf "$TEST_TEMP_DIR"
    fi
}


# Helper assertions
assert_file_exists() {
    [ -f "$1" ] || {
        echo "Expected file to exist: $1" >&2
        return 1
    }
}


assert_equals() {
    [ "$1" = "$2" ] || {
        echo "Expected: $2" >&2
        echo "Actual: $1" >&2
        return 1
    }
}

6.2 Unit Tests

# tests/validators.bats - Unit tests for validators


#!/usr/bin/env bats


load test-helper


@test "validate_email accepts valid email" {
    run validate_email "user@example.com"
    [ "$status" -eq 0 ]
}


@test "validate_email rejects invalid email" {
    run validate_email "invalid-email"
    [ "$status" -eq 1 ]
}


@test "validate_email rejects empty string" {
    run validate_email ""
    [ "$status" -eq 1 ]
}


@test "validate_semver accepts valid versions" {
    run validate_semver "1.2.3"
    [ "$status" -eq 0 ]


    run validate_semver "1.0.0-alpha.1"
    [ "$status" -eq 0 ]


    run validate_semver "2.0.0+build.123"
    [ "$status" -eq 0 ]
}


@test "validate_semver rejects invalid versions" {
    run validate_semver "1.2"
    [ "$status" -eq 1 ]


    run validate_semver "v1.2.3"
    [ "$status" -eq 1 ]


    run validate_semver "1.2.3.4"
    [ "$status" -eq 1 ]
}


@test "sanitize_path prevents directory traversal" {
    run sanitize_path "../../../etc/passwd"
    [ "$status" -eq 1 ]
    [[ "$output" == *"Path traversal detected"* ]]
}


@test "sanitize_path allows valid paths" {
    mkdir -p "$TEST_TEMP_DIR/valid"
    run sanitize_path "$TEST_TEMP_DIR/valid"
    [ "$status" -eq 0 ]
}

6.3 Integration Tests

# tests/deploy-integration.bats - Integration tests


#!/usr/bin/env bats


load test-helper


setup() {
    # Setup test environment
    export TEST_ENV="test"
    export APP_DIR="$TEST_TEMP_DIR/app"
    export BACKUP_DIR="$TEST_TEMP_DIR/backup"


    mkdir -p "$APP_DIR" "$BACKUP_DIR"


    # Create mock application
    echo "v1.0.0" > "$APP_DIR/version.txt"
}


@test "deploy performs backup before deployment" {
    # Mock deployment function
    deploy() {
        backup_current_version || return 1
        install_new_version || return 1
    }


    run deploy
    [ "$status" -eq 0 ]


    # Verify backup was created
    backup_count=$(find "$BACKUP_DIR" -type f | wc -l)
    [ "$backup_count" -gt 0 ]
}


@test "deploy rolls back on failure" {
    # Mock that fails
    install_new_version() {
        return 1
    }


    deploy_with_rollback() {
        backup_current_version || return 1


        if ! install_new_version; then
            rollback_to_backup
            return 1
        fi
    }


    run deploy_with_rollback
    [ "$status" -eq 1 ]


    # Verify original version restored
    version=$(cat "$APP_DIR/version.txt")
    [ "$version" = "v1.0.0" ]
}


@test "deploy validates post-deployment state" {
    deploy_and_validate() {
        install_new_version || return 1
        validate_deployment || return 1
    }


    run deploy_and_validate
    [ "$status" -eq 0 ]
}

6.4 Mock and Stub Patterns

# tests/mocks.bash - Mock utilities


#!/usr/bin/env bash


# Mock external commands
mock_command() {
    local cmd="$1"
    local return_code="${2:-0}"
    local output="${3:-}"


    # Create mock function
    eval "
    $cmd() {
        echo \"$output\"
        return $return_code
    }
    export -f $cmd
    "
}


# Stub with recorded calls
declare -A STUB_CALLS


stub_command() {
    local cmd="$1"
    shift
    local -a responses=("$@")


    STUB_CALLS[$cmd]=0


    eval "
    $cmd() {
        local call_count=\${STUB_CALLS[$cmd]}
        STUB_CALLS[$cmd]=\$((call_count + 1))


        if [ \$call_count -lt ${#responses[@]} ]; then
            echo \"\${responses[\$call_count]}\"
        fi
    }
    export -f $cmd
    "
}


# Verify stub was called
verify_stub_called() {
    local cmd="$1"
    local expected_calls="${2:-1}"
    local actual_calls="${STUB_CALLS[$cmd]:-0}"


    if [ "$actual_calls" -ne "$expected_calls" ]; then
        echo "Expected $cmd to be called $expected_calls times, was called $actual_calls times" >&2
        return 1
    fi
}


# Example usage in tests
@test "deploy calls backup before install" {
    # Stub the commands
    stub_command backup_current_version "backup-success"
    stub_command install_new_version "install-success"


    # Run deployment
    run deploy


    # Verify order of calls
    verify_stub_called backup_current_version 1
    verify_stub_called install_new_version 1
}

7. Documentation Standards

7.1 Inline Documentation

#!/usr/bin/env bash
#
# deploy-service.sh - Deploy application services to production
#
# This script orchestrates the deployment of application services with
# zero-downtime rolling updates, automatic health checks, and rollback
# capabilities.
#
# USAGE:
#   deploy-service.sh [OPTIONS] <service> <version>
#
# ARGUMENTS:
#   service    Service name (api|worker|frontend)
#   version    Semantic version to deploy (e.g., 1.2.3)
#
# OPTIONS:
#   -e, --env ENV          Target environment (staging|production)
#   -r, --replicas N       Number of replicas (default: 3)
#   -t, --timeout SECONDS  Deployment timeout (default: 300)
#   -d, --dry-run          Preview changes without applying
#   -h, --help             Show this help message
#
# ENVIRONMENT VARIABLES:
#   AWS_REGION             AWS region for deployment
#   SLACK_WEBHOOK_URL      Slack notifications (optional)
#   ROLLBACK_ON_FAILURE    Auto-rollback on failure (default: true)
#
# EXAMPLES:
#   # Deploy API service to staging
#   deploy-service.sh --env staging api 1.2.3
#
#   # Dry-run production deployment
#   deploy-service.sh --env production --dry-run frontend 2.0.0
#
#   # Deploy with custom replicas and timeout
#   deploy-service.sh --replicas 5 --timeout 600 worker 1.5.0
#
# EXIT CODES:
#   0   Success
#   1   Deployment failed
#   2   Invalid arguments
#   3   Pre-deployment validation failed
#   4   Health check failed
#   5   Rollback failed
#
# DEPENDENCIES:
#   - aws-cli (>= 2.0)
#   - kubectl (>= 1.20)
#   - jq (>= 1.6)
#
# AUTHOR:
#   DevOps Team <devops@example.com>
#
# SEE ALSO:
#   rollback-service.sh, validate-deployment.sh
#


#######################################
# Deploy a service with rolling update
#
# Performs zero-downtime deployment with health checks and automatic
# rollback on failure. Creates backup of previous deployment state.
#
# Globals:
#   AWS_REGION
#   ENVIRONMENT
#   ROLLBACK_ON_FAILURE
#
# Arguments:
#   $1 - service: Service name to deploy
#   $2 - version: Semantic version string
#   $3 - replicas: Number of instances (optional, default: 3)
#
# Outputs:
#   Writes deployment progress to stdout
#   Writes errors to stderr
#   Writes deployment logs to /var/log/deploy/service-TIMESTAMP.log
#
# Returns:
#   0 if deployment succeeds
#   1 if deployment fails
#   4 if health check fails
#
# Example:
#   deploy_service "api" "1.2.3" 5
#######################################
deploy_service() {
    local service="$1"
    local version="$2"
    local replicas="${3:-3}"


    # Implementation...
}

7.2 README Template

# Script Name


> One-line description of what the script does


## Overview


Detailed description of the script's purpose, use cases, and benefits.


## Prerequisites


- Bash 4.0 or higher
- Required system packages: `git`, `jq`, `curl`
- AWS CLI configured with appropriate credentials
- Kubernetes cluster access


## Installation


```bash
# Clone repository
git clone https://github.com/org/scripts.git


# Install dependencies
./scripts/setup-dependencies.sh


# Verify installation
./scripts/deploy.sh --version

Usage

Basic Usage

Terminal window
./deploy.sh --env production api 1.2.3

Advanced Options

Terminal window
# Dry-run mode
./deploy.sh --dry-run --env production api 1.2.3


# Custom configuration
./deploy.sh --env production --replicas 5 --timeout 600 api 1.2.3


# With environment file
ENV_FILE=.env.production ./deploy.sh api 1.2.3

Configuration

Environment Variables

VariableRequiredDefaultDescription
AWS_REGIONYes-AWS region for deployment
ENVIRONMENTYes-Target environment
SLACK_WEBHOOK_URLNo-Slack notifications

Configuration Files

Scripts load configuration in this order (later overrides earlier):

  1. /etc/app/config.env - System-wide defaults
  2. ~/.config/app/config.env - User defaults
  3. .env - Project-specific
  4. ENV_FILE - Explicit override

Architecture

deploy.sh (main orchestration)
├── lib/
│   ├── logger.sh (logging utilities)
│   ├── aws.sh (AWS operations)
│   └── validators.sh (input validation)
└── deploy/
    ├── backup.sh (pre-deployment backup)
    ├── health-check.sh (service validation)
    └── rollback.sh (failure recovery)

Testing

Terminal window
# Run unit tests
bats tests/*.bats


# Run integration tests
bats tests/integration/*.bats


# Run with coverage
bats --tap tests/*.bats | tee test-results.tap

Troubleshooting

Common Issues

Problem: Deployment times out Solution: Increase timeout with --timeout 600

Problem: Health check fails Solution: Verify service endpoints with ./health-check.sh

Debug Mode

Terminal window
# Enable verbose logging
LOG_LEVEL=DEBUG ./deploy.sh ...


# Enable bash tracing
bash -x ./deploy.sh ...

Security

  • Never commit .env files with secrets
  • Use AWS Secrets Manager or similar for sensitive data
  • Verify checksums before deployment
  • Audit logs stored in /var/log/deploy/

Contributing

  1. Fork the repository
  2. Create feature branch (git checkout -b feature/amazing)
  3. Add tests for changes
  4. Ensure shellcheck passes
  5. Submit pull request

License

MIT License - see LICENSE file

Support

---


## 8. DevOps Integration Patterns


### 8.1 CI/CD Integration


**GitLab CI Example**:


```yaml
# .gitlab-ci.yml


variables:
  SHELLCHECK_VERSION: "0.9.0"


stages:
  - validate
  - test
  - deploy


# Shell script validation
shellcheck:
  stage: validate
  image: koalaman/shellcheck:v${SHELLCHECK_VERSION}
  script:
    - shellcheck scripts/**/*.sh
    - shellcheck lib/**/*.sh
  only:
    changes:
      - scripts/**/*.sh
      - lib/**/*.sh


# Unit tests
bats-tests:
  stage: test
  image: ubuntu:22.04
  before_script:
    - apt-get update && apt-get install -y bats git
  script:
    - bats tests/*.bats
  coverage: '/^Coverage: \d+\.\d+%/'
  artifacts:
    reports:
      junit: test-results.xml


# Integration tests
integration-tests:
  stage: test
  image: ubuntu:22.04
  services:
    - docker:dind
  script:
    - ./scripts/integration-test.sh
  only:
    - main
    - merge_requests


# Deploy to staging (automatic)
deploy-staging:
  stage: deploy
  script:
    - ./scripts/deploy.sh --env staging
  environment:
    name: staging
    url: https://staging.example.com
  only:
    - main


# Deploy to production (manual)
deploy-production:
  stage: deploy
  script:
    - ./scripts/deploy.sh --env production
  environment:
    name: production
    url: https://example.com
  when: manual
  only:
    - main

GitHub Actions Example:

.github/workflows/shell-scripts.yml
name: Shell Scripts CI


on:
  push:
    branches: [main]
  pull_request:
    paths:
      - 'scripts/**/*.sh'
      - 'lib/**/*.sh'


jobs:
  shellcheck:
    name: ShellCheck Validation
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3


      - name: Run ShellCheck
        uses: ludeeus/action-shellcheck@master
        with:
          scandir: './scripts'
          severity: warning


      - name: Check POSIX compliance
        run: |
          find scripts -name "*.sh" -exec shellcheck -s sh {} +


  bats-tests:
    name: BATS Unit Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3


      - name: Setup BATS
        uses: mig4/setup-bats@v1
        with:
          bats-version: 1.10.0


      - name: Run tests
        run: |
          bats tests/*.bats


      - name: Upload coverage
        uses: codecov/codecov-action@v3


  integration-tests:
    name: Integration Tests
    runs-on: ubuntu-latest
    needs: [shellcheck, bats-tests]
    steps:
      - uses: actions/checkout@v3


      - name: Run integration tests
        run: |
          chmod +x scripts/*.sh
          ./scripts/integration-test.sh


  deploy:
    name: Deploy
    runs-on: ubuntu-latest
    needs: [integration-tests]
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v3


      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1


      - name: Deploy to staging
        run: |
          ./scripts/deploy.sh --env staging

8.2 Container Integration

Dockerfile with scripts:

# Dockerfile for script execution environment


FROM ubuntu:22.04


# Install dependencies
RUN apt-get update && apt-get install -y \
    bash \
    curl \
    git \
    jq \
    awscli \
    && rm -rf /var/lib/apt/lists/*


# Create non-root user
RUN useradd -m -s /bin/bash scriptuser && \
    mkdir -p /app /var/log/scripts && \
    chown -R scriptuser:scriptuser /app /var/log/scripts


# Copy scripts
COPY --chown=scriptuser:scriptuser scripts/ /app/scripts/
COPY --chown=scriptuser:scriptuser lib/ /app/lib/


# Set permissions
RUN find /app/scripts -name "*.sh" -exec chmod +x {} \; && \
    chmod 700 /app/scripts /app/lib


# Switch to non-root user
USER scriptuser
WORKDIR /app


# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD /app/scripts/health-check.sh || exit 1


# Default command
ENTRYPOINT ["/app/scripts/entrypoint.sh"]
CMD ["--help"]

Docker Compose with scripts:

docker-compose.yml
version: '3.8'


services:
  deploy-runner:
    build:
      context: .
      dockerfile: Dockerfile
    image: deploy-runner:latest
    environment:
      - AWS_REGION=${AWS_REGION}
      - ENVIRONMENT=${ENVIRONMENT:-dev}
      - LOG_LEVEL=${LOG_LEVEL:-INFO}
    volumes:
      # Mount scripts for development
      - ./scripts:/app/scripts:ro
      - ./lib:/app/lib:ro
      # Persistent logs
      - deploy-logs:/var/log/scripts
    secrets:
      - aws_credentials
    networks:
      - deploy-network


secrets:
  aws_credentials:
    file: ~/.aws/credentials


volumes:
  deploy-logs:


networks:
  deploy-network:
    driver: bridge

8.3 Kubernetes Jobs

k8s/deploy-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: deploy-job
  namespace: ci-cd
spec:
  ttlSecondsAfterFinished: 3600
  backoffLimit: 3


  template:
    metadata:
      labels:
        app: deploy-runner
    spec:
      restartPolicy: Never


      serviceAccountName: deploy-service-account


      initContainers:
      - name: validate-environment
        image: deploy-runner:latest
        command: ["/app/scripts/validate-env.sh"]
        env:
        - name: ENVIRONMENT
          valueFrom:
            configMapKeyRef:
              name: deploy-config
              key: environment


      containers:
      - name: deploy
        image: deploy-runner:latest
        command: ["/app/scripts/deploy.sh"]
        args:
          - "--env"
          - "$(ENVIRONMENT)"
          - "$(SERVICE_NAME)"
          - "$(VERSION)"
        env:
        - name: ENVIRONMENT
          valueFrom:
            configMapKeyRef:
              name: deploy-config
              key: environment
        - name: SERVICE_NAME
          value: "api"
        - name: VERSION
          value: "1.2.3"
        - name: AWS_REGION
          valueFrom:
            configMapKeyRef:
              name: deploy-config
              key: aws_region
        volumeMounts:
        - name: scripts
          mountPath: /app/scripts
          readOnly: true
        - name: aws-credentials
          mountPath: /root/.aws
          readOnly: true
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"


      volumes:
      - name: scripts
        configMap:
          name: deploy-scripts
          defaultMode: 0755
      - name: aws-credentials
        secret:
          secretName: aws-credentials

8.4 Infrastructure as Code Integration

Terraform with provisioning scripts:

terraform/main.tf
resource "aws_instance" "app_server" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = var.instance_type


  user_data = templatefile("${path.module}/scripts/cloud-init.sh", {
    environment = var.environment
    app_version = var.app_version
    region      = var.aws_region
  })


  # Copy provisioning scripts
  provisioner "file" {
    source      = "${path.module}/../scripts/"
    destination = "/tmp/scripts"


    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = file(var.ssh_private_key)
      host        = self.public_ip
    }
  }


  # Execute setup script
  provisioner "remote-exec" {
    inline = [
      "chmod +x /tmp/scripts/*.sh",
      "sudo /tmp/scripts/setup-instance.sh --env ${var.environment}",
      "sudo /tmp/scripts/deploy-app.sh --version ${var.app_version}"
    ]


    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = file(var.ssh_private_key)
      host        = self.public_ip
    }
  }


  tags = {
    Name        = "app-server-${var.environment}"
    Environment = var.environment
    ManagedBy   = "terraform"
  }
}


# Null resource for script updates
resource "null_resource" "update_scripts" {
  triggers = {
    script_hash = filemd5("${path.module}/../scripts/deploy-app.sh")
  }


  provisioner "remote-exec" {
    inline = [
      "sudo /tmp/scripts/deploy-app.sh --version ${var.app_version}"
    ]


    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = file(var.ssh_private_key)
      host        = aws_instance.app_server.public_ip
    }
  }


  depends_on = [aws_instance.app_server]
}

9. Quality Gates & Validation

9.1 Pre-commit Hooks

#!/usr/bin/env bash
# .git/hooks/pre-commit - Validate scripts before commit


set -euo pipefail


echo "Running pre-commit checks for shell scripts..."


# Find all shell scripts in staging
mapfile -t scripts < <(git diff --cached --name-only --diff-filter=ACM | grep '\.sh$' || true)


if [ ${#scripts[@]} -eq 0 ]; then
    echo "No shell scripts to validate"
    exit 0
fi


# Track if any checks fail
checks_failed=0


# 1. ShellCheck validation
echo "→ Running ShellCheck..."
for script in "${scripts[@]}"; do
    if ! shellcheck "$script"; then
        echo "✗ ShellCheck failed: $script"
        checks_failed=1
    fi
done


# 2. Bash syntax check
echo "→ Validating syntax..."
for script in "${scripts[@]}"; do
    if ! bash -n "$script"; then
        echo "✗ Syntax check failed: $script"
        checks_failed=1
    fi
done


# 3. Check for common security issues
echo "→ Checking for security issues..."
for script in "${scripts[@]}"; do
    # Check for eval usage
    if grep -qE '\beval\s' "$script"; then
        echo "✗ Security: 'eval' usage detected in $script"
        checks_failed=1
    fi


    # Check for hardcoded secrets patterns
    if grep -qE '(password|secret|key)\s*=\s*["\047][^"\047]+["\047]' "$script"; then
        echo "✗ Security: Possible hardcoded secret in $script"
        checks_failed=1
    fi
done


# 4. Check for proper shebang
echo "→ Validating shebangs..."
for script in "${scripts[@]}"; do
    first_line=$(head -n1 "$script")
    if [[ ! "$first_line" =~ ^#!/(usr/bin/env bash|bin/(ba)?sh) ]]; then
        echo "✗ Invalid or missing shebang in $script"
        checks_failed=1
    fi
done


# 5. Check for TODO/FIXME with required context
echo "→ Checking TODOs..."
for script in "${scripts[@]}"; do
    if grep -qE 'TODO(?!\([a-z]+\))' "$script"; then
        echo "✗ TODO without assignee in $script (use: TODO(username))"
        checks_failed=1
    fi
done


if [ $checks_failed -eq 1 ]; then
    echo ""
    echo "✗ Pre-commit checks failed. Please fix the issues above."
    exit 1
fi


echo "✓ All pre-commit checks passed"
exit 0

9.2 Continuous Validation

#!/usr/bin/env bash
# scripts/validate-all.sh - Comprehensive validation suite


set -euo pipefail


SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"


# Load logger
source "${SCRIPT_DIR}/lib/logger.sh"


# Validation results
declare -A VALIDATION_RESULTS=()
TOTAL_CHECKS=0
PASSED_CHECKS=0


run_check() {
    local check_name="$1"
    shift
    local -a check_command=("$@")


    TOTAL_CHECKS=$((TOTAL_CHECKS + 1))


    log_info "Running: $check_name"


    if "${check_command[@]}"; then
        VALIDATION_RESULTS["$check_name"]="PASS"
        PASSED_CHECKS=$((PASSED_CHECKS + 1))
        log_info "✓ $check_name passed"
        return 0
    else
        VALIDATION_RESULTS["$check_name"]="FAIL"
        log_error "✗ $check_name failed"
        return 1
    fi
}


# 1. ShellCheck validation
validate_shellcheck() {
    log_info "ShellCheck validation"


    local scripts
    mapfile -t scripts < <(find "$PROJECT_ROOT/scripts" -name "*.sh")


    for script in "${scripts[@]}"; do
        shellcheck "$script" || return 1
    done
}


# 2. Unit tests
validate_unit_tests() {
    log_info "Running unit tests"
    bats "$PROJECT_ROOT/tests"/*.bats
}


# 3. Integration tests
validate_integration_tests() {
    log_info "Running integration tests"
    bats "$PROJECT_ROOT/tests/integration"/*.bats
}


# 4. Security audit
validate_security() {
    log_info "Security audit"


    local issues=0


    # Check for eval usage
    if grep -rE '\beval\s' "$PROJECT_ROOT/scripts" --include="*.sh"; then
        log_error "Found 'eval' usage (security risk)"
        issues=$((issues + 1))
    fi


    # Check for hardcoded secrets
    if grep -rE '(password|secret|key)\s*=\s*["\047][^"\047]+["\047]' \
        "$PROJECT_ROOT/scripts" --include="*.sh"; then
        log_error "Found possible hardcoded secrets"
        issues=$((issues + 1))
    fi


    # Check file permissions
    while IFS= read -r -d '' file; do
        local perms
        perms=$(stat -c %a "$file" 2>/dev/null || stat -f %A "$file")
        if [[ "$perms" =~ [2367]$ ]]; then
            log_error "Insecure permissions on $file: $perms"
            issues=$((issues + 1))
        fi
    done < <(find "$PROJECT_ROOT/scripts" -name "*.sh" -print0)


    return "$issues"
}


# 5. Documentation validation
validate_documentation() {
    log_info "Documentation validation"


    local scripts
    mapfile -t scripts < <(find "$PROJECT_ROOT/scripts" -name "*.sh" ! -path "*/lib/*")


    for script in "${scripts[@]}"; do
        # Check for usage function
        if ! grep -q '^usage()' "$script"; then
            log_warn "Missing usage() function in $script"
        fi


        # Check for header comments
        if ! head -n 10 "$script" | grep -q '^#.*Usage:'; then
            log_warn "Missing usage documentation in $script"
        fi
    done
}


# 6. Code quality metrics
validate_code_quality() {
    log_info "Code quality metrics"


    local scripts
    mapfile -t scripts < <(find "$PROJECT_ROOT/scripts" -name "*.sh")


    for script in "${scripts[@]}"; do
        local lines
        lines=$(wc -l < "$script")


        # Warn on large scripts (>500 lines)
        if [ "$lines" -gt 500 ]; then
            log_warn "Large script detected: $script ($lines lines) - consider refactoring"
        fi


        # Check cyclomatic complexity (approximate)
        local complexity
        complexity=$(grep -cE '\b(if|while|for|case)\b' "$script" || true)
        if [ "$complexity" -gt 20 ]; then
            log_warn "High complexity: $script (complexity: $complexity)"
        fi
    done
}


# Main validation
main() {
    log_info "Starting comprehensive validation"


    run_check "ShellCheck" validate_shellcheck
    run_check "Unit Tests" validate_unit_tests
    run_check "Integration Tests" validate_integration_tests
    run_check "Security Audit" validate_security
    run_check "Documentation" validate_documentation
    run_check "Code Quality" validate_code_quality


    # Summary
    echo ""
    log_info "Validation Summary"
    log_info "=================="
    log_info "Total checks: $TOTAL_CHECKS"
    log_info "Passed: $PASSED_CHECKS"
    log_info "Failed: $((TOTAL_CHECKS - PASSED_CHECKS))"


    # Detailed results
    for check in "${!VALIDATION_RESULTS[@]}"; do
        local result="${VALIDATION_RESULTS[$check]}"
        if [ "$result" = "PASS" ]; then
            log_info "✓ $check"
        else
            log_error "✗ $check"
        fi
    done


    # Exit with failure if any checks failed
    if [ "$PASSED_CHECKS" -ne "$TOTAL_CHECKS" ]; then
        log_error "Validation failed"
        return 1
    fi


    log_info "All validation checks passed"
    return 0
}


main "$@"

10. Common Patterns & Anti-Patterns

10.1 Good Patterns

#!/usr/bin/env bash
# good-patterns.sh - Examples of best practices


# ✅ GOOD: Early return pattern
process_file() {
    local file="$1"


    # Validate early, return early
    [ ! -f "$file" ] && { log_error "File not found: $file"; return 1; }
    [ ! -r "$file" ] && { log_error "File not readable: $file"; return 1; }


    # Main logic only executes if validations pass
    cat "$file" | process_content
}


# ✅ GOOD: Use functions for reusability
is_valid_semver() {
    local version="$1"
    [[ "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]
}


deploy_version() {
    local version="$1"


    if ! is_valid_semver "$version"; then
        log_error "Invalid version format: $version"
        return 1
    fi


    # Deployment logic...
}


# ✅ GOOD: Proper array handling
process_items() {
    local -a items=("$@")


    for item in "${items[@]}"; do
        echo "Processing: $item"
    done
}


# ✅ GOOD: Proper quoting
safe_file_operations() {
    local source_file="$1"
    local dest_dir="$2"


    # Always quote variables with spaces
    cp "$source_file" "$dest_dir/"


    # Array expansion with quotes
    local -a files=("file 1.txt" "file 2.txt")
    for file in "${files[@]}"; do
        echo "$file"
    done
}


# ✅ GOOD: Command substitution with error handling
get_git_commit() {
    local commit
    commit=$(git rev-parse HEAD 2>/dev/null) || {
        log_error "Not a git repository"
        return 1
    }
    echo "$commit"
}


# ✅ GOOD: Here-doc for multi-line content
generate_config() {
    cat << EOF
{
    "environment": "$ENVIRONMENT",
    "version": "$VERSION",
    "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
}
EOF
}


# ✅ GOOD: Proper signal handling
cleanup_on_exit() {
    local exit_code=$?


    if [ -n "${TEMP_DIR:-}" ]; then
        rm -rf "$TEMP_DIR"
    fi


    if [ -n "${PID_FILE:-}" ]; then
        rm -f "$PID_FILE"
    fi


    exit $exit_code
}
trap cleanup_on_exit EXIT INT TERM


# ✅ GOOD: Named pipes for complex pipelines
process_with_named_pipe() {
    local pipe
    pipe=$(mktemp -u)
    mkfifo "$pipe"


    # Producer
    producer > "$pipe" &
    local producer_pid=$!


    # Consumer
    consumer < "$pipe"


    # Cleanup
    wait "$producer_pid"
    rm -f "$pipe"
}


# ✅ GOOD: Parallel execution with proper waiting
parallel_process() {
    local -a pids=()


    for item in "${items[@]}"; do
        process_item "$item" &
        pids+=($!)
    done


    # Wait for all background jobs
    local failed=0
    for pid in "${pids[@]}"; do
        if ! wait "$pid"; then
            failed=$((failed + 1))
        fi
    done


    return "$failed"
}

10.2 Anti-Patterns to Avoid

#!/usr/bin/env bash
# anti-patterns.sh - Common mistakes to avoid


# ❌ BAD: Useless use of cat
bad_cat() {
    cat file.txt | grep "pattern"
}


# ✅ GOOD: Direct input redirection
good_grep() {
    grep "pattern" file.txt
}


# ❌ BAD: Parsing ls output
bad_loop() {
    for file in $(ls *.txt); do
        echo "$file"
    done
}


# ✅ GOOD: Use glob patterns
good_loop() {
    for file in *.txt; do
        [ -f "$file" ] || continue
        echo "$file"
    done
}


# ❌ BAD: Unquoted variables
bad_quoting() {
    local file=$1
    cat $file  # Breaks with spaces
}


# ✅ GOOD: Always quote variables
good_quoting() {
    local file="$1"
    cat "$file"
}


# ❌ BAD: Using eval
bad_eval() {
    local cmd="ls -la"
    eval $cmd  # Command injection risk
}


# ✅ GOOD: Direct execution
good_execution() {
    ls -la
}


# ❌ BAD: Ignoring errors
bad_error_handling() {
    important_command
    # Continues even if it fails
}


# ✅ GOOD: Check return codes
good_error_handling() {
    if ! important_command; then
        log_error "Command failed"
        return 1
    fi
}


# ❌ BAD: Using backticks
bad_substitution() {
    result=`command`
}


# ✅ GOOD: Use $()
good_substitution() {
    result=$(command)
}


# ❌ BAD: [[ vs [ confusion
bad_test() {
    if [ $var == "test" ]; then  # Wrong operator for [ ]
        echo "match"
    fi
}


# ✅ GOOD: Correct test syntax
good_test() {
    if [ "$var" = "test" ]; then  # POSIX-compatible
        echo "match"
    fi


    # Or use bash [[
    if [[ "$var" == "test" ]]; then  # Bash-specific
        echo "match"
    fi
}


# ❌ BAD: Creating security vulnerabilities
bad_security() {
    local user_input="$1"
    eval "echo $user_input"  # Command injection
    curl "http://api.example.com/$user_input"  # URL injection
}


# ✅ GOOD: Validate and sanitize input
good_security() {
    local user_input="$1"


    # Validate format
    if [[ ! "$user_input" =~ ^[a-zA-Z0-9_-]+$ ]]; then
        log_error "Invalid input format"
        return 1
    fi


    # Use safely
    echo "$user_input"
    curl "http://api.example.com/$(urlencode "$user_input")"
}


# ❌ BAD: Not checking command existence
bad_command_check() {
    jq '.key' file.json  # Fails if jq not installed
}


# ✅ GOOD: Check dependencies
good_command_check() {
    if ! command -v jq >/dev/null 2>&1; then
        log_error "jq is required but not installed"
        return 1
    fi


    jq '.key' file.json
}


# ❌ BAD: Global variables without readonly
bad_globals() {
    CONFIG_FILE="/etc/app.conf"  # Can be modified
}


# ✅ GOOD: Use readonly for constants
good_globals() {
    readonly CONFIG_FILE="/etc/app.conf"
}


# ❌ BAD: Complex pipelines without error handling
bad_pipeline() {
    cat file.txt | grep pattern | sed 's/old/new/' | sort
    # If any step fails, continues anyway
}


# ✅ GOOD: Check pipeline status
good_pipeline() {
    set -o pipefail
    if ! cat file.txt | grep pattern | sed 's/old/new/' | sort; then
        log_error "Pipeline failed"
        return 1
    fi
}

10.3 Performance Patterns

performance-patterns.sh
#!/usr/bin/env bash
# ✅ GOOD: Use built-in parameter expansion
fast_string_ops() {
    local str="hello_world_test"


    # Instead of: echo "$str" | sed 's/_/-/g'
    echo "${str//_/-}"


    # Instead of: echo "$str" | cut -d_ -f1
    echo "${str%%_*}"


    # Instead of: echo "$str" | awk '{print length}'
    echo "${#str}"
}


# ✅ GOOD: Batch operations
batch_process() {
    local -a files=()


    # Collect files first
    while IFS= read -r -d '' file; do
        files+=("$file")
    done < <(find . -name "*.txt" -print0)


    # Process in batch
    if [ ${#files[@]} -gt 0 ]; then
        tar czf backup.tar.gz "${files[@]}"
    fi
}


# ❌ BAD: Multiple calls in loop
slow_loop() {
    for file in *.txt; do
        grep -l "pattern" "$file"
    done
}


# ✅ GOOD: Single call with all files
fast_grep() {
    grep -l "pattern" *.txt
}


# ✅ GOOD: Use process substitution
fast_comparison() {
    # Instead of creating temp files
    diff <(command1) <(command2)
}


# ✅ GOOD: Parallel processing with GNU parallel
parallel_process() {
    # Install: apt-get install parallel
    parallel -j4 process_file ::: *.txt
}


# ✅ GOOD: Use appropriate tools
fast_json() {
    # Instead of: cat file.json | python -c "import json..."
    jq '.key' file.json  # Much faster
}

Appendix A: Quick Reference

Common Commands Cheat Sheet

Terminal window
# File operations
cp -a source dest           # Copy preserving attributes
mv -n source dest           # Move, no clobber
rm -rf dir/                 # Remove recursively
mkdir -p path/to/dir        # Create with parents
touch file                  # Create empty file
chmod 755 script.sh         # Make executable


# Text processing
grep -r "pattern" dir/      # Recursive search
sed -i 's/old/new/g' file   # In-place replace
awk '{print $1}' file       # Print first column
cut -d: -f1 file            # Cut by delimiter
sort -u file                # Sort unique
uniq -c file                # Count duplicates


# System operations
ps aux | grep process       # Find process
kill -TERM pid              # Graceful shutdown
kill -KILL pid              # Force kill
df -h                       # Disk usage
du -sh dir/                 # Directory size
free -h                     # Memory usage


# Network operations
curl -sf URL                # Silent fetch
wget -q URL                 # Quiet download
nc -zv host port            # Port check
ping -c1 host               # Single ping


# Compression
tar czf archive.tar.gz dir/ # Create tarball
tar xzf archive.tar.gz      # Extract tarball
gzip file                   # Compress file
gunzip file.gz              # Decompress file

ShellCheck Directives

Terminal window
# Disable specific warnings
# shellcheck disable=SC2034  # Variable appears unused
UNUSED_VAR="value"


# Disable for entire file
# shellcheck disable=SC1090  # Can't follow sourced file


# Source external files
# shellcheck source=lib/common.sh
source lib/common.sh


# Specify shell
# shellcheck shell=bash


# Exclude patterns
# shellcheck disable=SC2086  # Double quote to prevent globbing

Appendix B: Tool Installation

ShellCheck

Terminal window
# macOS
brew install shellcheck


# Ubuntu/Debian
apt-get install shellcheck


# From source
git clone https://github.com/koalaman/shellcheck
cd shellcheck && cabal install

BATS

Terminal window
# macOS
brew install bats-core


# Ubuntu/Debian
npm install -g bats


# From source
git clone https://github.com/bats-core/bats-core.git
cd bats-core && ./install.sh /usr/local

Additional Tools

Terminal window
# shfmt - Shell formatter
GO111MODULE=on go install mvdan.cc/sh/v3/cmd/shfmt@latest


# shellharden - Shell hardening
cargo install shellharden

Appendix C: Resources

Official Documentation

Style Guides

Learning Resources

Document Version: 1.0 Last Updated: 2026-01-08 Maintained By: DevOps Team License: MIT