cloud-cost-ops-toolkit

KO | EN

gitlite — search

#typescript #llm #android #machine-learning #rust #deep-learning #linux #nodejs #macos #awesome #cli #kubernetes #pytorch #mcp

cloud-cost-ops-toolkit

(★ 2)

Practical AWS cost‑reduction toolkit: daily spend reports, rightsizing findings, idle cleanup (safe, dry‑run first), S3 lifecycle enforcement, and cost‑aware CI (Infracost + policy‑as‑code). Real‑world scripts that can save 30–80% without hurting reliability.

cloud-cost-ops-toolkit Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Cloud Cost Ops Toolkit

> **Pragmatic AWS cost optimization for engineering leaders**

A comprehensive, production-ready toolkit for AWS cost visibility, optimization, and governance. Built with security-first principles, dry-run safety, and clear operational guidance.

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Terraform](https://img.shields.io/badge/Terraform-1.0+-blue.svg)](https://www.terraform.io/)
[![Python](https://img.shields.io/badge/Python-3.9+-green.svg)](https://www.python.org/)
[![AWS](https://img.shields.io/badge/AWS-Multiple%20Services-orange.svg)](https://aws.amazon.com/)

## 🎯 Why This Toolkit?

**Engineering leaders need actionable cost insights, not just dashboards.** This toolkit provides:

- **📊 Daily Cost Visibility**: Automated reports with trend analysis and alerts
- **⚡ Rightsizing Intelligence**: Data-driven recommendations for underutilized resources
- **🧹 Idle Resource Cleanup**: Safe identification and removal of unused resources
- **📋 Lifecycle Enforcement**: Automated S3 storage optimization and policy compliance
- **🛡️ Cost-Aware CI/CD**: Integrate cost impact analysis into your deployment pipeline
- **📏 Policy-as-Code**: Prevent costly misconfigurations before they happen

## 🚀 Quick Start

### Prerequisites

- AWS CLI configured with appropriate permissions
- Python 3.9+ and pip
- Terraform 1.0+ (for infrastructure examples)
- Make (for convenience commands)

### Installation

```bash
# Clone the repository
git clone https://github.com/your-org/cloud-cost-ops-toolkit.git
cd cloud-cost-ops-toolkit

# Install dependencies
make install

# Verify installation
make help
```

### Generate Your First Cost Report

```bash
# Display cost report in terminal (safe, read-only)
make report

# Generate detailed 30-day analysis with monthly granularity
python scripts/cost_report.py --days 30 --granularity MONTHLY

# Filter by specific services with custom region
python scripts/cost_report.py --days 7 --services ec2,s3,rds --region us-west-2
```

**🔒 Safe by Default**: All operations run in dry-run mode unless explicitly overridden.

## 📊 Core Features

### 1. Daily Cost Reports

Generate comprehensive cost reports with trend analysis:

```bash
# Terminal dashboard
make report

# Automated daily reports (GitHub Actions)
# See .github/workflows/daily-cost-report.yml

# Custom analysis with AWS profile
python scripts/cost_report.py --days 7 --profile production --granularity DAILY
```

**Sample Output**:
```
📊 Cost Summary
💰 Total Cost: $247.83
📅 Daily Average: $35.40
📈 Trend: Increasing
🥇 Top Service: EC2-Instance ($89.23)
```

### 2. Resource Rightsizing

Identify overprovisioned resources and optimization opportunities:

```bash
# Safe analysis (no changes made)
make rightsizing-dry
python scripts/rightsizing.py --formats json,csv --region us-west-2
python scripts/rightsizing.py --apply --formats md --profile production

# With resource tagging (non-destructive)
python scripts/rightsizing.py --apply --formats json,csv,md
```

**What it analyzes**:
- **EC2 instances**: Uses AWS Compute Optimizer recommendations  
- **Auto Scaling groups**: Right-sizing and capacity optimization
- **RDS instances**: Basic analysis (manual review recommended)

**Outputs**:
- JSON/CSV/Markdown reports with potential monthly savings
- Console table with top opportunities ranked by savings
- Resource tagging for tracking candidates (when using --apply)

**Key features**:
- Non-destructive by default (--apply only tags resources)
- Graceful handling of missing Compute Optimizer permissions
- Multi-format output for integration with other tools

## 🚀 Shift-left Cost Awareness

**Catch cost impacts before they hit production.** The toolkit integrates cost analysis directly into your development workflow, enabling teams to make informed decisions early in the development cycle.

### Automated PR Cost Analysis

Every pull request with infrastructure changes automatically gets cost impact analysis:

```yaml
# Triggered on PRs with Terraform changes
- terraform-examples/**
- **/*.tf files
```

**What you get**:
- 💰 **Monthly cost delta** with clear before/after comparison
- 📊 **Resource breakdown** showing cost per component  
- 🚨 **Threshold alerts** when changes exceed budget limits ($100 default)
- 💡 **Optimization suggestions** for cost reduction

### Setup Cost Analysis

1. **Get your free Infracost API key**:
   ```bash
   # Sign up at https://infracost.io (free for teams up to 10 users)
   # Get your API key from the dashboard
   ```

2. **Add to repository secrets**:
   ```bash
   # In GitHub: Settings → Secrets → Actions
   # Add secret: INFRACOST_API_KEY = your_api_key_here
   ```

3. **Configure cost threshold** (optional):
   ```yaml
   # In .github/workflows/infracost-pr-comment.yml
   env:
     COST_THRESHOLD: 100  # Monthly cost increase limit in USD
   ```

### Sample PR Comment

```markdown
## 💰 Infrastructure Cost Analysis

📈 Monthly Cost Change: +$45.20

### 📊 Cost Breakdown
┌─────────────────────────────────────────┬─────────────┬─────────────┬───────────────┐
│ Name                                    │ Monthly Qty │ Unit        │ Monthly Cost  │
├─────────────────────────────────────────┼─────────────┼─────────────┼───────────────┤
│ aws_instance.web_server                 │         744 │ hours       │       $12.40  │
│ aws_rds_instance.database               │         744 │ hours       │       $28.80  │
│ aws_lb.application                      │         744 │ hours       │        $4.00  │
└─────────────────────────────────────────┴─────────────┴─────────────┴───────────────┘

💡 Cost Optimization Tips:
- Consider using smaller instance sizes where appropriate
- Review if all resources are necessary for this change
```

**Benefits**:
- ✅ **No surprises** - See cost impact before merging
- ✅ **Team awareness** - Everyone sees the cost implications  
- ✅ **Budget protection** - Automatic alerts on threshold breaches
- ✅ **Zero setup friction** - Works without API key (shows setup guidance)

### 3. Idle Resource Cleanup

Find and optionally remove unused resources with enterprise-grade safety controls:

```bash
# Identify idle resources (safe - default behavior)
make cleanup-dry

# Target specific age threshold and region
python scripts/cleanup_idle.py --max-age-days 14 --region us-west-2

# Actually delete resources (requires explicit confirmation)
python scripts/cleanup_idle.py --no-dry-run --confirm-delete
```

**🛡️ Safety Guarantees**:
- **Default dry-run mode** - No changes made unless explicitly confirmed
- **Limited deletion scope** - Only unattached EBS volumes and unused EIPs can be deleted
- **Double confirmation** - Requires both `--no-dry-run` AND `--confirm-delete` flags
- **Module-level safety switch** - `ALLOW_DESTRUCTIVE_ACTIONS` constant can disable all deletions
- **Resource re-inspection** - Resources are verified immediately before deletion
- **Never deletes** - RDS snapshots, S3 buckets, EC2 instances, or any other resource types

**Resource types detected**:
- **Unattached EBS volumes** (older than threshold) - ✅ Can be safely deleted
- **Unused Elastic IP addresses** (not associated with instances) - ✅ Can be safely deleted  
- **Old RDS snapshots** (exceeding retention period) - 🔒 Read-only analysis
- **S3 buckets without lifecycle policies** (cost optimization opportunity) - 🔒 Read-only analysis
- **Stopped EC2 instances** (in non-production environments, older than threshold) - 🔒 Read-only analysis

### 4. S3 Lifecycle Optimization

Automate storage cost reduction:

```bash
# Analyze S3 lifecycle opportunities
python scripts/ensure_s3_lifecycle.py --format table --region us-east-1

# Apply lifecycle policies (with confirmation)
python scripts/ensure_s3_lifecycle.py --apply --transition-days 30,90,365 --profile production
```

**Savings potential**: 30-70% reduction in S3 storage costs through automatic transitions.

### 5. Policy-as-Code Governance

**Prevent costly misconfigurations before they reach production.** Enforce cost optimization and governance policies automatically using Open Policy Agent (OPA) and Conftest.

```bash
# Test all policies against your Terraform
make policy-test

# Test specific directories
conftest test --policy policies/conftest terraform-examples/s3-lifecycle/
```

**Built-in Policies:**

**Cost Optimization:**
- ✅ **Required cost allocation tags**: Environment, Owner, CostCenter, Project
- ✅ **S3 lifecycle policies**: All buckets must have lifecycle rules for cost optimization
- ✅ **Large instance justification**: Large EC2 instances require justification tags
- ✅ **Resource naming conventions**: Standardized naming for better cost tracking

**Security & Compliance:**
- ✅ **Encryption requirements**: EBS and RDS encryption mandatory
- ✅ **Backup policies**: RDS instances must have backup retention enabled
- ✅ **Public access prevention**: S3 buckets must block public access

**Environment-Specific:**
- ✅ **Development restrictions**: Expensive resources blocked in dev environments
- ✅ **Production safeguards**: Deletion protection required for production databases

#### Policy Override Example

For legitimate exceptions, create a `.conftest-overrides.rego` file:

```rego
package terraform

# Override: Allow untagged resources for demo purposes
exception[resource_address] {
    input.resource_changes[_].address == resource_address
    resource_address == "aws_s3_bucket.demo_bucket"
}

# Override: Allow expensive dev resources for testing
exception[resource_address] {
    input.resource_changes[_].address == resource_address
    resource_address == "aws_nat_gateway.load_test"
    input.resource_changes[_].change.after.tags["TestingJustification"]
}
```

**Policy Integration:**
- 🔄 **GitHub Actions**: Automatic policy validation on PRs
- 📊 **CI/CD Integration**: Fails fast on policy violations
- 🎯 **Cost Prevention**: Blocks expensive misconfigurations early
- 📋 **Clear Messages**: Helpful violation messages with fix suggestions

### 6. Cost-Aware CI/CD

Integrate cost analysis into your deployment pipeline:

```yaml
# GitHub Actions integration (see .github/workflows/)
- name: Infracost Analysis
  uses: infracost/actions/comment@v1
  with:
    path: terraform/
    
- name: Policy Validation
  run: make policy-test
```

## 🏗️ Infrastructure Examples

Ready-to-deploy Terraform modules demonstrating cost optimization patterns:

### S3 Lifecycle Management

```bash
cd terraform-examples/s3-lifecycle
terraform init && terraform apply
```

**Features**:
- Automatic transitions (Standard → IA → Glacier → Deep Archive)
- Cost monitoring and alerts
- Multipart upload cleanup
- Versioning lifecycle rules

### Tagging Guardrails

```bash
cd terraform-examples/tagging-guardrails
terraform apply
```

**Enforces**:
- Required cost allocation tags
- AWS Config compliance rules
- Automated violation notifications
- Cost category creation

### RDS Autopause

```bash
cd terraform-examples/rds-autopause
terraform apply
```

**Provides**:
- Scheduled start/stop automation
- Lambda-based instance management
- Cost savings calculation (40-70% typical)
- SNS notifications

## 🛡️ Policy as Code

Prevent costly misconfigurations with Conftest policies:

```bash
# Validate Terraform plans
make policy-test

# Test against example violations
conftest test --policy policies/conftest policies/examples/
```

**Policies enforce**:
- ✅ Required cost allocation tags
- ✅ S3 lifecycle policy requirements
- ✅ RDS backup and encryption
- ✅ EC2 instance sizing justification
- ✅ Environment-appropriate resource types

## 📈 Dashboard & Visualization

### Athena-based Cost Analysis

Set up comprehensive cost dashboards:

```bash
# Follow the quickstart guide
cat dashboards/quickstart-athena-glue.md
```

**Capabilities**:
- Daily/weekly/monthly cost trends
- Service and environment breakdown
- Untagged resource identification
- Storage optimization opportunities

### Charts and Reports

Generate visual reports:

```python
from scripts.utils.charts import CostChartGenerator

generator = CostChartGenerator()
generator.create_daily_cost_trend(cost_data)
generator.create_service_breakdown_pie(service_costs)
```

## 🔧 Configuration

### Environment Variables

```bash
# Optional: Specify AWS profile
export AWS_PROFILE=your-profile

# Optional: Default output directory
export COST_OPS_OUTPUT_DIR=./reports

# Optional: Slack webhook for notifications
export SLACK_WEBHOOK_URL=https://hooks.slack.com/...
```

### Configuration File

Create `config.yaml` for customization:

```yaml
aws:
  region: us-east-1
  profile: default

cost_thresholds:
  daily_limit: 1000
  monthly_trend: 20

output:
  format: json
  directory: reports

notifications:
  slack_webhook: ${SLACK_WEBHOOK_URL}
  email: ops-team@company.com
```

## 🚨 Safety & Security

### Enhanced Safety Controls

- **🛡️ Triple-layer safety system** - Module constants, CLI validation, and runtime checks
- **🔒 Conservative by default** - All operations run in dry-run mode unless explicitly overridden
- **⚠️ Hard-stops for destructive actions** - Multiple confirmation flags required
- **🧪 Comprehensive test coverage** - 14+ dedicated safety tests ensure controls work
- **📊 Detailed safety reporting** - Every operation logs safety status and constraints
- **🚫 Never delete critical resources** - RDS snapshots, S3 buckets, and EC2 instances are protected

### Deletion Safety Matrix

| Resource Type | Detection | Deletion | Safety Level |
|---------------|-----------|----------|--------------|
| EBS Volumes (unattached) | ✅ | ⚠️ Controlled | Double confirmation required |
| Elastic IPs (unused) | ✅ | ⚠️ Controlled | Double confirmation required |  
| RDS Snapshots | ✅ | 🚫 Never | Read-only analysis only |
| S3 Buckets | ✅ | 🚫 Never | Read-only analysis only |
| EC2 Instances | ✅ | 🚫 Never | Read-only analysis only |

### Enterprise-Grade Security

- ✅ **Professional vulnerability reporting** with clear escalation procedures
- ✅ **Dependency pinning** with `constraints.txt` for reproducible builds
- ✅ **Security scanning** integrated into CI/CD pipeline
- ✅ **No secrets committed** to repository with comprehensive `.gitignore`
- ✅ **Least-privilege IAM policies** with exact permission specifications
- ✅ **Graceful credential handling** with multiple authentication methods
- ✅ **Code review requirements** with security-focused guidelines
- ✅ **Continuous security monitoring** with automated vulnerability detection

### Required IAM Permissions

Minimal permissions for cost analysis:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ce:GetCostAndUsage",
        "ce:GetDimensionValues",
        "ec2:DescribeInstances",
        "cloudwatch:GetMetricStatistics",
        "s3:ListAllMyBuckets",
        "s3:GetBucketLocation"
      ],
      "Resource": "*"
    }
  ]
}
```

## 📚 Documentation

### Comprehensive Guides
- **[Quickstart Guide](docs/quickstart.md)**: Get started in 15 minutes with complete setup instructions
- **[How We Saved 80%](docs/how-we-saved-80-percent.md)**: Real-world case study with ROI analysis
- **[Runbooks](docs/runbooks.md)**: Daily, weekly, and monthly operational procedures
- **[Dashboard Setup](dashboards/quickstart-athena-glue.md)**: Complete visualization guide

### Professional Development Standards
- **[Contributing Guidelines](CONTRIBUTING.md)**: Conventional commits, code style, testing requirements
- **[Security Policy](SECURITY.md)**: Vulnerability reporting, credential management, best practices
- **[Changelog](CHANGELOG.md)**: Detailed release notes and version history

### Quality Assurance
- **Production-ready code** with comprehensive testing and linting
- **Type hints and docstrings** for maintainable code
- **Professional documentation** following industry standards

## 🎯 Use Cases

### For Engineering Leaders

- **Cost Visibility**: Daily automated reports with actionable insights
- **Budget Management**: Proactive alerts and trend analysis
- **Resource Governance**: Policy enforcement and compliance monitoring
- **Team Accountability**: Cost allocation by team, project, and environment

### For DevOps Teams

- **CI/CD Integration**: Cost impact analysis in pull requests
- **Automated Optimization**: Scheduled cleanup and lifecycle management
- **Infrastructure as Code**: Terraform modules with cost optimization built-in
- **Monitoring**: Real-time cost anomaly detection

### For FinOps Teams

- **Cost Attribution**: Detailed tagging and allocation reporting
- **Optimization Opportunities**: Data-driven rightsizing recommendations
- **Compliance**: Automated policy enforcement and violation reporting
- **ROI Tracking**: Before/after cost analysis for optimization initiatives

## 🛠️ Development

### Running Tests

```bash
# Run all tests with coverage
make test

# Run quick tests (no AWS calls required)
make test-quick

# Run tests with detailed coverage report
make test-coverage

# Run linting and code quality checks
make lint

# Format code automatically
make format

# Run security analysis
make security

# Run comprehensive CI checks
make ci-lint && make ci-test && make ci-security
```

**Testing Infrastructure:**
- **30+ comprehensive tests** covering utilities, safety controls, and CLI behavior
- **14 dedicated safety tests** ensuring deletion controls work correctly
- **Mock-based AWS testing** for safe CI/CD execution without real resources
- **85%+ code coverage** with detailed reporting and HTML output
- **No AWS credentials required** for quick testing and CI validation
- **Safety-first test philosophy** - All destructive operations are thoroughly tested

### Project Structure

```
cloud-cost-ops-toolkit/
├── scripts/                 # Core Python scripts
│   ├── cost_report.py      # Daily cost reporting
│   ├── rightsizing.py      # Resource optimization
│   ├── cleanup_idle.py     # Idle resource cleanup
│   ├── ensure_s3_lifecycle.py # S3 lifecycle automation
│   ├── notify_slack.py     # Slack notifications
│   └── utils/              # Shared utilities
├── terraform-examples/     # Infrastructure examples
│   ├── s3-lifecycle/       # S3 cost optimization
│   ├── tagging-guardrails/ # Cost allocation enforcement
│   └── rds-autopause/      # Database scheduling
├── policies/               # Policy as code
│   ├── conftest/          # Terraform policies
│   └── examples/          # Violation examples
├── tests/                  # Test suite
│   ├── test_utils.py      # Unit tests for utilities
│   ├── test_cleanup_safety.py # Safety control tests
│   └── __init__.py        # Test package
├── .github/workflows/      # CI/CD automation
├── docs/                  # Documentation
├── requirements.txt       # Python dependencies
├── constraints.txt        # Dependency version pins
└── Makefile               # Build automation
```

## 🤝 Contributing

We welcome contributions! This project follows professional development standards with comprehensive guidelines.

### Quick Contribution Steps

1. **Fork the repository** and read [CONTRIBUTING.md](CONTRIBUTING.md)
2. **Create a feature branch**: `git checkout -b feature/amazing-optimization`
3. **Follow conventional commits**: Use standard commit message format
4. **Write tests**: Maintain the 85%+ coverage standard
5. **Run quality checks**: `make test && make lint && make policy-test`
6. **Submit a pull request** with detailed description

### Development Standards
- ✅ **Conventional Commits** for clear change tracking
- ✅ **Comprehensive testing** with mock AWS interactions
- ✅ **Code quality enforcement** with Black, flake8, and mypy
- ✅ **Security review** for all changes
- ✅ **Documentation updates** for user-facing changes

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- **AWS Cost Explorer API** for comprehensive cost data access
- **AWS Compute Optimizer** for ML-driven rightsizing recommendations
- **Infracost** for infrastructure cost estimation and PR integration
- **Conftest & Open Policy Agent** for policy-as-code validation
- **Rich** for beautiful terminal output and user experience
- **pytest** for comprehensive testing framework with safety controls
- **Black, flake8, mypy** for code quality and consistency
- **safety, bandit** for security scanning and vulnerability detection
- The **FinOps community** for cost optimization best practices and standards

## 📧 Support

- **Documentation**: Start with [docs/quickstart.md](docs/quickstart.md)
- **Issues**: Report bugs and request features via GitHub Issues
- **Discussions**: Join the conversation in GitHub Discussions
- **Security**: Report security issues via [SECURITY.md](SECURITY.md)

---

**🎉 Ready to optimize your AWS costs?** This toolkit is production-ready and has been thoroughly tested with comprehensive documentation.

**✅ Start here:**
1. `make install` - Set up your virtual environment  
2. `make test-quick` - Verify everything works (no AWS credentials needed)
3. `make report` - Generate your first cost analysis
4. Review the optimization opportunities and follow the [quickstart guide](docs/quickstart.md)

**🔒 Safe by Default**: All operations show you what they *would* do before making any changes. The toolkit prioritizes safety and provides clear operational guidance for production use.

cloud-cost-ops-toolkit

cloud-cost-ops-toolkit Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

cloud-cost-ops-toolkit

cloud-cost-ops-toolkit Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README