Test Suite Organization

This directory contains a comprehensive test suite organized for different CI/CD scenarios. The tests are carefully separated to ensure safe CI execution while still providing real API validation.

🎯 Test Categories

1. Unit Tests (`unit` marker)

Purpose: Fast, isolated tests with no external dependencies
CI Safe: ✅ Always safe to run
Cost: 🆓 Free
When to run: Every commit, every PR
Examples: Model validation, utility functions, core logic

2. Integration Tests (`integration` marker)

Purpose: Component integration with mocked external services
CI Safe: ✅ Safe when excluding real_api marker
Cost: 🆓 Free (uses mocks)
When to run: Every commit, every PR
Examples: Service integrations with mocked APIs

3. Real API Tests (`real_api` marker)

Purpose: End-to-end validation with actual API services
CI Safe: ❌ Only run manually or on schedule
Cost: 💰 Costs money (API calls)
When to run: Manual trigger, scheduled validation
Examples: OpenAI API calls, AWS Bedrock integration

🚀 Running Tests

Quick Start

# Run all safe tests (recommended for CI)
python run_tests.py --mode mock

# Run only unit tests (fastest)
python run_tests.py --mode unit

# Run integration tests with mocks
python run_tests.py --mode integration

CI/CD Modes

# CI Safe modes (no API calls, no costs)
python run_tests.py --mode unit           # Unit tests only
python run_tests.py --mode integration    # Integration tests with mocks
python run_tests.py --mode mock           # All safe tests (default)
python run_tests.py --mode all-safe       # All safe tests (explicit)

# Manual/Scheduled modes (cost money)
python run_tests.py --mode real-api       # Real API tests (with confirmation)
python run_tests.py --mode all            # Everything (with confirmation)

Direct pytest Usage

# Safe for CI (no API calls)
pytest -m "not real_api"                  # All safe tests
pytest -m "unit"                          # Unit tests only
pytest -m "integration and not real_api"  # Integration tests with mocks

# Costs money (manual only)
pytest -m "real_api"                      # Real API tests
pytest -m "openai_integration"            # OpenAI specific tests
pytest -m "bedrock_integration"           # Bedrock specific tests

🔧 GitHub Actions Integration

CI Workflow (Automatic)

File: .github/workflows/ci-tests.yml

# Runs on every push/PR - SAFE
- Push to main/develop
- Pull requests
- Multiple Python versions
- Coverage reporting
- No API calls, no costs

Real API Workflow (Manual)

File: .github/workflows/real-api-tests.yml

# Manual trigger only - COSTS MONEY
- workflow_dispatch with confirmation
- Optional scheduled runs
- Uses secrets for API keys
- Cost acknowledgment required

📊 Test Organization

Directory Structure

tests/
├── test_*.py                    # Main test files
├── api/                         # API-specific tests
├── core/                        # Core functionality tests
├── services/                    # Service layer tests
├── utils/                       # Utility tests
├── conftest.py                  # Test configuration
├── README.md                    # This file
└── README_REAL_API_TESTS.md     # Real API test details

Test File Organization

File	Type	Markers	Description
`test_models.py`	Unit	`unit`	Data model validation
`test_auth.py`	Unit	`unit`	Authentication logic
`test_chat.py`	Mixed	`integration`, some `real_api`	Chat functionality
`test_bedrock_chat.py`	Mixed	`integration`, some `real_api`	Bedrock integration
`test_real_api_integration.py`	Real API	`real_api`	Comprehensive API tests

🔒 Safety Features

1. Marker Protection

Tests with real_api marker only run when explicitly requested
Default test runs exclude expensive tests
Clear separation between mock and real tests

2. Cost Warnings

Interactive prompts before running expensive tests
Clear cost estimates for different test modes
Confirmation required for real API tests

3. CI/CD Safety

GitHub Actions workflows separated by cost/safety
Real API tests require manual trigger and confirmation
Environment variable validation before API calls

📋 Test Markers Reference

Marker	Purpose	CI Safe	Cost	Usage
`unit`	Fast unit tests	✅	🆓	`-m unit`
`integration`	Integration with mocks	✅	🆓	`-m integration`
`real_api`	Real API calls	❌	💰	`-m real_api`
`openai_integration`	OpenAI specific	❌	💰	`-m openai_integration`
`bedrock_integration`	Bedrock specific	❌	💰	`-m bedrock_integration`
`slow`	Slow tests	✅	🆓	`-m "not slow"`

🎮 Development Workflow

During Development

# Quick validation
python run_tests.py --mode unit --verbose

# Full local validation (safe)
python run_tests.py --mode mock --verbose

# Before committing
python run_tests.py --mode all-safe

Before Release

# Validate real API integration (costs money)
python run_tests.py --mode real-api

CI/CD Pipeline

# Automated (safe)
- Every commit: unit tests
- Every PR: all safe tests
- Merge to main: full safe test suite

# Manual (costs money)  
- Release validation: real API tests
- Weekly scheduled: comprehensive API tests

🛡️ Best Practices

For CI/CD

Never run real API tests automatically - always require manual trigger
Use cost confirmations - require explicit acknowledgment of costs
Set spending limits - configure API key spending limits
Monitor usage - set up billing alerts for API usage
Separate environments - use different API keys for testing vs production

For Development

Start with unit tests - write fast, isolated tests first
Mock external dependencies - use mocks for integration tests
Validate with real APIs sparingly - only when necessary
Document cost implications - clearly mark expensive tests
Use minimal API calls - keep real API tests lightweight

For Testing

Layer your tests - unit → integration → real API
Mark tests appropriately - use correct markers for each test
Handle credentials safely - never commit API keys
Test error scenarios - include negative test cases
Monitor test costs - track spending on API calls

🔍 Troubleshooting

Common Issues

Tests not running in CI
- Check that you’re using safe markers (not real_api)
- Verify GitHub Actions workflow configuration
Real API tests failing
- Check API credentials in environment/secrets
- Verify API key permissions and limits
- Check service availability and quotas
Unexpected costs
- Ensure real API tests have real_api marker
- Check that CI workflows exclude expensive tests
- Review API usage dashboards

Debugging Commands

# Check test discovery
pytest --collect-only

# Dry run without execution
pytest --collect-only -m real_api

# Run with maximum verbosity
python run_tests.py --mode unit --verbose -v

# Check marker filtering
pytest -m "not real_api" --collect-only

📈 Continuous Improvement

Metrics to Track

Test execution time (aim for fast CI)
API usage costs (minimize for validation)
Test coverage (maintain high coverage with safe tests)
Flaky test rates (especially for real API tests)

Regular Reviews

Weekly: Review test execution times
Monthly: Review API costs and usage patterns
Quarterly: Review test organization and effectiveness
Before releases: Validate real API functionality

This organization ensures safe, fast CI execution while maintaining comprehensive validation through carefully managed real API testing.