feat: add Test Coverage Optimizer recipe (#5118) (#5272)

Signed-off-by: Shivam Gawali <shivamgawali585@gmail.com> Co-authored-by: Rizel Scarlett <rizel@tbd.email>
2025-10-22 00:59:06 +05:30
parent 574ca2b290
commit 5a35f0952e
1 changed files with 530 additions and 0 deletions
@@ -0,0 +1,530 @@
 # ============================================================================
 # RECIPE METADATA
 # ============================================================================
 # Test Coverage Optimizer - An intermediate recipe that uses Developer and
 # Memory extensions to analyze test coverage patterns, learn from existing
 # tests, and generate intelligent test suggestions that improve over time.
 version: 1.0.0
 title: Test Coverage Optimizer
 description: |
  Analyzes test coverage patterns, learns from existing tests, and generates
  targeted test suggestions to improve coverage systematically. Uses memory
  to learn testing patterns and improve suggestions over time.
 author:
  contact: shiv669
 # ============================================================================
 # ACTIVITIES - User-Facing Workflow Steps
 # ============================================================================
 # These describe what the user will experience during recipe execution
 activities:
  - Scan project test files and coverage metrics
  - Analyze existing test patterns and methodologies
  - Retrieve learned testing patterns from memory
  - Identify critical code gaps and missing test coverage
  - Generate targeted, practical test suggestions based on learned patterns
  - Create test file templates with concrete examples
  - Store new patterns for continuous improvement
 # ============================================================================
 # INSTRUCTIONS - Role and Capabilities
 # ============================================================================
 # explains the role and key capabilities
 instructions: |
  You are a Test Coverage Optimization Specialist that helps development teams
  improve test coverage systematically and intelligently.
  Your goal is to analyze existing test patterns, identify critical gaps,
  and suggest high-impact new tests that developers can implement immediately.
  Key capabilities you possess:
  - Scan codebases for test files across multiple frameworks (pytest, jest, unittest, go-test)
  - Analyze existing test patterns and methodologies to understand team practices
  - Identify code paths that lack test coverage using static analysis
  - Remember and learn from previous testing patterns and improvements
  - Generate practical, ready-to-use test code with explanations
  - Track and improve suggestions based on feedback and patterns
  - Create customizable test templates that match team coding standards
  IMPORTANT: Always start by checking Memory for any saved testing patterns,
  team preferences, and previous coverage analysis for this project.
  This ensures your suggestions improve over time and match team standards.
 # ============================================================================
 # PARAMETERS - User Configuration Options
 # ============================================================================
 # These are the inputs users can provide to customize recipe behavior
 parameters:
  # Parameter 1: Project location
  - key: project_path
    input_type: string
    requirement: optional
    default: "."
    description: "Path to project root directory (where source code is located)"
  # Parameter 2: Testing framework selection
  - key: test_framework
    input_type: string
    requirement: optional
    default: "auto"
    description: "Testing framework to target: 'jest' (JavaScript), 'pytest' (Python), 'unittest' (Python), 'go-test' (Go), or 'auto' for auto-detection"
  # Parameter 3: Coverage target
  - key: coverage_threshold
    input_type: string
    requirement: optional
    default: "80"
    description: "Target coverage percentage (0-100). Used to identify gap size and prioritize tests."
  # Parameter 4: Focus areas for analysis
  - key: focus_areas
    input_type: string
    requirement: optional
    default: "all"
    description: "Comma-separated focus areas: 'core-logic' (primary business logic), 'edge-cases' (boundary conditions), 'error-handling' (exceptions), 'integration' (API/module interactions), or 'all' for comprehensive analysis"
  # Parameter 5: Template generation toggle
  - key: generate_templates
    input_type: string
    requirement: optional
    default: "true"
    description: "Whether to generate test file templates with examples (true/false). Set to false if you only want suggestions without code templates."
 # ============================================================================
 # EXTENSIONS - AI Tools Working Together
 # ============================================================================
 # Exactly 2 extensions that integrate sequentially for intermediate recipe
 extensions:
  # Extension 1: Memory - Context and pattern storage (runs first)
  - type: builtin
    name: memory
    display_name: Memory
    timeout: 300
    bundled: true
    description: |
      Stores and retrieves learned testing patterns, previous coverage analysis,
      and team preferences. Allows the recipe to improve suggestions over time
      by remembering what worked well in past runs.
  # Extension 2: Developer - Code analysis and generation (uses Memory's context)
  - type: builtin
    name: developer
    display_name: Developer
    timeout: 600
    bundled: true
    description: |
      Scans test files, analyzes coverage metrics, identifies code gaps,
      and generates new test code based on analysis and learned patterns
      from Memory. This is the primary tool for all file and code operations.
 # ============================================================================
 # PROMPT - Detailed Workflow with Extension Integration
 # ============================================================================
 # Step-by-step instructions showing how Memory and Developer extensions work together
 prompt: |
  You are analyzing test coverage for the project at {{ project_path }} to generate
  intelligent test suggestions that improve coverage to {{ coverage_threshold }}%.
  # =========================================================================
  # STEP 1: LOAD CONTEXT FROM MEMORY
  # =========================================================================
  # Purpose: Retrieve previously learned patterns to inform current analysis
  # Extension: Memory (loads first)
  Step 1: Load Testing Patterns from Memory
    First, retrieve any previously learned testing patterns for this project:
    - Retrieve key "{{ project_path }}_testing_style" - the team's preferred test structure
    - Retrieve key "{{ project_path }}_gap_patterns" - common gap patterns from past analysis
    - Retrieve key "{{ project_path }}_preferred_frameworks" - frameworks the team uses
    - Retrieve key "testing_best_practices" - general testing best practices library
    If this is your first time analyzing this project, you'll start fresh
    and build up knowledge over time. Store any retrieved patterns in your
    context for use in the generation steps below.
  # =========================================================================
  # STEP 2: DETECT TEST FRAMEWORK
  # =========================================================================
  # Purpose: Identify which testing framework is used in this project
  # Extension: Developer (file scanning)
  Step 2: Detect Testing Framework
    {% if test_framework == "auto" %}
    Since test_framework is set to 'auto', detect the framework automatically:
      a) Search for test file patterns in {{ project_path }}:
         - .test.js, .spec.js, __tests__/*.js files → JavaScript/Jest
         - _test.py, test_*.py, tests/*.py files → Python/pytest or unittest
         - *_test.go, *_test.go files → Go/go-test
         - *.test.ts, *.spec.ts files → TypeScript/Jest
      b) Check for configuration files:
         - package.json with "jest" dependency → Jest
         - pytest.ini, setup.cfg, pyproject.toml → pytest
         - go.mod → go test
         - unittest imports in test files → Python unittest
      c) Report the detected framework with confidence level
         Example: "Detected: pytest (high confidence - found pytest.ini)"
    {% else %}
    Use the specified framework: {{ test_framework }}
    Verify that test files for {{ test_framework }} exist in {{ project_path }}
    If no test files found, suggest creating a basic test structure for {{ test_framework }}
    {% endif %}
  # =========================================================================
  # STEP 3: ANALYZE CURRENT TEST COVERAGE
  # =========================================================================
  # Purpose: Understand current state and identify coverage gaps
  # Extension: Developer (code scanning and analysis)
  Step 3: Analyze Current Test Coverage
    Scan the project at {{ project_path }} for test coverage information:
    a) Test File Inventory:
       - Find all test files matching the detected/specified framework
       - Count total number of test files and individual test cases
       - Calculate ratio: lines of test code vs lines of application code
       - List the main test directories found
    b) Coverage Data Collection:
       - Look for coverage reports in common locations:
         * Python: .coverage, coverage.xml, htmlcov/, .coverage.*
         * JavaScript: coverage/, coverage.json, coverage-final.json
         * Go: coverage.out, coverage.html
       - If coverage reports exist, parse them to get:
         * Overall coverage percentage
         * Per-file coverage breakdown
         * Uncovered line numbers
       - If NO coverage reports found:
         * Estimate coverage by analyzing test file imports and function calls
         * Identify source files that have no corresponding test files
         * Note: "No coverage data found - will identify gaps by analysis"
    c) Gap Analysis:
       - Compare current coverage to target {{ coverage_threshold }}%
       - Identify files with coverage below threshold
       - List top 10 files/modules with lowest coverage
       - Focus analysis on {{ focus_areas }}
       - Prioritize gaps by impact:
         * Critical: Core business logic with <50% coverage
         * High: Error handling and edge cases with <70% coverage
         * Medium: Helper functions and utilities with <80% coverage
  # =========================================================================
  # STEP 4: IDENTIFY GAPS USING LEARNED PATTERNS
  # =========================================================================
  # Purpose: Use Memory's patterns to make smarter gap identification
  # Extension: Both (Developer analysis + Memory context)
  Step 4: Identify Testing Gaps (Using Learned Patterns)
    Using both the coverage analysis AND the learned patterns from Step 1:
    a) Apply learned patterns to current analysis:
       - If Memory contains testing style preferences, apply them
       - Check if current gaps match previously identified gap patterns
       - Consider team's preferred test granularity (unit vs integration)
       - Note patterns like: "Team prefers testing error cases separately"
    b) Categorize and prioritize missing tests by {{ focus_areas }}:
       {% if focus_areas == "all" or "core-logic" in focus_areas %}
       - Core Logic Gaps:
         * Functions/methods with no test coverage
         * Business logic paths not exercised by tests
         * Main workflows missing test scenarios
       {% endif %}
       {% if focus_areas == "all" or "edge-cases" in focus_areas %}
       - Edge Case Gaps:
         * Boundary conditions (empty inputs, max values, null/undefined)
         * Unusual input combinations
         * Race conditions or timing-sensitive code
       {% endif %}
       {% if focus_areas == "all" or "error-handling" in focus_areas %}
       - Error Handling Gaps:
         * Exception paths not tested
         * Error callbacks or error states
         * Validation failure scenarios
       {% endif %}
       {% if focus_areas == "all" or "integration" in focus_areas %}
       - Integration Gaps:
         * API endpoint tests
         * Database interaction tests
         * External service mock/integration tests
       {% endif %}
    c) Create prioritized list:
       - Rank gaps by: (impact × likelihood × ease of testing)
       - Mark which gaps align with learned patterns
       - Identify quick wins (high impact, easy to test)
  # =========================================================================
  # STEP 5: GENERATE TEST SUGGESTIONS
  # =========================================================================
  # Purpose: Create specific, actionable test recommendations
  # Extension: Developer (code generation using learned context)
  Step 5: Generate Test Suggestions
    For each identified gap, create specific test suggestions:
    a) For each high-priority gap:
       - Write a clear, descriptive test name following framework conventions
       - Explain what this test should verify
       - Provide the expected test structure for {{ test_framework }}
       - Include a brief code example showing the test skeleton
       - If learned patterns exist, match the team's testing style
       Example format:
       ```
       Test: test_user_authentication_with_invalid_credentials
       Purpose: Verify that authentication fails gracefully with wrong password
       Priority: Critical (core-logic, error-handling)
       Estimated effort: 10 minutes
       Code example for {{ test_framework }}:
       [Framework-specific test code here]
       ```
    b) Include implementation context:
       - Why this test is important (coverage gap, risk mitigation)
       - Which code path it covers
       - Expected assertions and edge cases to test
       - Dependencies or mocks needed
    c) Organize suggestions by priority:
       - Critical (must-have for {{ coverage_threshold }}% target)
       - High (important for robust coverage)
       - Medium (nice-to-have, improves confidence)
    d) Limit initial suggestions:
       - Provide top 5-10 most impactful tests
       - Note: "Implement these first, then re-run for more suggestions"
  # =========================================================================
  # STEP 6: CREATE TEST TEMPLATES (Optional)
  # =========================================================================
  # Purpose: Generate ready-to-use test file templates
  # Extension: Developer (template creation)
  Step 6: Generate Test Templates
    {% if generate_templates == "true" %}
    Create test file templates that developers can immediately use:
    a) For files with NO existing tests:
       - Create a complete new test file template
       - Include proper imports for {{ test_framework }}
       - Add setup/teardown methods if applicable
       - Provide 2-3 example test functions with placeholders
       - Match the coding style from Memory if available
    b) For files with SOME existing tests:
       - Generate test functions to add to existing test files
       - Match the existing test file's style and structure
       - Include comments indicating where to insert the new tests
    c) Include in templates:
       - Proper test file naming (matching framework conventions)
       - Required imports and dependencies
       - Mock/fixture setup if needed
       - Clear TODOs for values to fill in
       - Example assertions showing expected patterns
    d) Output format:
       - Provide templates as copyable code blocks
       - Include file paths where templates should be saved
       - Add instructions for running the tests
    Example output:
    ```
    # File: tests/test_user_authentication.py
    # Framework: pytest
    # Purpose: Tests for user authentication module
    import pytest
    from myapp.auth import authenticate_user
    def test_authentication_success():
        # TODO: Replace with actual test data
        user = {"username": "test_user", "password": "correct_password"}
        result = authenticate_user(user)
        assert result.success is True
        assert result.user_id is not None
    def test_authentication_invalid_password():
        # Test critical error path
        user = {"username": "test_user", "password": "wrong_password"}
        result = authenticate_user(user)
        assert result.success is False
        assert result.error == "Invalid credentials"
    ```
    {% else %}
    Template generation is disabled. Providing suggestions only (see Step 5).
    {% endif %}
  # =========================================================================
  # STEP 7: STORE PATTERNS IN MEMORY
  # =========================================================================
  # Purpose: Learn from this analysis to improve future runs
  # Extension: Memory (pattern storage)
  Step 7: Store Analysis Results and Patterns in Memory
    Update Memory with findings from this analysis:
    a) Store project-specific patterns:
       - Save key "{{ project_path }}_testing_style":
         * Detected framework: {{ test_framework }}
         * Common test patterns observed (e.g., "uses fixtures", "prefers mocks")
         * Naming conventions detected
         * Code style preferences
       - Save key "{{ project_path }}_gap_patterns":
         * Types of gaps found this run
         * Areas consistently lacking coverage
         * Common missing test scenarios
       - Save key "{{ project_path }}_coverage_baseline":
         * Current coverage: [calculated percentage]%
         * Target coverage: {{ coverage_threshold }}%
         * Date of analysis
         * Number of tests suggested
    b) Update general knowledge:
       - Save key "testing_best_practices":
         * Effective test patterns encountered
         * Framework-specific tips learned
         * Common pitfalls to avoid
    c) Store metadata for tracking:
       - Analysis timestamp
       - Framework detected/used
       - Focus areas analyzed
       - Number of suggestions generated
       - Estimated coverage improvement
    Note: This stored information will be used in Step 1 of the next run
    to provide better, more contextual suggestions.
  # =========================================================================
  # STEP 8: PRESENT FINAL REPORT
  # =========================================================================
  # Purpose: Clearly communicate results and next steps to user
  # Extension: Both (summarized results)
  Step 8: Present Comprehensive Report
    Provide a well-structured final report to the user:
    a) Coverage Summary:
       ```
       📊 Test Coverage Analysis for {{ project_path }}
       Current Coverage: [calculated percentage]% (based on analysis/reports)
       Target Coverage: {{ coverage_threshold }}%
       Coverage Gap: [target minus current]% (target - current)
       Framework Detected: {{ test_framework }}
       Focus Areas: {{ focus_areas }}
       Tests Analyzed: [count] test files, [count] test cases
       ```
    b) Top Recommended Tests:
       List the 5-10 highest priority test suggestions:
       ```
       🎯 High-Priority Test Suggestions:
       1. [CRITICAL] test_user_login_with_invalid_credentials
          - What: Verify authentication fails gracefully
          - Why: Core security logic, currently untested
          - Effort: ~10 minutes
          - Coverage gain: +2-3%
       2. [CRITICAL] test_data_validation_edge_cases
          - What: Test empty/null input handling
          - Why: Common source of production bugs
          - Effort: ~15 minutes
          - Coverage gain: +1-2%
       [... continue for top suggestions]
       ```
    c) Implementation Guidance:
       ```
       📝 Next Steps:
       1. Start with CRITICAL tests (biggest impact)
       2. {% if generate_templates == "true" %}
          Use the templates provided above - copy and customize
          {% else %}
          Create test files based on suggestions above
          {% endif %}
       3. Run tests to verify they work: {{ test_framework }} [command]
       4. Run coverage again to see improvement
       5. Re-run this recipe for additional suggestions
       💡 Pro Tips:
       - Implement tests iteratively (don't try to do everything at once)
       - Run coverage after each batch to track progress
       - For {{ coverage_threshold }}% target, focus on core-logic first
       - This recipe learns - patterns will improve with each run
       ```
    d) Coverage Projection:
       ```
       📈 Estimated Impact:
       If you implement the top 5 suggested tests:
       - Estimated coverage increase: +[percentage]%
       - New coverage projection: [new total]%
       - Remaining gap to target: [remaining gap]%
       Tests needed to reach {{ coverage_threshold }}%: approximately [number] more tests
       ```
    e) Learned Patterns Summary:
       ```
       🧠 Memory Updated:
       - Stored testing style preferences for {{ project_path }}
       - Saved [count] gap patterns for future reference
       - Updated coverage baseline
       - Future runs will provide better suggestions based on this analysis
       ```
  # =========================================================================
  # COMPLETION
  # =========================================================================
  Your analysis is complete! The user now has:
  - Clear understanding of current coverage gaps
  - Prioritized list of tests to implement
  {% if generate_templates == "true" %}
  - Ready-to-use test templates
  {% endif %}
  - Improved future suggestions through Memory learning
  Remind the user to re-run this recipe after implementing tests to:
  1. See coverage improvements
  2. Get additional suggestions based on learned patterns
  3. Continue iterating toward {{ coverage_threshold }}% coverage