feat: add A/B test framework generator recipe (#5378)
Signed-off-by: Shreyansh Singh Gautam <shreyanshrewa@gmail.com>
This commit is contained in:
Shreyansh Singh Gautam
committed by
GitHub
GitHub
parent
c3578b708f
commit
491eabfc81
@@ -0,0 +1,268 @@
|
|||||||
|
version: 1.0.0
|
||||||
|
title: A/B Test Framework Generator
|
||||||
|
description: An advanced recipe that generates complete A/B testing infrastructure for web applications, including variant setup, tracking code, statistical analysis, and interactive reporting dashboard with intelligent framework detection and multi-stage orchestration
|
||||||
|
author:
|
||||||
|
contact: scaler
|
||||||
|
|
||||||
|
activities:
|
||||||
|
- Detect web application framework and project structure
|
||||||
|
- Generate A/B test variant implementation templates
|
||||||
|
- Create tracking and analytics integration code
|
||||||
|
- Set up experiment configuration and user bucketing
|
||||||
|
- Implement statistical significance analysis framework
|
||||||
|
- Generate interactive reporting dashboard with real-time metrics
|
||||||
|
- Create comprehensive documentation and setup guide
|
||||||
|
- Optionally commit changes and create pull request
|
||||||
|
|
||||||
|
instructions: |
|
||||||
|
You are an A/B Test Framework Generator that creates complete testing infrastructure for web applications.
|
||||||
|
|
||||||
|
Your capabilities:
|
||||||
|
1. Detect web frameworks (React, Vue, Angular, vanilla JS) and adapt implementations
|
||||||
|
2. Generate variant-specific code templates with proper randomization
|
||||||
|
3. Create tracking event handlers and analytics integration
|
||||||
|
4. Set up statistical analysis framework for significance testing
|
||||||
|
5. Build interactive dashboards for real-time experiment monitoring
|
||||||
|
6. Orchestrate multiple sub-recipes for specialized tasks
|
||||||
|
7. Handle parameter passing and conditional logic based on framework type
|
||||||
|
|
||||||
|
Focus on:
|
||||||
|
- Production-ready A/B testing infrastructure
|
||||||
|
- Statistical rigor with proper significance testing
|
||||||
|
- Framework-specific implementations
|
||||||
|
- Real-time monitoring and reporting
|
||||||
|
- Comprehensive documentation and setup guides
|
||||||
|
- Manual file operations (users will need to commit changes themselves)
|
||||||
|
|
||||||
|
parameters:
|
||||||
|
- key: project_path
|
||||||
|
input_type: string
|
||||||
|
requirement: required
|
||||||
|
description: Path to the web application project directory to add A/B testing infrastructure
|
||||||
|
|
||||||
|
- key: framework
|
||||||
|
input_type: string
|
||||||
|
requirement: optional
|
||||||
|
default: "auto"
|
||||||
|
description: Web framework type - options are 'auto', 'react', 'vue', 'angular', 'vanilla'
|
||||||
|
|
||||||
|
- key: test_name
|
||||||
|
input_type: string
|
||||||
|
requirement: required
|
||||||
|
description: Name of the A/B test (e.g., 'button-color-test', 'checkout-flow-test')
|
||||||
|
|
||||||
|
- key: variants
|
||||||
|
input_type: string
|
||||||
|
requirement: required
|
||||||
|
description: Comma-separated variant names (e.g., 'control,variant-a,variant-b')
|
||||||
|
|
||||||
|
- key: metrics
|
||||||
|
input_type: string
|
||||||
|
requirement: required
|
||||||
|
description: Comma-separated metrics to track (e.g., 'conversion,engagement,bounce-rate,click-through')
|
||||||
|
|
||||||
|
- key: sample_size
|
||||||
|
input_type: string
|
||||||
|
requirement: optional
|
||||||
|
default: "1000"
|
||||||
|
description: Minimum sample size per variant for statistical significance
|
||||||
|
|
||||||
|
- key: confidence_level
|
||||||
|
input_type: string
|
||||||
|
requirement: optional
|
||||||
|
default: "95"
|
||||||
|
description: Statistical confidence level for significance testing (90, 95, 99)
|
||||||
|
|
||||||
|
- key: include_dashboard
|
||||||
|
input_type: string
|
||||||
|
requirement: optional
|
||||||
|
default: "true"
|
||||||
|
description: Whether to generate interactive reporting dashboard (true/false)
|
||||||
|
|
||||||
|
|
||||||
|
sub_recipes:
|
||||||
|
- name: "experiment_tracker"
|
||||||
|
path: "./subrecipes/experiment-tracker.yaml"
|
||||||
|
values:
|
||||||
|
test_name: "{{ test_name }}"
|
||||||
|
variants: "{{ variants }}"
|
||||||
|
metrics: "{{ metrics }}"
|
||||||
|
framework: "{{ framework }}"
|
||||||
|
|
||||||
|
- name: "statistical_analyzer"
|
||||||
|
path: "./subrecipes/ab-test-statistical-analyzer.yaml"
|
||||||
|
values:
|
||||||
|
sample_size: "{{ sample_size }}"
|
||||||
|
confidence_level: "{{ confidence_level }}"
|
||||||
|
metrics: "{{ metrics }}"
|
||||||
|
|
||||||
|
- name: "dashboard_generator"
|
||||||
|
path: "./subrecipes/ab-test-dashboard-generator.yaml"
|
||||||
|
values:
|
||||||
|
test_name: "{{ test_name }}"
|
||||||
|
variants: "{{ variants }}"
|
||||||
|
metrics: "{{ metrics }}"
|
||||||
|
include_dashboard: "{{ include_dashboard }}"
|
||||||
|
|
||||||
|
extensions:
|
||||||
|
- type: builtin
|
||||||
|
name: developer
|
||||||
|
display_name: Developer
|
||||||
|
timeout: 600
|
||||||
|
bundled: true
|
||||||
|
description: For file operations, code generation, and framework detection
|
||||||
|
|
||||||
|
- type: builtin
|
||||||
|
name: memory
|
||||||
|
display_name: Memory
|
||||||
|
timeout: 300
|
||||||
|
bundled: true
|
||||||
|
description: For storing experiment configurations and tracking patterns across sessions
|
||||||
|
|
||||||
|
prompt: |
|
||||||
|
Generate complete A/B testing infrastructure for {{ project_path }} with test "{{ test_name }}" and variants: {{ variants }}.
|
||||||
|
|
||||||
|
CRITICAL: Handle file paths correctly for all operating systems.
|
||||||
|
- Detect the operating system (Windows/Linux/Mac)
|
||||||
|
- Use appropriate path separators (/ for Unix, \\ for Windows)
|
||||||
|
- Be careful to avoid escaping of slash or backslash characters
|
||||||
|
- Use os.path.join() or pathlib.Path for cross-platform paths
|
||||||
|
- Create A/B test directories if they don't exist
|
||||||
|
|
||||||
|
Workflow:
|
||||||
|
1. Framework Detection & Project Analysis
|
||||||
|
- Detect web framework in {{ project_path }}:
|
||||||
|
* Look for package.json with React/Vue/Angular dependencies
|
||||||
|
* Check for framework-specific files (src/, components/, etc.)
|
||||||
|
* Identify build system (webpack, vite, rollup, etc.)
|
||||||
|
* Store framework detection results in memory
|
||||||
|
- Analyze project structure for integration points:
|
||||||
|
* Identify entry points and main components
|
||||||
|
* Check for existing analytics/tracking setup
|
||||||
|
* Determine state management approach
|
||||||
|
* Note CSS framework and styling approach
|
||||||
|
|
||||||
|
2. Experiment Configuration Setup
|
||||||
|
- Create experiment configuration structure:
|
||||||
|
* Generate experiment config JSON/YAML file
|
||||||
|
* Define variant specifications and traffic allocation
|
||||||
|
* Set up user bucketing and randomization logic
|
||||||
|
* Configure metrics tracking definitions
|
||||||
|
* Store configuration in memory for sub-recipe use
|
||||||
|
|
||||||
|
3. Variant Implementation Templates
|
||||||
|
{% if framework == "react" or framework == "auto" %}
|
||||||
|
- Generate React-specific templates:
|
||||||
|
* A/B test hook (useABTest) for component variants
|
||||||
|
* Higher-order component for variant wrapping
|
||||||
|
* Context provider for experiment state management
|
||||||
|
* TypeScript definitions for type safety
|
||||||
|
{% endif %}
|
||||||
|
{% if framework == "vue" or framework == "auto" %}
|
||||||
|
- Generate Vue-specific templates:
|
||||||
|
* Vue composable for A/B test logic
|
||||||
|
* Mixin for component variant handling
|
||||||
|
* Plugin for global experiment management
|
||||||
|
* TypeScript support for Vue 3
|
||||||
|
{% endif %}
|
||||||
|
{% if framework == "angular" or framework == "auto" %}
|
||||||
|
- Generate Angular-specific templates:
|
||||||
|
* Service for experiment management
|
||||||
|
* Directive for variant rendering
|
||||||
|
* Guard for experiment-based routing
|
||||||
|
* Module configuration
|
||||||
|
{% endif %}
|
||||||
|
{% if framework == "vanilla" or framework == "auto" %}
|
||||||
|
- Generate vanilla JS templates:
|
||||||
|
* Core A/B test library
|
||||||
|
* DOM manipulation utilities
|
||||||
|
* Event tracking helpers
|
||||||
|
* Browser compatibility layer
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
4. Tracking & Analytics Integration
|
||||||
|
- Create tracking event handlers:
|
||||||
|
* Variant assignment tracking
|
||||||
|
* Conversion event tracking
|
||||||
|
* User behavior analytics
|
||||||
|
* Performance metrics collection
|
||||||
|
- Set up data collection pipeline:
|
||||||
|
* Local storage for user assignments
|
||||||
|
* Cookie-based persistence
|
||||||
|
* API endpoints for data submission
|
||||||
|
* Error handling and fallbacks
|
||||||
|
|
||||||
|
5. Run Experiment Tracker Sub-recipe
|
||||||
|
- Execute experiment_tracker sub-recipe with:
|
||||||
|
* test_name: {{ test_name }}
|
||||||
|
* variants: {{ variants }}
|
||||||
|
* metrics: {{ metrics }}
|
||||||
|
* framework: {{ framework }}
|
||||||
|
- Capture returned tracking code and configuration
|
||||||
|
- Store results in memory for dashboard generation
|
||||||
|
|
||||||
|
6. Statistical Analysis Framework
|
||||||
|
- Run statistical_analyzer sub-recipe with:
|
||||||
|
* sample_size: {{ sample_size }}
|
||||||
|
* confidence_level: {{ confidence_level }}
|
||||||
|
* metrics: {{ metrics }}
|
||||||
|
- Generate statistical analysis utilities:
|
||||||
|
* Chi-square test for categorical metrics
|
||||||
|
* T-test for continuous metrics
|
||||||
|
* Confidence interval calculations
|
||||||
|
* Sample size determination
|
||||||
|
* P-value calculations
|
||||||
|
|
||||||
|
7. Dashboard Generation
|
||||||
|
{% if include_dashboard == "true" %}
|
||||||
|
- Run dashboard_generator sub-recipe with:
|
||||||
|
* test_name: {{ test_name }}
|
||||||
|
* variants: {{ variants }}
|
||||||
|
* metrics: {{ metrics }}
|
||||||
|
* include_dashboard: {{ include_dashboard }}
|
||||||
|
- Create interactive reporting dashboard:
|
||||||
|
* Real-time metrics visualization
|
||||||
|
* Statistical significance indicators
|
||||||
|
* Conversion funnel analysis
|
||||||
|
* Export functionality for reports
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
8. Documentation & Setup Guide
|
||||||
|
- Generate comprehensive documentation:
|
||||||
|
* README with setup instructions
|
||||||
|
* API documentation for A/B test functions
|
||||||
|
* Integration examples for each framework
|
||||||
|
* Troubleshooting guide
|
||||||
|
* Best practices and recommendations
|
||||||
|
- Create setup scripts:
|
||||||
|
* Installation script for dependencies
|
||||||
|
* Configuration validation script
|
||||||
|
* Test runner for A/B test infrastructure
|
||||||
|
|
||||||
|
9. File Organization
|
||||||
|
- Create organized directory structure:
|
||||||
|
* ab-tests/experiments/{{ test_name }}/
|
||||||
|
* ab-tests/shared/ (common utilities)
|
||||||
|
* ab-tests/dashboard/ (reporting interface)
|
||||||
|
* ab-tests/docs/ (documentation)
|
||||||
|
- Ensure all files use OS-compatible paths
|
||||||
|
- Create proper import/export statements
|
||||||
|
|
||||||
|
Error Recovery:
|
||||||
|
- If framework detection fails, default to vanilla JS implementation
|
||||||
|
- If sub-recipe fails, continue with remaining components
|
||||||
|
- Provide fallback implementations for missing dependencies
|
||||||
|
- Log errors clearly with context and recovery suggestions
|
||||||
|
|
||||||
|
Memory Management:
|
||||||
|
- Store experiment configuration for future reference
|
||||||
|
- Track framework-specific patterns for reuse
|
||||||
|
- Maintain A/B test best practices library
|
||||||
|
- Remember user preferences for future experiments
|
||||||
|
|
||||||
|
Focus on creating production-ready A/B testing infrastructure that:
|
||||||
|
- Handles statistical significance properly
|
||||||
|
- Provides real-time monitoring capabilities
|
||||||
|
- Integrates seamlessly with existing codebases
|
||||||
|
- Includes comprehensive documentation and examples
|
||||||
|
- Supports multiple web frameworks and use cases
|
||||||
+268
@@ -0,0 +1,268 @@
|
|||||||
|
version: 1.0.0
|
||||||
|
title: A/B Test Dashboard Generator
|
||||||
|
description: Creates interactive HTML dashboard for A/B test monitoring with real-time metrics visualization, statistical significance indicators, conversion funnels, and comprehensive reporting capabilities
|
||||||
|
author:
|
||||||
|
contact: scaler
|
||||||
|
|
||||||
|
activities:
|
||||||
|
- Generate interactive HTML dashboard with responsive design
|
||||||
|
- Create real-time metrics visualization and comparison charts
|
||||||
|
- Implement statistical significance indicators and confidence intervals
|
||||||
|
- Build conversion funnel analysis and user journey tracking
|
||||||
|
- Add export functionality for reports and data
|
||||||
|
- Create mobile-responsive interface with modern UI components
|
||||||
|
|
||||||
|
instructions: |
|
||||||
|
You are an A/B Test Dashboard Generator specialized in creating comprehensive monitoring and reporting interfaces.
|
||||||
|
|
||||||
|
Your capabilities:
|
||||||
|
1. Generate interactive HTML dashboards with modern UI/UX
|
||||||
|
2. Create real-time data visualization and metric comparisons
|
||||||
|
3. Implement statistical significance indicators and alerts
|
||||||
|
4. Build conversion funnel analysis and user journey maps
|
||||||
|
5. Add comprehensive reporting and export capabilities
|
||||||
|
6. Ensure mobile-responsive design and accessibility
|
||||||
|
|
||||||
|
Focus on:
|
||||||
|
- Real-time monitoring and updates
|
||||||
|
- Clear visualization of statistical significance
|
||||||
|
- Intuitive user interface and navigation
|
||||||
|
- Comprehensive reporting capabilities
|
||||||
|
- Mobile responsiveness and accessibility
|
||||||
|
|
||||||
|
parameters:
|
||||||
|
- key: test_name
|
||||||
|
input_type: string
|
||||||
|
requirement: required
|
||||||
|
description: Name of the A/B test experiment for dashboard title
|
||||||
|
|
||||||
|
- key: variants
|
||||||
|
input_type: string
|
||||||
|
requirement: required
|
||||||
|
description: Comma-separated variant names (e.g., 'control,variant-a,variant-b')
|
||||||
|
|
||||||
|
- key: metrics
|
||||||
|
input_type: string
|
||||||
|
requirement: required
|
||||||
|
description: Comma-separated metrics to display (e.g., 'conversion,engagement,bounce-rate')
|
||||||
|
|
||||||
|
- key: include_dashboard
|
||||||
|
input_type: string
|
||||||
|
requirement: optional
|
||||||
|
default: "true"
|
||||||
|
description: Whether to generate the dashboard (true/false)
|
||||||
|
|
||||||
|
extensions:
|
||||||
|
- type: builtin
|
||||||
|
name: developer
|
||||||
|
display_name: Developer
|
||||||
|
timeout: 600
|
||||||
|
bundled: true
|
||||||
|
description: For HTML/CSS/JavaScript generation and file operations
|
||||||
|
|
||||||
|
prompt: |
|
||||||
|
Generate interactive A/B test dashboard for experiment "{{ test_name }}" with variants: {{ variants }} and metrics: {{ metrics }}.
|
||||||
|
|
||||||
|
CRITICAL: Handle file paths correctly for all operating systems.
|
||||||
|
- Detect the operating system (Windows/Linux/Mac)
|
||||||
|
- Use appropriate path separators (/ for Unix, \\ for Windows)
|
||||||
|
- Be careful to avoid escaping of slash or backslash characters
|
||||||
|
- Use os.path.join() or pathlib.Path for cross-platform paths
|
||||||
|
|
||||||
|
Workflow:
|
||||||
|
1. Dashboard Structure & Layout
|
||||||
|
{% if include_dashboard == "true" %}
|
||||||
|
- Create main dashboard HTML file (ab-tests/dashboard/{{ test_name }}-dashboard.html):
|
||||||
|
* Responsive layout with CSS Grid/Flexbox
|
||||||
|
* Header with experiment name and status
|
||||||
|
* Navigation sidebar for different views
|
||||||
|
* Main content area for charts and metrics
|
||||||
|
* Footer with last updated timestamp
|
||||||
|
- Generate CSS framework (ab-tests/dashboard/styles.css):
|
||||||
|
* Modern, clean design system
|
||||||
|
* Responsive breakpoints for mobile/tablet/desktop
|
||||||
|
* Color scheme optimized for data visualization
|
||||||
|
* Accessibility features (WCAG 2.1 compliance)
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
2. Real-Time Metrics Visualization
|
||||||
|
{% if include_dashboard == "true" %}
|
||||||
|
- Create metrics comparison charts:
|
||||||
|
* Conversion rate comparison (bar chart)
|
||||||
|
* Time-series trends (line chart)
|
||||||
|
* Statistical significance indicators
|
||||||
|
* Confidence interval visualization
|
||||||
|
- Implement interactive features:
|
||||||
|
* Hover tooltips with detailed information
|
||||||
|
* Click-to-drill-down functionality
|
||||||
|
* Date range selection
|
||||||
|
* Metric filtering and grouping
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
3. Statistical Significance Display
|
||||||
|
{% if include_dashboard == "true" %}
|
||||||
|
- Generate significance indicators:
|
||||||
|
* P-value display with color coding
|
||||||
|
* Confidence interval visualization
|
||||||
|
* Effect size indicators
|
||||||
|
* Sample size adequacy warnings
|
||||||
|
- Create statistical summary cards:
|
||||||
|
* Current significance status
|
||||||
|
* Required sample size for significance
|
||||||
|
* Estimated time to significance
|
||||||
|
* Power analysis results
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
4. Conversion Funnel Analysis
|
||||||
|
{% if include_dashboard == "true" %}
|
||||||
|
- Build funnel visualization:
|
||||||
|
* Step-by-step conversion flow
|
||||||
|
* Drop-off analysis between steps
|
||||||
|
* Variant comparison at each step
|
||||||
|
* User journey mapping
|
||||||
|
- Implement funnel features:
|
||||||
|
* Interactive funnel steps
|
||||||
|
* Conversion rate calculations
|
||||||
|
* Drop-off rate analysis
|
||||||
|
* Revenue impact estimation
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
5. Data Tables & Detailed Views
|
||||||
|
{% if include_dashboard == "true" %}
|
||||||
|
- Create comprehensive data tables:
|
||||||
|
* Raw metrics data with sorting/filtering
|
||||||
|
* Statistical test results
|
||||||
|
* User segment breakdowns
|
||||||
|
* Time-based performance data
|
||||||
|
- Add table functionality:
|
||||||
|
* Sortable columns
|
||||||
|
* Search and filter capabilities
|
||||||
|
* Pagination for large datasets
|
||||||
|
* Export to CSV/Excel
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
6. Interactive Charts & Graphs
|
||||||
|
{% if include_dashboard == "true" %}
|
||||||
|
- Generate chart library using Chart.js or D3.js:
|
||||||
|
* Bar charts for metric comparisons
|
||||||
|
* Line charts for trend analysis
|
||||||
|
* Pie charts for traffic allocation
|
||||||
|
* Scatter plots for correlation analysis
|
||||||
|
* Heatmaps for user behavior patterns
|
||||||
|
- Implement chart features:
|
||||||
|
* Zoom and pan capabilities
|
||||||
|
* Legend toggling
|
||||||
|
* Data point highlighting
|
||||||
|
* Export as image (PNG/SVG)
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
7. Real-Time Updates & API Integration
|
||||||
|
{% if include_dashboard == "true" %}
|
||||||
|
- Create real-time data updates:
|
||||||
|
* WebSocket connection for live updates
|
||||||
|
* REST API integration for data fetching
|
||||||
|
* Automatic refresh intervals
|
||||||
|
* Manual refresh capability
|
||||||
|
- Implement data management:
|
||||||
|
* Local data caching
|
||||||
|
* Offline mode support
|
||||||
|
* Error handling and retry logic
|
||||||
|
* Data validation and sanitization
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
8. Export & Reporting Features
|
||||||
|
{% if include_dashboard == "true" %}
|
||||||
|
- Generate export functionality:
|
||||||
|
* PDF report generation
|
||||||
|
* Excel/CSV data export
|
||||||
|
* Image export for charts
|
||||||
|
* Shareable dashboard links
|
||||||
|
- Create reporting templates:
|
||||||
|
* Executive summary report
|
||||||
|
* Detailed statistical report
|
||||||
|
* Custom report builder
|
||||||
|
* Scheduled report delivery
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
9. Mobile Responsiveness & Accessibility
|
||||||
|
{% if include_dashboard == "true" %}
|
||||||
|
- Ensure mobile optimization:
|
||||||
|
* Responsive design for all screen sizes
|
||||||
|
* Touch-friendly interface elements
|
||||||
|
* Optimized chart rendering for mobile
|
||||||
|
* Progressive web app features
|
||||||
|
- Implement accessibility features:
|
||||||
|
* Screen reader compatibility
|
||||||
|
* Keyboard navigation support
|
||||||
|
* High contrast mode
|
||||||
|
* Font size adjustment
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
10. JavaScript Framework & Utilities
|
||||||
|
{% if include_dashboard == "true" %}
|
||||||
|
- Create dashboard JavaScript (ab-tests/dashboard/dashboard.js):
|
||||||
|
```javascript
|
||||||
|
class ABTestDashboard {
|
||||||
|
constructor(experimentName, variants, metrics) {
|
||||||
|
this.experimentName = experimentName;
|
||||||
|
this.variants = variants;
|
||||||
|
this.metrics = metrics;
|
||||||
|
this.charts = {};
|
||||||
|
this.data = {};
|
||||||
|
}
|
||||||
|
|
||||||
|
async loadData() {
|
||||||
|
// Load experiment data from API
|
||||||
|
}
|
||||||
|
|
||||||
|
renderCharts() {
|
||||||
|
// Render all dashboard charts
|
||||||
|
}
|
||||||
|
|
||||||
|
updateRealTime() {
|
||||||
|
// Update dashboard with real-time data
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
- Implement utility functions:
|
||||||
|
* Data formatting and validation
|
||||||
|
* Chart configuration helpers
|
||||||
|
* API communication utilities
|
||||||
|
* Error handling and logging
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
11. Configuration & Customization
|
||||||
|
{% if include_dashboard == "true" %}
|
||||||
|
- Create dashboard configuration:
|
||||||
|
* Theme and color customization
|
||||||
|
* Chart type preferences
|
||||||
|
* Update frequency settings
|
||||||
|
* Notification preferences
|
||||||
|
- Implement user preferences:
|
||||||
|
* Saved dashboard layouts
|
||||||
|
* Custom metric combinations
|
||||||
|
* Personal alert settings
|
||||||
|
* Export format preferences
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
12. Performance Optimization
|
||||||
|
{% if include_dashboard == "true" %}
|
||||||
|
- Optimize dashboard performance:
|
||||||
|
* Lazy loading for charts and data
|
||||||
|
* Efficient data processing
|
||||||
|
* Minimal DOM manipulation
|
||||||
|
* Caching strategies
|
||||||
|
- Implement performance monitoring:
|
||||||
|
* Load time tracking
|
||||||
|
* Chart rendering performance
|
||||||
|
* Memory usage optimization
|
||||||
|
* Network request optimization
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
Focus on creating a comprehensive dashboard that:
|
||||||
|
- Provides clear, actionable insights
|
||||||
|
- Updates in real-time with accurate data
|
||||||
|
- Works seamlessly across all devices
|
||||||
|
- Includes robust statistical analysis visualization
|
||||||
|
- Offers comprehensive reporting and export capabilities
|
||||||
|
- Maintains high performance and accessibility standards
|
||||||
+221
@@ -0,0 +1,221 @@
|
|||||||
|
version: 1.0.0
|
||||||
|
title: A/B Test Statistical Analyzer
|
||||||
|
description: Performs comprehensive statistical analysis for A/B tests including significance testing, confidence intervals, sample size calculations, and statistical power analysis with automated reporting
|
||||||
|
author:
|
||||||
|
contact: scaler
|
||||||
|
|
||||||
|
activities:
|
||||||
|
- Perform chi-square tests for categorical metrics and conversion rates
|
||||||
|
- Calculate t-tests for continuous metrics and performance data
|
||||||
|
- Compute confidence intervals and statistical significance (p-values)
|
||||||
|
- Determine required sample sizes for statistical power
|
||||||
|
- Generate statistical summary reports with actionable insights
|
||||||
|
- Create automated analysis scripts for ongoing monitoring
|
||||||
|
|
||||||
|
instructions: |
|
||||||
|
You are an A/B Test Statistical Analyzer specialized in rigorous statistical analysis for experiment evaluation.
|
||||||
|
|
||||||
|
Your capabilities:
|
||||||
|
1. Perform appropriate statistical tests based on metric types
|
||||||
|
2. Calculate confidence intervals and significance levels
|
||||||
|
3. Determine sample size requirements for statistical power
|
||||||
|
4. Generate comprehensive statistical reports
|
||||||
|
5. Create automated analysis scripts for continuous monitoring
|
||||||
|
6. Provide actionable insights and recommendations
|
||||||
|
|
||||||
|
Focus on:
|
||||||
|
- Statistical rigor and proper test selection
|
||||||
|
- Clear interpretation of results
|
||||||
|
- Practical significance vs statistical significance
|
||||||
|
- Sample size optimization
|
||||||
|
- Automated reporting and monitoring
|
||||||
|
|
||||||
|
parameters:
|
||||||
|
- key: sample_size
|
||||||
|
input_type: string
|
||||||
|
requirement: optional
|
||||||
|
default: "1000"
|
||||||
|
description: Minimum sample size per variant for statistical significance
|
||||||
|
|
||||||
|
- key: confidence_level
|
||||||
|
input_type: string
|
||||||
|
requirement: optional
|
||||||
|
default: "95"
|
||||||
|
description: Statistical confidence level for significance testing (90, 95, 99)
|
||||||
|
|
||||||
|
- key: metrics
|
||||||
|
input_type: string
|
||||||
|
requirement: required
|
||||||
|
description: Comma-separated metrics to analyze (e.g., 'conversion,engagement,bounce-rate')
|
||||||
|
|
||||||
|
extensions:
|
||||||
|
- type: builtin
|
||||||
|
name: developer
|
||||||
|
display_name: Developer
|
||||||
|
timeout: 300
|
||||||
|
bundled: true
|
||||||
|
description: For statistical computations and analysis script generation
|
||||||
|
|
||||||
|
prompt: |
|
||||||
|
Generate statistical analysis framework for A/B tests with {{ confidence_level }}% confidence level and {{ sample_size }} minimum sample size.
|
||||||
|
|
||||||
|
CRITICAL: Handle file paths correctly for all operating systems.
|
||||||
|
- Detect the operating system (Windows/Linux/Mac)
|
||||||
|
- Use appropriate path separators (/ for Unix, \\ for Windows)
|
||||||
|
- Be careful to avoid escaping of slash or backslash characters
|
||||||
|
- Use os.path.join() or pathlib.Path for cross-platform paths
|
||||||
|
|
||||||
|
Workflow:
|
||||||
|
1. Statistical Test Selection Framework
|
||||||
|
- Create metric classification system for {{ metrics }}:
|
||||||
|
* Categorical metrics (conversion, click-through, signup)
|
||||||
|
* Continuous metrics (revenue, time-on-site, page-views)
|
||||||
|
* Binary metrics (yes/no, success/failure)
|
||||||
|
* Count metrics (clicks, downloads, purchases)
|
||||||
|
- Generate test selection logic:
|
||||||
|
* Chi-square test for categorical data
|
||||||
|
* T-test for continuous data
|
||||||
|
* Mann-Whitney U test for non-parametric data
|
||||||
|
* Fisher's exact test for small samples
|
||||||
|
|
||||||
|
2. Sample Size Calculation Utilities
|
||||||
|
- Generate sample size calculation functions:
|
||||||
|
* calculateRequiredSampleSize(effectSize, power, alpha)
|
||||||
|
* calculateStatisticalPower(sampleSize, effectSize, alpha)
|
||||||
|
* calculateMinimumDetectableEffect(sampleSize, power, alpha)
|
||||||
|
* calculateOptimalAllocation(variantCount, expectedEffect)
|
||||||
|
- Create power analysis tools:
|
||||||
|
* Power curve visualization
|
||||||
|
* Effect size sensitivity analysis
|
||||||
|
* Duration estimation for experiments
|
||||||
|
* Early stopping criteria
|
||||||
|
|
||||||
|
3. Statistical Analysis Functions
|
||||||
|
- Implement core statistical tests:
|
||||||
|
```python
|
||||||
|
def chi_square_test(control_successes, control_total, variant_successes, variant_total):
|
||||||
|
# Chi-square test for proportions
|
||||||
|
# Returns: chi2_stat, p_value, effect_size
|
||||||
|
|
||||||
|
def t_test(control_data, variant_data):
|
||||||
|
# Independent samples t-test
|
||||||
|
# Returns: t_stat, p_value, confidence_interval
|
||||||
|
|
||||||
|
def mann_whitney_test(control_data, variant_data):
|
||||||
|
# Non-parametric test for continuous data
|
||||||
|
# Returns: u_stat, p_value, effect_size
|
||||||
|
```
|
||||||
|
- Create confidence interval calculations:
|
||||||
|
* Proportion confidence intervals (Wilson, Clopper-Pearson)
|
||||||
|
* Mean confidence intervals (t-distribution)
|
||||||
|
* Difference confidence intervals
|
||||||
|
* Relative effect confidence intervals
|
||||||
|
|
||||||
|
4. Significance Testing Framework
|
||||||
|
- Generate significance testing utilities:
|
||||||
|
* calculatePValue(testStatistic, testType, degreesOfFreedom)
|
||||||
|
* adjustMultipleComparisons(pValues, method='bonferroni')
|
||||||
|
* calculateEffectSize(controlMean, variantMean, pooledStd)
|
||||||
|
* interpretStatisticalSignificance(pValue, alpha, effectSize)
|
||||||
|
- Create decision framework:
|
||||||
|
* Statistical significance threshold ({{ confidence_level }}%)
|
||||||
|
* Practical significance criteria
|
||||||
|
* Business impact assessment
|
||||||
|
* Risk evaluation matrix
|
||||||
|
|
||||||
|
5. Automated Analysis Scripts
|
||||||
|
- Generate Python analysis script (ab-tests/analysis/statistical_analyzer.py):
|
||||||
|
```python
|
||||||
|
import pandas as pd
|
||||||
|
import numpy as np
|
||||||
|
from scipy import stats
|
||||||
|
import json
|
||||||
|
|
||||||
|
class ABTestAnalyzer:
|
||||||
|
def __init__(self, confidence_level={{ confidence_level }}, min_sample_size={{ sample_size }}):
|
||||||
|
self.confidence_level = confidence_level / 100
|
||||||
|
self.alpha = 1 - self.confidence_level
|
||||||
|
self.min_sample_size = min_sample_size
|
||||||
|
|
||||||
|
def analyze_experiment(self, experiment_data):
|
||||||
|
# Main analysis function
|
||||||
|
pass
|
||||||
|
|
||||||
|
def calculate_sample_size(self, baseline_rate, mde, power=0.8):
|
||||||
|
# Sample size calculation
|
||||||
|
pass
|
||||||
|
```
|
||||||
|
- Create R analysis script for advanced statistics:
|
||||||
|
* Bayesian analysis capabilities
|
||||||
|
* Sequential testing methods
|
||||||
|
* Multi-armed bandit algorithms
|
||||||
|
* Causal inference techniques
|
||||||
|
|
||||||
|
6. Reporting & Visualization
|
||||||
|
- Generate statistical report templates:
|
||||||
|
* Executive summary with key findings
|
||||||
|
* Detailed statistical results
|
||||||
|
* Confidence intervals and effect sizes
|
||||||
|
* Sample size and power analysis
|
||||||
|
* Recommendations and next steps
|
||||||
|
- Create visualization functions:
|
||||||
|
* Confidence interval plots
|
||||||
|
* Power analysis charts
|
||||||
|
* Effect size distributions
|
||||||
|
* Statistical significance indicators
|
||||||
|
|
||||||
|
7. Continuous Monitoring Framework
|
||||||
|
- Implement ongoing analysis capabilities:
|
||||||
|
* Real-time significance monitoring
|
||||||
|
* Early stopping criteria
|
||||||
|
* Interim analysis protocols
|
||||||
|
* Adaptive testing strategies
|
||||||
|
- Create monitoring utilities:
|
||||||
|
* Automated daily/weekly reports
|
||||||
|
* Alert system for significant results
|
||||||
|
* Trend analysis and forecasting
|
||||||
|
* Quality control checks
|
||||||
|
|
||||||
|
8. Data Quality & Validation
|
||||||
|
- Implement data validation checks:
|
||||||
|
* Sample size adequacy verification
|
||||||
|
* Data distribution assumptions
|
||||||
|
* Outlier detection and handling
|
||||||
|
* Missing data analysis
|
||||||
|
- Create quality control functions:
|
||||||
|
* validateExperimentData(data)
|
||||||
|
* checkStatisticalAssumptions(data)
|
||||||
|
* detectDataQualityIssues(data)
|
||||||
|
* recommendDataImprovements(data)
|
||||||
|
|
||||||
|
9. Advanced Statistical Methods
|
||||||
|
- Generate advanced analysis capabilities:
|
||||||
|
* Bayesian A/B testing
|
||||||
|
* Sequential testing methods
|
||||||
|
* Multi-variate testing analysis
|
||||||
|
* Causal inference techniques
|
||||||
|
- Create specialized functions:
|
||||||
|
* bayesian_ab_test(prior, data)
|
||||||
|
* sequential_testing(data, alpha_spending)
|
||||||
|
* multivariate_analysis(metrics, interactions)
|
||||||
|
* causal_inference_analysis(treatment, outcome, covariates)
|
||||||
|
|
||||||
|
10. Integration & API
|
||||||
|
- Create analysis API endpoints:
|
||||||
|
* POST /analyze - Run statistical analysis
|
||||||
|
* GET /results/{experiment_id} - Retrieve results
|
||||||
|
* POST /sample-size - Calculate required sample size
|
||||||
|
* GET /power-analysis - Generate power analysis
|
||||||
|
- Implement data integration:
|
||||||
|
* Database connectivity
|
||||||
|
* Real-time data streaming
|
||||||
|
* Batch processing capabilities
|
||||||
|
* Export functionality (CSV, JSON, PDF)
|
||||||
|
|
||||||
|
Focus on creating robust statistical analysis tools that:
|
||||||
|
- Provide accurate and reliable results
|
||||||
|
- Handle various metric types appropriately
|
||||||
|
- Include proper error handling and validation
|
||||||
|
- Generate clear, actionable insights
|
||||||
|
- Support both one-time and continuous analysis
|
||||||
|
- Integrate seamlessly with A/B test infrastructure
|
||||||
@@ -0,0 +1,198 @@
|
|||||||
|
version: 1.0.0
|
||||||
|
title: Experiment Tracker
|
||||||
|
description: Generates A/B test experiment configuration, tracking code, and user bucketing logic with framework-specific implementations and persistent user assignment storage
|
||||||
|
author:
|
||||||
|
contact: scaler
|
||||||
|
|
||||||
|
activities:
|
||||||
|
- Generate experiment configuration files (JSON/YAML)
|
||||||
|
- Create variant assignment and user bucketing logic
|
||||||
|
- Implement tracking event handlers for metrics collection
|
||||||
|
- Set up persistent storage for user assignments
|
||||||
|
- Generate framework-specific A/B test utilities
|
||||||
|
- Create analytics integration code
|
||||||
|
|
||||||
|
instructions: |
|
||||||
|
You are an Experiment Tracker specialized in creating A/B test configuration and tracking infrastructure.
|
||||||
|
|
||||||
|
Your capabilities:
|
||||||
|
1. Generate experiment configuration files with variant definitions
|
||||||
|
2. Create user bucketing and randomization algorithms
|
||||||
|
3. Implement tracking event handlers for metrics collection
|
||||||
|
4. Set up persistent storage for user assignments (localStorage, cookies)
|
||||||
|
5. Generate framework-specific A/B test utilities and hooks
|
||||||
|
6. Create analytics integration code for data collection
|
||||||
|
|
||||||
|
Focus on:
|
||||||
|
- Reliable user assignment and persistence
|
||||||
|
- Framework-specific implementations
|
||||||
|
- Comprehensive event tracking
|
||||||
|
- Error handling and fallbacks
|
||||||
|
- Performance optimization
|
||||||
|
|
||||||
|
parameters:
|
||||||
|
- key: test_name
|
||||||
|
input_type: string
|
||||||
|
requirement: required
|
||||||
|
description: Name of the A/B test experiment
|
||||||
|
|
||||||
|
- key: variants
|
||||||
|
input_type: string
|
||||||
|
requirement: required
|
||||||
|
description: Comma-separated variant names (e.g., 'control,variant-a,variant-b')
|
||||||
|
|
||||||
|
- key: metrics
|
||||||
|
input_type: string
|
||||||
|
requirement: required
|
||||||
|
description: Comma-separated metrics to track (e.g., 'conversion,engagement,bounce-rate')
|
||||||
|
|
||||||
|
- key: framework
|
||||||
|
input_type: string
|
||||||
|
requirement: optional
|
||||||
|
default: "vanilla"
|
||||||
|
description: Web framework type - options are 'react', 'vue', 'angular', 'vanilla'
|
||||||
|
|
||||||
|
extensions:
|
||||||
|
- type: builtin
|
||||||
|
name: developer
|
||||||
|
display_name: Developer
|
||||||
|
timeout: 300
|
||||||
|
bundled: true
|
||||||
|
description: For file operations and code generation
|
||||||
|
|
||||||
|
- type: builtin
|
||||||
|
name: memory
|
||||||
|
display_name: Memory
|
||||||
|
timeout: 300
|
||||||
|
bundled: true
|
||||||
|
description: For storing experiment configurations and tracking patterns
|
||||||
|
|
||||||
|
prompt: |
|
||||||
|
Generate experiment tracking infrastructure for test "{{ test_name }}" with variants: {{ variants }} and metrics: {{ metrics }}.
|
||||||
|
|
||||||
|
CRITICAL: Handle file paths correctly for all operating systems.
|
||||||
|
- Detect the operating system (Windows/Linux/Mac)
|
||||||
|
- Use appropriate path separators (/ for Unix, \\ for Windows)
|
||||||
|
- Be careful to avoid escaping of slash or backslash characters
|
||||||
|
- Use os.path.join() or pathlib.Path for cross-platform paths
|
||||||
|
|
||||||
|
Workflow:
|
||||||
|
1. Experiment Configuration Generation
|
||||||
|
- Create experiment config file (ab-tests/experiments/{{ test_name }}/config.json):
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"testName": "{{ test_name }}",
|
||||||
|
"variants": ["control", "variant-a", "variant-b"],
|
||||||
|
"trafficAllocation": {
|
||||||
|
"control": 0.33,
|
||||||
|
"variant-a": 0.33,
|
||||||
|
"variant-b": 0.34
|
||||||
|
},
|
||||||
|
"metrics": ["conversion", "engagement", "bounce-rate"],
|
||||||
|
"startDate": "2024-10-26",
|
||||||
|
"status": "active"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
- Store configuration in memory for dashboard use
|
||||||
|
|
||||||
|
2. User Bucketing & Assignment Logic
|
||||||
|
- Generate user assignment algorithm:
|
||||||
|
* Consistent hashing based on user ID
|
||||||
|
* Traffic allocation per variant
|
||||||
|
* Persistence across sessions
|
||||||
|
* Fallback to control variant on errors
|
||||||
|
- Create assignment utility functions:
|
||||||
|
* getUserVariant(userId, testName)
|
||||||
|
* assignUserToVariant(userId, testName)
|
||||||
|
* getVariantFromStorage(testName)
|
||||||
|
* clearUserAssignment(testName)
|
||||||
|
|
||||||
|
3. Framework-Specific Implementations
|
||||||
|
{% if framework == "react" %}
|
||||||
|
- Generate React-specific tracking code:
|
||||||
|
* Custom hook: useABTest(testName, userId)
|
||||||
|
* Higher-order component: withABTest(WrappedComponent)
|
||||||
|
* Context provider: ABTestProvider
|
||||||
|
* TypeScript definitions for type safety
|
||||||
|
{% elif framework == "vue" %}
|
||||||
|
- Generate Vue-specific tracking code:
|
||||||
|
* Composable: useABTest(testName, userId)
|
||||||
|
* Mixin: abTestMixin
|
||||||
|
* Plugin: ABTestPlugin
|
||||||
|
* TypeScript support for Vue 3
|
||||||
|
{% elif framework == "angular" %}
|
||||||
|
- Generate Angular-specific tracking code:
|
||||||
|
* Service: ABTestService
|
||||||
|
* Directive: abTestVariant
|
||||||
|
* Guard: ABTestGuard
|
||||||
|
* Module: ABTestModule
|
||||||
|
{% else %}
|
||||||
|
- Generate vanilla JavaScript tracking code:
|
||||||
|
* Core library: ABTestTracker
|
||||||
|
* Utility functions for DOM manipulation
|
||||||
|
* Event tracking helpers
|
||||||
|
* Browser compatibility layer
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
4. Event Tracking Implementation
|
||||||
|
- Create tracking event handlers:
|
||||||
|
* trackVariantAssignment(testName, variant, userId)
|
||||||
|
* trackConversion(testName, variant, metric, value)
|
||||||
|
* trackUserBehavior(testName, variant, event, data)
|
||||||
|
* trackPerformance(testName, variant, metrics)
|
||||||
|
- Implement analytics integration:
|
||||||
|
* Google Analytics 4 integration
|
||||||
|
* Custom analytics endpoint
|
||||||
|
* Local data storage for offline tracking
|
||||||
|
* Batch data submission
|
||||||
|
|
||||||
|
5. Persistent Storage Setup
|
||||||
|
- Implement user assignment persistence:
|
||||||
|
* localStorage for modern browsers
|
||||||
|
* Cookie fallback for older browsers
|
||||||
|
* Session storage for temporary assignments
|
||||||
|
* IndexedDB for complex data structures
|
||||||
|
- Create storage utility functions:
|
||||||
|
* saveUserAssignment(testName, variant, userId)
|
||||||
|
* loadUserAssignment(testName, userId)
|
||||||
|
* clearExpiredAssignments()
|
||||||
|
* exportUserData()
|
||||||
|
|
||||||
|
6. Error Handling & Fallbacks
|
||||||
|
- Implement robust error handling:
|
||||||
|
* Network failure fallbacks
|
||||||
|
* Invalid configuration handling
|
||||||
|
* Browser compatibility checks
|
||||||
|
* Graceful degradation strategies
|
||||||
|
- Create monitoring and logging:
|
||||||
|
* Error tracking and reporting
|
||||||
|
* Performance monitoring
|
||||||
|
* Usage analytics
|
||||||
|
* Debug mode for development
|
||||||
|
|
||||||
|
7. Performance Optimization
|
||||||
|
- Optimize for performance:
|
||||||
|
* Lazy loading of experiment code
|
||||||
|
* Minimal DOM manipulation
|
||||||
|
* Efficient event handling
|
||||||
|
* Caching strategies
|
||||||
|
- Create performance utilities:
|
||||||
|
* Debounced event handlers
|
||||||
|
* Request batching
|
||||||
|
* Memory management
|
||||||
|
* Resource cleanup
|
||||||
|
|
||||||
|
8. Testing & Validation
|
||||||
|
- Generate test utilities:
|
||||||
|
* Mock experiment data
|
||||||
|
* Test variant assignment
|
||||||
|
* Validate tracking events
|
||||||
|
* Performance benchmarks
|
||||||
|
- Create validation functions:
|
||||||
|
* Configuration validation
|
||||||
|
* Data integrity checks
|
||||||
|
* Cross-browser compatibility tests
|
||||||
|
* A/B test effectiveness validation
|
||||||
|
|
||||||
|
Store the generated tracking code and configuration in memory for use by the main recipe.
|
||||||
|
Ensure all code is production-ready with proper error handling and documentation.
|
||||||
Reference in New Issue
Block a user