SO Campaign Manager Documentation¶
Welcome to the SO Campaign Manager documentation! This package provides a comprehensive workflow orchestration system for managing and executing mapmaking campaigns on high-performance computing resources, specifically designed for the Simons Observatory project.
Getting Started:
- Installation
- Quick Start
- Tutorial
- Tutorial Overview
- Prerequisites
- Tutorial 1: Your First Campaign
- Tutorial 2: Multiple Workflows in Parallel
- Tutorial 3: Null Test Campaigns
- Tutorial 4: Resource Optimization
- Tutorial 5: Advanced Configuration Patterns
- Tutorial 6: Monitoring and Debugging
- Tutorial 7: Testing Before Production
- Tutorial 8: Programmatic Usage
- Next Steps
User Documentation:
Advanced Topics:
- Architecture
- Advanced Topics
- Custom Workflow Development
- Custom Planner Development
- Custom Enactor Development
- Resource Prediction and Slurmise Integration
- Multi-Campaign Orchestration
- Performance Optimization
- Advanced Configuration Techniques
- Integration with Other Tools
- Custom Callbacks and Hooks
- Distributed Campaign Management
- Best Practices for Advanced Usage
- Conclusion
API Reference:
Development:
What is SO Campaign Manager?¶
SO Campaign Manager is a sophisticated workflow orchestration system that enables efficient execution of mapmaking campaigns on HPC clusters. It combines intelligent scheduling, resource optimization, and seamless SLURM integration to maximize throughput while meeting campaign deadlines.
Key Features¶
Intelligent Scheduling: Uses HEFT (Heterogeneous Earliest Finish Time) algorithm to optimize workflow execution
Campaign Management: Organize and execute multiple workflows with deadline-based policies
Resource Management: Automatic resource allocation and QoS selection for SLURM
Workflow Support: Built-in support for ML mapmaking, null tests, and SAT simulations
SLURM Integration: Seamless job submission and monitoring via RADICAL-Pilot
Flexible Configuration: TOML-based hierarchical configuration system
Extensible Architecture: Easy to add custom workflows, planners, and execution backends
Resource Prediction: ML-based resource estimation via Slurmise integration
Dependency Management: Automatic workflow dependency resolution
Use Cases¶
SO Campaign Manager is designed for:
Mapmaking Campaigns: Process large volumes of time-ordered data into maps
Null Test Validation: Run comprehensive null test suites for systematic error detection
Multi-Band Analysis: Coordinate processing across multiple frequency bands
Resource-Constrained Execution: Optimize resource usage within budget constraints
Long-Running Campaigns: Manage campaigns spanning days or weeks
