Prefect

Prefect is a workflow orchestration platform that turns Python functions into observable, resilient, and schedulable tasks. It is widely used for ML pipelines, data processing, and batch inference workflows that need retry logic, scheduling, and monitoring.

Key use cases include:

• ML pipeline orchestration — schedule and monitor model training, evaluation, and deployment jobs
• Data pipelines — ETL/ELT workflows with automatic retries and failure handling
• Batch inference — scheduled LLM processing of large datasets
• RAG pipeline maintenance — scheduled document ingestion and index refresh
• Model retraining — trigger retraining based on data drift or schedules
• Caching — avoid recomputing expensive steps when inputs have not changed

Prefect is used by ML and data teams who need their Python workflows to be reliable and observable in production. Unlike heavier orchestrators (Airflow), Prefect adds orchestration to existing Python code with minimal refactoring — just add decorators.

Available as Prefect Cloud (hosted) or Prefect Server (self-hosted, open source).

1. Install Prefect:

   pip install prefect

2. Add decorators to your functions:

   from prefect import flow, task
   
   @task(retries=3)
   def extract_data():
       return {'data': [1, 2, 3]}
   
   @task
   def train_model(data):
       return 'model_v1'
   
   @flow
   def ml_pipeline():
       data = extract_data()
       model = train_model(data)
       return model
   
   ml_pipeline()

3. Start the local UI:

   prefect server start
   # Opens at http://localhost:4200

4. For production, sign up at Prefect Cloud and deploy with flow.serve() or flow.deploy().

Pricing: Prefect Server is free and open source (Apache 2.0). Prefect Cloud offers a free tier (limited task runs), with Pro plans starting at $500/month. See prefect.io/pricing.