MLflow

MLflow is an open-source platform for managing the full ML lifecycle — from experiment tracking and model packaging to versioning and deployment. Originally built for traditional ML, it now includes first-class support for LLM tracking and evaluation.

Key use cases include:

• Experiment tracking — log parameters, metrics, and artifacts for every training run
• Model registry — version models and manage stage transitions (staging, production, archived)
• Model packaging — package models in a standard format (MLmodel) that deploys anywhere
• LLM tracking — log prompts, completions, token usage, and evaluation metrics
• Model deployment — serve models as REST APIs with built-in inference servers
• Evaluation — run automated evaluations on models with built-in and custom metrics

MLflow is used by data science and ML engineering teams who need a vendor-neutral, open-source platform for experiment management and model lifecycle. It is the most widely adopted open-source MLOps tool and integrates with virtually every ML framework.

Managed by the Linux Foundation and backed by Databricks, MLflow has a massive community and extensive integrations.

1. Install MLflow:

   pip install mlflow

2. Start the tracking UI:

   mlflow ui
   # Opens at http://localhost:5000

3. Add tracking to your code:

   import mlflow
   
   with mlflow.start_run():
       mlflow.log_param('learning_rate', 0.01)
       mlflow.log_metric('accuracy', 0.95)
       mlflow.log_artifact('model.pkl')

4. For LLM tracking:

   mlflow.openai.autolog()
   # All OpenAI calls are now automatically logged

5. For team use, deploy the tracking server or use Databricks Managed MLflow.

Pricing: MLflow is free and open source (Apache 2.0). Databricks Managed MLflow is included with Databricks workspace pricing. Self-hosted server costs depend on your infrastructure.