Making MLflow Easier to Use with Codex and Claude

Context

Regulated industries like banking get their models audited regularly, and one of the things they need is to be able to recreate any model at any point in the future. Sounds simple, but in reality it's a pain. Most teams have DevOps sorted for their code, but the equivalent for data (DataOps) and models (MLOps) tends to be a lot less mature.

Challenge

Databricks handles much of this natively. Delta can query a table as it existed at a particular version or date. The catch is that time travel is bounded by retention, so audit work cannot assume every old version will remain available. We logged the exact table version against each training run so the model’s data lineage was explicit.

Managed MLflow covers experiment tracking, model versions and lifecycle management through the registry. It also records the libraries and runtime used for a run. That matters because recreating a model requires the original environment as well as its data and weights.

Approach

To make the registry easier to use, we built a skill for Codex and Claude that used the Databricks MCP to find the best-fit model for a task. A second skill let model builders update and deploy models through the MCP. Together they gave teams a consistent way to use champion/challenger aliases.

MLflow also has tools for GenAI models, but we had not explored those in depth at this point.

Results

• Consistent run metadata and parameter logging
• Comparable model metrics across candidate approaches
• Reproducible experiment history for handover and review
• Support for model validation requirements
• MLflow tracking used for 80% of new models across four teams
• New-user onboarding reduced to around 10 minutes