Driving compliance through Codex/Claude Skills and MCP connections for MLflow

Overview

Regulated industries like banking get their models audited regularly, and one of the things they need is to be able to recreate any model at any point in the future. Sounds simple, but in reality it's a pain. Most teams have DevOps sorted for their code, but the equivalent for data (DataOps) and models (MLOps) tends to be a lot less mature.

Databricks handles a good chunk of this natively. On the data side, Delta lets you query a table as of a given version or date and get the data back exactly as it was at that time. That's tick box number one. The one caveat is that time travel is bounded by retention by default, so for proper audit-grade reproducibility you can't just rely on it going back forever. You either manage retention deliberately, or better still, log the exact table version against each training run so the data lineage is pinned down per model.

MLOps used to be the part that cost teams a lot of time and money to get right. Databricks managed MLflow makes it much easier, letting you track experiments, version models, and manage their lifecycle through the registry. It also covers the bit people often forget, which is the environment. Recreating a model isn't just about the data and the weights, it's about the libraries and runtime it was built on, and MLflow logs those dependencies with each run, so you can rebuild the whole stack rather than just part of it.

To make it easier for people to use the right models, we built a skill (works with both Codex and Claude) that uses the Databricks MCP to return the model best suited to whatever they're trying to do. For the people building models, we built the other side of it, a skill that uses the MCP to update and deploy them. That gave us a consistent way of doing model development going forward, especially around champion/challenger aliasing.

LLMOps is still pretty fresh. MLflow has similar bits in place for GenAI models now, but that's not something we've dug into much yet.

Key Achievements

• Consistent run metadata and parameter logging
• Comparable model metrics across candidate approaches
• Reproducible experiment history for handover and review
• Ease of supporting model validation requirements