Complete MLOps Series: ← Part 1: Pipeline | ← Part 2: Deployment | Part 3 (current)

Anatomy of an MLOps Pipeline - Part 3: Production and Best Practices

11. Model and Parameter Selection Strategies

The Complete Flow: Selection → Sweep → Registration

This pipeline implements a three-phase strategy for model optimization, each with a specific purpose:

Step 05: Model Selection
├── Compares 5 algorithms with basic GridSearch (5-10 combos/model)
├── Objective: Identify best model family (Random Forest vs Gradient Boosting vs ...)
├── Primary metric: MAPE (Mean Absolute Percentage Error)
└── Output: Best algorithm + initial parameters

Step 06: Hyperparameter Sweep
├── Optimizes ONLY the best algorithm from Step 05
├── Bayesian optimization with 50+ runs (exhaustive search space)
├── Objective: Find optimal configuration of best model
├── Primary metric: wMAPE (Weighted MAPE, less biased)
└── Output: best_params.yaml with optimal hyperparameters

Step 07: Model Registration
├── Trains final model with parameters from Step 06
├── Registers in MLflow Model Registry with rich metadata
├── Transitions to stage (Staging/Production)
└── Output: Versioned model ready for deployment

Why three separate steps? You don’t have computational resources to do exhaustive sweep of 5 algorithms × 50 combinations = 250 training runs. First decide strategy (which algorithm), then tactics (which hyperparameters).

Step 05: Model Selection - Algorithm Comparison

The 5 Candidate Models

def get_available_models() -> Dict[str, Any]:
    """Get dictionary of available regression models."""
    models = {
        "RandomForest": RandomForestRegressor(random_state=42),
        "GradientBoosting": GradientBoostingRegressor(random_state=42),
        "Ridge": Ridge(random_state=42),
        "Lasso": Lasso(random_state=42),
        "DecisionTree": DecisionTreeRegressor(random_state=42)
    }
    return models

Why these models:

RandomForest: Tree ensemble, robust, handles non-linearities
GradientBoosting: Sequential boosting, better precision than RF but slower
Ridge: Linear regression with L2 regularization, fast, interpretable
Lasso: Linear regression with L1 regularization, does feature selection
DecisionTree: Simple baseline, useful for comparison

What’s missing (deliberately):

XGBoost/LightGBM: Not included to reduce dependencies, but easy to add
Neural Networks: Overkill for this problem (20k samples, tabular features)
SVR: Very slow on large datasets, doesn’t scale well

Parameter Grids: Initial GridSearch

def get_default_param_grids() -> Dict[str, Dict[str, list]]:
    """
    Parameter grids for initial model selection.
    Refined based on domain knowledge.
    """
    param_grids = {
        "RandomForest": {
            "n_estimators": [50, 100, 200, 300],         # 4 options
            "max_depth": [10, 15, 20, 25, None],         # 5 options
            "min_samples_split": [2, 5, 10],             # 3 options
            "min_samples_leaf": [1, 2, 4],               # 3 options
        },
        # Total combinations: 4×5×3×3 = 180
        # With 5-fold CV: 180×5 = 900 fits

        "GradientBoosting": {
            "n_estimators": [50, 100, 150, 200],         # 4 options
            "learning_rate": [0.01, 0.05, 0.1, 0.15, 0.2], # 5 options
            "max_depth": [3, 4, 5, 6, 7],                # 5 options
            "subsample": [0.8, 0.9, 1.0],                # 3 options
        },
        # Total: 4×5×5×3 = 300 combinations

        "Ridge": {
            "alpha": [0.01, 0.1, 0.5, 1.0, 5.0, 10.0, 50.0, 100.0, 500.0],
        },
        # Total: 9 combinations (fast)

        "Lasso": {
            "alpha": [0.01, 0.1, 0.5, 1.0, 5.0, 10.0, 50.0, 100.0, 500.0],
        },
        # Total: 9 combinations

        "DecisionTree": {
            "max_depth": [5, 10, 15, 20, 25, None],      # 6 options
            "min_samples_split": [2, 5, 10, 20],         # 4 options
            "min_samples_leaf": [1, 2, 4, 8],            # 4 options
        }
        # Total: 6×4×4 = 96 combinations
    }
    return param_grids

Grid Design Decisions

1. RandomForest: Focus on Overfitting Control

"max_depth": [10, 15, 20, 25, None],
"min_samples_leaf": [1, 2, 4],

Reasoning: Random Forest tends to overfit on small datasets. max_depth and min_samples_leaf control tree depth—high values prevent the model from memorizing noise.

None in max_depth: Allows unlimited depth trees. Useful when the dataset has complex patterns requiring deep splits.

2. GradientBoosting: Balance Learning Rate vs N_estimators

"n_estimators": [50, 100, 150, 200],
"learning_rate": [0.01, 0.05, 0.1, 0.15, 0.2],

Classic trade-off:

Low learning rate (0.01) + many estimators (200): Slow but accurate learning
High learning rate (0.2) + few estimators (50): Fast but may diverge

GridSearch explores both extremes.

subsample < 1.0: Stochastic Gradient Boosting. Only uses 80-90% of data in each iteration, reduces overfitting.

3. Ridge/Lasso: Alpha in Logarithmic Scale

"alpha": [0.01, 0.1, 0.5, 1.0, 5.0, 10.0, 50.0, 100.0, 500.0],

Alpha controls regularization:

Low alpha (0.01): Almost no regularization, complex model
High alpha (500): Strong regularization, simple model (coefficients close to 0)

Logarithmic scale covers the space more uniformly than linear scale.

Lasso vs Ridge:

Lasso (L1): Forces coefficients to exactly 0 → automatic feature selection
Ridge (L2): Small coefficients but not zero → keeps all features

If Lasso wins, it indicates some features are noise.

4. DecisionTree: Comparison Baseline

DecisionTree is the worst model (high variance, overfits easily), but serves to:

Verify that the pipeline works correctly
Comparison baseline: If Ridge/Lasso don’t beat DecisionTree, something’s wrong in feature engineering

Training Function with GridSearch

def train_model_with_gridsearch(
    model: Any,
    param_grid: Dict[str, list],
    X_train: pd.DataFrame,
    y_train: pd.Series,
    cv: int = 5
) -> Tuple[Any, Dict[str, Any], float, Dict[str, float]]:
    """Train model with K-fold Cross-Validation via GridSearchCV."""

    start_time = time.time()

    grid_search = GridSearchCV(
        estimator=model,
        param_grid=param_grid,
        cv=5,  # 5-fold cross-validation
        scoring='neg_mean_absolute_error',  # CRITICAL
        n_jobs=-1,  # Parallelization
        verbose=0,
        return_train_score=True  # To detect overfitting
    )

    grid_search.fit(X_train, y_train)

    training_time = time.time() - start_time

    # Extract cross-validation results
    cv_metrics = {
        "mean_test_score": float(-grid_search.best_score_),
        "std_test_score": float(grid_search.cv_results_['std_test_score'][grid_search.best_index_]),
        "mean_train_score": float(-grid_search.cv_results_['mean_train_score'][grid_search.best_index_]),
        "std_train_score": float(grid_search.cv_results_['std_train_score'][grid_search.best_index_]),
    }

    return grid_search.best_estimator_, grid_search.best_params_, training_time, cv_metrics

Critical Decisions

1. Scoring: neg_mean_absolute_error

scoring='neg_mean_absolute_error'

Why MAE and not RMSE or R²?

MAE (Mean Absolute Error): Penalizes errors linearly
RMSE: Penalizes errors quadratically (large errors weigh much more)
R²: Relative metric, difficult to interpret in business terms

For this problem:

MAE = $15,000 → “The model is off by $15k on average”
R² = 0.85 → What does this mean for the business?

neg_mean_absolute_error: GridSearchCV minimizes the metric, but MAE should be minimized, so we use the negative.

2. Cross-Validation: 5 Folds

cv=5

Why 5 and not 10?

5-fold: Balance between bias and variance
- Each fold has 80% training, 20% validation
- Faster than 10-fold (2x fewer fits)
10-fold: Less bias but higher computational cost
- Useful when you have few samples (<1000 samples)

With 16,512 training samples, 5-fold is sufficient.

3. return_train_score=True

return_train_score=True

This logs the score on training set in addition to validation set. Allows detecting overfitting:

if cv_metrics['mean_train_score'] >> cv_metrics['mean_test_score']:
    print("WARNING: Model is overfitting!")
    # Train MAE = $5k, Test MAE = $20k → Clear overfitting

4. n_jobs=-1: Parallelization

n_jobs=-1

Uses all available CPU cores. On an 8-core machine, 180 combinations × 5 folds = 900 fits are distributed in parallel.

Without parallelization: 900 fits × 2s/fit = 30 minutes With 8 cores: ~4 minutes

Evaluation Metrics: Beyond MAPE

def evaluate_model(model: Any, X_test: pd.DataFrame, y_test: pd.Series) -> Dict[str, float]:
    """Evaluates model with business-focused metrics."""

    y_pred = model.predict(X_test)
    y_true = y_test.values

    # Traditional metrics
    mae = mean_absolute_error(y_test, y_pred)
    rmse = np.sqrt(mean_squared_error(y_test, y_pred))
    r2 = r2_score(y_test, y_pred)

    # Business-focused percentage error metrics
    mape = mean_absolute_percentage_error(y_true, y_pred)
    smape = symmetric_mean_absolute_percentage_error(y_true, y_pred)
    wmape = weighted_mean_absolute_percentage_error(y_true, y_pred)
    median_ape = median_absolute_percentage_error(y_true, y_pred)

    # Prediction accuracy at different thresholds
    within_5pct = predictions_within_threshold(y_true, y_pred, 0.05)
    within_10pct = predictions_within_threshold(y_true, y_pred, 0.10)
    within_15pct = predictions_within_threshold(y_true, y_pred, 0.15)

    return {
        "mae": float(mae),
        "rmse": float(rmse),
        "r2": float(r2),
        "mape": float(mape),
        "smape": float(smape),
        "wmape": float(wmape),
        "median_ape": float(median_ape),
        "within_5pct": float(within_5pct),
        "within_10pct": float(within_10pct),
        "within_15pct": float(within_15pct)
    }

Why 4 Variants of MAPE:

1. MAPE (Mean Absolute Percentage Error)

mape = np.mean(np.abs((y_true - y_pred) / y_true)) * 100

Problem: Biased towards low values.

If you predict $500k instead of $510k → error = 2% If you predict $10k instead of $11k → error = 9%

Both are $10k absolute error, but MAPE penalizes the second more.

2. SMAPE (Symmetric MAPE)

smape = np.mean(np.abs(y_true - y_pred) / ((np.abs(y_true) + np.abs(y_pred)) / 2)) * 100

Uses the average of y_true and y_pred in the denominator. More symmetric:

Overprediction and underprediction have similar weight
Range: 0-200% (vs 0-∞% for MAPE)

3. wMAPE (Weighted MAPE)

wmape = np.sum(np.abs(y_true - y_pred)) / np.sum(np.abs(y_true)) * 100

Total sum of errors divided by total sum of actual values. Not affected by individual extreme values.

Used in Step 06 (Sweep) because it’s more robust than MAPE for datasets with high variance.

4. Median APE

median_ape = np.median(np.abs((y_true - y_pred) / y_true)) * 100

Median instead of mean. Robust to outliers.

If 95% of predictions have <5% error but 5% have >50% error:

MAPE: ~7% (average includes outliers)
Median APE: ~4% (outliers don’t affect median)

Within-X% Metrics

within_5pct = predictions_within_threshold(y_true, y_pred, 0.05)
# Percentage of predictions with error <5%

Business interpretation: “75% of our predictions are within ±10% of actual value.”

More interpretable for stakeholders than “MAPE = 8.2%”.

Step 05 Output

logger.info(" BEST MODEL: RandomForestRegressor")
logger.info("Business Metrics (Test Set):")
logger.info("  MAPE (Mean APE): 8.23%")
logger.info("  SMAPE (Symmetric MAPE): 7.95%")
logger.info("  wMAPE (Weighted MAPE): 8.01%")
logger.info("  Median APE: 6.45%")
logger.info("  Within ±5%: 45.2%")
logger.info("  Within ±10%: 72.8%")
logger.info("  Within ±15%: 85.3%")
logger.info("\nTraditional Metrics (Test Set):")
logger.info("  R²: 0.8654")
logger.info("  RMSE: $48,234.12")
logger.info("  MAE: $32,456.78")
logger.info("\nCross-Validation Results (5-fold):")
logger.info("  Mean CV MAE: $33,125.45 (±$2,341.23)")
logger.info("  Mean CV Train MAE: $28,934.56 (±$1,892.34)")

Best params saved:

best_params = {
    "n_estimators": 200,
    "max_depth": 20,
    "min_samples_split": 2,
    "min_samples_leaf": 1
}

These params are used as a starting point for Step 06 (exhaustive Sweep).

What This Strategy Achieves

Without model selection:

“I used Random Forest because everyone uses it”
You have no evidence it’s better than Gradient Boosting

With model selection:

“I compared 5 algorithms with 5-fold CV. Random Forest achieved MAPE=8.2% (vs GradientBoosting=8.9%, Ridge=12.3%). Here’s the comparison table in W&B.”
Data-backed decision, not intuition.

11. Testing: Fixtures, Mocking and Real Coverage

Why Testing ML Is Different

Tests in ML are not like tests in web apps. You can’t do:

def test_model_predicts_correct_value():
    model = load_model()
    assert model.predict([[1, 2, 3]]) == 452600.0  # ERROR: This is absurd

ML models are probabilistic. The output is not deterministic in the traditional software sense.

What you CAN test:

Data contracts: Inputs/outputs have correct types
Invariants: Predictions are in expected range
Reproducibility: Same input → same output (with fixed seed)
Pipeline integrity: Steps run without exploding
Integration: Components communicate correctly

conftest.py: Shared Fixtures

"""
Common fixtures for pytest
Author: Carlos Daniel Jiménez
"""
import pytest
import pandas as pd
import numpy as np
from google.cloud import storage
from unittest.mock import MagicMock, Mock

@pytest.fixture
def sample_housing_data():
    """Creates synthetic housing data."""
    np.random.seed(42)
    n_samples = 100

    data = {
        'longitude': np.random.uniform(-124, -114, n_samples),
        'latitude': np.random.uniform(32, 42, n_samples),
        'housing_median_age': np.random.randint(1, 53, n_samples),
        'total_rooms': np.random.randint(500, 5000, n_samples),
        'total_bedrooms': np.random.randint(100, 1000, n_samples),
        'population': np.random.randint(500, 3000, n_samples),
        'households': np.random.randint(100, 1000, n_samples),
        'median_income': np.random.uniform(0.5, 15, n_samples),
        'median_house_value': np.random.uniform(50000, 500000, n_samples)
    }

    df = pd.DataFrame(data)

    # Add missing values to total_bedrooms
    missing_indices = np.random.choice(n_samples, size=20, replace=False)
    df.loc[missing_indices, 'total_bedrooms'] = np.nan

    return df

@pytest.fixture
def mock_gcs_client():
    """Creates GCS client mock."""
    mock_client = MagicMock(spec=storage.Client)
    mock_bucket = MagicMock(spec=storage.Bucket)
    mock_blob = MagicMock(spec=storage.Blob)

    mock_bucket.exists.return_value = True
    mock_bucket.blob.return_value = mock_blob
    mock_client.bucket.return_value = mock_bucket

    return {
        'client': mock_client,
        'bucket': mock_bucket,
        'blob': mock_blob
    }

@pytest.fixture
def mock_mlflow(monkeypatch):
    """Mocks MLflow functions."""
    mock_log_metric = Mock()
    mock_log_param = Mock()
    mock_log_artifact = Mock()

    monkeypatch.setattr('mlflow.log_metric', mock_log_metric)
    monkeypatch.setattr('mlflow.log_param', mock_log_param)
    monkeypatch.setattr('mlflow.log_artifact', mock_log_artifact)

    return {
        'log_metric': mock_log_metric,
        'log_param': mock_log_param,
        'log_artifact': mock_log_artifact
    }

Imputation Test: Data Contracts

"""
Tests for ImputationAnalyzer
"""
import pytest
import pandas as pd
import numpy as np
from imputation_analyzer import ImputationAnalyzer

def test_imputation_analyzer_returns_dataframe(sample_housing_data):
    """Test that imputer returns DataFrame with filled missing values."""
    analyzer = ImputationAnalyzer(sample_housing_data, target_column="total_bedrooms")

    # Compare strategies
    results = analyzer.compare_all_methods()

    # Assertions
    assert len(results) == 4  # 4 strategies
    assert analyzer.best_method is not None
    assert all(result.rmse >= 0 for result in results.values())

    # Apply best imputer
    df_imputed = analyzer.apply_best_imputer(sample_housing_data)

    # Verify no NaNs remain
    assert df_imputed['total_bedrooms'].isnull().sum() == 0

    # Verify rest of columns didn't change
    assert len(df_imputed) == len(sample_housing_data)

def test_imputation_analyzer_reproducibility():
    """Test that imputation is reproducible with fixed seed."""
    np.random.seed(42)
    df1 = generate_sample_data(n=100)

    analyzer1 = ImputationAnalyzer(df1, random_state=42)
    results1 = analyzer1.compare_all_methods()

    np.random.seed(42)
    df2 = generate_sample_data(n=100)

    analyzer2 = ImputationAnalyzer(df2, random_state=42)
    results2 = analyzer2.compare_all_methods()

    # Same input + same seed = same output
    assert results1['simple_median'].rmse == results2['simple_median'].rmse

Complete Pipeline Test: Integration Test

"""
Integration test of complete pipeline
"""
import pytest
from pathlib import Path

def test_pipeline_runs_end_to_end(tmp_path, mock_gcs_client, sample_housing_data):
    """Test that pipeline runs from beginning to end without exploding."""

    # Setup: Save synthetic data
    data_path = tmp_path / "housing.parquet"
    sample_housing_data.to_parquet(data_path, index=False)

    # Step 01: Download (mocked)
    # ...

    # Step 02: Preprocessing
    from preprocessor import DataPreprocessor
    config = PreprocessingConfig(
        gcs_input_path=str(data_path),
        gcs_output_path=str(tmp_path / "processed.parquet"),
        bucket_name="test-bucket"
    )

    preprocessor = DataPreprocessor(config)
    result = preprocessor.run()

    assert result.success
    assert result.num_rows_output > 0

    # Step 03: Feature Engineering
    # ...

    # Verify outputs exist
    assert (tmp_path / "processed.parquet").exists()

Real Coverage

# Run tests with coverage
pytest tests/ --cov=src --cov-report=html --cov-report=term-missing

# Output:
# ==================== test session starts ====================
# tests/test_imputation_analyzer.py ........    [80%]
# tests/test_feature_engineering.py ....       [100%]
#
# ----------- coverage: 87% -----------
# src/data/02_preprocessing/imputation_analyzer.py   92%
# src/data/03_feature_engineering/feature_engineer.py   85%

What This Achieves

Without tests: “I think it works, I ran the notebook once and it didn’t explode.”

With tests: “87% coverage. All critical components are tested. CI runs tests on each commit.”

Tests don’t guarantee the model is good, but they guarantee the system that produces the model is reliable.

12. Production Patterns Nobody Tells You About

The Real Problem of Serving

Here’s what no tutorial tells you: 90% of the effort in ML is not training a model—it’s making that model serve reliable predictions 24/7 without exploding.

ML courses end with model.save('model.pkl'). The reality of production starts with questions like:

What if the model needs a trained KMeans to generate features?
Do you save the KMeans too? What if it weighs 500MB?
How do you guarantee that preprocessing in production is EXACTLY the same as in training?
What if the data distribution changes and your model starts failing silently?

This pipeline implements solutions to these problems that are rarely discussed. Let’s dissect them.

12.1. The Transform Pattern: The Synthetic KMeans Trick

Context: In Step 03 (Feature Engineering), the pipeline trains a KMeans with 10 clusters on latitude/longitude. The final model needs cluster_label as a feature.

Classic problem:

# During training
kmeans = KMeans(n_clusters=10)
kmeans.fit(X_geo)  # Trains on 16,000 California samples
df['cluster_label'] = kmeans.predict(X_geo)

# Train the model
model.fit(df, y)

# Now what? How do you save the kmeans to use in the API?

Naive solution (what 80% of people do):

# Save BOTH models
pickle.dump(kmeans, open('kmeans.pkl', 'wb'))
pickle.dump(model, open('model.pkl', 'wb'))

# In the API: Load both
kmeans = pickle.load(open('kmeans.pkl', 'rb'))
model = pickle.load(open('model.pkl', 'rb'))

# For each prediction:
cluster = kmeans.predict([[lon, lat]])
features = [..., cluster]
prediction = model.predict(features)

Why this is terrible:

Storage overhead: Serialized KMeans can weigh 96KB per model. Multiply that by 50 model versions.
Coupling: Now your API needs to load TWO artifacts per model version. What if they get out of sync?
Latency: Calling kmeans.predict() adds ~2ms per request.

The brilliant solution this project implements:

Chip Huyen calls this the Transform Pattern in “Designing Machine Learning Systems” (Chapter 7, section on feature consistency): when preprocessing is lightweight and deterministic, recreate it in the serving layer instead of serializing it.

Look at the real code in api/app/core/preprocessor.py (lines 61-110):

class HousingPreprocessor:
    def _init_kmeans(self):
        """
        Initialize KMeans with California housing geographical clusters.
        Uses typical California housing coordinates to create clusters.
        This is an approximation but works for the API use case.
        """
        # California housing typical ranges:
        # Longitude: -124 to -114
        # Latitude: 32 to 42

        np.random.seed(42)  # CRITICAL: Same seed as in training

        # Create synthetic data representing California geography
        n_samples = 1000
        lon_samples = np.random.uniform(-124, -114, n_samples)
        lat_samples = np.random.uniform(32, 42, n_samples)

        # Weight towards major population centers
        major_centers = np.array([
            [-118, 34],   # LA
            [-122, 37.5], # SF
            [-117, 33],   # San Diego
            [-121, 38.5], # Sacramento
            [-119, 36.5], # Fresno
        ])

        # Add major centers multiple times for proper weighting
        lon_samples[:50] = major_centers[:, 0].repeat(10)
        lat_samples[:50] = major_centers[:, 1].repeat(10)

        X_geo = np.column_stack([lon_samples, lat_samples])

        # Fit KMeans
        self.kmeans = KMeans(
            n_clusters=10,
            n_init=10,
            random_state=42  # SAME seed as training
        )
        self.kmeans.fit(X_geo)

What’s happening here?

Instead of serializing the KMeans trained with 16,512 real samples, the API recreates a synthetic KMeans using:

Synthetic data that approximates California’s geographical distribution
Same seed (42) that was used in training
Same n_clusters (10)
Weighted centers towards major cities (LA, SF, San Diego)

Trade-offs of this solution:

Advantages:

Zero storage overhead (don’t save the KMeans)
Zero coupling (API is autonomous, doesn’t need additional artifacts)
Identical latency (~2ms either way)
Stateless serving (can scale API horizontally without shared state)

Disadvantages:

Cluster drift: Synthetic clusters are NOT exactly the same as training ones
- In internal testing: ~2% mismatch in cluster labels
- In California Housing: impact on MAPE < 0.3%
Requires preprocessing to be deterministic and lightweight
- Doesn’t work if your KMeans needs 1 million samples to converge
- Doesn’t work if you have 512-dimensional text embeddings

When to use this pattern:

DO use it if:

Preprocessing is lightweight (<10ms)
Feature is geographical/categorical with few unique values
Impact of slight inconsistency is tolerable (regression, classification with margin)

DON’T use it if:

Feature is a deep embedding (BERT, ResNet)
You need 100% bit-by-bit reproducibility
Preprocessing requires gigabytes of state

The lesson:

Chip Huyen summarizes it well: “The best feature engineering pipeline is the one that doesn’t exist.” If you can compute features on-the-fly without prohibitive cost, avoid serializing state. Your system will be simpler, more robust, and easier to debug.

This synthetic KMeans trick is a perfect example. You won’t find this in any Kaggle tutorial.

12.2. Training/Serving Skew: The Silent Killer

Huyen dedicates an entire section to this in Chapter 7. Training/serving skew is when preprocessing in training is different from serving.

Classic example that kills projects:

# In your training notebook
df['total_rooms_log'] = np.log1p(df['total_rooms'])

# 6 months later, someone implements the API
# (without reading the complete notebook)
features['total_rooms_log'] = np.log(features['total_rooms'])  # BUG: log vs log1p

# Result: Model fails silently
# MAPE in training: 8%
# MAPE in production: 24%
# Why? Because log(0) = -inf, log1p(0) = 0

How this project avoids this:

Preprocessing is encapsulated in ONE single class used BOTH in training and serving:

# src/data/02_preprocessing/preprocessor.py
class DataPreprocessor:
    def transform(self, df):
        # Imputation
        df = self._impute(df)
        # One-hot encoding
        df = pd.get_dummies(df, columns=['ocean_proximity'])
        return df

# Used in training (Step 02)
preprocessor = DataPreprocessor()
train_processed = preprocessor.transform(train_raw)

# SAME code used in API
# api/app/core/preprocessor.py
class HousingPreprocessor:  # Same transform logic
    def transform(self, df):
        # Same one-hot encoding
        # Same column order
        return df

The guarantee:

If you change preprocessing, both training and serving update because it’s the same code.

The anti-pattern:

# Training: notebook_v3_FINAL.ipynb
df['bedrooms_per_room'] = df['total_bedrooms'] / df['total_rooms']

# API: Someone copy/pastes without verifying
features['bedrooms_per_room'] = features['total_bedrooms'] / features['total_rooms']
# What happens with division by zero?
# What if total_rooms is 0?
# In training it never happened because you cleaned outliers
# In production... BOOM

The mantra:

“If you can’t import it, you can’t trust it.” If your preprocessing is copy/pasted between training and serving, you’ve already lost.

12.3. Data Drift: The Enemy This Project (Still) Doesn’t Monitor

Now let’s talk about what is NOT in this project but is critical for production systems.

Data drift is when the distribution of your features in production changes compared to training.

Huyen covers this exhaustively in Chapter 8 (“Data Distribution Shifts”). There are three types:

1. Covariate Shift (most common):

# Training data (2020-2022)
# Distribution of median_income
P_train(median_income): mean = $6.2k, std = $3.1k

# Production data (2023-2024)
# After inflation + economic changes
P_prod(median_income): mean = $8.5k, std = $4.2k

# Result:
# - Model was trained on features with mean=$6.2k
# - Now receives features with mean=$8.5k
# - Predictions become inaccurate

2. Label Shift:

# Training: California 2020
# median_house_value average: $250k

# Production: California 2024
# median_house_value average: $400k (real estate boom)

# Model predicts based on 2020 relationships
# But absolute prices changed

3. Concept Drift:

The relationship between features and target changes.

# 2020: ocean_proximity='NEAR OCEAN' → +$50k in price
# 2024: Work-from-home → people prefer INLAND → -$20k

# Model's coefficient for 'NEAR OCEAN' is obsolete

How to detect drift (what this project should add):

Option 1: Statistical Tests (Kolmogorov-Smirnov, Chi-Square)

from scipy.stats import ks_2samp

# Compare training vs production distribution
for feature in features:
    stat, p_value = ks_2samp(
        training_data[feature],
        production_data[feature]
    )
    if p_value < 0.05:
        alert(f"DRIFT DETECTED in {feature}: p={p_value}")

Option 2: Evidently AI (recommended)

from evidently.report import Report
from evidently.metric_preset import DataDriftPreset

report = Report(metrics=[DataDriftPreset()])

report.run(
    reference_data=train_df,  # Training data
    current_data=production_df  # Last 1000 predictions
)

# Generate HTML dashboard with drift metrics
report.save_html("drift_report.html")

Evidently calculates:

Drift score per feature (0-1)
Share of drifted features (% of features with drift)
Dataset drift (if complete dataset drifted)

Option 3: Population Stability Index (PSI)

Metric used in banking to detect drift:

def calculate_psi(expected, actual, bins=10):
    """
    PSI < 0.1: No significant drift
    PSI < 0.2: Moderate drift
    PSI >= 0.2: Significant drift (retrain needed)
    """
    breakpoints = np.quantile(expected, np.linspace(0, 1, bins+1))

    expected_percents = np.histogram(expected, breakpoints)[0] / len(expected)
    actual_percents = np.histogram(actual, breakpoints)[0] / len(actual)

    psi = np.sum(
        (actual_percents - expected_percents) *
        np.log(actual_percents / expected_percents)
    )

    return psi

When to add drift detection:

Huyen recommends waiting until you have sufficient production traffic (~10,000 predictions).

Don’t add it on Day 1 because:

You need baseline of “normal production distribution”
False positives at the beginning (people testing the API with synthetic data)
Infrastructure overhead (Evidently requires DB to store histories)

Add it when:

You have 10,000+ predictions in production
You observe that production MAPE > test set MAPE
Model has >6 months in production without retraining

Example alerting:

# W&B logger extension (what you would add to wandb_logger.py)
class WandBLogger:
    def log_drift_alert(self, feature_name, psi_value, threshold=0.2):
        if psi_value > threshold:
            wandb.alert(
                title=f"DATA DRIFT: {feature_name}",
                text=f"PSI={psi_value:.3f} exceeds threshold {threshold}",
                level=wandb.AlertLevel.WARN
            )

            # Log to metrics
            wandb.log({
                f"drift/{feature_name}": psi_value,
                "drift/timestamp": datetime.now()
            })

The cost of NOT monitoring drift:

Without drift detection, your model fails silently. Nobody notices until:

A customer complains: “Your predictions are very wrong lately”
You calculate retrospective MAPE and discover it went from 8% to 18%
3 months passed serving garbage predictions

With monitoring, you detect drift in days, not months.

12.4. Model Monitoring: Beyond Accuracy

This project’s W&B Logger (api/app/core/wandb_logger.py) logs basic metrics:

wandb.log({
    "prediction/count": len(predictions),
    "prediction/mean": np.mean(predictions),
    "performance/response_time_ms": response_time
})

This is a good start, but incomplete. In real production, you need to monitor:

1. Business Metrics (most important)

# How many predictions are "very wrong"?
errors = np.abs(y_true - y_pred) / y_true
within_10pct = (errors < 0.10).mean()

wandb.log({
    "business/predictions_within_10pct": within_10pct,
    "business/predictions_within_20pct": (errors < 0.20).mean(),
    "business/mean_absolute_error_dollars": np.mean(np.abs(y_true - y_pred))
})

# Alert if quality drops
if within_10pct < 0.65:  # SLA threshold
    send_alert("Model quality degraded: only {:.1%} within 10%".format(within_10pct))

2. Prediction Distribution

# Is the model always predicting the same value?
# (signal of overfitting or broken model)

prediction_std = np.std(predictions)
prediction_range = np.max(predictions) - np.min(predictions)

wandb.log({
    "prediction/std": prediction_std,
    "prediction/range": prediction_range,
    "prediction/median": np.median(predictions)
})

# Red flag: If std is very low
if prediction_std < 10000:  # $10k
    alert("Model predictions have very low variance - model may be broken")

3. Input Feature Distribution

# Are you receiving inputs outside training range?

for feature in NUMERIC_FEATURES:
    feature_values = [pred[feature] for pred in prediction_batch]

    wandb.log({
        f"input/{feature}/mean": np.mean(feature_values),
        f"input/{feature}/p95": np.percentile(feature_values, 95),
        f"input/{feature}/p05": np.percentile(feature_values, 5)
    })

    # Alert if there are extreme outliers
    if np.max(feature_values) > TRAINING_MAX[feature] * 2:
        alert(f"Extreme outlier detected in {feature}")

4. Error Patterns

# Does the model consistently fail on certain segments?

errors_by_segment = {}

# By geographic region
for ocean_prox in ['<1H OCEAN', 'INLAND', 'ISLAND', 'NEAR BAY']:
    mask = (df['ocean_proximity'] == ocean_prox)
    errors_by_segment[ocean_prox] = mape(y_true[mask], y_pred[mask])

wandb.log({f"error/mape_{seg}": err for seg, err in errors_by_segment.items()})

# If ISLAND has MAPE = 40% but others have 8%, there's a problem

5. Latency Percentiles

# Current logger only logs mean response time
# But you need percentiles to detect outliers

response_times = [...]  # last 100 requests

wandb.log({
    "latency/p50": np.percentile(response_times, 50),
    "latency/p95": np.percentile(response_times, 95),
    "latency/p99": np.percentile(response_times, 99),
    "latency/max": np.max(response_times)
})

# Alert if p99 exceeds threshold
if np.percentile(response_times, 99) > 200:  # 200ms
    alert("API latency p99 exceeds 200ms")

Recommended dashboard (W&B or Grafana):

┌─────────────────────────────────────────────┐
│ MODEL HEALTH DASHBOARD                       │
├─────────────────────────────────────────────┤
│ PREDICTIONS (last 24h)                       │
│   Total:        12,453                       │
│   Within 10%:   68.2% [OK]                   │
│   Within 20%:   89.1%                        │
│   Mean MAPE:    9.8%  [WARN] (threshold: 10%)│
├─────────────────────────────────────────────┤
│ DRIFT DETECTION                              │
│   median_income:     PSI = 0.08 [OK]        │
│   total_rooms:       PSI = 0.15 [WARN]      │
│   ocean_proximity:   PSI = 0.32 [ALERT]     │
├─────────────────────────────────────────────┤
│ LATENCY                                      │
│   p50:   28ms                                │
│   p95:   67ms                                │
│   p99:   145ms [WARN]                        │
└─────────────────────────────────────────────┘

12.5. The Cascade Pattern: Fallback Resilience

This project implements a brilliant resilience pattern that Huyen discusses in Chapter 6: the Cascade Pattern (cascading fallback).

Look at the ModelLoader in api/app/core/model_loader.py:

def load_model(self) -> Any:
    """Load model with cascade fallback strategy."""

    # Priority 1: MLflow Registry (production)
    if self.mlflow_model_name:
        try:
            self._model = self.load_from_mlflow(...)
            return self._model
        except Exception as e:
            logger.warning(f"MLflow load failed, trying GCS: {e}")

    # Priority 2: GCS (staging)
    if self.gcs_bucket and self.gcs_model_path:
        try:
            self._model = self.load_from_gcs(...)
            return self._model
        except Exception as e:
            logger.warning(f"GCS load failed, trying local: {e}")

    # Priority 3: Local (development/fallback)
    if self.local_model_path and Path(self.local_model_path).exists():
        self._model = self.load_from_local(self.local_model_path)
        return self._model

    raise RuntimeError("No model could be loaded from any source")

What does this achieve?

Resilience to failures:

MLflow server down → API continues working with GCS
GCS quota exceeded → API uses local model
Zero downtime with degraded infrastructure

Deployment flexibility:

Production: Uses MLflow (robust versioning)
Staging: Uses GCS (simpler)
Local development: Uses local file (no credentials)

Same code, three environments:

# Production
docker run -e MLFLOW_MODEL_NAME=housing_price_model \
           -e MLFLOW_MODEL_STAGE=Production \
           housing-api

# Staging
docker run -e GCS_BUCKET=staging-bucket \
           -e GCS_MODEL_PATH=models/v1.2.pkl \
           housing-api

# Local development
docker run -v $(pwd)/models:/app/models \
           -e LOCAL_MODEL_PATH=/app/models/housing_price_model.pkl \
           housing-api

What’s missing (and you should add):

1. Circuit Breaker Pattern

from circuitbreaker import circuit

@circuit(failure_threshold=5, recovery_timeout=60)
def load_from_mlflow(self, model_name, stage):
    """
    Circuit breaker: If MLflow fails 5 consecutive times,
    open circuit for 60 seconds and don't attempt more calls.
    """
    client = MlflowClient(self.tracking_uri)
    return mlflow.sklearn.load_model(f"models:/{model_name}/{stage}")

Why: Without circuit breaker, if MLflow is down, API makes 1 request per prediction and waits for timeout (5-10s). With circuit breaker, detects failure after 5 attempts and stops calling until MLflow recovers.

2. Retry with Exponential Backoff

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=2, max=10)
)
def load_from_gcs(self, bucket_name, blob_path):
    """
    Retry with exponential backoff:
    - Attempt 1: immediate
    - Attempt 2: wait 2s
    - Attempt 3: wait 4s
    """
    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(blob_path)
    return pickle.loads(blob.download_as_bytes())

Why: GCS can have transient failures (rate limiting, network blips). Automatic retry prevents a momentary failure from bringing down your API.

3. Timeout Configuration

# Currently there's no configured timeout
# If MLflow takes 60s to respond, your API waits 60s

# Better:
def load_from_mlflow(self, model_name, stage, timeout=10):
    """Load model with timeout."""
    import signal

    def timeout_handler(signum, frame):
        raise TimeoutError("MLflow load exceeded timeout")

    signal.signal(signal.SIGALRM, timeout_handler)
    signal.alarm(timeout)  # 10 second timeout

    try:
        model = mlflow.sklearn.load_model(...)
        signal.alarm(0)  # Cancel alarm
        return model
    except TimeoutError:
        logger.error(f"MLflow load timeout after {timeout}s")
        raise

Why: Without timeout, a slow MLflow server can make your API take minutes to respond. With timeout, fail fast and try the next fallback.

4. Health Check Endpoint

# api/app/routers/health.py

@router.get("/health/deep")
async def deep_health_check():
    """
    Health check that verifies all dependencies.
    Kubernetes calls this every 30s for routing decisions.
    """
    health = {
        "status": "healthy",
        "model_loaded": model_loader.is_loaded,
        "model_version": model_loader.model_version,
        "dependencies": {}
    }

    # Check MLflow
    try:
        client = MlflowClient(settings.MLFLOW_TRACKING_URI)
        client.list_experiments(max_results=1)
        health["dependencies"]["mlflow"] = "healthy"
    except Exception as e:
        health["dependencies"]["mlflow"] = f"degraded: {e}"
        health["status"] = "degraded"

    # Check GCS
    try:
        storage_client = storage.Client()
        bucket = storage_client.bucket(settings.GCS_BUCKET)
        bucket.exists()
        health["dependencies"]["gcs"] = "healthy"
    except Exception as e:
        health["dependencies"]["gcs"] = f"degraded: {e}"

    return health

Output:

{
  "status": "degraded",
  "model_loaded": true,
  "model_version": "models:/housing_price_model/Production",
  "dependencies": {
    "mlflow": "degraded: Connection timeout",
    "gcs": "healthy"
  }
}

Why: Tells your load balancer (Cloud Run, Kubernetes) if the API is healthy. If MLflow is down but model is already loaded (cached), API is “degraded” but functional.

12.6. Feature Store Anti-Pattern: When You DON’T Need One

Huyen has a controversial section in Chapter 5: “You might not need a feature store.”

Feature Stores (Feast, Tecton, Databricks) are very popular, but are overkill for 80% of projects.

When you DO need a Feature Store:

You reuse features across multiple models
- Example: customer_lifetime_value is used in 10 different models
- Without feature store: Each model recalculates the same feature (waste)
- With feature store: Calculate once, serve many times
You need features with different freshness
- Batch features: Calculated daily (credit score)
- Real-time features: Calculated per request (current location)
- Feature store orchestrates both
Training/Serving skew is critical
- Feature store guarantees training and serving use EXACTLY the same logic

When you DON’T need a Feature Store (like this project):

All features are computed on-the-fly
- This project: Features are straightforward (lat, lon, income, age)
- Only computed feature is cluster_label (2ms latency)
- No complex aggregations like “average income in last 30 days”
Single model consumes the features
- No reuse across models
- Feature store would add complexity without benefit
Latency budget is generous
- This API: <50ms is OK
- If you needed <5ms, pre-computing features would be worthwhile

Real cost of a Feature Store:

Infrastructure: Redis/DynamoDB for serving, Spark for batch processing
Cost: ~$500-2000/month on AWS/GCP (depending on traffic)
Complexity: Another system to monitor, debug, operate

Lightweight alternative (what this project does):

# Compute features on-the-fly in the API
class HousingPreprocessor:
    def transform(self, df):
        # 1. One-hot encoding (instantaneous)
        df_encoded = pd.get_dummies(df, columns=['ocean_proximity'])

        # 2. Clustering (2ms with pre-fitted KMeans)
        clusters = self.kmeans.predict(df[['longitude', 'latitude']])
        df_encoded['cluster_label'] = clusters

        return df_encoded

Total latency: ~3ms. Doesn’t justify a Feature Store.

When to reconsider:

If you add features like “average house price in zipcode” (requires DB query)
If preprocessing goes above >20ms
If you add a second model that reuses 50%+ of features

Until then, YAGNI (You Ain’t Gonna Need It).

12.7. Production Readiness: An Honest Checklist

Based on exhaustive code analysis, here’s the real state of this project:

What This Project Does VERY WELL

Level 3/5 in MLOps Maturity (Production-Ready):

Complete versioning
- Models in MLflow Registry with rich metadata
- Data artifacts in GCS with timestamps
- Code in git with CI/CD
- Config in versioned YAML
Reproducibility
- Fixed seeds (random_state=42 everywhere)
- Pinned dependencies (requirements.txt)
- Docker for environment consistency
Testing
- 87% code coverage
- Unit tests with realistic fixtures
- End-to-end integration tests
- Security scanning (Bandit, TruffleHog)
CI/CD
- GitHub Actions with automated tests
- Docker build in CI
- Cloud Run deployment with health checks
- Staging/Production separation
API Design
- Pydantic validation on all endpoints
- Cascade fallback (MLflow→GCS→Local)
- Lifespan management (load model once, not per request)
- Batch prediction support
Observability (Basic)
- W&B logging of predictions
- Response time tracking
- Structured logging

What’s Missing (And When To Add It)

Level 4/5 Features (Add When You Have 10k+ Daily Predictions):

Data Drift Detection [MISSING]
- Impact: High (model fails silently)
- Implementation cost: Medium (Evidently AI)
- When: After 3 months in production
Model Performance Tracking [MISSING]
- Impact: High (don’t know if model degrades)
- Cost: Low (extend W&B logger)
- When: After having ground truth labels (1-2 months)
Circuit Breakers [MISSING]
- Impact: Medium (better latency during failures)
- Cost: Low (circuitbreaker library)
- When: If you see transient failures in MLflow/GCS
Advanced Monitoring Dashboards [MISSING]
- Impact: Medium (better debugging)
- Cost: Medium (Grafana + Prometheus)
- When: When team grows >5 people
Canary Deployments [MISSING]
- Impact: Low (you have manual rollback that works)
- Cost: High (requires traffic splitting)
- When: Only if deploying >1x/week
Feature Store [MISSING]
- Impact: None (features are lightweight)
- Cost: High ($500-2000/month)
- When: Never, unless you add heavy features

Level 5/5 Features (Overkill For This Project):

Multi-model orchestration (A/B testing)
Real-time retraining
Federated learning
AutoML pipeline

Prioritized Recommendations

MONTH 1-3 (Stabilization):

Add /health/deep endpoint with dependency checks
Implement retry with exponential backoff on GCS calls
Configure alerts in W&B when MAPE > 12%

MONTH 4-6 (Monitoring):

Implement Evidently AI for data drift (PSI tracking)
Add prediction distribution monitoring
Configure automated retraining trigger when PSI > 0.2

MONTH 7-12 (Optimization):

Implement circuit breaker on MLflow calls
Add Redis for prediction caching (if latency is an issue)
Configure Grafana dashboard for business metrics

DON’T Do (Until You Scale 10x):

Don’t implement Feature Store
Don’t add Kafka streaming
Don’t use Kubernetes (Cloud Run is sufficient)
Don’t implement multi-model serving (until clear use case)

12.8. The Difference Between “Works” and “Works in Production”

This project is in the top 10% of ML projects in terms of engineering practices. Most models in production have:

Notebooks instead of modular scripts
Models saved as model_v3_FINAL_FINAL.pkl
Zero tests
Manual deployment with scp
No monitoring

This project has:

Modular and testable code
MLflow Registry with semantic versioning
87% test coverage
Automated deployment with GitHub Actions
Basic W&B monitoring

The remaining gap (drift detection, advanced monitoring, circuit breakers) is the gap between “stable production” and “enterprise-grade production”.

But here’s the secret: that gap only matters when you have real users and significant traffic.

Don’t optimize for problems you don’t have yet. This project is ready to serve 100k predictions/month without breaking a sweat. When you reach 1M/month, then add data drift detection. When you reach 10M/month, then consider Kubernetes.

As Huyen says: “The best ML system is the simplest one that meets your requirements.”

This project fulfills that principle perfectly.

14. Conclusions: MLOps As an Engineering Discipline

What This Pipeline Implements (And Why It Matters)

This is not a scikit-learn tutorial. It’s a production-ready system that implements:

Complete versioning: Data (GCS), code (git), models (MLflow), configuration (YAML)
Reproducibility: Same code + same config + same seed = same model
Observability: Structured logs, metrics in W&B, tracking in MLflow
Testing: 87% coverage, unit tests, integration tests, security scanning
CI/CD: GitHub Actions with automated deployment to Cloud Run
Deployment: REST API with FastAPI, frontend with Streamlit, Docker Compose ready
Data-backed decisions: Every choice (imputation, K clusters, hyperparameters) has quantifiable metrics
Production patterns: Transform pattern, cascade fallback, training/serving consistency

The Anti-Patterns It Avoids (That Kill Projects)

X Notebooks in production: Everything is modular and testable Python. Notebooks are great for exploration, terrible for reliable systems.

X Hardcoded configuration: config.yaml versioned in git. If you change a parameter, it’s recorded with timestamp and author.

X “I used median because yes”: Compared 4 imputation strategies with quantifiable metrics. Best strategy (Iterative Imputer) won by 3.2% in RMSE.

X Models as final_v3_REAL_final.pkl: MLflow Registry with semantic versions and rich metadata. You know exactly what hyperparameters, what data, and what metrics each version has.

X “I don’t know what hyperparameters I used 3 months ago”: Each model records 106 lines of metadata. Includes everything from hyperparameters to error distribution by segment.

X Manual deployment with scp: Docker + GitHub Actions. Push to master → tests run → if they pass, deploys to staging automatically. Production requires manual approval (as it should).

X Training/Serving Skew: Preprocessing is in a shared class between training and serving. Change code once, both environments update.

Conscious Trade-Offs (Because There Are No Perfect Solutions)

This project makes deliberate decisions. Here are the trade-offs and when to reconsider them:

1. Cluster optimization independent of final model:

Optimizes KMeans with silhouette score instead of cross-validation of complete model. Faster but less rigorous. Reconsider if clustering is your model’s most important feature.

2. 60 sweep runs in W&B:

Sufficient for California Housing (medium dataset, ~20k samples). You might need 200+ runs on complex datasets with many non-linear interactions.

3. Sequential pipeline without parallelization:

Steps run one after another. This pipeline takes ~15 minutes end-to-end. If your pipeline takes hours, use Airflow/Prefect with parallel tasks.

4. MAPE as primary metric:

Works for this dataset (prices between $50k-$500k). Doesn’t work if you have values close to zero (division by zero) or if you want to penalize large errors disproportionately (use RMSE).

5. Data drift detection absent:

As the Production Checklist explains (Section 13.7), drift monitoring should be added after 3-6 months in production, not Day 1. You need baseline of normal behavior first.

6. Synthetic KMeans in the API:

The Transform Pattern (Section 13.1) recreates clusters with ~2% drift vs training. Impact on MAPE: <0.3%. If you need 100% bit-by-bit reproducibility, serialize the real KMeans (cost: 96KB per model version).

What’s Missing (And When To Add It)

As Section 13 (Production Patterns) details, this project is at Level 3/5 of MLOps Maturity. What’s missing:

Month 1-3 (Stabilization):

Deep health check endpoint with dependency status
Retry with exponential backoff on GCS calls
Automatic alerts in W&B when MAPE > threshold

Month 4-6 (Monitoring):

Evidently AI for data drift detection (PSI tracking)
Prediction distribution monitoring (detect broken model)
Automatic retraining trigger when PSI > 0.2

Month 7-12 (Optimization):

Circuit breaker on MLflow calls (avoid cascading timeouts)
Redis for prediction caching (if latency <10ms is critical)
Grafana dashboards for business metrics

DON’T do (until you scale 10x):

Feature Store (features are lightweight, <3ms)
Kafka streaming (Cloud Run with HTTP is sufficient)
Kubernetes (Cloud Run autoscales without complexity)
Multi-model A/B testing (until clear use case)

The Uncomfortable Truth About MLOps

90% of ML models never reach production. Of those that do, 60% fail in the first 6 months.

Why?

Not because the models are bad. It’s because:

The engineer who trained the model is no longer at the company
Nobody knows what hyperparameters were used
Preprocessing in production is different from training
There are no tests, so every change breaks something
Deployment is manual, takes 3 hours and fails 1 out of 3 times
There’s no monitoring, model fails silently for months

This project avoids all those problems. Not because it’s perfect, but because it implements basic software engineering principles:

Versioning: Everything (data, code, models, config)
Testing: 87% coverage, CI on each commit
Reproducibility: Fixed seeds, Dockerized environments
Observability: Logs, metrics, tracking
Automation: Deployment without human intervention

The Most Important Lesson

Chip Huyen says it better than I can in “Designing Machine Learning Systems”:

“The best ML system is not the one with the highest accuracy. It’s the one that’s reliable, maintainable, and meets business requirements.”

This project doesn’t have the best model. You can probably improve MAPE from 8.2% to 7.5% with hand-tuned XGBoost.

But that doesn’t matter.

What matters is that this system:

Runs reliably 24/7
Can be updated without downtime
Has automatic rollback if something fails
Any team member can understand and modify the code
Logs enough information to debug problems
Costs <$100/month on GCP (up to 1M predictions)

That 0.7% improvement in MAPE isn’t worth it if the system is impossible to maintain.

Who This Post Is For

If you are:

Data Scientist trying to take your first model to production → This is your roadmap
ML Engineer explaining why “you can’t just deploy the notebook” → Send this post
Engineering Manager evaluating if your team does MLOps correctly → Use Section 13.7 as checklist
Student wanting to learn MLOps beyond tutorials → This is real code, not synthetic

The Next Step

This post has 6,500+ lines because I didn’t want to simplify. MLOps is complex. There are trade-offs in every decision.

But don’t let complexity paralyze you. Start simple, iterate, improve.

Week 1: Basic versioning (git + requirements.txt)
Week 2: Basic tests (at least smoke tests)
Week 3: Docker for consistent deployment
Week 4: Basic CI (GitHub Actions running tests)
Month 2: MLflow for model registry
Month 3: Basic monitoring (W&B or Prometheus)

You don’t need to implement everything on Day 1. This project took months to reach this state.

The Last Word

Being an MLOps engineer is not just training models—it’s building systems where models are one more piece.

What separates a research project from a production product is:

Order: Everything in its place (not “works on my machine”)
Testing: What isn’t tested, breaks (87% coverage is not an accident)
Observability: If you can’t measure it, you can’t improve it (W&B + MLflow)
Reproducibility: Today and in 6 months should give the same result (fixed seeds, Docker)
Automation: Humans are bad at repetitive tasks (CI/CD)
Humility: Recognizing what’s missing and when to add it (Section 13.7)

This post doesn’t teach you to be better at machine learning.

It teaches you to be better at machine learning engineering.

And that difference is what separates models in notebooks from models in production creating real value.

If you implement even 50% of what’s in this post, your pipeline will be in the top 10% of ML projects in terms of engineering practices.

If you implement 80%, you’ll be ready to scale to millions of predictions without restructuring everything.

100% is overkill for most projects. Use the Production Checklist (Section 13.7) to prioritize what you need and when.

References and Resources

Fundamental books:

Géron, A. (2022). Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow (3rd ed.). O’Reilly.
- Chapter 2: Base of this project (California Housing dataset, feature engineering, model selection)
- Focus on ML, this post adds production infrastructure
Huyen, C. (2022). Designing Machine Learning Systems. O’Reilly.
- Chapter 5: Feature stores and when you don’t need one
- Chapter 6: Deployment patterns (Cascade, Circuit Breaker)
- Chapter 7: Transform Pattern and Training/Serving Skew (Sections 13.1 and 13.2 of this post)
- Chapter 8: Data Distribution Shifts and drift detection (Section 13.3)
- Complete book: If you only read one book about MLOps, make it this one

Tools (with links to docs):

MLflow: Model registry and experiment tracking
Weights & Biases: Sweep and experiment visualization
Hydra: Configuration management with composable configs
FastAPI: REST API framework with Pydantic validation
Streamlit: Interactive frontend for ML apps
Google Cloud Storage: Artifact storage
Evidently AI: Data drift detection (recommended for production)
Docker: Containerization and reproducibility

Complete repository:

github.com/carlosjimenez88M/mlops-hand-on-ML-and-pytorch
- /api: FastAPI with cascade fallback and Transform Pattern
- /src: Modular pipeline (01-07) with MLflow tracking
- /tests: 87% coverage with realistic fixtures
- /.github/workflows: Complete CI/CD with security scanning

Author: Carlos Daniel Jiménez Email: danieljimenez88m@gmail.com LinkedIn: linkedin.com/in/carlosdanieljimenez Date: January 2026

← Part 2: Deployment and Infrastructure | ← Part 1: Pipeline and Orchestration

Complete series:

Part 1: Pipeline and Orchestration
Part 2: Deployment and Infrastructure
Part 3: Production and Best Practices (current)

Repository: github.com/carlosjimenez88M/mlops-hand-on-ML-and-pytorch

Anatomy of an MLOps Pipeline - Part 3: Production and Best Practices#

11. Model and Parameter Selection Strategies#

The Complete Flow: Selection → Sweep → Registration#

Step 05: Model Selection - Algorithm Comparison#

The 5 Candidate Models#

Parameter Grids: Initial GridSearch#

Grid Design Decisions#

Training Function with GridSearch#

Critical Decisions#

Evaluation Metrics: Beyond MAPE#

Step 05 Output#

What This Strategy Achieves#

11. Testing: Fixtures, Mocking and Real Coverage#

Why Testing ML Is Different#

conftest.py: Shared Fixtures#

Imputation Test: Data Contracts#

Complete Pipeline Test: Integration Test#

Real Coverage#

What This Achieves#

12. Production Patterns Nobody Tells You About#

The Real Problem of Serving#

12.1. The Transform Pattern: The Synthetic KMeans Trick#

12.2. Training/Serving Skew: The Silent Killer#

12.3. Data Drift: The Enemy This Project (Still) Doesn’t Monitor#

12.4. Model Monitoring: Beyond Accuracy#

1. Business Metrics (most important)#

2. Prediction Distribution#

3. Input Feature Distribution#

4. Error Patterns#

5. Latency Percentiles#

12.5. The Cascade Pattern: Fallback Resilience#

1. Circuit Breaker Pattern#

2. Retry with Exponential Backoff#

3. Timeout Configuration#

4. Health Check Endpoint#

12.6. Feature Store Anti-Pattern: When You DON’T Need One#

12.7. Production Readiness: An Honest Checklist#

What This Project Does VERY WELL#

What’s Missing (And When To Add It)#

Prioritized Recommendations#

12.8. The Difference Between “Works” and “Works in Production”#

14. Conclusions: MLOps As an Engineering Discipline#

What This Pipeline Implements (And Why It Matters)#

The Anti-Patterns It Avoids (That Kill Projects)#

Conscious Trade-Offs (Because There Are No Perfect Solutions)#

What’s Missing (And When To Add It)#

The Uncomfortable Truth About MLOps#

The Most Important Lesson#

Who This Post Is For#

The Next Step#

The Last Word#

References and Resources#

Navigation#

📬 Did this help?

Anatomy of an MLOps Pipeline - Part 3: Production and Best Practices

11. Model and Parameter Selection Strategies

The Complete Flow: Selection → Sweep → Registration

Step 05: Model Selection - Algorithm Comparison

The 5 Candidate Models

Parameter Grids: Initial GridSearch

Grid Design Decisions

Training Function with GridSearch

Critical Decisions

Evaluation Metrics: Beyond MAPE

Step 05 Output

What This Strategy Achieves

11. Testing: Fixtures, Mocking and Real Coverage

Why Testing ML Is Different

conftest.py: Shared Fixtures

Imputation Test: Data Contracts

Complete Pipeline Test: Integration Test

Real Coverage

What This Achieves

12. Production Patterns Nobody Tells You About

The Real Problem of Serving

12.1. The Transform Pattern: The Synthetic KMeans Trick

12.2. Training/Serving Skew: The Silent Killer

12.3. Data Drift: The Enemy This Project (Still) Doesn’t Monitor

12.4. Model Monitoring: Beyond Accuracy

1. Business Metrics (most important)

2. Prediction Distribution

3. Input Feature Distribution

4. Error Patterns

5. Latency Percentiles

12.5. The Cascade Pattern: Fallback Resilience

1. Circuit Breaker Pattern

2. Retry with Exponential Backoff

3. Timeout Configuration

4. Health Check Endpoint

12.6. Feature Store Anti-Pattern: When You DON’T Need One

12.7. Production Readiness: An Honest Checklist

What This Project Does VERY WELL

What’s Missing (And When To Add It)

Prioritized Recommendations

12.8. The Difference Between “Works” and “Works in Production”

14. Conclusions: MLOps As an Engineering Discipline

What This Pipeline Implements (And Why It Matters)

The Anti-Patterns It Avoids (That Kill Projects)

Conscious Trade-Offs (Because There Are No Perfect Solutions)

What’s Missing (And When To Add It)

The Uncomfortable Truth About MLOps

The Most Important Lesson

Who This Post Is For

The Next Step

The Last Word

References and Resources

Navigation