Data Engineering
Jul 17, 20268 min read
XGBoost vs LightGBM: Benchmarking Tabular Data Processing Speed & Accuracy
Histogram-based decision trees vs depth-wise growth: why LightGBM trains 7x faster on 10M row datasets with equal AUC-ROC scores.
Data Engineering
Field notes on Apache Iceberg vs Delta, Debezium CDC, Snowflake clustering, Parquet dictionary encoding, Flink, and dbt.
Data Engineering
Jul 16, 20268 min read
MLOps Model Registry Patterns: Tracking Artifacts with MLflow & S3
Automating model lineage, hyperparameter logging, and staging-to-production promotion gates in CI/CD pipelines.
Data Engineering
Jul 15, 20268 min read
Data Drift Detection in Production ML: Population Stability Index (PSI) & Monitoring
Catching covariate shift and concept drift before model accuracy degrades using PSI metrics and Evidently AI dashboards.
Data Engineering
Jul 14, 202610 min read
Building CDC Ingestion with Apache Debezium, Kafka, and StarRocks
Streaming PostgreSQL WAL changes into StarRocks real-time analytics engine with sub-second end-to-end latency.
Data Engineering
Jul 13, 202610 min read
Apache Iceberg vs Delta Lake: Comparing Table Formats for Cloud Data Lakes
Comparing hidden partitioning, ACID transactions, time-travel queries, and multi-engine interoperability across Snowflake and Spark.
Data Engineering
Jul 12, 20268 min read
Parquet File Format Deep Dive: Row Groups, Dictionary Encoding, and Snappy Compression
Why Parquet files achieve 80%+ compression ratios over CSV: inspecting columnar binary structure, footer metadata, and dictionary encoding.
Data Engineering
Jul 11, 20269 min read
Real-Time Stream Processing with Apache Flink & Kafka: Event Time vs Processing Time
Handling out-of-order event streams, tumbling vs sliding windows, and watermarking strategies for sub-second streaming analytics.