Practice 02
Data engineering and data science
A data platform is judged on one thing: whether the number in the board pack matches the number in the operational system. Everything else is plumbing in service of that.
At a glance
We build ingestion, storage, modelling and serving layers with lineage and tests attached — then the models that make them earn their keep.
Tools we use most
- Airbyte
- Debezium
- Kafka
- Airflow
- Dagster
- dbt
- Apache Iceberg
- Trino
- Snowflake
- BigQuery
- Databricks
- DuckDB
- Great Expectations
- Feast
- MLflow
Related work
- One warehouse for 40+ lender schemas
- Demand forecasting for a 240-store chain
- Unified billing, loyalty and recommendations for a retail chain
- A renewal calendar built from contract extraction
What we get called in for
- Forty source systems, forty different definitions of an active customer.
- The warehouse bill tripled and nobody can point at which query did it.
- Reports break every time a partner changes a column, and you find out from the CFO.
- A data science team is spending 70% of its time on extraction instead of models.
- You need a model in production, not a notebook that works on one laptop.
- Pricing, loyalty or campaign rules live in someone's head or a spreadsheet, and the business team cannot self-serve a change without breaking something.
How we build it
Map sources before choosing tools
We inventory every feed: format, cadence, volume, ownership, and how often it changes shape. Tool selection is the last decision, not the first.
Land raw, model deliberately
Raw immutable landing zone, then staged and conformed layers with tests at each boundary. When a partner ships a bad file, you can reprocess instead of reconstruct.
Contract-test the schemas
Every source gets an expected schema and a drift check. New column: warn. Removed column: fail loudly, before it reaches a dashboard.
One semantic layer, one definition
Metrics defined once, versioned in code, consumed by BI, the API and the ML feature store. This is what makes numbers agree.
Ship models like software
Feature pipelines that run the same in training and serving, model registry, shadow deployment, and monitoring on both drift and business outcome.
What you get
- Source inventory with volumes, cadence, owners and change history
- Ingestion pipelines with schema contracts and quarantine handling
- Lakehouse or warehouse modelling with tests, lineage and documentation
- Semantic layer with versioned metric definitions
- Cost model per pipeline, plus the three queries doing most of the damage
- Production ML: feature store, registry, deployment and drift monitoring
- Self-serve configuration for business rules — pricing, loyalty, campaigns — with staged approval, so non-engineers can change parameters without an engineer in the loop
Further reading
Next step
Tell us what you're trying to ship.
Send the brief, the RFP, or three messy sentences about the problem. You get a written point of view from an architect within two working days — not a sales deck.