DESK A
Pipeline Build Bays
Pipeline Build Bays are where an extraction pipeline is physically assembled, wired and tested. A bay holds a source connector, a normalisation stage, a validation gate and one or more sink writers, and none of them are allowed to reach production until the whole chain has run end to end against a captured sample. We treat the bay as a workshop bench rather than a shell script: the connectors are documented, the normalisers are versioned, and the contract between stages is written down before any code is merged.
Every build bay ends with a runnable artefact that the client can execute, inspect and restart without our engineers present. We include a runbook listing the environment variables, credentials scope, retry behaviour and expected throughput, because a pipeline that only runs on the machine of the person who wrote it is not a pipeline. Where an existing system already extracts data, we usually rebuild in parallel and cut over once the new bay matches the old output on a full historical replay.
Build bays are also where we decide how much of the parse belongs in code and how much belongs in configuration. Column renames, unit conversions and null policies are configuration; joining, deduplication and enrichment logic are code. Keeping that boundary explicit means the client can change a mapping without a release, and can change behaviour without touching a spreadsheet.
DESK X
Schema Map Runs
Schema Map Runs take the payloads captured during a build and reconcile every field against the target model. The output is a mapping ledger that records source path, target column, type, nullability, transformation and the name of the reviewer who approved it. That ledger is the single source of truth for what a number in the warehouse actually means, and it is versioned alongside the pipeline code.
We run maps in two modes. A discovery run is exploratory and surface every field in a sample plus fields that appear intermittently, including nested structures, repeated arrays and loosely typed numbers arriving as strings. A reconciliation run is strict and fails when a mapped field is missing, changes type or exceeds an agreed value range. Clients use discovery runs when onboarding a new supplier and reconciliation runs on every scheduled refresh.
Because the ledger is replayable, a question about last year does not require last year employees. We can re-apply the mapping that was active at the time and reproduce the transform that produced a given row, which is often the difference between closing an audit and losing a week to guesswork.
DESK P
QA Harness Desks
QA Harness Desks hold the regression fixtures for every pipeline we ship. A harness desk contains a frozen sample of each feed shape, the expected output after mapping, and a set of negative cases that must be rejected. When a connector version changes or a normaliser is edited, the harness proves that the transformation still matches the signed map or tells us precisely where it drifted.
We write harnesses as plain test suites in the language of the surrounding stack, not as proprietary assertions inside a closed tool. This matters at handover: the client team can read the cases, add a case when a supplier changes a rule, and run the suite in their own continuous integration. The harness becomes part of the client estate rather than a service we must be paid to operate.
Each desk also carries a small holdout set that the build team does not see during development. Holdout cases catch the subtle overfitting that happens when a parser is tuned to one sample file and quietly fails on the next delivery. For pipelines touching money, identity or regulated reporting, the holdout set is reviewed with the client compliance contact before signoff.
DESK Y
Error Log Windows
Error Log Windows give operators an ordered view of what failed, where it failed, what the pipeline did next, and which record was affected. We design the log schema before we design any dashboard. Each entry carries a stage identifier, a record key, a severity and a short remedy hint, and entries are grouped by run so a single morning incident reads as one narrative instead of a scattered stream.
The window is written for the person on call at an awkward hour. It shows the last successful checkpoint first, then the first failure, then the count of records held back by the failure. It avoids noisy stack dumps in the primary view and keeps the full trace one click away. Paging rules are agreed with the client operations team so that a retryable network blip does not wake anyone, while a mapping breach does.
Over time the window becomes an operational record. Recurring entries reveal which supplier feeds are structurally unstable, which transformations are under specified, and which retry policies are too optimistic. We review that history at each Source Refresh Review so the log feeds directly back into the map.
DESK Q
Source Refresh Reviews
Source Refresh Reviews examine upstream feeds on an agreed cadence, comparing fresh sample payloads against the map that was signed. Suppliers change column order, rename fields, add optional blocks and switch encodings without always telling the receiving team. A refresh review catches those changes in a controlled comparison rather than in a failed nightly load.
Each review produces a short note: what changed, which mapped fields are affected, which harness cases now fail, and whether the change is ignored, absorbed by a configuration update, or requires a new pipeline release. The note names an owner on each side so the decision does not sit in an inbox. Where a supplier has a published contract, we compare the live payload against both the contract and our map, because drift between those two is the usual source of silent breakage.
The cadence itself is a design choice. High value feeds are reviewed weekly, stable archives monthly, and dormant feeds on demand before any replay. The schedule is written into the release ledger so nobody has to remember it.
DESK S
Delivery Signoff Sittings
Delivery Signoff Sittings close the trace. Every artefact is reviewed in one sitting: the pipeline contract, the mapping ledger at its current version, the harness results including holdout cases, the error log window, and the most recent source refresh note. The meeting is short because the evidence already exists, and it is honest because nothing is declared live on a verbal confirmation.
We record what was tested, by whom, on which data snapshot, and which known limitations were accepted. If a limitation is accepted, it enters the release ledger with a review date. This is the part clients value most after the first year, when a report is challenged and the question becomes what exactly did the system guarantee on the day the figure was produced.
The sitting also sets the next engagement frame. Outstanding work is either scheduled into a build bay or explicitly dropped, and the refresh cadence is re-confirmed. A signoff that leaves ambiguity is treated as incomplete, and we say so in the minutes rather than pass it through.