Reducing the number of collectors while maintaining data integrity

Where sprawl comes from

Workflow-scoped design. Each new integration gets its own collectors, so the same customer, item, or GL object is collected three or four times on different schedules. Two consequences:

• Duplicated collection volume
• The same object holding different values depending on which workflow ran last

The consolidation pattern

Treat collectors as shared, reusable assets rather than workflow components:

• Identify objects collected by more than one workflow
• Build a single collector per source object, writing to a shared schema
• Where a workflow needs a narrower slice, apply that narrowing in the query, not by building a second collector

Then make sure every consumer reads the same collection

Sharing a collector is only half the work. Unless collection is sequenced ahead of consumption, workflows can still read different states of the same object. Two ways to guarantee it, in order of preference:

• Combine workflows that share collectors into a single workflow. Place the Run Collector step at the head, then all dependent query and target steps after it. Sequential execution means every step downstream reads data collected in that same run. This is the cleanest option and the only one that delivers both wins at once — one collector definition and one collection per run.
• Chain the workflows where they must stay separate: different cadences, different owners, or a need for independent failure isolation. Run the shared collector once at the head of the chain and have downstream workflows consume that collection rather than re-running it.

Which to choose

Supporting controls

• Filters on the shared collector must be the union of what every consumer needs, not the intersection. A collector previously filtered to one entity now serving a workflow that needs two will cause that workflow to silently under-report. Widen the collector, narrow in the query
• Be deliberate about Schema Update Type, since multiple queries now depend on that schema staying stable
• Add a Run Credential Test at the start of each workflow. This matters more once merged, since credential failure now stops several integrations rather than one.

Validate before retiring anything

• Run the consolidated collector alongside the existing ones for at least one full cycle
• Compare record counts and control totals
• Add a Data Quality Report against the consolidated collector so drift surfaces on its own
• Retire duplicates only once that comparison is clean

State plainly to the client

Consolidation increases coupling. A merged workflow that fails partway leaves every downstream step unrun, and earlier steps are not rolled back. This is a deliberate trade: a visible, total failure is easier to detect and recover than intermittent disagreement between integrations reading different data.