Skip to content
English
Contact us
  • There are no suggestions because the search field is empty.

Data collection optimization

The goal is to collect the smallest dataset that still supports the transformation, using selection criteria that are recomputed on every run rather than carried forward between runs.

Restrict the fields

  • Collect only the fields your queries and data targets actually reference
  • Fields collected "just in case" persist in the schema and inflate every subsequent run

Restrict the population

  • Apply collector filters to limit records by entity, location, status, or dimension
  • Filtering at the collector keeps the data out of DataBlend entirely, rather than discarding it later in the query

Restrict the time window

  • For transactional or activity data, combine a filter with a date parameter against Created Date or Modified Date
  • Example: start of last month through today, rather than the full history of the object

Prefer Relative Date parameters for anything scheduled

Relative Date parameters collect within a relative window such as start of the first quarter through end of the last quarter, rather than fixed Month/Day/Year values. Two reasons this matters:

  • The window shifts automatically with each run, so nobody has to update it manually
  • Because the window is recalculated from the run date each time rather than advanced by the previous run, a failed run doesn't create a permanent gap

Build deliberate overlap into the window, a rolling 30 days for a daily workflow, not a rolling 24 hours, so the next successful run recovers anything a failed run missed. Size the overlap to your tolerance for a missed cycle, not to the minimum that works when everything succeeds.

Reduce how often collectors run at all

Where a workflow only needs to act when something has changed, structure it as a gate:

  1. Collect the minimum data needed to detect change
  2. Run a query to evaluate it
  3. Use a Boolean parameter to enable or disable the remaining steps based on the result

For example, an invoice workflow collects recent invoice data, checks whether any records are new, then switches the downstream collection and target steps on or off accordingly.

To validate that any of this worked, each workflow execution reports the total amount of data scanned, compare before and after.