Data Processing

Overview

The data processing views provide visibility into updates made to Luminate's various datasets. Use these views to kick off downstream queries once new data arrives, and to potentially skip them if nothing has changed.

These are read-only views in the EXTRACT_S schema of your shared database. Substitute your own database name wherever this page writes <LISTING_DB>.

Processing Coverage

Views

VW_DATA_AVAILABILITY_DS

SELECT * FROM <LISTING_DB>.EXTRACT_S.VW_DATA_AVAILABILITY_DS
WHERE country_code = 'US';

VW_MARKET_SHARE_AVAILABILITY_DS

VW_CHART_AVAILABILITY_DS

Datasets

Integration

Behavior

VW_DATA_AVAILABILITY_DS and VW_MARKET_SHARE_AVAILABILITY_DS hold current state: the latest data processed for a dataset and country, and when that processing happened.

SELECT dataset, country_code, latest_week_id, latest_report_date, modified_at
FROM <LISTING_DB>.EXTRACT_S.VW_DATA_AVAILABILITY_DS
WHERE dataset = 'DASHBOARD'
AND country_code = 'US';

MODIFIED_AT advances whenever processing completes. Compare it against the value from your last run: if it has advanced, there is new data to pull. LATEST_WEEK_ID and LATEST_REPORT_DATE tell you the newest week and report date now covered.

VW_CHART_AVAILABILITY_DS is more granular and holds full history — one row per chart and week. Look up the chart and week you need:

SELECT chart_id, week_id
FROM <LISTING_DB>.EXTRACT_S.VW_CHART_AVAILABILITY_DS
WHERE chart_id = 'CR01184'
AND week_id  = 202633;

Scheduling

Run the check on an interval rather than at a fixed time of day, so your load starts once processing has completed, including when it completes later than usual. The check can be enabled via several mechanisms, including Snowflake tasks.

The DASHBOARD, ANALYSIS, and MARKET_SHARE datasets are updated once per day, while the CHART dataset can be updated several times in a day, so VW_CHART_AVAILABILITY_DS is worth checking on a shorter interval.

Guidelines

Compare MODIFIED_AT to detect new updates. It advances every time processing completes, including when data for an existing report date is reprocessed. LATEST_WEEK_ID and LATEST_REPORT_DATE describe what is now covered, and they do not change when an existing date is reprocessed, so a check based on them alone will miss those rounds.

Do not create streams on these views. They are secure views without change tracking, so streams are not supported. Compare MODIFIED_AT as described above.

Timestamps are UTC. LATEST_WEEK_ID and LATEST_REPORT_DATE are calendar values in Luminate reporting terms and are not timezone-shifted.

Support

Questions about these views, or which of them you have access to, go to [email protected].