DEV Community

Cover image for #AWS - Increase your Quicksight SPICE data refresh frequency.
Gururajan Padmanaban
Gururajan Padmanaban

Posted on

#AWS - Increase your Quicksight SPICE data refresh frequency.

Scenario:

Let us say I want to fetch the data from the source (Jira) and push it to SPICE and render it in Quicksight Dashboards.

Requirement:
Push the data every 30 Mins once.

*Quicksight supports the following: *

  • Full refresh
  • Incremental refresh

Full refresh:

  • Process - Old data is replaced with new data.
  • Frequency - Every 1 Hr once
  • Refresh count - 24 / Day

Incremental refresh:

  • Process - New data get appended to the dataset.
  • Frequency - Every 15 Min once
  • Refresh count - 96 / Day

Issue:

  • We need to push the data every 30 Min once.
  • It is going to be a FULL_REFRESH
  • When it comes to Full Refresh Quicksight only supports Hourly refresh.

Solution:

We can leverage API support from AWS.

  • Package - Python Boto 3
  • Class - Quicksight.client
  • Method - create_ingestion
  • Process - You can manually refresh datasets by starting new SPICE ingestion.
  • Refresh cycle: Each 24-hour period is measured starting 24 hours before the current date and time.

Limitations:

  • Enterprise edition accounts 32 times in a 24-hour period.
  • Standard edition accounts 8 times in a 24-hour period.

Sample code:

Python - Boto for AWS:

import boto3
client = boto3.client('quicksight')

response = client.create_ingestion(
    DataSetId='string',
    IngestionId='string',
    AwsAccountId='string',
    IngestionType='INCREMENTAL_REFRESH'|'FULL_REFRESH'
)
Enter fullscreen mode Exit fullscreen mode

awswrangler:

import awswrangler as wr
wr.quicksight.cancel_ingestion(ingestion_id="jira_data_sample_refresh", dataset_name="jira_db")

Enter fullscreen mode Exit fullscreen mode

CLI:

aws quicksight create-ingestion --data-set-id dataSetId --ingestion-id jira_data_sample_ingestion --aws-account-id AwsAccountId --region us-east-1

Enter fullscreen mode Exit fullscreen mode

API:

PUT /accounts/AwsAccountId/data-sets/DataSetId/ingestions/IngestionId HTTP/1.1
Content-type: application/json

{
   "IngestionType": "string"
}
Enter fullscreen mode Exit fullscreen mode

Conclusion:

Using this approach we can achieve 56 Full Refreshes for our dataset also we can go one step further and get the peak hours of our source tool (Jira) and configure the data refresh accordingly. This way we can even achieve a refresh frequency of 10 Min once.

Ref:

Top comments (0)