How to Automate API Reports in Assurance

Automation is generally a very good thing! A number of Assurance users are now using automated data analysis tools such as Microsoft PowerBI to pull down complex report data from the Assurance API.

In this article, you will learn about:

• Timeout Errors and How They Happen
• The Solution to Timeout Errors
• Using the Assurance API to Integrate with PowerBI
• Generating Large API Reports Using the Snapshot Endpoint
• Automating API Reports Using the Snapshot Scheduler

Timeout Errors and How They Happen

A potential issue with this approach is network timeouts. When an Assurance report is generated and downloaded via the API, it appears to the client as a single operation. Actually it's multiple operations behind the scenes, but for practical purposes it boils down to three:

1. Retrieving the report data from the database (across multiple tables and data models);
2. Processing the data so it is merged together into one contiguous block of data;
3. Returning the data to the client.

Each of these three operations takes time, and they all must complete before your PowerBI instance closes the connection and successfully imports data.

Unless PowerBI is set up with increased timeout values, if any of these steps takes longer than a certain internal limit, you'll likely see a 499 error. Immediately retrying is unlikely to fix the issue.

What's likely to cause this to happen? One (or both) of two things:

• A complex report (for example, one with lots of dependent, nested or calculated fields)
• A large report (one that has lots of responses to it, and/or lots of fields generally)

A complex report and a large report are not the same thing. A complex report will take a long time to retrieve and process the data, and a large one will take a long time to return it over the API. Two different possible issues, but with the same result - a timeout error.

Over time the issue tends to get worse, as reports accumulate more and more data. Adding some date filtering to your report can help (and we encourage this too), but we do acknowledge that there are some reports for which that's not appropriate or desirable.

The Solution to Timeout Errors

To enable customers to retrieve their data much faster and more reliably, we have recently implemented the snapshot endpoint. It's faster because the API endpoint does not have to process and return the data in the same request. This two-step pattern in APIs is increasingly common, as customers want to return increasingly large amounts of data.

We've also released a new feature called the snapshot scheduler, which is an automatic remote agent that processes your report data in the background, refreshing it on a preset schedule.

This makes it easier for customers doing automation with PowerBI (and other similar Business Intelligence tools), because they do not have to perform the initial request to refresh the snapshot. Instead, the remote agent does it for them. The returned data also contains a timestamp that shows exactly when the snapshot was last refreshed.

Using the Assurance API to Integrate with PowerBI

For instructions on how to use the Assurance API to integrate with PowerBI, please refer to our Support Hub article: How to Use the Assurance API to Integrate with PowerBI – Support Hub (zendesk.com)

Generating Large API Reports Using the Snapshot Endpoint

For instructions on how to generate large API reports using the snapshot endpoint, please refer to our Support Hub article: How to Generate Large API Reports in Assurance – Support Hub (zendesk.com)

Automating API Reports Using the Snapshot Scheduler

Once you have set up your API Report, you can decide which reports will be automated via the API and select the refresh schedule from a range of available options.

Please Note: This will only work on reports that have been selected as Available in API (which is switched off by default).

You can choose to refresh reports on the following schedule:

• Every 3 hours (starting from midnight, at the top of the hour)
• Every 6 hours (starting from midnight, at the top of the hour)
• Every 12 hours (starting from midnight, at the top of the hour)
• Every day (at midnight)
• Every week (at midnight on Monday)
• Every month (at midnight on the first day of the month)

The scheduler will keep on refreshing the report, until or unless the Snapshot schedule is switched back to None.

Important notes:

• A refresh on the cache can be triggered anytime you want, by using the API endpoint.
• That said, the cache will only be refreshed if:
  • It's manually triggered; OR
  • It's set to refresh on a schedule.
• Existing endpoints are not being retired - they will continue to work.
• If you do not require completely fresh data every time, it is best to use the cached data, as it is much faster to retrieve and less likely to cause timeout issues with the server.

• The v1 endpoint for Assurance reports is deprecated as of 1 December 2023. While it will continue to work through the new year, we strongly encourage anyone using the Assurance API reports to adopt the v2 version of the endpoint. (Just substitute 'v2' for 'v1' in the URL - everything else is identical.)