Paginate AWS Systems Manager REST API in SSIS

SSIS PowerPack
, ,
1

When working with AWS Systems Manager (SSM) APIs, you may encounter paginated responses where only a subset of data is returned in each API call. To fetch all the data, you need to implement pagination in your API calls. This guide will show you how to configure ZappySys JSON Source to handle pagination for the DescribeInstanceInformation operation of AWS SSM.

Step 1: Configure the JSON Source

  1. Access Mode: Set the Access Mode to File path or web URL .
  2. Enter Path or Web URL: Enter the API endpoint URL. For example: https://enter-url .
  3. HTTP Request Method: Set the method to POST .
  4. Body (Request Data): Provide the JSON body for the request. Use placeholders for pagination tokens, such as:
{
   "MaxResults": 50,
   "NextToken": "{$pagetoken$}"
}
  1. Body Content Type: Select JSON (application/json) .

Step 2: Enable Pagination

  1. Go to the Pagination tab.
  2. Paging Mode: Select Response Attribute Mode - Read next page information from response .
  3. Next Link/Cursor Expression: Enter the JSONPath expression to extract the pagination token from the API response. For example:
$.NextToken
  1. Stop Indicator Attribute: Leave this blank unless your API provides a specific attribute or value to signal the end of the data.

Step 3: Configure Advanced Pagination Settings

  1. Go to the Advanced Pagination tab.
  2. Enable Page Token for Body: Check the box to enable page token usage in the request body.
  3. First Page Body: Define the JSON body for the first request. For example:
{
   "MaxResults": 50
}
  1. Next Page Body: Define the JSON body for subsequent requests, using the `$pagetoken placeholder. For example:
{
   "MaxResults": 50,
   "NextToken": "{$pagetoken$}"
}

Step 4: Test and Validate

  1. After configuring the settings, click Preview Data to test your setup.
  2. Verify that the API fetches all pages of data by automatically appending the NextToken in subsequent requests.

Additional Tips

  • If the API does not return a NextToken in the response, ZappySys will stop fetching additional pages.
  • Use the Read Console Output link in the JSON Source to debug and view detailed logs of the API requests and responses.
  • Adjust the MaxResults parameter based on your API’s limits.

By following these steps, you can efficiently handle paginated responses from AWS SSM using ZappySys JSON Source.