When utilizing the JSON Source or XML Source with API Pagination configuration, you may encounter the following error:
Error: System.Net.WebException: The remote server returned an error: (429) Too Many Requests
Cause:
The error message “(429) Too Many Requests.” indicates that the server you’re making API requests to has rate-limiting policies in place, and you’ve exceeded the allowed number of requests within a specific time frame.
To handle this situation, you should implement strategies to respect the rate limits set by the API service. Here are some general guidelines:
- Rate Limiting:
Check the API documentation of the service you are trying to connect to. Understand their rate-limiting policies and make sure you are not exceeding the allowed limits. - Backoff and Retry:
Implement a backoff and retry mechanism in your code. When you receive a 429 error, wait for some time before making the next request. Gradually increase the wait time if consecutive requests continue to receive 429 errors. - Check API Documentation:
- Refer to the documentation of the API you’re using to understand the specific rate-limiting policies and guidelines.
- Some APIs may include information about the headers in the response that provide details on the rate limits.
To address this issue:
Here are several options to address and resolve this error (outlined below):
-
Throttling (API Rate Limit)
If pagination is present in the JSON/XML Source, we can utilize the Throttling option.
On the Throttling tab, we can define the wait time for the next request in milliseconds (ms). Depending on the specifics of the API service, you may need to adjust the frequency of your requests.
e.g. Let’s say your API has a rate limit of 100 calls per minute, we should configure the throttling value to be 600. The calculation (60000 milliseconds / 100 calls) results in a delay of 600 milliseconds between each API call, effectively ensuring compliance with the rate limit.
-
Error Retry Settings in HTTP/OAuth Connection.
Many API restricts you from calling too many API calls too soon (just to save their servers from being overwhelmed) - Example. Let’s think this way if you fetching a million rows via paginated response and the very last page fails then you won’t be happy because now you have to fetch everything again. To solve this ZappySys provides retry options on HTTP / OAuth and REST API connection managers.Generally, when you exceed the rate limit, the server sends you error code 429 but it can be something else too, or multiple codes are returned. In the below example, we use 503. Check your API documentation to know the exact code you should retry.
To set the retry options go to Connection > Error Retry Settings (e.g. HTTP / OAuth / REST connection).
To retry on multiple codes use Pipe as a delimiter. Example: 429|503
Effectively managing API rate limits is a crucial aspect of successful data integration. With ZappySys providing powerful tools for API connectivity, users can navigate rate-limiting challenges by configuring throttling values in accordance with the specific requirements of the APIs they interact with.