Treat every external API as a dependency that will become slow, unavailable or incompatible. Isolate it behind an adapter, validate contracts, set explicit timeouts, retry only safe transient failures, make operations idempotent, queue long work and provide observable recovery.
This is a SaaS development, AI automation and web engineering problem. A successful happy-path demo does not prove the workflow can survive real dependency behaviour.
AWS recommends explicit connection and request timeouts for remote calls and describes backoff, retries and circuit breakers as patterns for graceful failure Source: AWS Well-Architected. Configure them from the workflow's needs, not library defaults.
Map the dependency boundary
List every call, webhook, file transfer, token exchange and callback involving the external service. Record owner, endpoint, purpose, data, rate limit, timeout, retry, version and failure consequence.
Trace what the workflow does before and after the call. Identify customer-visible and irreversible steps.
Create a dependency register with provider, endpoint, owner, authentication, timeout, response contract, retry policy and recovery route. Our AI CRM integration guide provides an integration example, rather than a ready-made register.
Put the provider behind an adapter
Create an internal interface that expresses the business operation rather than spreading provider-specific fields throughout the application. Translate requests, responses and errors in one controlled layer.
This reduces the blast radius of a field rename or provider replacement. It also gives tests a stable seam.
Do not leak raw provider error messages or objects into customer interfaces.
Define a versioned contract
Specify required and optional fields, types, allowed values, limits, status semantics and error shapes. Validate incoming and outgoing payloads.
Reject or quarantine malformed responses rather than allowing undefined values to fail later. Tolerate additive optional fields where safe.
Track provider version and schema changes through release notes and automated contract tests.
For customer-system integrations, our AI CRM integration glossary explains the connected workflow. Document the precise API request and response contract separately; a general integration description is not a compatibility guarantee.
Set explicit timeouts
Use separate connection and request timeouts based on user tolerance, queue deadlines and downstream resource cost. A 30-second default may be too long for an interactive page and too short for a batch job.
Cancel or abandon work safely when the deadline expires. Propagate an overall deadline so nested calls do not each consume the full budget.
AWS notes that timeouts free client resources and allow a decision to back off, retry or open a circuit Source: AWS Well-Architected.
Classify failures
Distinguish authentication, permission, validation, not found, conflict, rate limit, transient server error, timeout, malformed response and permanent business rejection.
Map each class to retry, user correction, operator review, fallback or stop. Do not retry a rejected invalid request endlessly.
Preserve provider request IDs and a safe internal correlation ID for support.
Retry only safe operations
Retry transient failures with bounded exponential backoff and jitter. Respect Retry-After headers and provider limits. Cap total attempts and elapsed time.
Before retrying a create, payment, message or booking, ensure the operation is idempotent. Network failure may hide a successful provider action.
Avoid coordinated retry storms when the dependency recovers.
Make operations idempotent
Generate a stable idempotency key for one logical operation and reuse it across retries. Store the outcome so repeated delivery does not create duplicate charges, records or messages.
AWS describes idempotency tokens as a way for repeated identical requests to have the same effect as one request Source: AWS Well-Architected.
Define key lifetime, collision handling and behaviour when payloads differ under the same key.
Decouple with queues
Move non-interactive and long-running work into a durable queue. Acknowledge the customer action after the business has safely accepted responsibility, not merely after an in-memory task starts.
Set visibility timeouts, retry policy, ordering needs, concurrency and retention. Use a dead-letter queue for work that exceeds attempts.
Before launch, document how to identify failed work, inspect its last confirmed step, prevent duplicate effects, replay safely and verify the final result. Our sales and marketing AI workflows guide offers workflow context, while this recovery record should remain specific to your system.
Use circuit breakers and bulkheads
Open a circuit after a defined failure pattern to stop hammering a broken provider. Probe recovery cautiously and close after stable success.
Separate worker pools, queues or resource limits so one failing integration does not consume every thread or connection. Prioritise critical workflows.
Expose circuit state to operators and customer-facing status where appropriate.
Design a truthful fallback
Decide whether the workflow can use cached data, defer completion, switch provider or require manual processing. Mark stale data clearly and limit its use.
Do not return “completed” when the external action remains uncertain. Use pending, delayed or needs-review states with a reference and next step.
Provide operators with replay, cancel and reconcile controls protected by permissions.
Handle webhooks defensively
Verify signatures, timestamps and source where supported. Acknowledge quickly, enqueue processing and make handlers idempotent.
Expect duplicates, out-of-order delivery, delayed events and missing callbacks. Reconcile periodically against the provider when the business consequence requires it.
Store the minimal raw evidence needed for investigation with access and retention controls.
Manage rate limits
Track per-account and global quotas. Throttle before the provider rejects every request, distribute work fairly and use caching or batching where semantics allow.
Communicate capacity constraints to product and sales. A workflow designed for ten daily calls may fail at a customer launch producing ten thousand.
Monitor remaining quota and reset times.
Recalculate limits when customer volume or provider plans change.
Make failure observable
Measure latency percentiles, success by operation, timeout, retry, circuit state, queue depth, oldest job, dead letters, schema errors and reconciliation differences.
Alert on customer and business impact rather than every isolated provider error. Include correlation IDs without leaking secrets or personal data.
Create a dependency dashboard and incident runbook.
Test failure deliberately
Simulate slow connections, timeouts, 429s, 5xx errors, invalid JSON, missing fields, new fields, changed enums, duplicate callbacks, out-of-order events and partial success.
Test provider sandbox differences and production-like limits. Verify customer messages, retries, dead letters, manual recovery and audit history.
Run contract tests on a schedule as well as during releases.
My take: integration code is reliability code
My take is that an API client is not complete when it parses a successful response. It is complete when the business can explain and recover every ambiguous outcome without duplicating harm.
Design the failure state as a product state, not a hidden exception.
Prepare for provider change
Monitor deprecation notices, status feeds, certificates, SDKs and terms. Assign a vendor owner and upgrade deadlines.
Keep export and reconciliation capability. Avoid provider-specific assumptions in core records where an internal representation can remain stable.
Review exit and migration risk before the dependency becomes critical.
Frequently asked questions
How many times should an API call retry?
There is no universal count. Base attempts on failure type, deadline, provider guidance, idempotency and the cost of retrying.
What is a circuit breaker?
It temporarily stops calls after a failure threshold, allowing the dependency and your system to recover before cautious probes resume.
Should every integration use a queue?
No. Use queues for durable asynchronous work and load smoothing. Interactive reads may need a direct call with timeout and fallback.
How do we handle an unknown outcome?
Mark it pending or needs review, reconcile with the provider using a stable operation ID, and never blindly repeat a consequential action.
Make external failure a recoverable state
If your business needs resilient integration architecture, get in touch for SaaS development and automation.