How to Build an ERP Integration Layer Without Losing Your Mind

Every ERP integration project starts the same way: "We just need to sync orders from Shopify to SAP." Two months later, you're handling partial shipments, backorders, multi-currency conversion, tax calculation edge cases, and the integration engineer who built it is the only person who understands how it works.

After building ERP integration layers for manufacturers, distributors, and e-commerce companies running SAP, Oracle, Dynamics 365, NetSuite, and Epicor, here's the architecture that keeps integrations maintainable at scale — and the shortcuts that create technical debt you'll pay for every month.

Why ERP integration is uniquely difficult

The data model problem

ERPs have the most complex data models in enterprise software. A single sales order in SAP touches 30+ tables (VBAK, VBAP, VBPA, VBKD, KONV, LIKP, LIPS, VBRK, VBRP — and that's just the standard ones). The relationships between these tables encode decades of business logic: pricing conditions, partner functions, delivery splits, billing blocks, and credit management rules.

External systems (Shopify, Salesforce, custom apps) have simple data models by comparison. A Shopify order has a customer, line items, and a shipping address. Mapping this simple structure to SAP's complex structure is where 80% of integration effort goes.

The real-time vs. batch tension

ERPs were built for batch processing. MRP runs nightly. Financial postings batch hourly. Inventory snapshots update every 15 minutes. External systems expect real-time: an e-commerce order should appear in the ERP immediately, inventory levels should update on the website in seconds, shipping confirmations should trigger customer notifications instantly.

Bridging this fundamental architectural gap — batch ERP meets real-time world — is the core challenge of modern ERP integration.

The customization tax

No two ERP installations are the same. Every company customizes their ERP — custom fields, custom tables, custom workflows, custom validation rules. An integration built for "standard SAP" will break the first time it encounters a custom mandatory field, a custom pricing procedure, or a custom approval workflow.

This means ERP integrations can't be truly generic. Even the best iPaaS connectors handle 70-80% of the integration; the remaining 20-30% requires custom configuration for your specific ERP setup.

The architecture that works: event-driven integration with a canonical data model

After building dozens of ERP integration layers, we've converged on an architecture pattern that handles the complexity while remaining maintainable. It has four components:

Component 1: Canonical data model

Instead of translating directly between each source system and the ERP (Shopify format → SAP format, Salesforce format → SAP format), define a canonical data model that represents your business objects in a system-agnostic format.

Source systems → Canonical model → Target systems

Shopify order → Canonical order → SAP sales order
WooCommerce order → Canonical order → SAP sales order
Manual order entry → Canonical order → SAP sales order

The canonical model is your data format. Every integration translates to/from this format, never directly between source and target systems.

Why this matters: When you add a new source system (moving from Shopify to BigCommerce), you only build one translator (BigCommerce → canonical). All downstream integrations (canonical → SAP, canonical → warehouse, canonical → shipping) remain unchanged. Without a canonical model, adding a new source system requires rebuilding every downstream integration.

How to design a canonical model:

• Start with your ERP's data model as the base (it's the most complex, and you need to map to it anyway)
• Simplify — remove ERP-specific technical fields (SAP client, change pointers, internal IDs)
• Add fields from external systems that the ERP doesn't natively support (e-commerce session ID, marketing attribution data)
• Version the schema — your canonical model will evolve as you add integrations
• Use JSON Schema or Protocol Buffers for formal schema definition with validation

Component 2: Event bus

Instead of polling systems for changes (checking every 5 minutes if new orders arrived), use an event-driven architecture where systems publish events when data changes, and integrations subscribe to the events they care about.

Shopify → "order_created" event → Event bus → Order integration subscribes → Creates SAP order
SAP → "order_shipped" event → Event bus → Notification integration subscribes → Sends email

Why events beat polling:

• Latency: Events arrive in seconds. Polling at best arrives at the next poll interval (5-60 minutes).
• Efficiency: Events only fire when something changes. Polling checks every interval whether or not anything changed, wasting API calls and compute.
• Scalability: Events can be processed in parallel by multiple consumers. Polling is inherently sequential.

Event bus options:

• Apache Kafka: For high-volume (100K+ events/day), mission-critical integration. Offers persistence, replay, and exactly-once semantics. Operational overhead is significant.
• AWS EventBridge / Google Pub/Sub / Azure Event Grid: Managed event buses with lower operational overhead. Good for most mid-market use cases.
• Redis Streams: Lightweight, fast, works well for moderate volumes (10K-100K events/day). Good for teams already running Redis.

Component 3: Integration workers

Integration workers are the processes that actually do the work — consuming events, transforming data, calling APIs, and handling errors. Each worker handles one integration flow (e.g., "Shopify order → SAP sales order").

Worker design principles:

• Idempotent: Processing the same event twice must produce the same result (not a duplicate record). Use unique business keys (order number, not internal IDs) for deduplication.
• Retryable: Failed processing should be automatically retried with exponential backoff. After N retries, route to a dead-letter queue for manual investigation.
• Observable: Every worker logs its input event, processing steps, output, and any errors. Structured logging (JSON) with correlation IDs that trace an event through the entire processing pipeline.
• Independent: Workers should not depend on each other. If the shipping integration is down, the order integration should continue processing.

Component 4: Error handling and monitoring

This is where most ERP integrations fail. The happy path works fine — it's the error handling that determines whether the integration is production-grade.

Error categories:

• Transient errors: API timeouts, rate limits, network blips. Handle with automatic retry (3-5 attempts with exponential backoff).
• Data errors: Invalid data (missing required field, invalid format). Route to a dead-letter queue with the error details. Alert the integration team for manual fix-and-replay.
• Business logic errors: The ERP rejects the data (credit blocked, material not available, pricing error). These require business decision — route to a business user queue, not IT.
• Schema errors: The source or target system changed its API schema. These break the integration entirely — alert immediately, require code fix.

Monitoring must-haves:

• Processing lag: Time between event publication and successful processing. Alert if lag exceeds threshold (30 seconds for real-time, 15 minutes for near-real-time).
• Error rate: Percentage of events that fail processing. Alert above 2% for transient errors, above 0.1% for data errors.
• Dead-letter queue depth: Number of events waiting for manual intervention. Alert if growing (indicates a systematic problem, not individual errors).
• Volume anomalies: Sudden drops in event volume (indicates source system is broken) or spikes (indicates replay or data import).

The iPaaS decision: when to buy vs. build

Use an iPaaS (Workato, Boomi, MuleSoft, Celigo) when:

• Your integration patterns are standard (ERP ↔ CRM sync, e-commerce order flow)
• Your volume is moderate (under 50K transactions/day per integration)
• Your team doesn't include dedicated integration engineers
• You value speed to market over customization (iPaaS integrations deploy in weeks, not months)
• Your ERP is a major platform with pre-built connectors (SAP, NetSuite, Dynamics 365)

iPaaS strengths: Pre-built connectors for major systems, visual flow builders, built-in error handling and monitoring, managed infrastructure.

iPaaS weaknesses: Vendor lock-in, limited customization for complex transformations, pricing that scales with volume (can become expensive at high transaction volumes), performance limitations for real-time requirements.

Cost: $2K-$15K/month for mid-market usage + $50K-$200K implementation.

Build custom when:

• Your ERP is heavily customized with non-standard APIs
• You have high-volume real-time requirements (over 50K transactions/day, sub-second latency)
• Your transformation logic is complex (multi-step, conditional, requires external lookups)
• You need full control over error handling, retry logic, and data persistence
• You have engineering capacity to build and maintain

Custom strengths: Full control, optimal performance, no vendor lock-in, no per-transaction pricing.

Custom weaknesses: Higher upfront cost, requires dedicated maintenance, you own all the operational complexity.

Cost: $100K-$400K initial development + 0.5-1 FTE ongoing maintenance.

The hybrid approach (what most mid-market companies should do):

Use an iPaaS for the 80% of integrations that are standard (CRM sync, basic order flow, data replication). Build custom for the 20% that are complex (custom ERP modules, high-volume real-time flows, complex transformation logic).

Connect the iPaaS and custom integrations through the same event bus and canonical data model so they interoperate cleanly.

ERP-specific integration patterns

SAP integration

For SAP S/4HANA:

• Use OData V4 APIs for CRUD operations (create sales orders, update materials, read inventory)
• Use SAP Event Mesh or custom ABAP events for change notifications
• Use IDocs for EDI integration (856 ASNs, 810 invoices, 850 POs)

For SAP ECC (legacy):

• Use BAPIs (Business Application Programming Interfaces) for transactions
• Use IDocs for async document exchange
• Use RFC (Remote Function Call) for real-time queries
• Avoid custom ABAP programs when standard BAPIs exist

SAP-specific pitfall: SAP's transaction model requires explicit commits. A BAPI call that creates a sales order doesn't actually commit the order until you call BAPI_TRANSACTION_COMMIT. If your integration doesn't handle this, orders appear to succeed but silently disappear.

NetSuite integration

• Use SuiteScript 2.0 RESTlets for custom API endpoints
• Use SuiteTalk (SOAP API) for standard CRUD operations
• Use SuiteFlow for workflow-triggered integrations
• Use Celigo (iPaaS specifically designed for NetSuite) for standard patterns

NetSuite-specific pitfall: NetSuite's governance limits restrict how many API calls you can make per script execution (10,000 units per script). High-volume integrations need to be designed around these limits, using scheduled scripts for batch processing and RESTlets for real-time queries.

Microsoft Dynamics 365 integration

• Use Dataverse Web API (OData) for CRUD operations
• Use Azure Service Bus for event-driven integration
• Use Power Automate for low-volume workflow-triggered integrations
• Use Logic Apps for cloud-to-cloud integration

Dynamics-specific pitfall: Dataverse has a 5,000-record limit per query. Paginating through large datasets requires handling continuation tokens, and the API performance degrades significantly for queries touching 100K+ records.

Testing ERP integrations (the part everyone skips)

ERP integration testing is tedious and frequently under-invested. Here's what you actually need to test:

Happy path testing

• Standard transactions (create order, update inventory, ship order)
• Various product types (simple, configured, BOM items)
• Multiple currencies, languages, and organizational units

Error path testing

• What happens when the ERP is down? (Events should queue, not drop)
• What happens when a required field is missing? (Clear error message, not silent failure)
• What happens when the same event is processed twice? (Idempotent behavior, not duplicates)
• What happens when the ERP rejects the data? (Business error routing, not retry loop)

Volume testing

• Process 10x expected daily volume in 1 hour
• Process 100x expected peak volume to find breaking points
• Run for 72 hours continuously to find memory leaks and connection pool exhaustion

Regression testing

• After any ERP upgrade or configuration change, re-run the full integration test suite
• After any integration code change, run the full test suite
• Automate as much as possible — manual integration testing is unreliable and expensive

The maintenance reality

ERP integrations are never "done." They require ongoing maintenance for:

• ERP updates: SAP and Oracle ship quarterly patches that can change API behavior
• External system changes: Shopify, Salesforce, and other SaaS platforms update their APIs 2-4 times per year
• Business changes: New products, new pricing structures, new customers, new warehouses all require integration adjustments
• Volume growth: Integrations designed for 100 orders/day may need re-architecture at 1,000 orders/day

Budget 15-25% of the initial integration cost per year for maintenance. This isn't optional — undermaintained integrations accumulate technical debt that eventually causes a production failure (usually at the worst possible time, like during your busiest sales month).

---

Building an ERP integration layer? Talk to our enterprise engineering team — we've built integration architectures for SAP, Oracle, Dynamics 365, and NetSuite across manufacturing and distribution.