configuring-exports

<!-- TIER:1 -->

Configuring Exports

An export is the data source in a Celigo integration. It connects to an external system and pulls data into the pipeline. Exports serve two roles:

• Source -- the starting point that fetches the primary batch of records
• Lookup -- a mid-flow enrichment step (isLookup: true) that fetches additional data per-record during processing

Both roles are used across flows, APIs, and tools.

Beyond fetching data, exports also handle post-retrieval processing before records enter the pipeline:

• Output filter -- expression-based filtering to skip records that don't match criteria
• Transform -- Transformation 2.0 expression rules to reshape/flatten response data before mapping
• preSavePage hook -- JavaScript processing on the full page of records before they enter the pipeline
• One-to-many -- when used as a lookup, fan out child records from a parent. Set oneToMany: true and pathToMany to the child array path so each child triggers a separate lookup. Once fanned out, the array element itself is the record -- see One-to-many fan-out
• Response mapping -- when used as a lookup, extract fields from the lookup response back into the record. Configured on the flow's pageProcessors[] entry, but planned when building the lookup export. The response contains a data array and an errors array. Use data[0].fieldName when you expect a single result (e.g., fetching one order by ID). When multiple results are expected, map the whole array with extract: "data" (downstream steps then read it as a normal JSON array, or fan out over it with one-to-many), or use a lists entry with data[*].fieldName extracts to build a reshaped array. Do not put data[*].fieldName in a top-level fields[].extract -- the wildcard is silently ignored there and nothing is merged. See writing-mappings > Response Mapping Reference. Response mapping uses Transformation 1.0 syntax (extract/generate pairs), not the newer expression-based transforms
• postResponseMap hook -- JavaScript processing after response mapping merges the lookup response back into the record. Configured on the flow's pageProcessors[] entry, but planned when building the lookup export

Export Execution Pipeline

When a flow runs, each export executes this pipeline in strict order:

1. API request / query / file read -- fetches raw data from the external system
2. Response parsing -- resourcePath extracts the record array from the response body or file (e.g., http.response.resourcePath for HTTP, file.json.resourcePath for JSON files, XPath for XML)
3. Transformation (optional) -- transform reshapes individual records after extraction (Transformation 2.0)
4. Output filter (optional) -- discards records that don't match filter expression rules
5. preSavePage hook (optional) -- JavaScript processing on the full page of records

Key distinction: resourcePath tells the export WHERE to find records in the response. Transforms reshape WHAT each record looks like after extraction. When a user says "extract records from X" or "treat each X as a separate record", that's almost always a resourcePath change, not a transform. Use transforms when you need to flatten nested objects, rename fields, or restructure individual records.

Three Categories of Export

Not all exports work the same way. Before building, understand which category you need:

Listeners

Receive data pushed to Celigo from an external system. No polling, no scheduling -- the source system sends data when events happen.

• WebhookExport -- inbound HTTP listener (no connection required)
• AS2Export -- AS2 EDI file reception
• Distributed exports (type: "distributed") -- real-time event-driven push for NetSuite (via SuiteScript) and Salesforce (via streaming API). The platform installs listeners in the source system that fire when records change.
• Change data capture (type: "stream") -- MongoDB change streams that tail the oplog for real-time record changes.

When to use: The source system supports outbound webhooks, push notifications, or change data capture and you want real-time processing.

File Transfers

Read files from a remote location, then either parse them into records or transfer them as blobs.

• FTPExport / S3Export / FileSystemExport -- fetch files from FTP/SFTP, S3, or local filesystem
• HTTPExport with http.type: "file" -- fetch files over HTTP from cloud storage APIs (Google Drive, Box, Dropbox, Azure Blob Storage). The HTTP connector handles auth; the file{} config handles parsing.
• NetSuiteExport with netsuite.type: "file" -- fetch and parse files (CSV, JSON, XLSX, XML, EDI) from the NetSuite file cabinet
• Parsed mode (file.output: "records") -- CSV, XML, JSON, XLSX, EDI files are parsed into individual records
• Blob mode (type: "blob") -- binary files transferred as-is without parsing. Supported on HTTPExport, NetSuiteExport, SalesforceExport, FTPExport, and S3Export.

When to use: The source system drops files (CSV, EDI, XML, etc.) into a directory, bucket, file cabinet, or cloud storage rather than exposing a record-based API.

Record-Based Exports

Actively fetch batches of records from an API or database on a schedule.

• HTTPExport -- REST/GraphQL APIs
• NetSuiteExport -- saved searches, restlets, SuiteQL
• SalesforceExport -- SOQL/Bulk queries
• RDBMSExport -- SQL SELECT queries
• MongodbExport, JDBCExport, DynamodbExport -- other databases
• WrapperExport -- custom stack (Walmart, BigCommerce)

When to use: You need to poll an API or query a database for records on a schedule (full fetch or delta/incremental).

Quick Reference

Adaptor Decision Matrix

Your data comes from...	Use adaptorType	Category	Read schema
REST or GraphQL API	HTTPExport	Record-based	http.yml
Files over HTTP (Google Drive, Box, Dropbox, Azure Blob)	HTTPExport with http.type: "file"	File transfer	http.yml + file.yml
NetSuite (any method)	NetSuiteExport	Record-based	netsuite.yml
Salesforce objects	SalesforceExport	Record-based	salesforce.yml
SQL database	RDBMSExport	Record-based	rdbms.yml
MongoDB	MongodbExport	Record-based	mongodb.yml
JDBC database	JDBCExport	Record-based	jdbc.yml
DynamoDB	DynamodbExport	Record-based	dynamodb.yml
Files on FTP/SFTP	FTPExport	File transfer	ftp.yml + file.yml
Files on S3	S3Export	File transfer	s3.yml + file.yml
Webhooks / push events	WebhookExport	Listener	webhook.yml
AS2 EDI messages	AS2Export	Listener	as2.yml
Manual file upload	SimpleExport	File transfer	simple.yml
Local filesystem	FileSystemExport	File transfer	filesystem.yml + file.yml
Pre-built stack connector	WrapperExport	Record-based	wrapper.yml

Raw HTTP is the fallback, not the default. Pick the most specific match, in order:

1. Native adaptor -- if the application has its own row (NetSuite, Salesforce, databases, FTP/S3), use it. Do not build an HTTPExport against that app's REST API.
2. Pre-built HTTP connector -- for any other REST/GraphQL app, check the 550+ connector catalog before writing HTTP config (see Check for a pre-built connector). The step is still an HTTPExport, but it runs on a connector-backed connection and takes its endpoint config from the connector.
3. Manual HTTP -- hand-write the config from public API docs only when no connector exists or it doesn't cover the endpoint you need.

adaptorType is case-sensitive: HTTPExport, not httpExport.

Minimum Required Fields

Every export needs at minimum:

• name -- human-readable label
• adaptorType -- from the matrix above
• _connectionId -- except WebhookExport and SimpleExport
• Adaptor config block -- http{}, netsuite{}, ftp{}, salesforce{}, rdbms{}, etc.

Which Schemas to Read

1. Always: request.yml (base fields for all exports)
2. Plus: the adaptor-specific file from the matrix above (e.g., http.yml for HTTPExport)
3. If file-based: also file.yml (CSV, XML, JSON, XLSX, EDI parsing config)
4. If delta/incremental: check delta.yml or Handlebars URI pattern ({{{lastExportDateTime}}})
5. If cloning: clone-request.yml, clone-response.yml

Schema Index

All schemas are in references/schemas/:

• Base fields (all exports): request.yml
• Response shape: response.yml
• Adaptor-specific config:
  • http.yml -- HTTP/REST/GraphQL
  • netsuite.yml -- NetSuite (restlet, saved search, SuiteQL, file cabinet)
  • salesforce.yml -- Salesforce (SOQL, bulk)
  • ftp.yml -- FTP/SFTP
  • s3.yml -- Amazon S3
  • rdbms.yml -- SQL databases
  • mongodb.yml -- MongoDB
  • jdbc.yml -- JDBC databases
  • dynamodb.yml -- DynamoDB
  • as2.yml -- AS2 EDI
  • wrapper.yml -- custom stack connectors
  • filesystem.yml -- local filesystem
  • simple.yml -- data loader / manual upload
• File parsing: file.yml (CSV, XML, JSON, XLSX, EDI)
• Operational modes: delta.yml, webhook.yml, distributed.yml, once.yml
• Mock output: mock-output.yml
• Clone: clone-request.yml, clone-response.yml

Related Skills

• configuring-connections > Quick Reference -- connection types, auth methods, iClients
• writing-mappings > Transformation 2.0 -- reshape export output before mapping
• writing-scripts > Data Pipeline Hooks -- preSavePage, postResponseMap hooks
• writing-handlebars > Quick Reference -- dynamic values in URIs, filters, delta tokens
• building-flows > How to Build a Flow -- wiring exports into flows
• troubleshooting-flows > Diagnostic Workflow -- diagnosing export-related failures

<!-- TIER:2 -->

How to Build an Export

1. Identify the target application

What system are you pulling data from? This determines everything -- adaptor type, connection type, and configuration shape.

2. Check for existing patterns

Before building from scratch, look at what already exists:

# Search across the entire account for related resources
celigo account search "<keyword>"

# Show what an existing export uses (connection) and what uses it (flows)
celigo account dependencies export <id>

# Find orphaned exports not referenced by any flow
celigo account lint

# Check if a similar export already exists in the account
celigo exports list | grep -i "<application-name>"

# Search the marketplace for pre-built integration templates
celigo templates marketplace

# Preview a template to see its export configuration
celigo templates preview <id> --model Export
celigo templates preview <id> --summary

The account index auto-refreshes when stale (>4 hours). Force a fresh snapshot with celigo account snapshot.

Existing exports in the account are the best reference -- they show proven patterns for that specific customer's setup. Marketplace templates may provide a complete pre-built integration you can install rather than building from scratch.

3. Check for a pre-built connector

Always run this check before writing any HTTP config. Celigo maintains 550+ HTTP connector definitions and 590+ trading partner connectors. These provide pre-configured auth, base URLs, and endpoint definitions for common applications. Connectors are set on the connection, not the export -- but they determine what the export can do. Hand-write a manual HTTPExport from public API docs only when this search comes up empty or the connector doesn't cover the endpoint you need.

# Search HTTP connectors (REST APIs: Shopify, Stripe, HubSpot, etc.)
celigo http-connectors list | grep -i "<application-name>"
celigo http-connectors get <id> --full    # see endpoints, resources, auth config

# Drill into the endpoints the connector defines for exports
celigo http-connectors catalog <id> --resource-type export --published-only
celigo http-connectors endpoint-detail <id> --resource-type export --resource-id <rid> --endpoint-id <epid>

# Search trading partner connectors (EDI, AS2, VAN)
celigo tp-connectors list

If an HTTP connector exists for your target app, create the connection from it (http._httpConnectorId -- see configuring-connections > Check for a pre-built connector and global iClient) and take the export's relativeURI, method, pagination, and response paths from the connector's endpoint metadata rather than reconstructing them from public API docs. The connector-reference fields on the export itself (http._httpConnectorEndpointId, http._httpConnectorVersionId, http._httpConnectorResourceId) are read-only -- the platform sets them; what you control is the connection and the endpoint config you copy from the connector.

If a trading partner connector exists (EDI/AS2), reference it on the export via ftp._tpConnectorId (FTP exports) or as2._tpConnectorId (AS2 exports). You may also need to set _ediProfileId on the export for EDI document validation.

4. Query metadata for the target system

For NetSuite, Salesforce, and RDBMS connections, you can discover available record types and fields directly from the live system:

# List available record types / sObjects / tables
# NetSuite also returns saved searches alongside record types
celigo metadata types <connectionId>

# List fields for a specific entity type
celigo metadata fields <connectionId> <entityType>

This tells you what data is available to export before you write any configuration.

• NetSuite: metadata types returns both record types and saved searches (with IDs you need for netsuite.restlet.searchId). metadata fields returns field IDs, names, types, and group — including sublist fields you'll need for mapping.lists[].generate on the import side.
• Salesforce: metadata types returns sObjects with queryable/createable flags. metadata fields returns fields, types, and relationship names — use these to discover child objects for distributed.relatedLists[] and relationship field names for cross-object queries.
• RDBMS: metadata types returns table names. metadata fields returns column names and types for a given table — use these when writing SQL queries or building field mappings.

5. Determine the category

Is this a listener (real-time push from the source), a file transfer (fetch and parse/transfer files), or a record-based export (poll an API or query a database)? This narrows which adaptor types and modes apply.

6. Choose the right adaptor type

Use the Adaptor Decision Matrix in Quick Reference above to select the correct adaptorType for your target system.

7. Build the export JSON

Use the Schema Index and Which Schemas to Read in Quick Reference above. Read request.yml for base fields, then the adaptor-specific schema, plus file.yml if file-based and delta.yml if incremental.

Export Design Decisions

A few design choices recur when building exports. Each has a defensible default once the framing is clear.

Delta vs one-time vs full sync

The export's type field selects the sync behavior:

• Delta (type: "delta") -- pulls only records created or modified since the last successful run. The default for ongoing scheduled syncs when the source exposes a usable "last modified" timestamp. Non-HTTP adaptors set the timestamp field via delta.dateField; HTTP exports instead embed {{{lastExportDateTime}}} in the relativeURI or body. See delta.yml.
• One-time (type: "once") -- processes each record exactly once via a tracking flag: each run selects records where once.booleanField is false, then sets it to true after a page succeeds so later runs skip them. Use for backfills and migrations, or when the source has no reliable timestamp but its records can carry a processed flag. See once.yml.
• Full (neither delta nor once mode) -- re-pulls the entire dataset every run. Use when the source has no usable modification timestamp, the dataset is small enough that re-pulling is cheap, or business logic requires a fresh snapshot each run.

When the request is vague ("sync customers"), confirm which kind of sync is intended before building. Delta is a reasonable default when the source exposes a timestamp field; full is reasonable for small static datasets.

Listener/webhook vs scheduled export

Both are starting steps (see Three Categories of Export); the choice is driven by what the source supports and the latency budget, not preference:

• Reach for a listener (WebhookExport, or NetSuite/Salesforce type: "distributed") when the source pushes events and the flow needs to react quickly ("when X happens, do Y").
• Reach for a scheduled export when the source has no push mechanism, or when batch timing at off-peak hours is acceptable.

NetSuite and Salesforce support both for many record types. Mixing them on one flow is a common, good pattern -- a listener handles low-latency reactions while a scheduled export runs as a safety net for backfills, end-of-day reconciliation, and catching up after a webhook outage.

Lookup export vs separate scheduled export

The distinguishing question is when the data is needed:

• A lookup export (isLookup: true) runs per in-flight record, mid-pipeline, keyed off the upstream record -- fetching the customer for a specific order, or inventory for a specific SKU.
• A scheduled export runs once per flow run as a starting point, producing the first batch of records the flow processes.

If the request is "for each X, look up Y", it's a lookup. If it's "every hour, pull all Y", it's a scheduled export.

One-to-many fan-out -- the array element IS the record

With oneToMany: true and pathToMany set to a child array path, each element of that array triggers its own lookup. Once fanned out, the element becomes the record: templates reference the element's own fields as {{record.variantId}} -- not {{variantId}}, and not {{record.lineItems.variantId}}. The array wrapper is gone; you are inside one element.

Three consequences worth knowing before you debug the wrong thing:

• The build-time preview warning is expected. Previewing a fanned-out lookup in isolation reports "<field> not defined in the model" because no upstream record is bound yet. That is not a broken template -- don't "fix" a correct {{record.X}} reference because of it.
• Response mapping merges per element automatically. To get looked-up values back onto each element, author a normal top-level fields response mapping; Celigo merges each result into its corresponding fanned-out element.
• Two anti-patterns. Don't target the array with a lists entry (that nests a new array inside each element), and don't attempt the per-element merge in postResponseMap (it sees the page of parent records, not per-element results).

Source-side transform vs destination-side mapping

Both reshape data, but in opposite directions:

• A transform on a source export reshapes records as they enter the flow -- flattening nested responses, or aligning multiple sources to a common shape (see Export Execution Pipeline).
• A mapping on a downstream import reshapes records as they leave the flow toward a destination.

Don't add a transform to "match a destination" -- that's the destination import mapping's job. Transforms are for entry reshaping; mappings are for exit reshaping.

Async APIs (submit, poll, fetch)

Most APIs return data in the same call and need none of this. Some APIs only acknowledge a request (an HTTP 202, a job ticket, a feed or document id) and process it in the background -- Amazon SP-API feeds, large report generators, bulk extract and file-conversion jobs. For those, attach an async helper to the export via http._asyncHelperId. The helper teaches the step the submit-poll-fetch pattern; it is part of the export, not something managed on its own, and bundles three pieces:

1. A status export (required) -- run on each poll to ask "is it done yet?". Configure the status path to read in the response, the case-sensitive in-progress / done / done-without-data / error value lists (taken from the API's docs), and the initial wait and poll wait intervals in minutes.
2. A result export (optional, usually present) -- fetches the final payload once status reports done.
3. Initial-submission handling -- where to find the job ticket in the first acknowledgement: "same as status" when the acknowledgement is itself shaped like a status response, otherwise a resource path (plus transform rules for non-JSON acknowledgements, e.g. Amazon's XML).

Two constraints shape the design: the status and result exports must be ordinary synchronous exports (an async helper cannot nest another), and the async-configured step cannot carry its own transform, output filter, or preSavePage hook -- put any reshaping or filtering on the dedicated result export instead. The same pattern applies symmetrically to imports writing to asynchronous destinations (_asyncHelperId on the import).

Reach for an async helper only when the API genuinely forces the fire-and-check-back shape. Adding one to a synchronous API is pure overhead -- extra polling plus a status and result export to maintain.

CLI Commands

# CRUD
celigo exports list
celigo exports get <id>
celigo exports create < export.json
celigo exports update <id> < export.json
celigo exports set <id> key=value [key2=value2 ...]
celigo exports delete <id>

# Invoke (test-run an export, see what data comes back)
celigo exports invoke [id] [--all]

# Clone and connection management
echo '{"connectionMap":{"oldConnId":"newConnId"}}' | celigo exports clone <id>
celigo exports replace-connection <id> <newConnectionId>

# Discovery
celigo account search "<keyword>"
celigo templates marketplace
celigo templates preview <id> --model Export
celigo templates preview <id> --summary
celigo http-connectors list
celigo http-connectors catalog <id> --resource-type export --published-only
celigo http-connectors endpoint-detail <id> --resource-type export --resource-id <rid> --endpoint-id <epid>
celigo tp-connectors list
celigo metadata types <connectionId>
celigo metadata fields <connectionId> <entityType>

# Debug
celigo exports enable-debug <id> [--duration <minutes>]
celigo exports disable-debug <id>

<!-- TIER:3 -->

Pre-Submit Checklist

Before creating or updating an export, verify:

• adaptorType is exact -- case-sensitive, matches the Adaptor Decision Matrix (e.g., HTTPExport, not httpExport or HttpExport)
• Pre-built connector was checked -- for HTTP exports, celigo http-connectors list found no connector for the app (or the connector lacks the endpoint) before any hand-written relativeURI
• _connectionId is valid -- points to an existing, online connection of the correct type. Not needed for WebhookExport or SimpleExport
• Adaptor config block is present -- http{}, netsuite{}, ftp{}, etc. matches the adaptorType
• resourcePath or query is correct -- wrong path silently returns 0 records with no error
• Pagination is configured -- for HTTP exports, set http.paging if the API returns paginated results
• Delta/incremental is configured -- if using delta, check delta.dateField or Handlebars {{{lastExportDateTime}}} in the URI
• File parsing matches the format -- if file-based, file.type matches the actual file format (csv, json, xml, xlsx, edi)
• mockOutput format is correct -- { "page_of_records": [{ "record": {...} }] }, not a plain array
• No rest: block -- rest: creates a legacy RESTExport. Use only http: for new exports
• Output filter syntax is valid -- if using an output filter expression, test it against sample data
• Lookup config is complete -- if isLookup: true, ensure response mapping is planned for the flow's pageProcessors[] entry

Gotchas

1. PUT erases omitted fields. Always GET first, modify, then PUT. The set command handles this.
2. Including a rest: block creates a legacy RESTExport. Use only http: for new exports.
3. Wrong resourcePath produces 0 records with no error. First thing to check when an export succeeds but returns nothing.
4. mockOutput format is { "page_of_records": [{ "record": {...} }] }. Not a plain array.
5. HTTP delta exports use Handlebars ({{{lastExportDateTime}}} in relativeURI), not delta.dateField.
6. NetSuite saved searches need netsuite.restlet.searchId. Use celigo metadata types <connectionId> to find the search ID.
7. File exports require the file{} block. Without it, file-based exports return raw bytes instead of parsed records.
8. Webhook exports have no _connectionId. Setting one causes validation errors.
9. Distributed exports require type: "distributed" on the export AND distributed: true on the connection.
10. type: "once" needs a dedicated, writeable tracking flag. once.booleanField must be writeable by the export's connection, and no other process may update the same field -- a shared flag causes records to be skipped.
11. An async-helper export cannot carry its own transform, output filter, or preSavePage hook. Build that processing into the helper's result export instead. The status and result exports must themselves be plain synchronous exports -- an async helper cannot nest another. See Async APIs (submit, poll, fetch).

Common Errors

Error	Likely Cause	Fix
404 Not Found on export invoke	Wrong relativeURI or resourcePath	Verify the endpoint path against the API docs; check for missing path parameters
401 Unauthorized	Connection credentials expired or invalid	Run celigo connections ping <connId>; re-authorize OAuth connections
0 records exported (no error)	Wrong resourcePath, empty date range, or overly restrictive filter	Check resourcePath, widen delta window, test without output filter
Cannot read property of undefined in preSavePage	Script assumes a field exists that is missing from some records	Add null checks: if (record.field) before access
mockOutput is invalid	Wrong format -- used array instead of object	Use { "page_of_records": [{ "record": {...} }] }
Invalid adaptorType	Case mismatch or typo	Use exact casing from the Adaptor Decision Matrix
Connection is offline	Connection failed health check	Fix credentials, re-authorize, then celigo connections ping <id>
Rate limit exceeded / 429	Too many concurrent requests to the source API	Lower concurrencyLevel on the connection; add retry config
Timeout on large exports	Query returns too much data or API is slow	Add pagination, narrow the date range, or increase timeout settings
File parsing error	file.type doesn't match actual file format, or delimiter/encoding mismatch	Verify file.type, check file.csv.columnDelimiter, ensure correct encoding