doctrine-batch-processing

Doctrine Batch Processing (Symfony)

Use when

• Processing more rows than fit in memory: imports, exports, backfills.
• A command or worker runs out of memory iterating over entities.
• Updating or deleting many rows at once.
• Inserting large volumes through Doctrine or DBAL.

Default workflow

1. Measure the dataset size and the memory budget of the process.
2. Stream reads with toIterable(), or paginate by id (id > :lastId) for very large tables.
3. Flush and clear() every N items, and re-fetch anything needed after a clear.
4. Use DQL UPDATE/DELETE or DBAL for bulk writes that need no hydration.
5. Log memory usage and progress, and make the run resumable.

Guardrails

• Never findAll() on a table that grows.
• Entities loaded before a clear() are detached: do not modify or persist them.
• Bulk DQL and DBAL skip lifecycle events and listeners. Check what relies on them.
• On ORM 3, Query#iterate() and setSQLLogger() are gone: use toIterable() and a logging middleware.
• Paginate by id rather than by offset: offsets get slower and skip rows when data moves.

Progressive disclosure

• Use this file for execution posture and risk controls.
• Open references when deep implementation details are needed.

Output contract

• A batch loop with an explicit batch size and clear points.
• The choice between ORM, DQL bulk and DBAL for each write, with the reason.
• Memory measurements before and after on a realistic volume.

References

• reference.md