Job delivery and recovery
Processing a video requires both saving it in the database and delivering work to the worker. Delivery intentions are also recorded in the DB so an API or network interruption does not lose the job.
From upload to job
A signed URL is a time-limited URL for uploading the target file. The file goes to storage while the API manages permissions and state.
If delivery fails
external_tasks is an outbox that stores delivery intentions. Writing it in the same transaction as business data makes it possible to recover when the DB update succeeds but dispatch does not.
Delivery normally happens during API processing. TASK_SCHEDULER alarms handle failures and interruptions. Daily maintenance also performs recovery. Successful delivery is distinct from successful worker execution.
If the worker stops midway
SQS may redeliver the same job. The worker keeps an execution record for each job_id in job_executions.
- Completed jobs are not processed again.
- Running jobs acquire a time-limited execution lease.
- If processing stops, the job can run again after the lease expires.
- Follow-up job IDs are derived from the parent job so retries do not create distinct jobs.
This mechanism does not automatically make arbitrary external API operations execute exactly once. Each processing step must also define how it behaves on retry.
Where to look
| Area | Implementation |
|---|---|
| API delivery intentions and dispatch | external-tasks.ts |
| Scheduling recovery | task-scheduler.ts |
| Worker leases and duplicate protection | job_execution.py |
| Dispatching follow-up jobs | sqs_enqueue.py |
Related: State transitions, Change asynchronous processing, Troubleshooting.