The Complete Guide to Background Jobs and Task Queues

Background jobs are easy to start and surprisingly difficult to run reliably.

Calling sendEmail() after returning an HTTP response may look asynchronous, but it is not a durable background-job system. If the process crashes, the deployment restarts, or the machine disappears, that work may vanish. A real task queue records the work somewhere durable enough for another process to pick it up, retry it, and report what happened.

This guide explains the architecture and failure modes that matter in production, then builds a practical Node.js and TypeScript example with BullMQ and Redis.

Background jobs, task queues, and asynchronous code

A background job is work performed outside the request-response path. A task queue is the infrastructure that coordinates that work between producers and workers.

Those ideas are related, but they are not identical to ordinary asynchronous JavaScript:

• await, promises, and non-blocking I/O let one Node.js process use its time efficiently.
• setImmediate() or an unawaited promise schedules work only inside the current process.
• A task queue stores a description of the work so a worker can execute it independently, potentially on another machine and after the producer has exited.

A typical system has three parts:

1. Producer: the web application, API, or service that creates a job.
2. Queue: the broker or data store that holds pending jobs and coordinates delivery.
3. Worker: a separate process that reserves a job, performs the work, and reports success or failure.

The queue creates a boundary between accepting work and executing it. That boundary improves responsiveness and allows the web tier and worker tier to scale independently, but it also introduces distributed-systems problems: duplicate delivery, partial failure, retries, ordering, stale data, and operational backlogs.

What belongs in a background job?

Good candidates are operations that are slow, failure-prone, resource-intensive, or unnecessary for the immediate response:

• Sending email, push notifications, and webhooks
• Resizing images or transcoding video
• Importing large files and generating reports
• Synchronizing data with external services
• Rebuilding search indexes or caches
• Running scheduled cleanup and maintenance
• Processing analytics events

Keep work synchronous when the user needs its result before the request can be considered successful. Validation, authorization, and the database write that creates the requested resource usually belong in the request path. Payment authorization often does too when checkout cannot finish without knowing whether the payment was accepted. Follow-up work such as receipts, fulfillment notifications, and reconciliation can be queued.

A useful rule is:

Put the minimum required state change in the request path. Queue the independent consequences.

The life cycle of a job

Although queue products use different terminology, most jobs follow the same life cycle:

1. The producer serializes a small payload and enqueues it.
2. A worker reserves or receives the job.
3. The queue hides or locks the job while it is being processed.
4. The worker performs the operation.
5. On success, the worker acknowledges completion.
6. On failure, the job is retried, delayed, or retained for investigation.

The critical detail is that most practical systems provide at-least-once execution, not exactly-once execution. A worker can finish the external side effect and crash before acknowledging the job. The queue then makes the job available again, and another worker may execute it a second time.

That is why idempotency is not an optional optimization. It is part of the correctness model.

Do not assume strict FIFO ordering

A queue may store jobs in insertion order, but that does not mean jobs will finish in that order. Multiple workers, different execution times, priorities, retries, delayed jobs, and redelivery can all change the order observed by the application.

When ordering is a business requirement, define its scope precisely. You may need one active consumer, partitioning by an entity such as accountId, or sequence checks in the database. Global ordering usually reduces throughput and availability, so avoid requiring it unless the domain truly needs it.

Durability is configured, not automatic

A queue does not magically make work impossible to lose. Reliability depends on broker configuration, persistence, replication, producer acknowledgements, retention, and the way the application crosses the database-to-queue boundary. A durable queue can still lose a logical task if the database commit succeeds but enqueueing fails afterward. We will address that with the transactional outbox pattern.

A practical Node.js and TypeScript implementation

The following example uses BullMQ with Redis. Install bullmq and ioredis, run Redis, and provide a REDIS_URL environment variable.

The example queues a welcome email after registration. The job contains identifiers rather than a complete user object, which keeps the payload small and prevents stale or sensitive data from being copied into Redis.

Define the queue and job contract

// jobs/email-queue.ts
import { Queue } from 'bullmq'
import IORedis from 'ioredis'

export type EmailJobName = 'send-welcome-email'

export interface EmailJobData {
  schemaVersion: 1
  userId: string
  correlationId: string
}

const redisUrl = process.env.REDIS_URL

if (!redisUrl) {
  throw new Error('REDIS_URL is required')
}

// Producers should fail reasonably quickly when Redis is unavailable.
const producerConnection = new IORedis(redisUrl, {
  maxRetriesPerRequest: 1,
  enableOfflineQueue: false,
})

export const emailQueue = new Queue<EmailJobData, void, EmailJobName>(
  'email',
  {
    connection: producerConnection,
    defaultJobOptions: {
      attempts: 5,
      backoff: {
        type: 'exponential',
        delay: 1_000,
      },
      removeOnComplete: {
        age: 60 * 60,
        count: 1_000,
      },
      removeOnFail: {
        age: 7 * 24 * 60 * 60,
        count: 5_000,
      },
      stackTraceLimit: 20,
    },
  },
)

The producer and worker use different Redis connection behavior. An HTTP request should not wait forever when Redis is unavailable, while a long-running worker should normally keep reconnecting.

Enqueue a job

// jobs/enqueue-welcome-email.ts
import { randomUUID } from 'node:crypto'
import { emailQueue, type EmailJobData } from './email-queue.js'

export async function enqueueWelcomeEmail(
  userId: string,
  correlationId = randomUUID(),
): Promise<string> {
  const data: EmailJobData = {
    schemaVersion: 1,
    userId,
    correlationId,
  }

  const job = await emailQueue.add('send-welcome-email', data, {
    // BullMQ job IDs cannot contain a colon.
    // This suppresses duplicate enqueues while this job ID is retained.
    jobId: `welcome-email-${userId}`,
  })

  if (!job.id) {
    throw new Error('Queue returned a job without an ID')
  }

  return job.id
}

A custom job ID is useful for deduplication, but it is not a complete idempotency strategy. Once an old job is removed, the same ID can be added again. More importantly, a job that has already started can still be redelivered after a crash. The side effect itself must remain safe to repeat.

Process the job in a worker

// workers/email-worker.ts
import { UnrecoverableError, Worker, type Job } from 'bullmq'
import IORedis from 'ioredis'
import { type EmailJobData, type EmailJobName } from '../jobs/email-queue.js'
import { emailProvider } from '../services/email-provider.js'
import { userRepository } from '../repositories/user-repository.js'

const redisUrl = process.env.REDIS_URL

if (!redisUrl) {
  throw new Error('REDIS_URL is required')
}

const workerConnection = new IORedis(redisUrl, {
  maxRetriesPerRequest: null,
})

async function processEmailJob(
  job: Job<EmailJobData, void, EmailJobName>,
): Promise<void> {
  if (job.data.schemaVersion !== 1) {
    throw new UnrecoverableError(
      `Unsupported job schema: ${job.data.schemaVersion}`,
    )
  }

  const user = await userRepository.findById(job.data.userId)

  if (!user) {
    throw new UnrecoverableError(`User ${job.data.userId} no longer exists`)
  }

  await job.log(
    JSON.stringify({
      event: 'sending_welcome_email',
      userId: user.id,
      correlationId: job.data.correlationId,
      attempt: job.attemptsMade + 1,
    }),
  )

  try {
    await emailProvider.sendWelcomeEmail({
      to: user.email,
      // The provider should treat repeated requests with this key as one send.
      idempotencyKey: `welcome-email-${user.id}`,
      signal: AbortSignal.timeout(10_000),
    })
  } catch (error) {
    if (emailProvider.isPermanentError(error)) {
      throw new UnrecoverableError(
        error instanceof Error ? error.message : 'Permanent email error',
      )
    }

    // Throwing a normal Error lets BullMQ apply the retry policy.
    throw error instanceof Error ? error : new Error(String(error))
  }
}

export const emailWorker = new Worker<EmailJobData, void, EmailJobName>(
  'email',
  processEmailJob,
  {
    connection: workerConnection,
    concurrency: 20,
    limiter: {
      max: 50,
      duration: 1_000,
    },
  },
)

emailWorker.on('completed', (job) => {
  console.info('email_job_completed', { jobId: job.id })
})

emailWorker.on('failed', (job, error) => {
  console.error('email_job_failed', {
    jobId: job?.id,
    attemptsMade: job?.attemptsMade,
    error: error.message,
  })
})

emailWorker.on('error', (error) => {
  console.error('email_worker_error', { error })
})

async function shutdown(signal: NodeJS.Signals): Promise<void> {
  console.info('email_worker_shutting_down', { signal })
  await emailWorker.close()
  await workerConnection.quit()
}

for (const signal of ['SIGTERM', 'SIGINT'] as const) {
  process.once(signal, () => {
    void shutdown(signal)
      .then(() => process.exit(0))
      .catch((error: unknown) => {
        console.error('email_worker_shutdown_failed', { error })
        process.exit(1)
      })
  })
}

The worker has a timeout, explicit permanent-error handling, bounded retries, rate limiting, structured context, and graceful shutdown. Those details are not decoration; they determine how the system behaves during deployments and dependency failures.

Retry only failures that may succeed later

Retries are useful only when the next attempt has a meaningful chance of succeeding.

Failure	Typical response
Network timeout before a confirmed result	Retry with backoff, provided the operation is idempotent
HTTP 429 or temporary service overload	Honor Retry-After when available, then retry
Temporary database or broker outage	Retry with bounded backoff
Invalid payload or unsupported schema	Fail permanently
Missing resource that should never reappear	Fail permanently
Timeout after an external request may have succeeded	Reconcile using an idempotency key or provider status API before repeating

Use exponential backoff so a failing dependency is not hammered continuously. Add random jitter when your queue library or custom retry strategy supports it; otherwise thousands of jobs can wake at the same instant and create a thundering herd.

Retries must be bounded. After the final attempt, retain the failed job or route it to a dead-letter mechanism so an operator can inspect, correct, and redrive it. Infinite retries turn permanent defects into permanent load.

Idempotency, deduplication, and exactly-once myths

These terms are often mixed together:

• Deduplication prevents some duplicate jobs from entering or waiting in the queue.
• Idempotency makes repeated execution produce the same intended final state.
• Exactly-once delivery is not something an application should assume across a queue, database, and external API.

The unsafe pattern is "check, perform the side effect, then record success." Two workers can pass the check concurrently, or the process can crash after the side effect and before the success record is written.

For external operations such as charging a card or sending an email, the strongest practical solution is usually a stable idempotency key understood by the external provider:

// services/capture-payment.ts
export async function captureOrderPayment(orderId: string): Promise<void> {
  const order = await orderRepository.getForPayment(orderId)

  await paymentProvider.capture({
    paymentMethodId: order.paymentMethodId,
    amountInCents: order.amountInCents,
    idempotencyKey: `capture-order-${order.id}`,
  })

  await orderRepository.markPaid(order.id)
}

If the worker crashes after the provider accepts the charge, the retry uses the same key and retrieves or reuses the same provider-side operation instead of charging again.

When the dependency does not support idempotency keys, prefer naturally idempotent operations such as an upsert to a known value. Otherwise, design explicit reconciliation and accept that duplicate side effects may be possible. A database uniqueness constraint can protect database state, but it cannot atomically cover an unrelated external API call.

Also, background jobs commonly exist specifically to perform side effects. "Avoid side effects" is therefore the wrong rule. The better rule is: make side effects explicit, bounded, observable, and safe to repeat.

Close the database-to-queue gap with a transactional outbox

Consider this sequence:

1. Insert a user into PostgreSQL.
2. Enqueue the welcome-email job in Redis.

If the database commit succeeds and Redis is unavailable, the account exists but the job is missing. Reversing the order creates the opposite problem: the worker may run before the database transaction commits or after it rolls back.

A transactional outbox solves this dual-write problem by writing the business record and an outbox event in the same database transaction. A separate dispatcher publishes committed outbox events to the queue.

// services/register-user.ts
import { randomUUID } from 'node:crypto'
import { pool } from '../database.js'
import type { EmailJobData } from '../jobs/email-queue.js'

export async function registerUser(email: string): Promise<string> {
  const client = await pool.connect()

  try {
    await client.query('BEGIN')

    const userId = randomUUID()
    await client.query('INSERT INTO users (id, email) VALUES ($1, $2)', [
      userId,
      email,
    ])

    const eventId = randomUUID()
    const payload: EmailJobData = {
      schemaVersion: 1,
      userId,
      correlationId: eventId,
    }

    await client.query(
      `INSERT INTO outbox_events (id, topic, payload)
       VALUES ($1, $2, $3::jsonb)`,
      [eventId, 'send-welcome-email', JSON.stringify(payload)],
    )

    await client.query('COMMIT')
    return userId
  } catch (error) {
    await client.query('ROLLBACK')
    throw error
  } finally {
    client.release()
  }
}

A compact dispatcher can claim an unpublished row, enqueue it with the outbox ID as the queue job ID, and mark it published:

// workers/outbox-dispatcher.ts
import type { PoolClient } from 'pg'
import { pool } from '../database.js'
import {
  emailQueue,
  type EmailJobData,
  type EmailJobName,
} from '../jobs/email-queue.js'

interface OutboxEvent {
  id: string
  topic: EmailJobName
  payload: EmailJobData
}

export async function publishOneOutboxEvent(): Promise<boolean> {
  const client: PoolClient = await pool.connect()

  try {
    await client.query('BEGIN')

    const result = await client.query<OutboxEvent>(
      `SELECT id, topic, payload
       FROM outbox_events
       WHERE published_at IS NULL
       ORDER BY created_at
       FOR UPDATE SKIP LOCKED
       LIMIT 1`,
    )

    const event = result.rows[0]
    if (!event) {
      await client.query('COMMIT')
      return false
    }

    await emailQueue.add(event.topic, event.payload, {
      jobId: event.id,
    })

    await client.query(
      `UPDATE outbox_events
       SET published_at = NOW()
       WHERE id = $1`,
      [event.id],
    )

    await client.query('COMMIT')
    return true
  } catch (error) {
    await client.query('ROLLBACK')
    throw error
  } finally {
    client.release()
  }
}

A production dispatcher normally claims batches with a lease instead of holding a database transaction open during a network call, but the correctness principle is the same. A crash can still cause the event to be published more than once, so the worker remains idempotent. The outbox gives reliable eventual publication, not magical exactly-once execution.

Design job payloads as durable contracts

A queued payload may live longer than the deployment that created it. Treat it like a versioned API contract.

Keep payloads small and include:

• A schema version
• Stable resource identifiers
• A correlation or trace ID
• Only the data required to locate or perform the work

Avoid embedding large objects, secrets, access tokens, or unnecessary personal data. Do not enqueue a path such as /tmp/upload-123 when another worker or machine cannot access that filesystem. Store the file in shared object storage and pass its object key.

Decide how new workers handle old payload versions before deploying a breaking change. Common strategies are backward-compatible readers, explicit migration, or temporary workers for both versions.

Scheduled jobs, workflows, and batching

Scheduled and recurring jobs

A scheduler should create jobs; workers should execute them. This preserves the same retry, monitoring, and isolation model as one-off work.

With current BullMQ job schedulers, a recurring job can be upserted by a stable scheduler ID:

// jobs/schedule-cleanup.ts
import { Queue } from 'bullmq'
import IORedis from 'ioredis'

const connection = new IORedis(process.env.REDIS_URL!, {
  maxRetriesPerRequest: 1,
})

const maintenanceQueue = new Queue('maintenance', { connection })

await maintenanceQueue.upsertJobScheduler(
  'daily-expired-session-cleanup',
  { pattern: '0 0 3 * * *' },
  {
    name: 'delete-expired-sessions',
    data: { schemaVersion: 1 },
    opts: {
      attempts: 3,
      backoff: { type: 'exponential', delay: 5_000 },
    },
  },
)

For recurring work, define the timezone, daylight-saving behavior, missed-run policy, and overlap policy. If one run takes longer than its interval, decide whether the next run should wait, be skipped, or execute concurrently. The job itself should still be idempotent because scheduler failover or operator action can create duplicates.

Workflows and chains

Split a workflow when each step needs different retries, resource limits, or visibility. For example:

upload received → virus scan → image variants → CDN publish → database update

Do not replace a durable workflow with a chain of in-process promises when the process may restart. Use queue-native parent-child flows or a workflow engine when the sequence has long waits, compensation, human approval, or complex state transitions.

Batching

Batching can reduce API calls and database round trips, but an in-memory buffer inside one worker is not durable and does not coordinate across replicas. Prefer broker-supported bulk enqueue, database-backed batches, or a dedicated batch-aggregation job. Define how partial failures are retried so successful items are not repeated unnecessarily.

Scaling workers correctly in Node.js

For I/O-heavy jobs, increasing BullMQ worker concurrency lets one Node.js process make progress on other jobs while it waits on databases and network calls. Measure dependency capacity before increasing it: concurrency that overwhelms a database or vendor API reduces throughput rather than improving it.

CPU-heavy jobs are different. Image codecs, document parsing, compression, and large calculations can block the Node.js event loop. That delays queue lock renewal and may cause the job to be classified as stalled and processed again. Use separate worker processes, BullMQ sandboxed processors, or a worker_threads pool for CPU-bound JavaScript. Scale those workers according to available CPU cores rather than setting a large asynchronous concurrency value.

Separate queues are often useful for different workload classes:

• Fast, latency-sensitive jobs
• Slow or CPU-heavy jobs
• Jobs calling a rate-limited dependency
• Critical jobs with stricter operational handling

This prevents a flood of slow work from starving password-reset emails or other urgent tasks. Prefer separate queues over many priority levels when the workloads need different scaling, retry, or alerting policies.

Backpressure and rate limiting

A queue absorbs bursts; it does not create infinite downstream capacity. If jobs arrive faster than workers can safely process them, queue age grows until users experience stale results or retention limits are reached.

Apply backpressure at several layers:

• Limit worker concurrency to the capacity of dependencies.
• Rate-limit calls to external services.
• Reject, defer, or coalesce low-value work when the backlog is unhealthy.
• Deduplicate rapid updates when only the latest state matters.
• Scale using queue age and throughput, not CPU alone.

Queue depth is useful, but the age of the oldest ready job is often a better user-impact signal. Ten thousand one-millisecond jobs may be healthy; fifty jobs waiting for an hour are not.

Observability and operations

A production queue needs the same operational discipline as a database or API.

Track at least:

• Jobs enqueued, started, completed, failed, and retried
• Queue depth and age of the oldest job
• End-to-end latency from enqueue to completion
• Processing duration percentiles by job type
• Exhausted jobs and dead-letter or failed-set size
• Stalled jobs and lock-renewal problems
• Worker concurrency, utilization, memory, and event-loop lag

Every log entry should include the queue, job name, job ID, attempt number, and correlation ID. Do not log complete payloads when they contain personal data or secrets.

Alert on symptoms that require action: sustained queue age, a rising retry ratio, jobs exhausting attempts, repeated stalls, no active workers, or a producer enqueue failure rate above normal. A dashboard without alerts simply documents an outage after the fact.

Operational tooling should support pausing a queue, inspecting a job, retrying or redriving failed work, and safely deleting obsolete jobs. Keep enough completed and failed history for debugging, but configure retention so Redis or broker storage cannot grow without limit.

Choosing a queue technology

There is no universally best queue. Choose based on delivery model, operational constraints, ecosystem, and the surrounding architecture.

Technology	Strong fit	Important consideration
BullMQ with Redis	Node.js applications needing retries, delays, schedulers, rate limits, and workflows	You operate Redis correctly and design workers for at-least-once execution
RabbitMQ	Multi-language systems needing acknowledgements, routing, and broker-controlled delivery	Topology and durability settings require careful operation
Amazon SQS or Google Cloud Tasks	Teams that prefer managed infrastructure and cloud-native integration	Semantics, limits, observability, and local development are provider-specific
Kafka or pub/sub event platforms	Durable event streams, replay, fan-out, and multiple independent consumers	They model events differently from command-style task execution

Use a task queue when one piece of work should be handled by one logical worker. Use an event stream or pub/sub system when an event should be observed independently by multiple consumers. Some systems use both: an event is published, and each subscriber creates its own task for controlled execution.

Production checklist

Before launching a background-job system, verify that:

• The request path contains only work required for the immediate response.
• The producer handles enqueue failures explicitly.
• Database changes and required jobs use an outbox or another reliable publication mechanism.
• Job payloads are small, versioned, and free of unnecessary sensitive data.
• Workers are safe under duplicate execution.
• External side effects use stable idempotency keys when available.
• Retries are limited, classified, and delayed with backoff.
• Permanent failures are retained and alert an operator.
• Timeouts and cancellation are implemented for external calls.
• Worker concurrency matches the workload and dependency capacity.
• CPU-heavy work runs outside the main Node.js event loop.
• Queue age, retries, failures, and stalls are monitored.
• Deployments shut workers down gracefully.
• Operators can pause, inspect, retry, and redrive jobs.
• Retention and Redis or broker durability settings are tested, not assumed.

Conclusion

Background jobs improve responsiveness and scalability by moving independent work out of the request path. The queue itself, however, is only the beginning.

Reliable systems assume that jobs can be delivered more than once, that ordering can change, and that every boundary can fail. They use small versioned payloads, bounded retries, idempotent side effects, transactional outboxes, separate worker processes, graceful shutdown, and metrics based on user-visible delay.

The most important design principle is simple:

A job is not reliable because it was placed on a queue. It is reliable because the entire path—from database commit to enqueue, execution, side effect, acknowledgement, retry, and recovery—was designed for failure.