Message queues and background jobs explained, with a working example

When someone signs up, your app might send a welcome email, add them to a CRM, resize their profile photo and notify your team on Slack. If the sign-up request does all of that before answering, the visitor waits several seconds, and if the CRM is down, the sign-up fails. A background job fixes both: the request saves a note saying “this needs doing” and answers straight away, and a separate worker does the slow part.

The pieces#

• Producer: the code that creates a job, usually inside a request handler.
• Queue: where jobs wait. A database table, Redis, or a hosted service.
• Worker: a separate process that takes jobs off the queue and runs them.

Visitor ──► Web server ──► Queue ──► Worker ──► Email service, CRM, Slack…
               │
               └──► "Thanks for signing up!" (instantly)

What you gain:

• Fast responses. The visitor isn’t waiting for other companies’ servers.
• Resilience. If the email service is down, the job waits and retries; the sign-up still worked.
• Smoothing. A burst of 10,000 sign-ups becomes a queue the workers get through at a steady pace, instead of 10,000 simultaneous calls that trip someone’s rate limit.
• Scaling. More work? Run more workers, without touching the web servers.

What belongs in a background job#

A good rule: anything slow, anything that calls another company’s service, and anything the visitor doesn’t need to see the result of right now.

• Sending emails and SMS
• Generating PDFs, invoices and reports
• Resizing images and processing uploads
• Syncing to CRMs, accounting software and spreadsheets
• Processing incoming webhooks after acknowledging them
• AI calls that take more than a few seconds

Keep in the request what the visitor needs immediately: checking their input, saving their order, telling them it worked.

A small working queue#

You don’t need new infrastructure to start. This queue lives in a database table. It’s tested with Node.js’s built-in SQLite; the same design works in PostgreSQL.

CREATE TABLE jobs (
  id         INTEGER PRIMARY KEY,
  type       TEXT NOT NULL,
  payload    TEXT NOT NULL,
  status     TEXT NOT NULL DEFAULT 'queued',  -- queued, running, done, failed
  attempts   INTEGER NOT NULL DEFAULT 0,
  run_after  INTEGER NOT NULL DEFAULT (unixepoch()),
  last_error TEXT
);

The producer adds a row and moves on:

function enqueue(type, payload) {
  db.prepare("INSERT INTO jobs (type, payload) VALUES (?, ?)").run(type, JSON.stringify(payload));
}

// In the sign-up handler, after saving the user:
enqueue("sendWelcomeEmail", { email: user.email });

The worker claims the next job that’s due, runs it, and records what happened:

const handlers = {
  async sendWelcomeEmail({ email }) {
    await emailService.send({ to: email, template: "welcome" });
  },
};

async function workOnce() {
  // Claim one job in a single statement, so two workers can't take the same one.
  const job = db.prepare(`
    UPDATE jobs SET status = 'running', attempts = attempts + 1
    WHERE id = (
      SELECT id FROM jobs
      WHERE status = 'queued' AND run_after <= unixepoch()
      ORDER BY id LIMIT 1
    )
    RETURNING *
  `).get();
  if (!job) return false;

  try {
    await handlers[job.type](JSON.parse(job.payload));
    db.prepare("UPDATE jobs SET status = 'done' WHERE id = ?").run(job.id);
  } catch (err) {
    const giveUp = job.attempts >= 5;
    const delay = 2 ** job.attempts * 30; // 1 min, 2 min, 4 min...
    db.prepare(`
      UPDATE jobs SET status = ?, last_error = ?, run_after = unixepoch() + ?
      WHERE id = ?
    `).run(giveUp ? "failed" : "queued", err.message, delay, job.id);
  }
  return true;
}

// Run forever: work while there are jobs, then check again every second.
while (true) {
  if (!(await workOnce())) await new Promise((r) => setTimeout(r, 1000));
}

When a job fails, it goes back in the queue with a growing delay (exponential backoff): 1 minute, then 2, then 4. After 5 attempts it’s marked failed with its last error, so a person can look at it. That pile of failed jobs is what larger systems call a dead letter queue.

In PostgreSQL with several workers, claim jobs with SELECT ... FOR UPDATE SKIP LOCKED so workers skip rows another worker is already taking. You also need a way to recover jobs stuck in running if a worker crashes mid-job, for example by resetting any that have been running for longer than a timeout.

Jobs will run more than once#

This is the most important rule of queues. Almost all of them guarantee delivery at least once, not exactly once. A worker can finish sending the email and crash before marking the job done; the job is retried and the email goes twice.

So write every job to be safe to repeat, which is idempotency:

• Before sending, check whether you already have (welcome_email_sent_at on the user).
• Use INSERT ... ON CONFLICT DO NOTHING and unique keys instead of blind inserts.
• Pass idempotency keys to external APIs that accept them, such as payment providers.

Don’t lose jobs between the database and the queue#

If you save an order to the database and then push a job to a separate queue (like Redis), a crash between the two leaves an order with no confirmation email. The database-table queue above avoids this: insert the order and the job in the same transaction, and either both exist or neither does. With a separate queue, the standard fix is the outbox pattern: write the job to an outbox table in the same transaction, and have a small process move rows from there to the queue.

When to use a real queue tool#

A database table is a fine queue for thousands of jobs an hour. Move to a dedicated tool when you need more throughput, scheduling features or a dashboard:

For work that runs on a timetable rather than in response to an event, such as a nightly cleanup, use a scheduler instead; on Wix, that’s scheduled jobs with Velo.

Queues are one of the core tools in how to scale a web app. If you need emails, syncs or AI processing moved off the critical path of your site, I build that as part of custom API integration.