Guides / Prompting and shipping

How to Run Background Jobs in a Web App: Queues, Schedulers and Cron

How to run background and scheduled jobs in a web app: in-process tasks, database queues, hosted job services and external schedulers, with honest trade-offs.

Mythex Team · 2026-09-29 · 7 min read

To run background jobs in a web app, first split the work into two kinds. Triggered jobs (send a welcome email, process an upload) can be saved to a job queue, often a table in your database, and handled by a worker with retries. Scheduled jobs (a nightly report, reminders every 15 minutes) need something that fires on a timetable: a host's cron feature, or an external scheduler that calls a protected endpoint in your app. Always make jobs safe to run twice, record failures, and never rely on a timer inside a server that can sleep or restart.

This guide explains the options, when each fits, and how to set them up with an AI app builder, including on hosts that scale to zero.

What counts as a background job

Anything slow, unreliable or not needed for the user's immediate response:

• Sending emails, notifications and receipts
• Resizing images, transcoding video, generating PDFs
• Calling slow third-party APIs, including AI models for long tasks
• Importing or syncing data from another system
• Nightly clean-up, reports and reminders

Doing this inside the request makes pages slow and fragile: if the email provider is down, the user's signup fails. Moving it to the background means the user gets an instant response and the job can be retried.

The options

There's no single right answer. A small app often needs just a database queue plus an external scheduler.

Why hosts that scale to zero change the picture

Many modern hosts put your app to sleep when nobody is using it and wake it on the next request. That saves money, but a sleeping server runs no code: timers don't fire and workers don't poll. On these hosts, scheduled work has to start with an incoming request, and the easiest source of that request is an external scheduler. Each call wakes the app (and often its database), so calling every minute costs more than every 15 minutes. Choose the slowest schedule that meets the need.

Pattern 1: a database job queue

For triggered jobs:

1. Create a jobs table: type, payload (JSON), status (pending, running, done, failed), attempts, run_after, last_error, timestamps.
2. When something happens, insert a job row in the same transaction as the change that caused it, so they succeed or fail together.
3. A worker picks up due jobs, marks them running, does the work and marks them done.
4. On failure, increase attempts, set run_after later (for example 1, 5, then 30 minutes) and store the error. After a maximum, mark it failed and alert someone.

If more than one worker might run at once, jobs must be claimed safely. On Postgres, SELECT ... FOR UPDATE SKIP LOCKED lets each worker claim different rows without clashing.

What runs the worker? On an always-on server, a loop in a separate process. On a scale-to-zero host, the scheduled endpoint in Pattern 2 can process a batch of due jobs on each call.

Pattern 2: an external scheduler calling your app

For scheduled jobs:

1. Add an endpoint such as POST /api/cron/send-reminders.
2. Protect it: require a secret token in a header, stored as a secret on both sides, and reject anything else.
3. Make it idempotent: running it twice in the same minute must not send two reminders. Record what was sent.
4. Keep each call short: process a batch (say, 100 items) and let the next call pick up the rest.
5. Point a scheduler at it.

Scheduler choices:

• GitHub Actions scheduled workflows. Free within GitHub's usage limits. As of September 2026, GitHub's docs say the shortest interval is once every 5 minutes, runs can be delayed at busy times (especially the start of each hour), schedules use UTC by default, and scheduled workflows in a public repository are disabled after 60 days without repository activity.
• Hosted cron services. Many services exist whose only job is to call a URL on a schedule. Check reliability, minimum interval and alerting before you depend on one.
• Your cloud provider's scheduler. If you already use a large cloud provider, it will have a managed scheduler that can call an HTTPS endpoint.
• A hosted job or workflow service. These combine scheduling, queues and retries, and call your endpoints for each step. Compare pricing and limits on each vendor's site.

Pattern 3: let the provider do it

Sometimes the best background job is none at all:

• Payments. Stripe retries failed subscription payments and sends webhooks; you react instead of polling. See what a webhook is.
• Email. Providers queue and retry delivery once you've handed over the message.
• Bulk AI work. OpenAI and Anthropic offer batch APIs for non-urgent requests that you submit and collect later.

Prompts to give your AI builder

For a queue:

Add a background job queue using a jobs table in our database (type, payload JSON, status, attempts, run_after, last_error). When a user signs up, insert a send_welcome_email job in the same transaction instead of sending inline. Add a processJobs(limit) function that claims due jobs with SELECT ... FOR UPDATE SKIP LOCKED, runs them, and retries failures after 1, 5 and 30 minutes, marking them failed after 4 attempts.

For a scheduled trigger:

Add POST /api/cron/tick protected by a bearer token from the secret CRON_SECRET; reject any request without it. Each call runs processJobs(100) and sends booking reminders due in the next 24 hours that haven't been sent yet, recording sent reminders so nothing is sent twice. Return a JSON summary of what ran. Then write a GitHub Actions workflow that calls this endpoint every 15 minutes, reading the URL and token from repository secrets.

Common mistakes

• Relying on setInterval. It silently stops when the server sleeps or restarts, and doubles up when two copies run.
• An unprotected cron endpoint. Anyone who finds it can trigger your jobs, or your email bill.
• Jobs that aren't idempotent. Retries and duplicate scheduler calls will happen. Design for them.
• No failure visibility. A job that fails silently for a week is worse than one that fails loudly. Log errors and alert on repeated failures.
• Long runs in a single request. Requests time out. Process in batches.
• Scheduling too often. Every call on a scale-to-zero host wakes and bills the app. Match the interval to the real need.
• Time zones. "9am" means different things to your scheduler (often UTC) and your users. Store times in UTC and convert per user.
• Secrets in the workflow file. Keep tokens in the scheduler's secret store. See how to keep API keys safe.

Checklist

• Slow work moved out of user requests
• Jobs stored durably, not only in memory
• Retries with increasing delays and a maximum
• Every job safe to run twice
• Scheduled endpoint protected by a secret token
• Each run processes a bounded batch
• Failed jobs visible, with alerts
• Schedule interval chosen with hosting cost in mind
• Times stored in UTC

Background jobs in Mythex

Mythex's documentation does not describe scheduled or cron jobs, and published apps and their databases sleep when idle and wake on the next request (see Hosting limits). So don't rely on timers inside your app. The approach that works is the one above: ask the agent for a jobs table and a token-protected endpoint, then point an external scheduler at your published URL. Store the token as a project secret, and remember that each wake-up spends credits for the time the app and database are awake, so pick a sensible interval. If your app needs an API backend for this, the REST API backend recipe is a starting point, and how to build a REST API covers designing the endpoints. If always-on workers or bots are the core of your product, a host built for persistent processes may suit you better.