Differences

This shows you the differences between two versions of the page.

--- iqxwebnext:technical:web_jobs [2026/10/03 16:30] – created Gareth Johnstone
+++ iqxwebnext:technical:web_jobs [2026/10/05 20:02] (current) – Gareth Johnstone
@@ Line 2: / Line 2: @@
 # Web Jobs (JavaScript)
-A **Web Job** is a `.js` file that WebHub runs on request and hands back the result over HTTP — a stored procedure written in JavaScript. Jobs can query the IQX database, use a small set of approved npm modules, and return either JSON or a raw file (HTML, CSV, PDF…). They can be called from a browser, from iqxWEB, from a third-party integration, or on a cron schedule.
+A **Web Job** is a short JavaScript program that WebHub runs on request and answers over HTTP — think of it as a stored procedure written in JavaScript, or a small web page. A job can query the IQX database, call the public internet, use a short list of approved npm modules, and return JSON, a file (CSV, PDF…) or an HTML page. Jobs are called from browsers, from iqxWEB, from third-party webhooks, or on a cron schedule. Each run executes in its own worker thread with a runtime, memory and CPU limit, so a badly written job fails on its own without taking the node down.
-This page is about the JavaScript jobs served from `/api/webjobs`. The XML **JobRunner** jobs (`<Job>` files, `/api/wpk/job/<name>.xml`) are a different engine with their own page.
+Jobs are written and managed from the **Web Jobs** pages in iqxWEB (Owner → Web Jobs → *Manage* and *Scheduler*) and served from `/api/webjobs/<name>`.
-## How a job runs
+This page is about the JavaScript jobs served from `/api/webjobs`. The XML **JobRunner** jobs (`<Job>` files) are a different engine with their own page: they run inside IQX with a fixed vocabulary of commands, whereas a Web Job is ordinary JavaScript answering an HTTP request. They share no code, and one does not run the other.
-. A request arrives at `GET` or `POST /api/webjobs/<name>`. The name must match `[a-zA-Z0-9_-]+` (no dots, slashes or spaces) — anything else is a `HUB-007`.
+## Writing a job
-. Unless the name starts with `public_`, the caller must have a valid session (bearer token). A missing or bad token is refused before the file is even looked at.
-. WebHub reads `jobs/<name>.js` from its working directory (`/app/jobs` in the container, the service folder on Windows). No file → `SYS-404`.
-. The file's text is wrapped in a function and called with a fixed set of globals (below). It is **not** an ES module: there is no `import`, no top-level `await`, and `require` is WebHub's own controlled version.
-. The job does its work and calls `finish ( result )` exactly once. WebHub polls for that call every 100 ms and gives up after the runtime limit (10 s by default) with `JOB-003`.
-. The result is streamed back as **NDJSON**: one metadata line, then the data (see *What the caller receives*). The run is recorded in history, metrics and the daily log file.
-The function wrapper matters in practice. Because the body is plain script, the idiomatic shape of every job is an async IIFE:
+A job is the **body of an async function**. Use `await` and `return` directly; there is no wrapper to remember.
 ```javascript
 /*
-Author: IQX Limited
+Name: Vacancies by department
-Name: My Job
+Description: Live vacancies for one department, newest first
-Description: One line saying what it does
+Runtime: 10000
-Version: 1.0.0
-Last Updated: 2026-10-03
 */
-(async function (params, userID, finish) {
+const department = String(params.department || "").trim()
-  try {
+if (!department) return response.json({ error: "department is required" }, 400)
-    // ... work ...
-    finish({ success: true, data })
+const rows = await db.exec(
-  } catch (error) {
+  `SELECT TOP 50 VacancyID, Position, Town, WhenCreated
-    console.error("my-job failed:", error.message)
+     FROM Vacancy WHERE DepartmentID = ? AND Status = 'L'
-    finish({ success: false, error: error.message })
+    ORDER BY WhenCreated DESC`,
-  }
+  [department]
-})(params, userID, finish)
+)
+return { success: true, department, count: rows.length, vacancies: rows }
 ```
-The leading comment block is optional but parsed: `Key: value` lines become the job's metadata (`author`, `name`, `description`, `version`, `lastupdated`) shown in the jobs list.
+The rules that matter:
+- **The return value is the result.** A plain value or object is sent as JSON; a `response.*` value is sent as that response. A job that returns nothing fails with `JOB-004`.
+- **An uncaught error fails the run immediately** with `JOB-001` and the message — including a rejected promise nobody caught and a throw inside a timer. You no longer need a `try/catch` just to make failures prompt; catch only where you want to turn an error into a particular answer.
+- **`success: false` in a JSON result marks the run as failed.** The caller gets a `JOB-001` error *instead of* your payload, the run is counted as a failure, and a scheduled run is recorded as failed. Use it to mean "this did not work". For "no rows matched" or "not found", return normally, or `response.json ( body, 404 )`.
+- **The header comment is the job's contract.** `Key: value` lines become metadata (`Name`, `Description`, `Author`, `Version`, `Last Updated`), limits (`Runtime`, `Memory`, `CPU`), the access level (`Access`) and the kind of job (`Mode: page`). Don't put `*/` inside it.
+### The header, in full
+```
+/*
+Name: Staff PDF report              shown in the jobs list
+Description: One line on what it does and what it expects
+Author: IQX Limited
+Version: 1.2.0
+Last Updated: 2026-10-05
+Runtime: 60000                      ms of wall clock (default 10000)
+Memory: 512                         MB of heap (default 256)
+CPU: 20000                          ms of CPU time (default 5000)
+Access: owner                       public | user (default) | owner | key
+Mode: page                          when the job renders a browser page
+*/
+```
 ## What a job can see
+Only this. There is no `process`, no `fs`, no `fetch`, no `import`, and `require` knows only the approved list.
 | Global | What it is |
 |---|---|
-| `params` | The request parameters. `GET` → the query string (**every value is a string**, so `Number ( params.limit )` yourself); `POST` → the parsed JSON body. On a scheduled run it is the schedule's stored JSON. `enableLogger` is stripped before the job sees it. |
+| `params` | The request parameters. `GET` → the query string (**every value is a string**, so `Number ( params.limit )` yourself); `POST` → the parsed JSON or form body. A checkbox group arrives as a string for one tick and an array for several. On a scheduled run it is the schedule's stored JSON. The engine's own flags (`enableLogger`, `raw`, `_console`, `_debug`) are stripped before the job sees them. |
-| `userID` | The caller's `IQXNetUserID` for a session call; `null` for an unauthenticated `public_` call. Scheduled runs get the system user ID. |
+| `request` | `{ method, url, path, query, headers, ip, caller }`. `method` tells a page job a landing `GET` from a submitted `POST`; a scheduled run sees `"SCHEDULE"`. `caller` is who is calling: `{ kind, id, name, userClass }` where `kind` is `anonymous`, `user`, `system`, `schedule` or `apikey`. The `Authorization` header is withheld. |
-| `isAuthenticated` | `true` when `userID` is set. |
+| `userID`, `isAuthenticated` | The caller's `IQXNetUserID` and whether there is one — a signed-in user opening a public page is recognised. Scheduled runs get the system user. |
-| `getDB ( )` | Returns `{ exec ( sql, params? ) }`. `exec` runs the SQL as the pears (business-table) user with `?` placeholders bound from the array, and resolves to an **array of row objects** for a `SELECT`. Statements that return no rows resolve to an affected-rows result instead. A failing statement is logged with the job's prefix and rethrown, so wrap it in `try/catch`. |
+| `db.exec ( sql, params? )` | Runs the SQL as the pears (business-table) user with `?` placeholders bound from the array, and resolves to an **array of row objects** for a `SELECT` (`{ affectedRows }` for insert/update/delete). The connection carries the caller's `userstaffid`, so `ExternalCredentialGet ( )` and division-aware procedures work. A failing statement is logged with the job's prefix and rethrown with its SQL code. |
-| `finish ( result )` | Ends the job. A plain value/object is sent as JSON; a `rawResponse ( … )` is sent as-is. Only the first call counts — later calls log a warning and are ignored. |
+| `http ( url, options? )` | Outbound HTTP. `options`: `method`, `headers`, `body` (a string, bytes, or an object which is sent as JSON), `username`/`password` (basic auth), `timeoutMs`, `followRedirects`. Returns `{ status, ok, statusText, headers, text, bytes, json ( ) }`. Only `http`/`https` to the **public internet**: loopback, private ranges, link-local, the hub's own database server and anything on `WEBJOB_HTTP_DENY` are refused, every redirect hop is re-checked, and a site can narrow it to named hosts with `WEBJOB_HTTP_ALLOW`. 10 s timeout and 10 MB response by default, at most 50 calls per run. |
-| `rawResponse ( body, headers )` | Marks `body` (string, `Buffer` or `Uint8Array`) as a raw HTTP response with the given headers. Set `Content-Type`; anything other than `application/json` is sent as bytes. |
+| `require ( name )` | Loads an **approved** module synchronously: `const { format } = require ( "date-fns" )`. Approved today: `date-fns`, `@iqx-limited/quick-pdf`. Anything else throws (`JOB-002`). `await require ( … )` from older jobs still works. |
-| `require ( name )` | Loads an **approved** module. It is really a dynamic `import ( )`, so it returns a *promise* of the module namespace: `const { format } = await require ( "date-fns" )`. A module with a default export is at `.default`. Anything not on the list throws (`JOB-002`). |
+| `console` | `log`/`info`/`debug`/`warn`/`error`. Lines go to the server log prefixed `[WebJob:<name>]`, are kept with the run in history (see *Run history*), stream live to the run dialog, and can be shown under a page with `_console=1`. |
-| `console` | `log`/`info`/`debug`/`warn`/`error`, prefixed `[WebJob:<name>]` in the server log (`debug` level; `warn`/`error` as `warn`, so visible in production). With `enableLogger=1` the lines are also streamed to the caller. |
+| `response` | Builders for non-JSON answers — see *Responses*. |
-| `extendLimits ( cpu, memory, runtime )` | Logs the request. **At present it does not change the 10 s timeout** — the limit is read before the job's code runs — so treat it as documentation of intent until the engine is changed. |
+| `sign ( payload, ttlSeconds? )`, `verify ( token )` | A signed, expiring token (HMAC, keyed from the server secret). `verify` returns the payload or `null`. This is how a multi-step page proves that an earlier step passed — see *Serving pages*. Default TTL 15 minutes. |
-| `setTimeout` / `clearTimeout` / `setInterval` / `clearInterval`, `Date`, `Math`, `JSON`, `parseInt`, `parseFloat`, `isNaN`, `isFinite` | Passed through from Node. |
+| `breakpoint ( )` | Pauses the job when it is being run under the debugger; a no-op otherwise (so is a plain `debugger` statement). See *Debugging*. |
+| `limits` | The resolved `{ runtimeMs, memoryMb, cpuMs }` for this run. |
+| `Buffer`, `URL`, `URLSearchParams`, `TextEncoder`/`TextDecoder`, `atob`/`btoa`, `structuredClone`, `AbortController`, timers | Passed through. |
-Approved modules right now: `date-fns` and `@iqx-limited/quick-pdf` (HTML → PDF, `const { html2pdf } = await require ( "@iqx-limited/quick-pdf" )`). The list is `APPROVED_MODULES` in `src/routes/webJobs.ts`; a module must also be a dependency in WebHub's `package.json`.
+## Who may call a job
-The wrapper is a convenience, not a security boundary — the code runs in the WebHub process with its full privileges. That is why writing, uploading and editing jobs is limited to **OWNER** users, and why a `public_` job should be treated like any other unauthenticated endpoint (validate every parameter, bind every value, return the minimum).
+WebHub works out **who is calling** before anything runs: a signed-in user (from the bearer token or the iqxWEB session cookie), the hub's system token, the scheduler, or nobody. It then checks that against the job's **access level**, the `Access:` header line:
-## Success, failure and `success: false`
+| `Access:` | Who may run it |
+|---|---|
+| `public` | Anyone, with no credentials. Rate limited per caller (`WEBJOB_PUBLIC_RATE`, 60 a minute, with a `Retry-After`). A job named `public_…` is `public` when the header is absent. |
+| `user` | Any signed-in user. **The default** for every other job. |
+| `owner` | OWNER users only. |
+| `key` | Reserved for an API key pinned to this job (keys are planned, not yet issued). Owners can run it; everyone else is refused. |
-The HTTP status does not depend on what the job returns — with one exception. If the JSON result contains `success: false`, WebHub treats the run as **failed**: it is counted as a failure, recorded with the `error` text, and the caller gets a `JOB-001` error line *instead of* the job's own payload. Use that deliberately for "this did not work"; for "no rows matched", return `success: true` with an empty array.
+The system token and the scheduler always pass. A refused call gets `AUTH-SESSION-001` (401, nobody signed in) or `SYS-403` (signed in, but not enough). Inside the job, `request.caller.kind` says which of these happened, so one job can serve a visitor and a consultant differently, and history records the caller for every run.
-A synchronous throw at the top of the file fails the run at once (`JOB-001`), but an error thrown *inside* the async IIFE with no catch is just an unhandled rejection: nothing calls `finish`, so the job sits there until the 10 s limit and fails with `JOB-003`. Always catch and `finish ( { success: false, error } )` so the failure is immediate and explained.
+For an external system that must call a non-public job today, the honest options are the system token, or a `public` job that checks a signed token or shared secret in its parameters with `verify ( )`.
-| Code | Meaning |
+## Limits
-|---|---|
-| `JOB-001` | The job failed: it returned `success: false`, threw, or a query inside it failed. `detail` has the job's text. |
-| `JOB-002` | `require ( )` of a module that is not approved or not installed. |
-| `JOB-003` | Timed out (10 s). Usually a missing `finish` on some path, or an uncaught error. |
-| `JOB-004` | `finish ( )` was called with nothing. |
-Full table on **Web error codes**; every failure carries a `ref` to search the server logs with.
+Jobs are meant to be snappy. The defaults are 10 seconds of wall clock, 256 MB of heap and 5 seconds of CPU time. A job that needs more declares it in its header, which is deliberate: it is an acknowledgement that this job is slow, and it is visible in the jobs list (an *extended* badge).
+Each value is capped by a server maximum (`WEBJOB_MAX_RUNTIME_MS` 300 000, `WEBJOB_MAX_MEMORY_MB` 2048, `WEBJOB_MAX_CPU_MS` 120 000); a higher declaration is clamped and logged. The runtime limit terminates the worker. The memory limit is enforced by V8 on the worker's own heap. The CPU budget is measured per thread from the main thread while the job runs, so a synchronous busy-loop is caught by it rather than by the runtime limit. All three end the run with `JOB-003` and a message naming which limit was hit.
+Two more guards are per node rather than per job. At most `WEBJOB_MAX_CONCURRENT` (8) workers run at once; further requests queue for up to 15 seconds and are then refused with `SYS-429`. And callers without credentials are rate limited as above.
+One deployment caveat: a `--max-old-space-size` in the WebHub process's `NODE_OPTIONS` overrides per-worker heap limits. The server logs a clear warning on the first run if that is the case; remove the flag to get memory limits back.
+## How a run works
-## What the caller receives
+. A request arrives at `GET` or `POST /api/webjobs/<name>`. The name must match `[a-zA-Z0-9_-]+` or it is a `HUB-007`.
+. WebHub resolves the caller and checks the job's access level (previous section).
+. It reads the job's source from the job store (no such job → `SYS-404`), parses the header, resolves the limits, and starts a **worker thread** with an empty environment, its own heap, and a context holding exactly the globals above. Everything the job does to the outside world — `db`, `http`, `console` — is a message to the host, which applies the policy and does the real work.
+. The job runs; the host kills it on the runtime or CPU limit, V8 on the memory limit.
+. The run — parameters, result, console output, outcome, caller — is recorded in history, and the answer is sent back (next section).
-The response is `application/x-ndjson`-style: the first line is metadata, then the data.
+The worker is a containment boundary for mistakes, not a wall against a hostile author — someone who can write a job already has full SQL against the database, and `vm` contexts are escapable by design. That is why writing and uploading jobs is limited to **OWNER** users, and why a `public` job should treat every parameter as untrusted input.
+## Responses
+`return` a plain value for JSON. For anything else use `response`:
+```javascript
+return response.json({ status: "not_found" }, 404)         // JSON with a real status
+return response.html(page)                                  // text/html; charset=utf-8
+return response.text("ok")
+return response.csv(text, "staff.csv")                      // download
+return response.pdf(bytes, "report.pdf")                    // bytes or a base64 string
+return response.file(bytes, "x.png", "image/png")
+return response.redirect("/api/webjobs/public_next?token=" + token)   // 302 + Location
+return response.raw(body, { "Content-Type": "…" }, 200)
+```
+How that reaches the caller depends on who is asking:
+- **A browser** (a navigation, or `Accept: text/html`) gets an ordinary HTTP response: for a `response.*` result the status, the headers and the body; for a plain JSON result just the JSON, pretty-printed as `application/json`, so a job URL pasted into a tab shows its output rather than downloading a file. HTML pages get a job-page `Content-Security-Policy` that allows inline scripts and `https:` assets unless the job sets its own. A failure comes back as a JSON error object with the matching status.
+- **An API client and the iqxWEB run dialog** (`Accept: */*`, or `enableLogger=1`) get the **NDJSON** form: one metadata line, then the data.
 ```
-{"success":true,"jobId":"job_1759…","isMetaData":true,"type":"json","format":"application/json","contentLength":123,"runtime":84}
+{"success":true,"jobId":"job_1759…","isMetaData":true,"type":"json","format":"application/json","status":200,"contentLength":123,"runtime":84}
 {"contacts":[…],"count":3}
 ```
-For a `rawResponse` whose `Content-Type` is not JSON, the metadata line is followed by the raw bytes (HTML text, a PDF…). With `enableLogger=1` (query string on `GET`, body field on `POST`) each `console.*` call arrives first as `{"type":"log","level":"log","message":[…]}` lines, which is how the iqxWEB job page shows a live console. On failure the single line is `{"success":false,"error":"…","code":"JOB-001","ref":"…","runtime":…}` with the matching HTTP status.
+For a raw response the metadata line is followed by the bytes. With `enableLogger=1` each `console.*` call arrives first as a `{"type":"log",…}` line, which is how the run dialog shows a live console. A failure is a single line `{"success":false,"error":"…","code":"JOB-001","ref":"…","runtime":…}` with the matching HTTP status. A script that wants the plain response rather than the stream adds `?raw=1`.
 ```bash
-# public job, GET parameters
 curl "https://hub.example.com/api/webjobs/public_ping?who=gareth"
+curl -X POST "https://hub.example.com/api/webjobs/3cx" -H "Authorization: Bearer $TOKEN" \
-# authenticated job, POST parameters, streamed console
+     -H "Content-Type: application/json" -d '{"limit": 50, "enableLogger": 1}'
-curl -X POST "https://hub.example.com/api/webjobs/3cx" \
+curl -o report.pdf "https://hub.example.com/api/webjobs/pdf-report?raw=1" -H "Authorization: Bearer $TOKEN"
-  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-  -d '{"limit": 50, "enableLogger": 1}'
 ```
-## Managing jobs
+## Serving pages
-All of these live under `/api/webjobs` and (except where noted) need an **OWNER** session. The Owner web-jobs page in iqxWEB is built on them.
+A job can be a small web application: `GET` renders a page, the page `POST`s back to the same job (or another), and the job decides what to render next. There is no server-side session, so **state travels in the request**, and anything that must not be forged — "this person passed the date-of-birth check" — travels as a token from `sign ( )`. The pattern, from `public_deeplink`:
-| Method and path | What it does |
+```javascript
-|---|---|
+if (request.method !== "POST") {                 // landing: ask for the DOB
-| `GET /available` | Every `.js` in the jobs folder with its parsed metadata, run statistics and modified time. |
+  return page(T("dob", { uid, error: "" }))
-| `GET /<name>/code` | The source of one job. |
+}
-| `POST /save` `{ name, code }` | Write (or overwrite) `jobs/<name>.js`. |
+if (params.step === "dob") {                     // check it, mint a token, show the shifts
-| `POST /upload` (multipart: `file`, `name`) | Same, from an uploaded `.js` file (`application/javascript` or `text/javascript`). |
+  if (params.dob !== record.dateofbirth) return page(T("dob", { uid, error: "No match" }))
-| `DELETE /<name>` | Remove the file. |
+  const token = sign({ uid, step: "shifts" }, 15 * 60)
-| `POST /execute` (multipart: `file`, `params` JSON, `enableLogger` `"1"`) | Run an uploaded file **once without saving it** — the scratch-pad behind the editor's Run button. It runs as the caller, named `adhoc_<file>` in history. |
+  return page(T("shifts", { uid, token, shiftrows }))
-| `GET /running`, `POST /<jobId>/stop` | Jobs in flight on this node. *Stop* only removes the entry from the list; it cannot interrupt running JavaScript. |
+}
-| `GET /history?limit&offset&search`, `GET /<jobId>/logs`, `GET /<jobId>/output` | The last 1,000 runs with status, runtime, input and output. |
+if (params.step === "shifts") {                  // only a valid token gets here
-| `GET /metrics` | Totals, success rate, average/fastest/slowest runtime, failures today. |
+  const claim = verify(params.token)
-| `POST /cron` | Re-read the schedule table (below). |
+  if (!claim || claim.uid !== uid) return page(T("dob", { uid, error: "Session expired" }))
+  // … record the selection …
+  return page(T("done", { count }))
+}
+```
-History and metrics are kept in `node-persist` on each node, and every run is also appended to `logs/job_YYYY-MM-DD.json`. Both are **per node**: on a two-node site each WebHub has its own jobs folder, history and schedules, so a job saved through one node has to reach the other (shared volume, or save on both).
+To chain into a *different* job, redirect to it with the token in the URL (`response.redirect`) and `verify` on arrival; a browser can only be redirected with a `GET`, so the token is the only trustworthy thing it can carry.
-## Scheduled jobs
+Templates live in the database, not in files: a `WebJobTemplate ( Name, Html )` table read with `db.exec` and a few lines of `{{placeholder}}` replacement (the deep-link job shows one with escaped and raw placeholders). What you owe a public page, because the engine cannot do it for you: HTML-escape every value that came from the database or the request; check posted IDs against what you offered; keep tokens short-lived; and remember the rate limit is per caller, not per candidate.
-Schedules are rows in the database read through `NetV2GetWebJobSchedule`: an ID, a cron expression, the display name, the file name (without `.js`), a JSON string of parameters and an enabled flag. WebHub loads them at start-up and again on `POST /api/webjobs/cron`; a disabled row stops any running schedule. Each firing runs the job as the system user with the stored parameters (invalid JSON is logged and the job runs with `{}`), then writes the outcome back with `NetV2UpdateJobSchedule` (`pLastStatus` success/failed, `pLastRun`, `pNextRun`).
+Declare a page job with `Mode: page` in its header. The engine treats it like any other job, but the Owner page does the right thing with it: *Open* opens the page in a browser tab instead of the run dialog, the list shows a *page* badge, a *Pages* filter finds them, and *Debug* attaches to the real page (see *Debugging*). A job page is a real page with its own forms and scripts, so it always opens in its own tab — it is not embedded inside iqxWEB.
-A scheduled job has no caller to read its output, so make it report through its own effects — write a row, send an email — and use `success: false` for anything that should show as a failed run.
+To see what a page job logged while using it, add `_console=1` to its URL: the HTML comes back with a collapsible drawer underneath holding the job's console lines with timings, the run time, the method and parameters it saw, and the request `ref`. The drawer is only added for an **OWNER** session (or on a dev server with `WEBJOB_PAGE_CONSOLE=1`); a visitor adding the flag gets the page unchanged.
-## Examples
+## The Owner pages
-### 1. Minimal public job
+**Manage** lists every job on one line — name, badges (`public`, `page`, `extended`, and green *scheduled* / grey *schedule off*), description, last run with its result and a live *running · scheduled · 12 s* indicator when something is in flight — with filters (Public, Pages, Scheduled, Extended limits, Last run failed) and a search box. Expand a row for its limits, access, source and schedules. The **Run** button runs the job in a dialog that shows the console as it happens and then the result (JSON, a text or HTML preview, or a download); its menu offers *Run with parameters…* (remembered per job), *Open in browser*, *Debug…*, *Edit code*, *History* and *Delete*. **Create**, **Upload** (a `.job.js` file) and **Ad-hoc** (run a file once without saving it) are in the toolbar; the editor has *Run* and *Debug* buttons that act on the unsaved code.
-Anyone can call `GET /api/webjobs/public_ping?who=Gareth`. No database, no auth.
+**History** (per job, or per schedule from the Scheduler page) lists the recent runs with status, duration, who or what started them and the node that ran them; click one to replay its console output with timings and see the result it returned, or why it failed.
-```javascript
+**Scheduler** lists schedules the same way: a switch to enable or disable each one, what it runs and when in plain words, the last run (click it for that schedule's history) and the real next fire time, *Run now* with the schedule's parameters, and a menu for history, editing, running with other parameters and deleting. *Add schedule* saves a new schedule switched **off** unless *Enabled* is ticked.
-/*
-Author: IQX Limited
-Name: Ping
-Description: Smoke test that the Web Jobs engine is up
-Version: 1.0.0
-*/
-(async function (params, userID, finish) {
+## Debugging
-  finish({
-    success: true,
-    message: `pong, ${params.who || "stranger"}`,
-    at: new Date().toISOString(),
-    authenticated: isAuthenticated
-  })
-})(params, userID, finish)
-```
-### 2. Parameterised query with validation
+A job can be stepped through line by line from the Owner page: *Debug* in a job's Run menu, or the *Debug* button in the editor (which runs the unsaved code). The dialog shows the source with the paused line highlighted, every variable in scope at that point with its value (expand objects and arrays), the console so far, and **Step**, **Continue** and **Stop**.
-`POST /api/webjobs/vacancies-by-dept` with `{ "department": "NURS", "limit": 25 }`. Note the `?` binds — never splice a parameter into the SQL string — and that `TOP ?` is bound like any other value.
+Put a breakpoint where you want to stop:
 ```javascript
-/*
+const rows = await db.exec("SELECT TOP 5 * FROM Person")
-Author: IQX Limited
+breakpoint()            // or a plain `debugger` statement
-Name: Vacancies by department
+const names = rows.map(r => r.Name)
-Description: Live vacancies for one department, newest first
-Version: 1.0.0
-*/
-(async function (params, userID, finish) {
-  const department = String(params.department || "").trim()
-  const limit = Number(params.limit) || 50
-  if (!department) {
-    return finish({ success: false, error: "department is required" })
-  }
-  if (!Number.isInteger(limit) || limit < 1 || limit > 500) {
-    return finish({ success: false, error: "limit must be 1-500" })
-  }
-  const db = getDB()
-  try {
-    const rows = await db.exec(
-      `SELECT TOP ? VacancyID, Position, Town, Salary, WhenCreated
-         FROM Vacancy
-        WHERE DepartmentID = ? AND Status = 'L'
-        ORDER BY WhenCreated DESC`,
-      [limit, department]
-    )
-    finish({ success: true, department, count: rows.length, vacancies: rows })
-  } catch (error) {
-    console.error("vacancies-by-dept:", error.message)
-    finish({ success: false, error: "Query failed", details: error.message })
-  }
-})(params, userID, finish)
 ```
-### 3. Returning a file — CSV download
+Without any breakpoint the run pauses before the first line. Stepping pauses before each statement in the job body and inside `async` functions (including `if`/`for`/`try` blocks within them); code inside synchronous callbacks such as `rows.map ( r => … )` runs without pausing, because the pause is an `await`. A not-yet-initialised `let` shows as `<tdz>`, long strings and deep objects are truncated, and *Stop* ends the run with a `JOB-001` "Stopped by the debugger at line N". The runtime limit is held while the job is parked, so sitting on a breakpoint does not time the job out; a session left alone for 10 minutes is cleaned up.
-`rawResponse` turns the result into a download. The same pattern with `Content-Type: text/html` serves a report page, and with a `Buffer` from `html2pdf` serves a PDF.
+For a **page job**, *Debug* attaches to the real flow: the dialog opens *live*, waiting; **Open page** opens the job in a new tab with the debugger attached, and every request that tab makes — the landing `GET`, each form `POST` back, a redirect — runs under the debugger and pauses in the dialog while the browser waits for its page. The console shows each request arriving and being answered. The tab is tagged with the session (first in the URL as `_debug=`, then in a cookie scoped to `/api/webjobs` so the page's own posts keep it); other visitors are untouched. The switch at the bottom chooses between pausing on the first line of every request and only at `breakpoint ( )` / `debugger`. Closing the dialog detaches, stops any paused request, and the tab's next request runs normally.
-```javascript
+`breakpoint ( )` and `debugger` are **no-ops in a normal run** — a normal run is never instrumented and `breakpoint` is an empty function in the sandbox — so one left in a job does nothing in production; only an OWNER starting a debug session makes them pause.
-/*
-Author: IQX Limited
-Name: Staff CSV
-Description: Current staff list as a CSV file
-Version: 1.0.0
-*/
-(async function (params, userID, finish) {
+## Scheduled jobs
-  const csvCell = (v) => {
-    const s = v === null || v === undefined ? "" : String(v)
-    return /[",\n]/.test(s) ? `"${s.replace(/"/g, '""')}"` : s
-  }
-  try {
+Schedules are rows of the `IQXNetWebJobSchedule` table: an ID, a cron expression, a display name, the job's name, a JSON string of parameters and an enabled flag. The Scheduler page manages them through the `NetV2…JobSchedule` procedures as the signed-in user; WebHub itself reads and updates the table directly, because those procedures are gated on the caller's user class and the hub runs as the SYSTEM user. WebHub loads the enabled rows at start-up and on `POST /api/webjobs/cron` (which reports how many it loaded); a disabled or deleted row stops its cron job.
-    const rows = await getDB().exec(
-      "SELECT StaffID, Name, EMail, manager FROM Staff ORDER BY Name"
-    )
-    const header = rows.length ? Object.keys(rows[0]) : []
-    const lines = [header.join(",")]
-      .concat(rows.map((r) => header.map((k) => csvCell(r[k])).join(",")))
-    finish(
+Each firing runs the job as the system user with the stored parameters (invalid JSON is logged and the job runs with `{}`), sees `request.method === "SCHEDULE"` and `request.caller.kind === "schedule"`, is recorded in the job's history with the schedule's name, and writes `LastRun`, `NextRun` and `LastStatus` back to the row. A scheduled job has no caller to read its output, so it should report through its own effects and use `success: false` for anything that should show as a failed run — the console output and result are kept in history either way, which is where to look when a scheduled job starts going wrong.
-      rawResponse(lines.join("\r\n"), {
-        "Content-Type": "text/csv",
-        "Content-Disposition": `attachment; filename=staff_${Date.now()}.csv`
-      })
-    )
-  } catch (error) {
-    finish({ success: false, error: error.message })
-  }
-})(params, userID, finish)
-```
-### 4. Using an approved module and the caller's identity
+On a multi-node site every node loads every schedule today, so a schedule fires once per node.
-`date-fns` is awaited out of `require`. The job also refuses to run for an anonymous caller, which is only possible because its name does *not* start with `public_` — WebHub has already checked the session, so `userID` is always set here; the check is belt and braces.
+## Run history
-```javascript
+Every run — manual, scheduled, from the editor or under the debugger — is recorded with its parameters, its result (text results up to 64 KB), what it wrote to the console (up to 300 lines / 32 KB, with timings), how it ended, who or what started it, and the node it ran on. The History dialog replays this, and the per-job stats on the Manage page and the metrics cards are computed from it.
-/*
-Author: IQX Limited
-Name: My week
-Description: The calling consultant's placements starting in the next 7 days
-Version: 1.0.0
-*/
-(async function (params, userID, finish) {
+History lives in **MongoDB** by default (`WEBJOB_HISTORY_STORE=mongo`): the `webjob_runs` collection in the hub's own `IQXWebHub` database on the Mongo server WebHub already uses. It is not the recruitment `IQXWeb` database, and nothing goes to the IQX SQL database. Every node writes there and reads the same history; runs expire after `WEBJOB_HISTORY_RETENTION_DAYS` (90) via a TTL index. `WEBJOB_HISTORY_STORE=local` keeps a per-node `node-persist` file instead (newest `WEBJOB_HISTORY_MAX` runs).
-  if (!userID) {
-    return finish({ success: false, error: "Authentication required" })
-  }
-  try {
+## Where the jobs live
-    const { format, addDays } = await require("date-fns")
-    const from = format(new Date(), "yyyy-MM-dd")
-    const to = format(addDays(new Date(), 7), "yyyy-MM-dd")
-    const rows = await getDB().exec(
+`WEBJOB_STORE=file` (the default) keeps jobs as `jobs/<name>.job.js` on each node's disk, so on a multi-node or multi-container site a job saved through one node has to reach the others (shared volume, or save on both). A plain `.js` in the folder is not a job and is reported once in the log at start-up.
-      `SELECT p.PlacementID, p.StartDate, c.Name AS Company
-         FROM Placement p
-         JOIN Company c ON c.CompanyID = p.CompanyID
-         JOIN IQXNetUser u ON u.StaffID = p.StaffID
-        WHERE u.IQXNetUserID = ?
-          AND p.StartDate BETWEEN ? AND ?
-        ORDER BY p.StartDate`,
-      [userID, from, to]
-    )
-    finish({ success: true, from, to, placements: rows })
+`WEBJOB_STORE=db` keeps them in the `IQXNetWebJob` table instead (`Name`, `Code`, `UpdatedAt`, `UpdatedBy`), created on first use and read per run, so every node sees the same jobs the moment one is saved and nothing needs to be on disk. To move an existing node over: set the variable, restart, then `POST /api/webjobs/store/import-files` once (OWNER) to copy its folder into the table (`{"overwrite": true}` to replace rows that already exist). `GET /api/webjobs/store` says which store a node is using, and the Manage page shows it in the list header.
-  } catch (error) {
-    console.error("my-week:", error.message)
-    finish({ success: false, error: error.message })
-  }
-})(params, userID, finish)
-```
-### 5. A job written to be scheduled
+## Managing jobs over HTTP
-Scheduled with cron `0 7 * * 1-5` and parameters `{"days": 3}`. It has no caller, so it leaves a trail in the log and signals failure the only way a schedule can see — `success: false` — which `NetV2UpdateJobSchedule` records as a failed run.
+All under `/api/webjobs`, OWNER session unless noted. The Owner pages are built on them.
-```javascript
+| Method and path | What it does |
-/*
+|---|---|
-Author: IQX Limited
+| `GET /available` | Every job with metadata, resolved limits, `access`, `mode`, run statistics, where it is stored and when it changed. |
-Name: Expiring compliance reminder
+| `GET /<name>/code` | The source of one job. |
-Description: Flags candidate documents expiring within N days (default 3)
+| `POST /save` `{ name, code }` | Write (or overwrite) a job. |
-Version: 1.0.0
+| `POST /upload` (multipart `file`, `name`) | Same, from an uploaded `.job.js`. |
-*/
+| `DELETE /<name>` | Remove a job. |
+| `POST /execute` (multipart `file`, `params`, `enableLogger`) | Run an uploaded file once without saving it — the editor's Run and the Ad-hoc button. Always streams. |
+| `GET /running`, `POST /<jobId>/stop` | Runs in flight on this node (`jobName`, `source`, `runtime`); the Owner pages poll it. *Stop* removes the entry; the worker is killed by its own limits. |
+| `GET /history?limit&offset&search`, `GET /<jobId>/output`, `GET /<jobId>/logs` | Run history. `/output` returns a run's parameters, result, console lines and outcome. |
+| `GET /metrics` | Totals, success rate, runtimes, failures today, `workersRunning` / `workersQueued` / `workersMax`, and which stores this node uses. |
+| `POST /debug/<name>` `{ params, code?, stepFromStart?, method? }` | Run one simulated request under the debugger (streams `debug` / `pause` lines). |
+| `POST /debug/<name>/attach` `{ stepFromStart? }` | Attach the debugger to real requests tagged with the session it returns. |
+| `POST /debug-session/<id>/command` `{ command }` | `step`, `continue`, `stop`, `config` (`stepFromStart`) or `detach`. |
+| `GET /store`, `POST /store/import-files` | Which job store this node uses; copy its jobs folder into the database store. |
+| `GET /schedules/status` | The schedules this node has loaded, with each one's real next fire time. |
+| `POST /cron` | Re-read the schedule table; reports how many schedules loaded. |
-(async function (params, userID, finish) {
+## Error codes
-  const days = Number(params.days) || 3
-  const db = getDB()
-  try {
+| Code | Meaning |
-    const expiring = await db.exec(
+|---|---|
-      `SELECT PersonID, DocumentTypeID, ExpiryDate
+| `JOB-001` | The job failed: it threw, returned `success: false`, a query inside it failed, it has a syntax error, or it was stopped from the debugger. `detail` has the message. |
-         FROM PersonDocument
+| `JOB-002` | `require ( )` of a module that is not approved or not installed. |
-        WHERE ExpiryDate BETWEEN today() AND dateadd(day, ?, today())
+| `JOB-003` | A limit was hit: runtime, memory or CPU. The message says which. |
-          AND ReminderSent = 0`,
+| `JOB-004` | The job finished without a result (nothing was returned). |
-      [days]
+| `AUTH-SESSION-001`, `SYS-403` | The caller may not run this job (nobody signed in / not enough) — see *Who may call a job*. |
-    )
+| `SYS-429` | Too many: the rate limit for callers without credentials, or the node's worker queue is full. |
-    let flagged = 0
+Full table on **Web error codes**; every failure carries a `ref` to search the server logs with.
-    for (const doc of expiring) {
-      await db.exec(
-        `UPDATE PersonDocument SET ReminderSent = 1
-          WHERE PersonID = ? AND DocumentTypeID = ?`,
-        [doc.PersonID, doc.DocumentTypeID]
-      )
-      flagged++
-    }
-    console.log(`compliance reminder: ${flagged} document(s) flagged (${days} days)`)
-    finish({ success: true, flagged, days })
-  } catch (error) {
-    console.error("compliance reminder failed:", error.message)
-    finish({ success: false, error: error.message })
-  }
-})(params, userID, finish)
-```
-(Table and column names in examples 2, 4 and 5 are illustrative — check them against the schema before using.)
-## Things that bite
+## Trying it
-- **Top-level `await` does not work.** The file is a function body, not a module. Wrap in `(async function () { … })()` as every example does. The old README example without the wrapper will not run.
+The repo ships a small test pack; run each and compare with its header. `_test-limits` returns the resolved limits, and `?what=runtime`, `?what=cpu` or `?what=memory` shows each limit ending a run with `JOB-003`. `_test-failures` shows how runs fail and how they don't: `?mode=throw` (default), `async` and `timer` all end at once with `JOB-001`, `?mode=false` is a `success: false` failure, `?mode=404` is a 404 that is *not* a failure, `?mode=ok` succeeds and echoes the caller. `_test-sandbox` lists what the sandbox exposes — every probe should read `undefined` or refused. `_test-http` exercises the outbound network policy. `public_test-page` is a page job to open in a browser (inline script, a form that posts back, a console link), and `public_ping` is the smallest possible public job. The engine's unit tests are `npm run test:unit` (`src/routes/webJobEngine.test.ts`).
-- **`require` is async.** `const { format } = await require ( "date-fns" )`. Forgetting the `await` gives you a promise with no `format` on it.
-- **Call `finish` on every path**, including inside `catch`. A path that doesn't is a 10-second wait and a `JOB-003`.
-- **`success: false` replaces your payload** with an error for the caller. Don't use it for an empty-but-valid result.
-- **`GET` parameters are strings.** `params.limit === 50` is never true on a `GET`; convert first.
-- **The 10 s limit is real** and `extendLimits` does not currently raise it. Keep jobs short; for long work, write the outcome somewhere and poll.
-- **Two nodes, two folders.** Save or upload a job on both, or put `jobs/` on a shared volume; schedules fire on every node that has the row loaded.
-- **Hot reload is free.** The file is read on every request, so a saved change is live immediately — on that node.
-- **No `process`, `fs` or arbitrary `require`** are offered, and nothing in the wrapper stops a determined author reaching them. Jobs are trusted code; keep authoring to OWNER.
 ## For developers
-- Engine: `src/routes/webJobs.ts` (`executeWebJob`, `APPROVED_MODULES`, `rawResponse`). Routes, history, metrics and the cron scheduler: `src/routes/jobsManagement.ts`, mounted at `/api/webjobs` in `src/routes.ts`. Name and `public_` rules: `src/routes/webJobAuth.ts`.
+- `src/routes/webJobWorker.ts` — the JavaScript that runs inside the worker (kept as a string, started with `eval: true`): the sandbox, the async wrapper, the bridge stubs, `response`, `sign`/`verify`, `breakpoint`.
-- The scheduler is started from `server.ts` (`updateSchedulesFromDb`) at boot.
+- `src/routes/webJobEngine.ts` — the host: starts the worker (empty env, `resourceLimits`), answers bridge calls, enforces limits, the concurrency gate, `resolveJobLimits`, the debugger pause protocol. No database or Fastify imports, so it is unit-testable (`webJobEngine.test.ts`).
-- Error classification is `toJobError`; a new failure kind needs a `JOB-00n` entry in `@iqx-limited/web-errors` and a row on **Web error codes**.
+- `src/routes/webJobHttp.ts` — the network policy and `http ( )` implementation.
-- Tests: `src/test/webjobs.ts` (note it still calls `test` unauthenticated, which the `public_` rule now refuses — rename the fixture or add a token).
+- `src/routes/webJobDebug.ts` — the debugger's source instrumentation (acorn) and value serialiser.
+- `src/routes/webJobAuth.ts` — `JobCaller` and `resolveJobCaller` (the one place identity is read), the `Access:` policy, the rate limit. An API-key scheme is one new branch here.
+- `src/routes/webJobStore.ts` — where job source lives: the `jobs/` folder or the `IQXNetWebJob` table.
+- `src/routes/webJobHistory.ts` — run history and metrics: MongoDB (`IQXWebHub.webjob_runs`) or `node-persist`.
+- `src/routes/webJobs.ts` — `executeWebJob` wiring (store, metadata, `query` as pears with `userstaffid`, logging), `toJobError`, approved modules.
+- `src/routes/jobsManagement.ts` — the routes: raw mode vs NDJSON, the `_console` drawer, the debug sessions, the cron scheduler.
+- Settings are `WEBJOB_*` in `.env.sample`. A new failure kind needs a `JOB-00n` entry in `@iqx-limited/web-errors` and a row on **Web error codes** — memory and CPU currently share `JOB-003`.
+- Not done yet, by design: API keys (`Access: key`); scheduler on the cluster leader only; DNS-rebinding pinning in `http ( )`.
 </markdown>