HELP

Debugging

Check and fix a device running nodexOS from its terminal.

Debug the supervisor

Every Linux device runs nodexa-agent, the supervisor that registers the device, sends heartbeats, pulls new releases and runs your containers. When a device shows offline or a release doesn't arrive, start here. Open the device in the dashboard and go to Logs & Terminal to get a shell on it.

These commands are for nodexOS on Linux devices. ESP32 boards have no shell; use their Logs instead.

Is the supervisor running?
Look for Active: active (running). The last lines of the output show its most recent log messages.
```
systemctl status nodexa-agent.service --no-pager
```

What has it logged?

The last 20 lines from the current boot (-b), printed without a pager so they work in the dashboard terminal:

journalctl -u nodexa-agent.service -b --no-pager -n 20

Other useful views:

journalctl -u nodexa-agent.service -f                  # follow live (Ctrl+C to stop)
journalctl -u nodexa-agent.service -b -p warning       # this boot, warnings and errors only
journalctl -u nodexa-agent.service -b -1 --no-pager    # previous boot, e.g. after a crash

Restart it
```
systemctl restart nodexa-agent.service
```
systemd already restarts the supervisor 2 seconds after it crashes, and its 30-second watchdog restarts it if it hangs. If it crashes 5 times within a minute, systemd gives up and the status shows failed. Clear that and start it again once you've found the cause in the logs:
```
systemctl reset-failed nodexa-agent.service
systemctl start nodexa-agent.service
```
Check the rest of the OS
Lists any system service that failed to start:
```
systemctl --failed --no-pager
```

nodexOS services

Service	What it does
`nodexa-agent.service`	The supervisor: cloud connection, releases, containers, device actions
`nodexa-os-commit.service`	Marks a freshly updated OS as good once the supervisor runs on it; otherwise the device rolls back
`nodexa-expand-storage.service`	Grows the data partition to fill the disk, before the supervisor starts
`nodexa-flash-emmc.service`	BeagleBone: copies the SD card onto the onboard eMMC on first boot

Device commands (nodex)

On the device, nodex is a short name for nodexactl (nodexa works too). It's a different tool from the nodex CLI you push releases with, and it only talks to the local supervisor.

Everyday checks

nodex status                      # overall device status
nodex ps                          # containers and their state
nodex logs --tail 50 -f <name>    # a container's output (Ctrl+C to stop)
nodex exec -it <name> sh          # shell inside a running container
nodex network ls                  # container networks

Device

Command	What it does
`nodex status`	Overall device status
`nodex version`	nodexOS and supervisor versions
`nodex identity`	The device's stable identity
`nodex health`	Detailed system health
`nodex network`	Network interfaces and Tailscale VPN status

Containers

Command	What it does
`nodex ps`	List containers (same as `nodex containers`)
`nodex logs [--tail N] [-f] <name>`	A container's output; `-f` keeps streaming
`nodex exec [-it] <name> [cmd…]`	Run a command inside a running container; also `-d`, `-u <user>`, `-w <dir>`
`nodex stats <name>`	CPU and memory use of a container
`nodex inspect <name>`	Full details of a container or network
`nodex containers start\|stop <name>`	Start or stop a container
`nodex rm <name>`	Remove a container with its logs and volumes; this can't be undone

Images

Command	What it does
`nodex images`	Images pulled by deployments
`nodex rmi <name>`	Remove a cached image; running containers aren't affected

Networks

Command	What it does
`nodex network ls`	List container networks (also `network ps`)
`nodex network create <name>`	Create a network; options `--driver` (default `bridge`), `--subnet`, `--gateway`
`nodex network inspect <name>`	Network details
`nodex network rm <name>`	Remove a network

Supervisor updates

Command	What it does
`nodex update status`	State of the supervisor's own update
`nodex update apply <url\|file>`	Install a supervisor update; optional `--sha256` and `--version`
`nodex update rollback`	Go back to the supervisor version that shipped with the OS image

nodexOS specification

Base	Embedded Linux built with the Yocto Project (Poky, scarthgap release), systemd init
Supervisor	`nodexa-agent`, a systemd service with a 30-second watchdog
Containers	OCI containers run with `runc`; images pulled with skopeo and umoci. No Docker daemon on the device.
Root filesystem	Read-only. Changes made outside `/var/lib/nodexa` don't survive a reboot or update.
Persistent data	Separate `nodexa-data` partition at `/var/lib/nodexa`, kept across OS updates
OS updates	BeagleBone: A/B root slots. A new OS gets 3 boots to start the supervisor, or the device falls back to the previous slot.
Device identity	Derived from the hardware, so the same board keeps its device ID after a reflash
Remote access	Tailscale VPN client included
Hardware	BeagleBone Black and Variscite DART-6UL (both ARMv7, `linux/arm/v7`)

Where things live

/etc/nodexa/nodexa.conf          main config (read-only, baked into the image)
/run/nodexa/agent.sock           nodexa-agent's local API (what nodex talks to)
/var/lib/nodexa/                 persistent data partition, survives OS updates
  identity/identity.json         device identity
  state/state.json               boot count, first/last boot time
  containers/                    one OCI bundle per container
  cache/                         scratch space

Common problems

Device shows offline in the dashboard: Run systemctl status nodexa-agent.service. If it's running, check the connection with nodex network and look for connection errors in the supervisor's journal.
A new release never reaches the device: Watch the journal (journalctl -u nodexa-agent.service -f) around the next update check for pull errors. nodex images shows whether the new image arrived.
A container keeps restarting: nodex logs --tail 100 <name> shows why it exits, and nodex inspect <name> its configuration. exec format error means the image was built for the wrong CPU.
nodex says it can't reach the agent: The command talks to the supervisor over /run/nodexa/agent.sock, so the supervisor has to be running. Check it with systemctl status and restart it as above.