< back to series
DAY 9

Supply Chain Hardening

Mar 26, 2026

Update (March 29, 2026): All three v3 controls are confirmed operational. Tool execution runs as tito-toolbox (uid 997) via the tito-tools wrapper. The AppArmor profile is loaded in enforce mode. nftables UID 997 rules are active and persist across reboots via nftables.service. The implementation matches the architecture described below.

Context

On March 24, 2026, the litellm package on PyPI was backdoored. Version 1.82.8 shipped a .pth file that executed on every Python startup. It collected environment variables, SSH keys, and cloud credentials, encrypted them, and exfiltrated them to an attacker-controlled domain. The package was live for about three hours.

litellm is a common dependency. The prospecting tools on this machine use Python. This triggered a review of what v2 protected and what it did not.


What v2 covered

The v2 proxy holds real API keys under the openclaw-secrets OS user. A .pth file reading tito's environment would find proxy-managed placeholders and no usable API keys.

What v2 did not cover:

  1. SSH deploy keys still live in tito's ~/.ssh/. A malicious package could read them directly.
  2. Raw socket bypass: HTTP_PROXY is an environment variable. A package using Python's socket module directly ignores it and opens a connection that never touches mitmproxy.

Both of these are addressed in v3.


v3 changes

New OS user: tito-toolbox

Tool scripts no longer run as tito. A new system user is created:

bashsudo useradd --system --no-create-home --shell /usr/sbin/nologin --uid 997 tito-toolbox

UID 997. No shell. No home directory. Tool scripts are exec'd via sudo -u tito-toolbox from the tito-tools wrapper. Because tito-toolbox is a different OS user, it cannot read tito's files — including ~/.ssh/ — regardless of any other control.

bash# Verify: tito-toolbox cannot read tito's SSH key
sudo -u tito-toolbox cat /home/tito/.ssh/openclaw-tito-key   # → Permission denied

tito-tools wrapper

OpenClaw is configured to use /usr/local/bin/tito-tools as its exec shell. Every tool call goes through this wrapper before anything runs:

bash#!/bin/bash
# Shell wrapper: routes OpenClaw tool exec through tito-toolbox user
# Called by OpenClaw as: tito-tools -c "<command>"
# Also supports direct python invocation: tito-tools script.py

export TITO_TOOL_TIER=restricted
export HTTP_PROXY=http://127.0.0.1:18080
export HTTPS_PROXY=http://127.0.0.1:18080
export http_proxy=http://127.0.0.1:18080
export https_proxy=http://127.0.0.1:18080

if [ "$1" = "-c" ]; then
    shift
    exec sudo -u tito-toolbox /bin/bash -c "$*"
else
    exec sudo -u tito-toolbox /home/tito/.openclaw/workspace/.venv/bin/python3 "$@"
fi

The wrapper sets proxy env vars so HTTP from the tool tier goes through mitmproxy, sets TITO_TOOL_TIER=restricted as a tier tag, and hands execution to tito-toolbox via sudo. AppArmor and nftables enforce the boundary after the handoff.

nftables kernel-level network fence

The HTTP_PROXY bypass is closed by locking UID 997 at the kernel level. The relevant section of /etc/nftables.conf:

table inet filter {
    chain input {
        type filter hook input priority filter;
    }
    chain forward {
        type filter hook forward priority filter;
    }
    chain output {
        type filter hook output priority filter;
        # tito-toolbox: must route through local proxy only (doc 0019 v3)
        meta skuid "tito-toolbox" ip daddr 127.0.0.1 tcp dport 18080 accept
        meta skuid "tito-toolbox" reject
    }
}

These two rules say UID 997 may only open TCP connections to 127.0.0.1:18080. Everything else is rejected at the kernel before the packet hits the wire. A raw socket() call to any external host gets rejected before it ever becomes a real outbound connection.

nftables.service is enabled, so rules load automatically on boot.

AppArmor filesystem jail

The tito-tools wrapper runs under an AppArmor profile in enforce mode. This is the actual profile in /etc/apparmor.d/usr.local.bin.tito-tools:

#include <tunables/global>

/usr/local/bin/tito-tools {
  #include <abstractions/base>
  #include <abstractions/bash>
  #include <abstractions/python>
  #include <abstractions/nameservice>

  /usr/local/bin/tito-tools  r,
  /bin/bash                  ix,
  /usr/bin/bash              ix,

  # exec sudo unconfined — switches to tito-toolbox; nftables + OS perms take over
  /usr/bin/sudo              ux,

  /home/tito/.openclaw/workspace/.venv/bin/python3  ix,
  /usr/bin/python3.13                               ix,
  /usr/bin/python3                                  ix,

  /home/tito/.openclaw/workspace/.venv/             r,
  /home/tito/.openclaw/workspace/.venv/**           r,
  /home/tito/.openclaw/workspace/.venv/lib/**       mr,

  /home/tito/.openclaw/workspace/                   r,
  /home/tito/.openclaw/workspace/tools/             r,
  /home/tito/.openclaw/workspace/tools/**           r,
  /home/tito/.openclaw/workspace/data/              r,
  /home/tito/.openclaw/workspace/data/**            rw,
  /home/tito/.openclaw/workspace/memory/            r,
  /home/tito/.openclaw/workspace/memory/**          rw,
  /home/tito/.openclaw/workspace/*.md               rw,

  # Env config (read-only — proxy credentials)
  /etc/environment.d/openclaw.conf                  r,

  /tmp/**  rw,

  /lib/x86_64-linux-gnu/**        mr,
  /lib/**                         mr,
  /usr/lib/x86_64-linux-gnu/**    mr,
  /usr/lib/python3/**             r,
  /usr/lib/python3.13/**          r,
  /usr/local/lib/python3.13/**    r,

  network inet stream,
  network inet6 stream,

  /proc/*/mounts                              r,
  /proc/sys/kernel/ngroups_max                r,
  /sys/kernel/mm/transparent_hugepage/enabled r,

  deny /etc/shadow  rw,
  deny /root/**     rw,
  deny network raw,
}

Key points:

  • /home/tito/.ssh/ is not listed — denied by default (whitelist model)
  • deny network raw blocks raw socket creation as a second layer on top of nftables
  • network inet stream is allowed — but nftables restricts where those streams can go
  • /usr/bin/ssh and /usr/bin/git are not listed as executables — subprocess.run(['ssh', ...]) gets a denied exec
  • sudo is exec'd ux (unconfined) — once it switches to tito-toolbox, the OS user boundary and nftables take over

Proxy allowlist

Every request that reaches mitmproxy must target one of these hosts. Anything else gets a 403:

HostCredentials injected
places.googleapis.comX-Goog-Api-Key
striping-app.*.workers.devAuthorization, CF Access headers
api.cloudflare.comAuthorization (D1 token)
api.telegram.orgNone (token is in the URL path)
pypi.orgNone
files.pythonhosted.orgNone

Enforcement is purely domain-based. The proxy blocks any unlisted host regardless of what headers are present.

Note: api.telegram.org is in NO_PROXY in /etc/environment.d/openclaw.conf, so Telegram traffic from the gateway never hits mitmproxy at all. The entry in the inject rules table exists as documentation, not enforcement.

Gateway tier and NO_PROXY

The gateway process (tito user, Node.js) also reads HTTP_PROXY from the system environment. To keep AI provider API calls working, GitHub Copilot and ChatGPT are bypassed via NO_PROXY:

bashNO_PROXY=api.telegram.org,api.individual.githubcopilot.com,chatgpt.com

These three hosts go direct. Everything else from the gateway routes through the proxy and is subject to the domain allowlist.

SSRF policy on the gateway tier

The gateway has unrestricted outbound by design because it runs web_fetch, browser, and web_search. That creates an SSRF risk: a fetched page could try to steer the agent toward http://169.254.169.254/ (cloud metadata) or an internal LAN address.

The SSRF guard is enabled in openclaw.json:

json"browser": {
  "ssrfPolicy": {
    "dangerouslyAllowPrivateNetwork": false
  }
}

This blocks RFC1918 ranges, localhost, and link-local addresses from gateway-tier fetches. Public internet access is unaffected.


Full v3 architecture


Defense layers

A compromised pip package running under tito-toolbox needs to defeat all of the following independently:

LayerWhat it stops
OS user isolation (tito-toolbox)Reading tito's SSH keys and config files
nftables UID matchDirect outbound TCP — kernel-enforced, raw sockets included
AppArmor whitelistFilesystem reads outside workspace; exec of ssh/git/curl
Proxy allowlistHTTP to hosts not on the 6-host list

These controls stack. Getting past one does not automatically get past the rest.


What v3 still does not fix

Telegram bot token: TELEGRAM_BOT_TOKEN remains in tito's environment. Telegram embeds the token in the URL path — the proxy cannot inject a URL path segment, and api.telegram.org is in NO_PROXY. This is the one real credential the agent holds directly.

SSH key custody: Deploy keys live in tito's ~/.ssh/. The AppArmor profile prevents tito-toolbox from reading them, and nftables prevents exfiltration if they were read by gateway-tier code. The cleaner fix would be a dedicated deploy user with no overlap with tito's home directory. That migration is noted but not implemented in v3. Day 16 adds a narrow host-routed tito-git path for protected repo operations, but that is an operational bridge, not full SSH key isolation.

Tradeoffs

nftables over iptables: Both enforce kernel-level UID-based rules. nftables is the current Linux standard (iptables is legacy on Debian 12). Rules are defined in /etc/nftables.conf and loaded on boot by nftables.service. No extra persistence setup is required.

Separate OS user over only AppArmor: AppArmor profiles can be modified by root, and a profile miss would expose files. OS-level user separation is enforced by the kernel permission model. A process running as tito cannot override it. The two controls are independent.

tito-toolbox has no shell: A no-shell, no-home system user cannot be logged into interactively and has no init files that could be modified to change behavior. The only way to exec code as tito-toolbox is through the sudo -u tito-toolbox allowlist entry, which is controlled by tito's sudoers config.


What's next

Day 10 covers three data paths to Cloudflare: the CRM API, direct D1 SQL queries, and R2 aerial images.