Snoopy Runbook
Snoopy Logger Runbook & Cheatsheet
Snoopy is a small LD_PRELOAD library that interposes on execv()/execve() and logs every program execution, with arguments, to syslog. Part one deploys it; part two is the configuration reference: datasources, filter chains, outputs, and log routing.
Decide & pre-flight
Pick the right tool before deploying, then confirm the host can build or package-install it.
| Need | Use |
|---|---|
| See what a web app / cron / CI actually shells out to | snoopy |
| Readable “who ran what, where” trail across admins | snoopy |
| Tamper-resistant audit trail, compliance evidence | auditd |
| Catch a hostile user or malware evading logging | auditd / eBPF |
| Zero-config on statically linked binaries | auditd / eBPF |
Both is fine Snoopy is cheap and its output is far more readable than auditd EXECVE records. Running both is a common pattern: Snoopy for humans, auditd for evidence.
# Debian / Ubuntu apt-get install -y build-essential autoconf automake libtool git wget # Fedora / RHEL dnf install -y gcc make autoconf automake libtool git wget # Gentoo (build tools already present) emerge dev-build/autoconf dev-build/automake dev-build/libtool
Install
Three routes. Distro package is simplest; the upstream installer gets you the current release; source gives you build-time control over datasources and filters.
apt-get install -y snoopy # Debian / Ubuntu dnf install -y snoopy # Fedora
Caveat Distro packages often lag upstream and may ship older helper scripts (snoopy-enable / snoopy-disable) instead of snoopyctl. Check with snoopyctl version.
Downloads, builds, installs, and enables in one pass. Read the script before running it, as with any piped installer.
wget -q -O install-snoopy.sh \ https://github.com/a2o/snoopy/raw/install/install/install-snoopy.sh chmod 755 install-snoopy.sh # latest stable release sudo ./install-snoopy.sh stable # or track development master instead # sudo ./install-snoopy.sh git-master
Only datasources and filters compiled in are usable at runtime. Build everything unless you are trimming for size.
wget https://github.com/a2o/snoopy/releases/latest/download/snoopy.tar.gz tar -xzf snoopy.tar.gz && cd snoopy-* ./configure --sysconfdir=/etc --enable-everything make sudo make install sudo snoopyctl enable
Trimmed build Individual toggles exist, e.g. –enable-filtering –with-filter-chain=”only_uid:0″ to bake a default filter into the binary.
Enable
Adds libsnoopy.so to /etc/ld.so.preload so the dynamic linker preloads it into every dynamically linked process.
snoopyctl enable snoopyctl status cat /etc/ld.so.preload
Already-running processes Only newly executed programs pick Snoopy up. Reboot, or at minimum restart the services you care about, for full coverage.
Best way to test, or to watch exactly one application without logging the whole host.
# Everything this shell spawns gets logged; exit to stop LD_PRELOAD=/usr/local/lib/libsnoopy.so bash # Or wrap a single service unit: # Environment=LD_PRELOAD=/usr/local/lib/libsnoopy.so
A broken ld.so.preload can make a system unbootable or leave every command failing. Keep a root shell open and a backup on disk.
cp /etc/ld.so.preload /root/ld.so.preload.bak ls -l /usr/local/lib/libsnoopy.so # confirm the path is real # Recovery if things break: boot single-user / rescue, then # mv /etc/ld.so.preload /etc/ld.so.preload.off
Never put a path in /etc/ld.so.preload that does not exist or lives on a filesystem mounted after root. Every dynamically linked binary will emit loader errors.
Configure /etc/snoopy.ini
Snoopy re-reads its config on every invocation, so changes apply instantly with no restart.
[snoopy] ; Who ran what, from where message_format = "[uid:%{uid} user:%{username} login:%{login} sid:%{sid} tty:%{tty} cwd:%{cwd} bin:%{filename}]: %{cmdline}" ; devlog writes straight to /dev/log - the safe default output = devlog ; Own facility makes routing to a dedicated file trivial syslog_facility = LOG_LOCAL6 syslog_ident = "snoopy" syslog_level = LOG_INFO ; Drop the noisiest sources; no spaces allowed in the chain filter_chain = "exclude_spawns_of:cron"
systemd warning Do not set output = syslog on a systemd host. Upstream warns it can hang the system at boot, because syslog() blocks when journald is not yet running. Keep devlog.
Emit structured lines so Elasticsearch or Splunk parse them without a regex extraction.
message_format = "{\"ts\":\"%{datetime}\",\"host\":\"%{hostname}\",\"user\":\"%{username}\",\"uid\":%{uid},\"login\":\"%{login}\",\"tty\":\"%{tty}\",\"cwd\":\"%{cwd}\",\"bin\":\"%{filename}\",\"cmd\":\"%{cmdline}\"}"Escaping caveat Snoopy does not JSON-escape datasource values. A command containing a double quote or backslash produces a malformed line. Either keep the readable bracket format and parse it at ingest, or make your pipeline tolerant of bad JSON.
Log routing & rotation
By default Snoopy lands in the auth log alongside sudo and sshd. Split it into its own file so it can be rotated and shipped independently.
cat > /etc/rsyslog.d/40-snoopy.conf <<'EOF' :programname, isequal, "snoopy" /var/log/snoopy.log & stop EOF systemctl restart rsyslog
Snoopy is chatty. Without rotation a busy host will fill the log partition.
cat > /etc/logrotate.d/snoopy <<'EOF'
/var/log/snoopy.log {
daily
rotate 14
compress
delaycompress
missingok
notifempty
create 0640 root adm
sharedscripts
postrotate
systemctl kill -s HUP rsyslog.service 2>/dev/null || true
endscript
}
EOFjournalctl -t snoopy -f journalctl -t snoopy --since "1 hour ago" journalctl -t snoopy --since today | grep -F 'user:cpx'
Ship to a central collector
Local logs are erasable by whoever you are watching. Forwarding off-host is the single most valuable step for real visibility.
cat > /etc/rsyslog.d/45-snoopy-forward.conf <<'EOF' :programname, isequal, "snoopy" @@10.0.0.5:514 EOF systemctl restart rsyslog
Syntax @@host:port is TCP, @host:port is UDP. Use TCP, and TLS (RELP or gtls) if the traffic crosses an untrusted network.
# Splunk universal forwarder: monitor the dedicated file /opt/splunkforwarder/bin/splunk add monitor /var/log/snoopy.log \ -index linux_audit -sourcetype snoopy # Filebeat -> Elasticsearch # filebeat.inputs: # - type: filestream # paths: [ /var/log/snoopy.log ] # fields: { sourcetype: snoopy }
Search it Once indexed: index=linux_audit sourcetype=snoopy | stats count BY user, bin in SPL, or a terms aggregation on user in Elasticsearch.
Verify
snoopyctl status # enabled and loaded? snoopyctl conf # effective configuration snoopyctl version # Generate a distinctive marker, then look for it id snoopy-test-marker 2>/dev/null grep -F 'snoopy-test-marker' /var/log/snoopy.log | tail -n 3 # Confirm the preload is actually in place grep -F libsnoopy /etc/ld.so.preload
Disable & remove
snoopyctl disable # removes libsnoopy.so from /etc/ld.so.preload # Manual fallback if snoopyctl is gone sed -i '/libsnoopy/d' /etc/ld.so.preload unset LD_PRELOAD LD_PRELOAD_32 LD_PRELOAD_64 # Package removal apt-get remove --purge snoopy # or: dnf remove snoopy # Source install: run 'make uninstall' from the build directory
Running processes Programs already started keep the library mapped. A reboot forces every process to reload without it.
snoopyctl
| Action | Does |
|---|---|
| snoopyctl enable | Add libsnoopy.so to /etc/ld.so.preload. |
| snoopyctl disable | Remove it from /etc/ld.so.preload. |
| snoopyctl status | Detect whether Snoopy is enabled and loaded. |
| snoopyctl conf | Show the effective configuration. |
| snoopyctl version | Show the Snoopy version. |
| snoopyctl about | General information. |
| snoopyctl help | List available actions. |
Older builds Pre-2.4 packages ship snoopy-enable and snoopy-disable shell scripts instead of the snoopyctl binary.
Datasources for message_format
Referenced as %{name}, or %{name:arg} where the source takes arguments. Only sources compiled into your build are available.
| Datasource | Value |
|---|---|
| %{cmdline} | Full command line, with arguments. |
| %{filename} | Full path to the executable. |
| %{cwd} | Current working directory. |
| %{username} / %{uid} | Real user name / uid that executed the command. |
| %{eusername} / %{euid} | Effective user name / uid. |
| %{group} / %{gid} / %{egid} | Group name, gid, effective gid. |
| %{login} | Login name: getlogin_r(), then SUDO_USER, then LOGNAME. Survives sudo. |
| %{tty} / %{tty_uid} / %{tty_username} | Controlling TTY and its owner. |
| %{pid} / %{ppid} / %{sid} / %{tid} | Process, parent, session leader, thread IDs. |
| %{rpname} | Root process name of the executing process tree. |
| %{hostname} / %{domain} | System hostname / domain. |
| %{ipaddr} | IP address of the connected terminal (read from utmp). |
| %{datetime:fmt} | Date and time; strftime() format, defaults to ISO 8601. |
| %{timestamp} / _ms / _us | Unix timestamp and sub-second parts. |
| %{env:VAR} | One environment variable. |
| %{env_all} | All environment variables, comma separated. Very noisy. |
| %{systemd_unit_name} | systemd unit, from /proc/PID/cgroup. |
| %{cgroup:PATTERN} | Control group info by hierarchy id or controller name. |
| %{snoopy_version} | Snoopy version. |
| %{snoopy_literal:arg} | Returns its argument literally (useful as a static tag). |
Filter chain
| Filter | Effect |
|---|---|
| only_uid:LIST | Log only these UIDs. |
| exclude_uid:LIST | Log everything except these UIDs. |
| only_root | Log only root commands. |
| only_tty | Log only commands attached to a TTY (drops daemon noise). |
| exclude_spawns_of:LIST | Drop commands spawned inside these process trees. |
; Rules: filters separated by ';', args by ',', NO spaces anywhere in the chain filter_chain = "only_uid:0" ; root only filter_chain = "exclude_uid:0" ; everything but root filter_chain = "only_uid:1000,1001,1002" ; named humans only filter_chain = "exclude_spawns_of:cron,puppet" ; drop automation noise filter_chain = "only_tty;exclude_uid:0" ; interactive, non-root
Order matters The chain is evaluated left to right; a command must pass every filter to be logged.
Outputs
| Output | Behaviour |
|---|---|
| devlog | Default. Writes directly to /dev/log. Safe under systemd. |
| syslog | Legacy default. Do not use with systemd; can hang the system at boot. |
| file:PATH | Write straight to a file. Path may contain datasources. |
| socket:PATH | Write to a Unix socket at an absolute path. |
| devtty | Write to the current TTY via /dev/tty. |
| stdout / stderr | Debugging only. |
| devnull | Black hole; discards everything. |
output = devlog
output = file:/var/log/snoopy.log
output = file:/var/log/snoopy.log-%{datetime:%Y-%m-%d} ; daily files
output = file:/var/log/snoopy-%{username}.log
output = socket:/var/run/socket-for-snoopy.sockFile permissions With file: output, the target must be writable by every user whose commands you log, or their entries are silently lost. Snoopy will not create missing directories.
Config directives
All under the [snoopy] section of /etc/snoopy.ini. Re-read on every execution, so edits take effect immediately.
| Directive | Default | Notes |
|---|---|---|
| message_format | build-set | Log line template built from datasources. |
| filter_chain | “” (empty) | Semicolon-separated filters; no spaces. |
| output | devlog | Destination; see Outputs. |
| syslog_facility | LOG_AUTHPRIV | AUTH, AUTHPRIV, CRON, DAEMON, LOCAL0-7, USER, and so on. |
| syslog_ident | “snoopy” | Program name in the log; supports datasources. |
| syslog_level | LOG_INFO | EMERG through DEBUG. |
| error_logging | no | Enable only while debugging a format or datasource. |
| datasource_message_max_length | 2047 | Per-datasource cap, 255–1048575. Allocated on every invocation. |
| log_message_max_length | 16383 | Final message cap, 255–1048575. Same memory caveat. |
Troubleshooting
| Symptom | Check |
|---|---|
| No log entries at all | snoopyctl status, then confirm libsnoopy is in /etc/ld.so.preload and the .so path exists. |
| Only some commands logged | Statically linked binaries and shell builtins never appear. Snoopy only sees execv()/execve() in dynamically linked programs. |
| Nothing after enabling | Existing processes keep the old mapping. Restart the service, or reboot. |
| Logs missing from /var/log/snoopy.log | Facility/ident mismatch with the rsyslog rule. Cross-check snoopyctl conf against /etc/rsyslog.d/. |
| System hung at boot | output = syslog on systemd. Boot to rescue, set output = devlog. |
| Every command errors after enable | Bad path in /etc/ld.so.preload. Rescue boot, then mv /etc/ld.so.preload /etc/ld.so.preload.off. |
| Malformed format / empty fields | Set error_logging = yes temporarily; the datasource may not be compiled into this build. |
| Log partition filling | Add a filter chain and logrotate; drop %{env_all} if present. |
Optimize
Snoopy runs inside every exec, so its cost is paid per command. Keep it lean.
| Area | Lever |
|---|---|
| Volume | Filter at the source: exclude_spawns_of:cron and only_tty remove most of the noise before it is ever written. |
| Message size | Log only the fields you query. Never ship %{env_all} to production. |
| Memory | Both max-length settings allocate on every invocation. Raise them only if lines are being truncated. |
| Disk | Route to a dedicated file with its own logrotate policy; compress with delaycompress. |
| Parsing | Pick one stable format and keep it identical across hosts, so one ingest parser covers the fleet. |
| Coverage | Forward off-host. A local-only trail is worthless against anyone with root. |
| Scope | For a single noisy app, prefer per-service LD_PRELOAD over system-wide preload. |
| Build | Compile only the datasources and filters you use; unused ones are dead weight in every process. |
References
| Resource | Use | Link |
|---|---|---|
| a2o/snoopy | Upstream project | github.com/a2o/snoopy |
| snoopy.ini.in | Full annotated config reference | etc/snoopy.ini.in |
| install-from-source | Build flags and enable mechanics | doc/install-from-source.md |
| install-snoopy.sh | Automated installer script | install/install-snoopy.sh |