journald

Identity

• Service: systemd-journald
• Unit: systemd-journald.service
• Config: /etc/systemd/journald.conf, /etc/systemd/journald.conf.d/*.conf
• Persistent storage: /var/log/journal/ (survives reboot; exists only if created or Storage=persistent)
• Volatile storage: /run/log/journal/ (lost on reboot; always present while service runs)
• Format: binary (.journal files) — not directly grep-able
• Namespace support: /run/log/journal/<machine-id>/ per-namespace subdirectories

Key Operations

| Goal | Command | |------|---------| | Follow live logs | journalctl -f | | All logs since current boot | journalctl -b | | All logs from previous boot | journalctl -b -1 | | List available boots | journalctl --list-boots | | Logs for a unit | journalctl -u nginx.service | | Follow unit logs live | journalctl -fu nginx.service | | Filter by priority | journalctl -p err (emerg alert crit err warning notice info debug) | | Errors and above | journalctl -p err..emerg | | Since a time | journalctl --since "1 hour ago" | | Time range | journalctl --since "2025-03-01 10:00" --until "2025-03-01 11:00" | | By PID | journalctl _PID=1234 | | By UID | journalctl _UID=1000 | | By systemd slice/cgroup | journalctl _SYSTEMD_UNIT=user@1000.service | | Kernel messages only | journalctl -k | | Kernel messages this boot | journalctl -kb | | JSON output (one entry per line) | journalctl -o json | | JSON pretty-printed | journalctl -o json-pretty | | ISO timestamps | journalctl -o short-iso | | Show catalog explanations | journalctl -x or journalctl -xe | | Disk usage summary | journalctl --disk-usage | | Vacuum by size | journalctl --vacuum-size=500M | | Vacuum by time | journalctl --vacuum-time=30d | | Vacuum by file count | journalctl --vacuum-files=10 | | Verify log integrity | journalctl --verify | | Export to binary stream | journalctl -o export > journal.bin | | Import binary stream | systemd-journal-remote or journalctl --import journal.bin | | Show catalog entry for error | journalctl -x -u <unit> | | Send a test log message | systemd-cat echo "test message" | | Send with priority | echo "test" \| systemd-cat -p warning | | Pipe command output to journal | my-script 2>&1 \| systemd-cat -t my-script |

Expected State

• systemctl is-active systemd-journald returns active
• At least one of /var/log/journal/ or /run/log/journal/ exists and is readable
• journalctl -b 0 -n 1 returns at least one entry (current boot)
• Disk usage within configured bounds: journalctl --disk-usage

Health Checks

1. systemctl is-active systemd-journald → active
2. journalctl -b -n 5 --no-pager → recent log entries visible, no errors
3. journalctl --disk-usage → total size within SystemMaxUse limit

Common Failures

| Symptom | Likely cause | Check/Fix | |---------|-------------|-----------| | Logs lost after reboot | Volatile storage only; /var/log/journal/ absent | mkdir -p /var/log/journal && systemd-tmpfiles --create --prefix /var/log/journal && systemctl restart systemd-journald | | No journal files were found | Storage dir missing or wrong permissions | Check /run/log/journal/ and /var/log/journal/ exist; chown root:systemd-journal on the dir | | Journal growing without bound | SystemMaxUse not set; disk filling up | Set SystemMaxUse= in journald.conf; run journalctl --vacuum-size=500M to reclaim immediately | | Timestamps wrong or in UTC | Timezone mismatch | journalctl --utc to display in UTC explicitly; system timezone: timedatectl | | Remote journal not receiving logs | ForwardToSyslog or systemd-journal-remote misconfigured | Check receiver: systemctl status systemd-journal-remote.service; check JOURNAL_REMOTE_URL env | | Failed to determine timestamp on import | Journal file corrupt or truncated | journalctl --verify to check integrity; recover from backup | | Rate-limited: logs missing | RateLimitBurst exceeded by a noisy service | Increase RateLimitBurst or set RateLimitIntervalSec=0 to disable (risk: disk exhaustion) | | Permission denied reading journal | User not in systemd-journal group | usermod -aG systemd-journal <username> then re-login | | Sealed journal fails --verify | FSSEAL keys missing or rotated | journalctl --verify reports which files are affected; reseal or accept the gap |

Pain Points

• Logs are volatile by default: unless /var/log/journal/ exists (or Storage=persistent), the journal lives in /run/log/journal/ and is lost at every reboot. This is the most common support scenario.
• Disk usage must be bounded explicitly: by default the journal will use up to 10% of the filesystem. On a small root partition this can fill it. Set SystemMaxUse and RuntimeMaxUse in journald.conf.
• Binary format means no grep: you cannot grep -r /var/log/journal/. Use journalctl -g <pattern> (grep) or -o json | jq for structured filtering instead.
• Rate limiting can silently suppress logs: a service emitting bursts of messages will be rate-limited by default (RateLimitBurst=10000 per RateLimitIntervalSec=30s). Messages are dropped with a single systemd-journald[PID]: <service> suppressed N messages entry.
• Forwarding to syslog is opt-in: rsyslog and syslog-ng do not automatically receive journal messages. Set ForwardToSyslog=yes in journald.conf, or configure rsyslog with imjournal to read the journal directly.
• -b -1 requires persistent storage: journalctl -b -1 (previous boot) only works if /var/log/journal/ is present. With volatile storage, only the current boot is available.
• Catalog entries are ID-based: journalctl -x annotates entries that have a catalog entry (a MESSAGE_ID field). Not all entries have one — the annotation appears only when a matching catalog record exists.

References

See references/ for:

• journald.conf.annotated — full configuration file with every directive explained
• docs.md — official documentation links