Logs and journald for monitoring

Topic: Monitoring basics

Use journald (journalctl) to query and forward logs; use log aggregation to centralize logs from multiple hosts for search and alerting. Use this when setting up log-based monitoring or when correlating events across services.

Intent: How-to

Quick answer

  • Query journal: journalctl -u nginx -f (follow); journalctl -b (this boot); journalctl -p err (priority error and above); journalctl --since '1 hour ago'. Export: journalctl -o json or forward to a central system (e.g. Loki, Elastic, Fluentd).
  • Forward to central: configure journald to forward to a collector (systemd-journal-upload, or Fluent Bit reading from journal); collector sends to Loki, Elasticsearch, or cloud logging. Retain locally per Storage= in journald.conf.
  • Alert on logs: use a log pipeline that matches patterns (e.g. error, OOM, login failure) and triggers alerts. Correlate with metrics (e.g. high error rate and high latency).

Steps

  1. Query and filter

    journalctl -u service -f; journalctl -p err --since today; journalctl -b -1 for previous boot. Use -o short-iso for timestamps; export to file or pipe to grep for analysis.

  2. Forward to central

    Install Fluent Bit or similar; read from /var/log/journal or socket; send to Loki, Elasticsearch, or cloud. Ensure TLS and auth; retain local journal per policy (Storage=persistent in journald.conf).

  3. Retention and size

    In journald.conf: SystemMaxUse=, MaxRetentionSec=. Prevent disk fill; rotate or forward before trimming. Match retention to compliance or operational need.

  4. Alert on log patterns

    In log pipeline or SIEM: match 'out of memory', 'authentication failed', 'error'. Trigger alert or create incident. Link to runbook; reduce false positives by tuning pattern and context.

Summary

Use journalctl to query and filter logs; forward to a central system for search and alerting. Use this to set up log-based monitoring and to correlate logs with metrics.

Prerequisites

None.

Steps

Step 1: Query and filter

Use journalctl with unit, priority, and time filters; export when needed.

Step 2: Forward to central

Use a log shipper to send journal (or files) to a central store; use TLS and auth.

Step 3: Retention and size

Configure journald and central retention so disk and cost are under control.

Step 4: Alert on log patterns

Define patterns that indicate failure or security events; trigger alerts and link runbooks.

Verification

  • Logs are queryable and forwarded; retention is correct; log-based alerts fire when expected.

Troubleshooting

Journal empty — Check Storage=persistent; ensure journal dir exists and has space. Forward not working — Check collector config and network; verify credentials and endpoint.

Next steps

Continue to