Journalctl: How to Read and Edit Systemd Logs

Introduction

Systemd logs all Linux messages from the kernel and system processes. The journalctl command enables viewing and editing the systemd logs, making it a powerful tool for service and process debugging.

This guide shows how to read, control, and maintain systemd logs using journalctl through examples.

Prerequisites

• Access to the command line/terminal window.
• A text editor (such as nano) to edit the config file.
• A user with sudo privileges (see how to add a user to sudoers).

What Is Systemd?

Systemd is a Linux service and system manager. While users do not invoke systemd directly, the manager contains many tools and daemons to run individually for various system processes.

One of the most powerful systemd functionalities is the logging features. Systemd provides a centralized solution for logging all kernel and user processes through logs known as journals.

The journald daemon collects all the messages the system outputs and then creates journals, regardless of the program or process. The daemon gathers data from all available system resources and stores them in a binary format.

journalctl Syntax

The journalctl command queries and manipulates the journal data collected by the journald daemon. The tool is vital for system administrators and complements other Linux logging tools and Syslog server software solutions.

The command syntax is:

journalctl <options> <matches>

Without any parameters, the journalctl command outputs the entire journal contents starting from the oldest entry. The <match> is one or more space-separated arguments for filtering the output fields. The format is "FIELD=VALUE".

journalctl Options

The table below summarizes common journalctl options:

Check the journalctl manual page using the man command for a full list of options.

How to Read systemd

The following section outlines how to read systemd logs and use the various display options for the journalctl command. The output is different for every machine since records for every system are unique.

Display All Journal Entries

To show all journal entries, use the journalctl command without any options:

journalctl

The first line from the output shows the time range of the log data. The columns contain the following data in order from left to right:

• Date and time.
• Host.
• Log source.
• Log message.

The journal data contains many entries. Use the arrow keys (similar to the less command) to navigate.

Exit the journal by pressing q.

Show Most Recent Entries

The journalctl command shows the oldest entries by default. To jump to the pager end and display the most recent entries, use the -e option:

journalctl -e

The output shows the final 1000 entries to save space.

To control how many lines display in the output, use the -n option followed by the number of lines. For example, to show the five most recent journal entries, use:

journalctl -n 5

The -e option is unnecessary and implied by the -n option. Omitting the number shows the ten most recent entries by default.

Limit the Logs to the Specific Boot

To limit the logs to the current boot, use the -b tag without any parameters:

journalctl -b

Without any parameters, the command shows current boot logs.

Jump to a specific boot by adding an offset parameter. For example, show the previous boot logs with:

journalctl -b -1

Alternatively, show the oldest available boot log with:

journalctl -b +1

An alternative way to see a specific boot is to use a boot ID. Fetch the boot IDs using --list-boots with:

journalctl --list-boots

The first column displays the negative offset number, while the second column fetches the boot ID. Copy the ID and add it as a parameter to the command, for example:

journalctl -b cc07702b00884ec59312ece62604cac8

The output limits the log display to the provided ID instance.

Display Logs Within a Specific Time Window

Filter the journal by specifying a time limit. The two options for limiting since or until a specified time are:

journalctl -S <datetime>

journalctl -U <datetime>

Use the options individually or combine them to create a time window.

The command expects one of the following date and time formats:

• Specific date and time, for example, 2022-04-30 09:20:00. Omitting the time parameter defaults to 00:00:00.
• Strings, such as "yesterday", "today", "2 hours ago", or "now".

Below is an example journalctl command with a specific time window:

journalctl -S 2022-04-02 -U 2022-04-22

The command creates a time window from April 2nd, 2022, to April 22nd, 2022. The output shows journals that fall into that timeframe.

Alternatively, use a string pattern such as:

journalctl -S "50 minutes ago"

The output shows logs from the stated time up until the current time.

Display Logs By Specific systemd Unit

Filter the logs by the specific systemd unit using the -u tag and providing the unit name. For example, to filter only the Jenkins service unit records, run:

journalctl -u jenkins

The output shows the journal entries related to the specific systemd unit (in this case, Jenkins).

systemctl list-units

Display Kernel Messages

To display only the kernel journal log messages, use the -k option:

journalctl -k

The output shows the kernel messages only from the current boot, applying the -b tag. To find kernel logs from a different boot session, add the -b tag and search for a specific boot.

Follow Logs

Use the -f or --follow tag to print the most recent logs continuously:

journalctl -f

The output prints the logs as they generate in real-time. The option allows monitoring the logs with journalctl as they append.

To exit the viewer, press CTRL+C.

Filter Log Messages Based on Priority

Filter the log messages by priority using the following command:

journalctl -p <number or text priority>

The following priorities exist:

• Emergency - 0 or emerg
• Alert - 1 or alert
• Critical - 2 or crit
• Error - 3 or err
• Warning - 4 or warning
• Notice - 5 or notice
• Inform - 6 or info
• Debug - 7 or debug

A lower number indicates the highest priority messages. Specifying a single-level priority also shows all lower priority number (more critical) logs.

For example, to display alerts, use:

journalctl -p 1

Or alternatively:

journalctl -p alert

The output displays only messages at the alert level and more important ones (if any).

Filter Log Messages Based on a Specific User

To see logs for a user, fetch the user ID (UID) with:

id <user>

To fetch the ID for the current user, omit the <user>. The output shows the UID value for the given user. Use the UID journal field to filter log messages based on the specific user:

journalctl _UID=<UID>

The output filters the journal log based on the specified user ID.

man systemd.journal-fields

How to Edit systemd Log Output

An essential aspect of working with logs as a system administrator is formatting the log outputs. Systemd offers many methods to manipulate the visual result and fetch the data in the desired format.

Below are some standard output editing options and examples.

Output to Standard Out

The journalctl command displays the output using a pager. Disable the pager with:

journalctl --no-pager

The resulting output is in standard output (stdout). Use this option when parsing the log data with text editing tools or Bash scripts.

Truncate or Expand Output

The journalctl pager shows expanded journal events in the output. Pressing the right and left arrow keys helps navigate the text that doesn't fit the screen size.

To limit truncate the journalctl output, use the --no-full option:

journalctl --no-full

The output limits the lines to the screen size, adding ellipsis (...) to indicate a truncated display.

Output Formats

The journalctl command offers various options for output formats. The syntax for output format is:

journalctl -o <output format>

Some of the available formats include:

• cat - Displays only the message field.
• export - Outputs binary format, suitable for backups.
• short - Generates an output similar to classic Syslog files.
• short-precise - Displays time with microseconds.
• json - Formats journal entries into single-line JSON entries.
• json-pretty - Formats into JSON structures in multiple lines.

For example, to display using the json-pretty format, use:

journalctl -o json-pretty

Different formats allow using the log data in databases, script files, or parsing it through monitoring software.

Log Maintenance

Storing log data comes at a cost and takes up space. Below are some tips and tricks to discover the disk usage, maintain log data files, and free up space used by old log files.

Display Disk Usage

To check the journal disk usage, run the following command:

journalctl --disk-usage

The output shows the disk's total occupied space by archived and active journals.

Delete Old Logs

Delete old log archives by setting the desired size limit. The command requires sudo to delete the files /var/log/journal.

For example, set the size to 1M with:

sudo journalctl --vacuum-size=1M

Enter the sudo password and press Enter. The output prints the file names and sizes, and the last line shows the amount of freed memory.

Alternatively, delete archived logs based on time. Any files older than the set time delete and free up memory. For example, to delete files older than two months, run:

sudo journalctl --vacuum-time=2months

The time suffixes are s, m, h, days, months, weeks, or years.

Limit the Journal

The journal configuration file allows setting limits and controlling how much journald data takes up on disk. To edit the file, run:

sudo nano /etc/systemd/journald.conf

The file contains example configuration fields. The following parameters deal with the journal size and memory limits:

• SystemMaxUse - Maximum persistent storage used by the journal.
• SystemKeepFree - Amount of free space a journal leaves when adding entries to persistent storage.
• SystemMaxFileSize - Sets the maximum size for journal files in persistent storage.
• RuntimeMaxUse - Maximum volatile storage disk space.
• RuntimeKeepFree - Amount of free space for other uses when writing to volatile storage.
• RuntimeMaxFileSize - Sets the maximum size for journal files in volatile storage.

File size controls target archived files to reach the limits. Uncomment the lines and set the limits to gain better control over the machine storage and consumption resources.

Conclusion

This guide showed how to view, control, and manage systemd journal logs through examples. The journalctl command is a valuable tool that helps troubleshoot Linux services and discover system errors.