Using netconsole for Kernel Debugging

Development Environment

Client

• Development Board: FireFly-RK3399
• Linux 4.4
• IP: 192.168.137.110

Server

• VMware Workstation Pro 16, Ubuntu 18.04
• IP: 192.168.137.100
• MAC: 00:0c:29:c1:9c:ed

Introduction to netconsole

The netconsole module allows printk messages (information output to the console) to be sent over the network via UDP to a remote host.

Unlike a serial console, netconsole does not require a serial cable. It also makes it easy to collect kernel messages from multiple servers onto a single server.

When the serial port on a device is unavailable, using netconsole to save log information to a remote host is a convenient and viable option.

The netconsole module cannot be used for obtaining crash dumps or for general console input/output.

The netconsole module cannot capture network-related kernel panics or kernel information related to panics that occur between system startup and the initialization of the network and netconsole module.

Compiling netconsole

netconsole can be compiled as a separate module to be loaded manually, or it can be compiled directly into the kernel for automatic loading. This example uses the first method.

Compiling as a Standalone Module

By default, netconsole is not compiled into the kernel. It needs to be manually compiled into the kernel or as a module.

In the kernel directory, run `make menuconfig` and enable the following option:

Device Drivers > Network device support > Network console logging support

Recompile the kernel and flash it to the development board.

To compile the netconsole module manually:

make CONFIG_NETCONSOLE=m -C /home/zhongyi/code/rk3399_linux_release_v2.5.1_20210301/kernel M=/home/zhongyi/code/rk3399_linux_release_v2.5.1_20210301/kernel/drivers/net modules CROSS_COMPILE=/home/zhongyi/code/rk3399_linux_release_v2.5.1_20210301/prebuilts/gcc/linux-x86/aarch64/gcc-linaro-6.3.1-2017.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-

After this, the `netconsole.ko` file will be generated in the module directory.

Compiling into the Kernel

To enable netconsole in a custom kernel, enable the following kernel options (to compile as a module):

CONFIG_NETCONSOLE=m CONFIG_NETCONSOLE_DYNAMIC=y

If you compile netconsole directly into the kernel for automatic loading, you must pass the configuration through the kernel boot parameters and ensure the network interface card driver is loaded before the netconsole driver.

Loading netconsole

Transfer `netconsole.ko` to the development board via a shared directory and grant it execute permissions. You need to know the server's IP address and MAC address beforehand.

If the log server is on the same local network, get the destination MAC address by pinging it:

$ ping -c 1 192.168.1.103 > /dev/null $ arp -n 192.168.1.103 Address HWtype HWaddress Flags Mask Iface 192.168.1.103 ether 08:00:46:d4:1d:82 C eth0

If the log server is not on the same local network, get the gateway's MAC address:

$ netstat -rn | grep ^0.0.0.0 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0 $ ping -c 1 192.168.1.1 > /dev/null $ arp -n 192.168.1.1 Address HWtype HWaddress Flags Mask Iface 192.168.1.1 ether 00:0f:66:5b:2a:3c C

Execute the load command:

insmod netconsole.ko netconsole=6665@192.168.137.110/eth0,514@192.168.137.100/00:0c:29:c1:9c:ed

The parameters are defined as follows:

netconsole=[src-port]@[src-ip]/[dev],[tgt-port]@[tgt-ip]/[tgt-macaddr] where: src-port Source UDP port (defaults to 6665) src-ip Source IP address of the interface dev Network interface (e.g., eth0) tgt-port Target port for the logging agent (defaults to 6666) tgt-ip IP address of the logging agent tgt-macaddr Ethernet MAC address of the logging agent

rsyslog Server Setup

Open Port 514

With the client configured, set up rsyslog to run in server mode.

sudo vim /etc/rsyslog.conf

Uncomment the lines for UDP and TCP port binding:

# provides UDP syslog reception module(load="imudp") input(type="imudp" port="514") # provides TCP syslog reception module(load="imtcp") input(type="imtcp" port="514")

Test if the port is accessible:

? ~ sudo nc -vuz 192.168.137.110 514 Connection to 192.168.137.110 514 port [udp/syslog] succeeded!

Create a Reception Template

Create a template to instruct the rsyslog server on how to store incoming syslog messages. Add the template before the `GLOBAL DIRECTIVES` section:

$template remote-incoming-logs,"/var/log/%HOSTNAME%/%fromhost-ip%-%$YEAR%-%$MONTH%-%$DAY%.log" *.* ?remote-incoming-logs & ~

Received logs will be parsed using this template and stored in the `/var/log/` directory, with filenames following the convention: `%HOSTNAME%/%fromhost-ip%-%$YEAR%-%$MONTH%-%$DAY%.log`.

Afterward, save and close the file. Then, check the Rsyslog configuration for syntax errors:

rsyslogd -f /etc/rsyslog.conf -N1

You should see the following output:

rsyslogd: version 8.32.0, config validation run (level 1), master config /etc/rsyslog.conf rsyslogd: End of config validation run. Bye.

Configure the Startup File

Modify the `/etc/default/rsyslog` startup configuration file as root:

RSYSLOGD_OPTIONS="-m 0 -r"

The `-r` option allows the server to receive log messages from remote hosts. The `-m` option modifies the interval for internal mark messages (0 disables it). For example, `-m 180` adds a timestamp to the log file every 180 minutes.

Restart the Service

Restart the rsyslog service for the changes to take effect:

sudo systemctl restart rsyslog

Confirm that the service is listening on the configured port:

ss -tunelp | grep 514 udp UNCONN 0 0 0.0.0.0:514 0.0.0.0:* ino:178435 sk:2c <-> udp UNCONN 0 0 [::]:514 [::]:* ino:178436 sk:2d v6only:1 <-> tcp LISTEN 0 25 0.0.0.0:514 0.0.0.0:* ino:178439 sk:2e <-> tcp LISTEN 0 25 [::]:514 [::]:* ino:178440 sk:2f v6only:1 <->

Configure the Firewall for Rsyslog

If the UFW firewall service is running, allow the rsyslog port:

sudo ufw allow 514/tcp sudo ufw allow 514/udp

Testing the Output

The test uses the SysRq key on the client to output kernel information.

On the development board, edit `/etc/sysctl.conf` to enable SysRq:

kernel.sysrq=1

Run the following command to apply the change:

sysctl -p

Execute the following command to send kernel information to the console:

root@firefly:~/mnt/module# echo h > /proc/sysrq-trigger

On the server, you can see the generated log file in the `/var/log/192.168.137.110/` directory.

? 192.168.137.110 cat 192.168.137.110-2022-10-16.log 2022-10-16T22:35:54.777180+08:00 192.168.137.110 [10224.626165] sysrq: SysRq : 2022-10-16T22:35:54.777180+08:00 192.168.137.110 loglevel(0-9) 2022-10-16T22:35:54.777180+08:00 192.168.137.110 [10224.626165] sysrq: SysRq : 2022-10-16T22:35:54.777180+08:00 192.168.137.110 loglevel(0-9) 2022-10-16T22:35:54.777180+08:00 192.168.137.110 reboot(b) 2022-10-16T22:35:54.777180+08:00 192.168.137.110 crash(c) ... 2022-10-16T22:35:54.900989+08:00 192.168.137.110 dump-ftrace-buffer(z) 2022-10-16T22:35:54.900989+08:00 192.168.137.110 dump-ftrace-buffer(z) 2022-10-16T22:35:54.900989+08:00 192.168.137.110

If you lack permission to access the `192.168.137.110` directory, grant permissions with `sudo chmod 777 192.168.137.110`.

Other Listening Methods

netcat

On a host without syslogd running, you can use `netcat` or `socat` to receive messages from a remote host:

nc -u -l -p <port> netcat -u -l -p <port> socat udp-recv:<port> -

Example:

~ sudo netcat -l -p 514 -u [sudo] password for zhongyi: [ 263.748032] sysrq: SysRq : HELP : [ 263.748032] sysrq: SysRq : HELP : loglevel(0-9) loglevel(0-9) reboot(b) reboot(b) crash(c) crash(c) ...

dmesg

The `-n/--console-level` option controls the kernel message console output level. Set the output level to debug (-n 8):

dmesg -n 8

netconsole in U-Boot

netconsole in U-Boot functions similarly to network terminals like telnet in the kernel, using the network for input and output. This allows you to log into the device's U-Boot from a PC over the network and run commands.

The U-Boot netconsole implementation is in `drivers/net/netconsole.c`. To enable it, add `CONFIG_NETCONSOLE=y` to the `configs/firefly-rk3399_defconfig` file. Then recompile and flash U-Boot to the device.

Set the environment variable `nc` for communication. You can set `CONFIG_NETCONSOLE_BUFFER_SIZE` to override the default buffer size. Set the environment variable `ncip` to the remote IP address in the format `<ip_addr>:<port>`. If `<port>` is not set, it defaults to `6666`. The sending and receiving ports can be set separately with `ncinport` and `ncoutport`.

setenv ipaddr 192.168.137.110 setenv nc 'setenv stdout nc;setenv stdin nc' setenv ncip 192.168.137.100 saveenv run nc

U-Boot will enter netconsole mode, and output to the serial console will cease.

On the server, run:

./tools/netconsole 192.168.137.110

You can now run U-Boot commands from the PC.