Why is kworker consuming so many resources on Linux 3.0.0-12-server?
I found this thread on lkml that answers your question a little. (It seems even Linus himself was puzzled as to how to find out the origin of those threads.)
Basically, there are two ways of doing this:
$ echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
$ cat /sys/kernel/debug/tracing/trace_pipe > out.txt
(wait a few secs)
For this you will need ftrace to be compiled in your kernel, and to enable it with:
mount -t debugfs nodev /sys/kernel/debug
More information on the function tracer facilities of Linux is available in the ftrace.txt documentation.
This will output what threads are all doing, and is useful for tracing multiple small jobs.
cat /proc/THE_OFFENDING_KWORKER/stack
This will output the stack of a single thread doing a lot of work. It may allow you to find out what caused this specific thread to hog the CPU (for example). THE_OFFENDING_KWORKER is the pid of the kworker in the process list.
The solution is: I don't know how to find the cause. Nobody told me so far.
But talking with the BTRFS developers revealed a bug in the btrfs drivers when writing many many small files in a very short time period. This is an issue on kernels from 3.0 upto 3.1. Maybe it gets fixed in 3.2.
In the meantime I got a patch for the current kernel that solved the problem.
I had a similar issue; looking at the kworker's thread stack:
while true ; do clear ; grep -n ^ /proc/24910/stack | sort -rn | cut -d: -f 2- ; sleep 1 ; done
[<ffffffffffffffff>] 0xffffffffffffffff
[<ffffffff810908f0>] kthread+0x0/0xe0
[<ffffffff81576432>] ret_from_fork+0x42/0x70
[<ffffffff810908f0>] kthread+0x0/0xe0
[<ffffffff810909b1>] kthread+0xc1/0xe0
[<ffffffff8108b520>] worker_thread+0x0/0x550
[<ffffffff8108b573>] worker_thread+0x53/0x550
[<ffffffff8108aa4b>] process_one_work+0x14b/0x420
[<ffffffff81405a3d>] rpm_idle+0x1ad/0x220
[<ffffffff8140509d>] __rpm_callback+0x2d/0xb0
[<ffffffffa01aef16>] usb_runtime_idle+0x26/0x30 [usbcore]
[<ffffffffa01aeef0>] usb_runtime_idle+0x0/0x30 [usbcore]
[<ffffffff8140686c>] __pm_runtime_suspend+0x5c/0x90
[<ffffffff81405b19>] __pm_runtime_idle+0x69/0x90
[<ffffffff81405295>] rpm_suspend+0x105/0x620
[<ffffffff8140513f>] rpm_callback+0x1f/0x70
[<ffffffff8140509d>] __rpm_callback+0x2d/0xb0
[<ffffffffa01aee50>] usb_runtime_suspend+0x0/0x80 [usbcore]
[<ffffffffa01aee7e>] usb_runtime_suspend+0x2e/0x80 [usbcore]
[<ffffffffa01adc4f>] usb_suspend_both+0xef/0x1f0 [usbcore]
[<ffffffffa01adb06>] usb_resume_interface.isra.6+0xa6/0x100 [usbcore]
[<ffffffffa01a0c63>] hub_resume+0x23/0x60 [usbcore]
[<ffffffffa01a0636>] hub_activate+0xc6/0x5c0 [usbcore]
[<ffffffffa01a9d3f>] usb_kill_urb+0x3f/0xa0 [usbcore]
[<ffffffffa019d249>] hub_port_status+0xd9/0x120 [usbcore]
[<ffffffff81088a4f>] __queue_work+0x12f/0x340
[<ffffffff810888b6>] insert_work+0x46/0xb0
[<ffffffffa01aa6d4>] usb_control_msg+0xc4/0x110 [usbcore]
[<ffffffffa01aa55a>] usb_start_wait_urb+0x9a/0x150 [usbcore]
[<ffffffff810a36f7>] update_curr+0xd7/0x120
I figured it's the usb modules. I had previously on another machine blithely rmmod'd all usb and [uex]hci modules are realised that I had turned off the keyboard (not even ctrl-shift-sysrq-U !), so I ended up doing this:
MODS="uvcvideo ohci_hcd ehci_hcd xhci_hcd ohci_pci ehci_pci xhci_pci usbcore"
( echo $MODS $MODS | xargs -n 1 rmmod -v ; sleep 3 ; echo $MODS | xargs -n 1 modprobe -v ; )
root@hp:~# ( echo $MODS $MODS | xargs -n 1 rmmod -v ; sleep 3 ; echo $MODS | xargs -n 1 modprobe -v ; )
rmmod: ERROR: Module ohci_hcd is in use by: ohci_pci
rmmod: ERROR: Module ehci_hcd is in use by: ehci_pci
rmmod: ERROR: Module xhci_hcd is in use by: xhci_pci
rmmod: ERROR: Module uvcvideo is not currently loaded
rmmod: ERROR: Module ohci_pci is not currently loaded
rmmod: ERROR: Module ehci_pci is not currently loaded
rmmod: ERROR: Module xhci_pci is not currently loaded
insmod /lib/modules/4.1.0-2-amd64/kernel/drivers/media/usb/uvc/uvcvideo.ko
insmod /lib/modules/4.1.0-2-amd64/kernel/drivers/usb/host/ehci-hcd.ko
insmod /lib/modules/4.1.0-2-amd64/kernel/drivers/usb/host/ohci-hcd.ko
insmod /lib/modules/4.1.0-2-amd64/kernel/drivers/usb/host/xhci-hcd.ko
insmod /lib/modules/4.1.0-2-amd64/kernel/drivers/usb/host/ehci-pci.ko
insmod /lib/modules/4.1.0-2-amd64/kernel/drivers/usb/host/ohci-pci.ko
insmod /lib/modules/4.1.0-2-amd64/kernel/drivers/usb/host/xhci-pci.ko
did the trick:
grep -n ^ /proc/24910/stack | sort -rn | cut -d: -f 2-
[<ffffffffffffffff>] 0xffffffffffffffff
[<ffffffff810908f0>] kthread+0x0/0xe0
[<ffffffff81576432>] ret_from_fork+0x42/0x70
[<ffffffff810908f0>] kthread+0x0/0xe0
[<ffffffff810909b1>] kthread+0xc1/0xe0
[<ffffffff8108b520>] worker_thread+0x0/0x550
[<ffffffff8108b5ec>] worker_thread+0xcc/0x550
So my chief suspect is this gadget: RTL8723B* WIFI+Bluetooth Module. I'm wondering now if the power-management code realises it's the same device if it tries to e.g. powerdown an unused BT adapter.
context:
root@hp:~# lsusb
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 002: ID 0c45:651b Microdia
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 002: ID 0bda:b001 Realtek Semiconductor Corp.
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 009 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 008 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 007 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 006 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@hp:~# lsmod | grep usb
btusb 45056 0
btbcm 16384 1 btusb
btintel 16384 1 btusb
bluetooth 438272 5 bnep,btbcm,btusb,btintel
usbcore 200704 8 btusb,uvcvideo,ohci_hcd,ohci_pci,ehci_hcd,ehci_pci,xhci_hcd,xhci_pci
usb_common 16384 1 usbcore
root@hp:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux stable-updates (sid)
Release: stable-updates
Codename: sid
root@hp:~# uname -a
Linux hp 4.1.0-2-amd64 #1 SMP Debian 4.1.6-1 (2015-08-23) x86_64 GNU/Linux
root@hp:~# dmesg | tail -n 20
[97865.088740] usb 2-4: SerialNumber: HP Webcam
[97865.091557] uvcvideo: Found UVC 1.00 device HP Webcam (0c45:651b)
[97865.105948] input: HP Webcam as /devices/pci0000:00/0000:00:13.2/usb2/2-4/2-4:1.0/input/input17
[97865.189817] usb 3-3: new full-speed USB device number 2 using ohci-pci
[97865.350981] usb 3-3: No LPM exit latency info found, disabling LPM.
[97865.368958] usb 3-3: New USB device found, idVendor=0bda, idProduct=b001
[97865.368969] usb 3-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[97865.368976] usb 3-3: Product: Bluetooth Radio
[97865.368981] usb 3-3: Manufacturer: Realtek
[97865.368985] usb 3-3: SerialNumber: 00e04c000001
[97865.375859] Bluetooth: hci0: rtl: examining hci_ver=06 hci_rev=000b lmp_ver=06 lmp_subver=8723
[97865.375867] Bluetooth: hci0: rtl: loading rtl_bt/rtl8723b_fw.bin
[97865.375896] usb 3-3: firmware: failed to load rtl_bt/rtl8723b_fw.bin (-2)
[97865.375902] usb 3-3: Direct firmware load for rtl_bt/rtl8723b_fw.bin failed with error -2
[97865.375907] Bluetooth: hci0: Failed to load rtl_bt/rtl8723b_fw.bin
[97865.397812] Bluetooth: hci0: rtl: examining hci_ver=06 hci_rev=000b lmp_ver=06 lmp_subver=8723
[97865.397821] Bluetooth: hci0: rtl: loading rtl_bt/rtl8723b_fw.bin
[97865.397850] usb 3-3: firmware: failed to load rtl_bt/rtl8723b_fw.bin (-2)
[97865.397856] usb 3-3: Direct firmware load for rtl_bt/rtl8723b_fw.bin failed with error -2
[97865.397861] Bluetooth: hci0: Failed to load rtl_bt/rtl8723b_fw.bin