DeatheTongue

I reverted to 6.11.5 and continued to have crashes until I removed a bluetooth adapter. BT was attached and bluez-5.66-x86_64-1.txz installed and running to share into Home Assistant docker. At 1 month uptime on 6.11.5 and will probably be attempting 6.12.x again in the coming weeks. Will follow up to close the loop on my report.

Here it is. syslog-127.0.0.1.log

Diagnostics again, now with moar syslog. Uptime of 4 days, I think. The parity check had just finished the day before the hang. The interesting thing is these kernel faults are at 1500-ish and I didn't notice the full hang until around 2200. I have to completely remove power from the system to get it to reboot from this hang. Oct 27 10:14:07 Kong kernel: veth8c9ca72: renamed from eth0 Oct 27 10:14:09 Kong kernel: eth0: renamed from veth0df38af Oct 27 10:45:59 Kong kernel: docker0: port 19(veth05c1d17) entered disabled state Oct 27 10:45:59 Kong kernel: veth35adc40: renamed from eth0 Oct 27 10:45:59 Kong kernel: docker0: port 19(veth05c1d17) entered disabled state Oct 27 10:45:59 Kong kernel: device veth05c1d17 left promiscuous mode Oct 27 10:45:59 Kong kernel: docker0: port 19(veth05c1d17) entered disabled state Oct 27 10:46:01 Kong kernel: docker0: port 19(veth50db4f8) entered blocking state Oct 27 10:46:01 Kong kernel: docker0: port 19(veth50db4f8) entered disabled state Oct 27 10:46:01 Kong kernel: device veth50db4f8 entered promiscuous mode Oct 27 10:46:02 Kong kernel: eth0: renamed from veth9a08974 Oct 27 10:46:02 Kong kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Oct 27 10:46:02 Kong kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth50db4f8: link becomes ready Oct 27 10:46:02 Kong kernel: docker0: port 19(veth50db4f8) entered blocking state Oct 27 10:46:02 Kong kernel: docker0: port 19(veth50db4f8) entered forwarding state Oct 27 11:02:35 Kong monitor: Stop running nchan processes Oct 27 12:00:51 Kong rpc.mountd[12955]: v4.2 client detached: 0xa1fd6fb7653934a1 from "192.168.10.13:727" Oct 27 15:01:08 Kong kernel: ------------[ cut here ]------------ Oct 27 15:01:08 Kong kernel: WARNING: CPU: 14 PID: 0 at kernel/softirq.c:415 __do_softirq+0x256/0x288 Oct 27 15:01:08 Kong kernel: Modules linked in: rpcsec_gss_krb5 nvidia_uvm(PO) xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle iptable_mangle vhost_net tun vhost vhost_iotlb tap ipvlan veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xfs ip6table_nat nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) algif_hash algif_skcipher af_alg cmac bnep tcp_diag inet_diag iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs 8021q garp mrp bridge stp llc mlx4_en mlx4_core igb i2c_algo_bit nvidia_drm(PO) nvidia_modeset(PO) edac_mce_amd edac_core intel_rapl_msr intel_rapl_common iosf_mbi kvm_amd nvidia(PO) kvm btusb btrtl video Oct 27 15:01:08 Kong kernel: drm_kms_helper btbcm crct10dif_pclmul crc32_pclmul btintel crc32c_intel ghash_clmulni_intel sha512_ssse3 drm bluetooth aesni_intel mpt3sas crypto_simd backlight cryptd syscopyarea sysfillrect wmi_bmof mxm_wmi raid_class ecdh_generic i2c_piix4 rapl sysimgblt tpm_crb ecc k10temp ccp scsi_transport_sas i2c_core fb_sys_fops nvme ahci cp210x tpm_tis input_leds tpm_tis_core nvme_core led_class usbserial libahci corsair_psu wmi tpm button acpi_cpufreq unix [last unloaded: mlx4_core] Oct 27 15:01:08 Kong kernel: CPU: 14 PID: 0 Comm: swapper/14 Tainted: P O 6.1.49-Unraid #1 Oct 27 15:01:08 Kong kernel: Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 6203 07/27/2023 Oct 27 15:01:08 Kong kernel: RIP: 0010:__do_softirq+0x256/0x288 Oct 27 15:01:08 Kong kernel: Code: c0 75 09 41 ff cf 0f 85 2b fe ff ff e8 af 2d 47 ff 65 81 05 38 c9 41 7e 00 ff ff ff 65 8b 05 31 c9 41 7e a9 00 ff ff 00 74 02 <0f> 0b 8b 54 24 10 65 48 8b 04 25 c0 cb 01 00 81 60 2c ff f7 ff ff Oct 27 15:01:08 Kong kernel: RSP: 0018:ffffc90000534fa0 EFLAGS: 00010006 Oct 27 15:01:08 Kong kernel: RAX: 0000000080010001 RBX: 0000000000000000 RCX: 000000000000000a Oct 27 15:01:08 Kong kernel: RDX: 0000000000010101 RSI: ffffffff821626da RDI: ffffffff82117638 Oct 27 15:01:08 Kong kernel: RBP: ffffffff82206110 R08: 0000000000000000 R09: 0000000000010101 Oct 27 15:01:08 Kong kernel: R10: 0000000000000000 R11: ffffc90000534ff8 R12: 0000000000000009 Oct 27 15:01:08 Kong kernel: R13: 0000000000010101 R14: ffff8881003dde80 R15: 000000000000000a Oct 27 15:01:08 Kong kernel: FS: 0000000000000000(0000) GS:ffff888feeb80000(0000) knlGS:0000000000000000 Oct 27 15:01:08 Kong kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 27 15:01:08 Kong kernel: CR2: 000014ccf21cf410 CR3: 0000000850d50000 CR4: 0000000000350ee0 Oct 27 15:01:08 Kong kernel: Call Trace: Oct 27 15:01:08 Kong kernel: <IRQ> Oct 27 15:01:08 Kong kernel: ? __warn+0xab/0x122 Oct 27 15:01:08 Kong kernel: ? report_bug+0x109/0x17e Oct 27 15:01:08 Kong kernel: ? __do_softirq+0x256/0x288 Oct 27 15:01:08 Kong kernel: ? handle_bug+0x41/0x6f Oct 27 15:01:08 Kong kernel: ? exc_invalid_op+0x13/0x60 Oct 27 15:01:08 Kong kernel: ? asm_exc_invalid_op+0x16/0x20 Oct 27 15:01:08 Kong kernel: ? __do_softirq+0x256/0x288 Oct 27 15:01:08 Kong kernel: __irq_exit_rcu+0x5e/0xb8 Oct 27 15:01:08 Kong kernel: sysvec_apic_timer_interrupt+0x85/0xa6 Oct 27 15:01:08 Kong kernel: </IRQ> Oct 27 15:01:08 Kong kernel: <TASK> Oct 27 15:01:08 Kong kernel: asm_sysvec_apic_timer_interrupt+0x16/0x20 Oct 27 15:01:08 Kong kernel: RIP: 0010:cpuidle_enter_state+0x11d/0x202 Oct 27 15:01:08 Kong kernel: Code: 20 22 a0 ff 45 84 ff 74 1b 9c 58 0f 1f 40 00 0f ba e0 09 73 08 0f 0b fa 0f 1f 44 00 00 31 ff e8 4c e3 a4 ff fb 0f 1f 44 00 00 <45> 85 e4 0f 88 ba 00 00 00 48 8b 04 24 49 63 cc 48 6b d1 68 49 29 Oct 27 15:01:08 Kong kernel: RSP: 0018:ffffc900001c7e98 EFLAGS: 00000246 Oct 27 15:01:08 Kong kernel: RAX: ffff888feeb80000 RBX: ffff888108042400 RCX: 0000000000000000 Oct 27 15:01:08 Kong kernel: RDX: 0000a8bbb2035d2b RSI: ffffffff820ed4af RDI: ffffffff820ed9b8 Oct 27 15:01:08 Kong kernel: RBP: 0000000000000002 R08: 0000000000000002 R09: 0000000000000002 Oct 27 15:01:08 Kong kernel: R10: 0000000000000020 R11: 00000000000000ff R12: 0000000000000002 Oct 27 15:01:08 Kong kernel: R13: ffffffff82323720 R14: 0000a8bbb2035d2b R15: 0000000000000000 Oct 27 15:01:08 Kong kernel: ? cpuidle_enter_state+0xf7/0x202 Oct 27 15:01:08 Kong kernel: cpuidle_enter+0x2a/0x38 Oct 27 15:01:08 Kong kernel: do_idle+0x18d/0x1fb Oct 27 15:01:08 Kong kernel: cpu_startup_entry+0x1d/0x1f Oct 27 15:01:08 Kong kernel: start_secondary+0x101/0x101 Oct 27 15:01:08 Kong kernel: secondary_startup_64_no_verify+0xce/0xdb Oct 27 15:01:08 Kong kernel: </TASK> Oct 27 15:01:08 Kong kernel: ---[ end trace 0000000000000000 ]--- Oct 27 15:01:08 Kong kernel: BUG: scheduling while atomic: swapper/14/0/0x00010001 Oct 27 15:01:08 Kong kernel: Modules linked in: rpcsec_gss_krb5 nvidia_uvm(PO) xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle iptable_mangle vhost_net tun vhost vhost_iotlb tap ipvlan veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xfs ip6table_nat nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) algif_hash algif_skcipher af_alg cmac bnep tcp_diag inet_diag iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs 8021q garp mrp bridge stp llc mlx4_en mlx4_core igb i2c_algo_bit nvidia_drm(PO) nvidia_modeset(PO) edac_mce_amd edac_core intel_rapl_msr intel_rapl_common iosf_mbi kvm_amd nvidia(PO) kvm btusb btrtl video Oct 27 15:01:08 Kong kernel: drm_kms_helper btbcm crct10dif_pclmul crc32_pclmul btintel crc32c_intel ghash_clmulni_intel sha512_ssse3 drm bluetooth aesni_intel mpt3sas crypto_simd backlight cryptd syscopyarea sysfillrect wmi_bmof mxm_wmi raid_class ecdh_generic i2c_piix4 rapl sysimgblt tpm_crb ecc k10temp ccp scsi_transport_sas i2c_core fb_sys_fops nvme ahci cp210x tpm_tis input_leds tpm_tis_core nvme_core led_class usbserial libahci corsair_psu wmi tpm button acpi_cpufreq unix [last unloaded: mlx4_core] Oct 27 15:01:08 Kong kernel: Preemption disabled at: Oct 27 15:01:08 Kong kernel: [<0000000000000000>] 0x0 Oct 27 15:01:08 Kong kernel: CPU: 14 PID: 0 Comm: swapper/14 Tainted: P W O 6.1.49-Unraid #1 Oct 27 15:01:08 Kong kernel: Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 6203 07/27/2023 Oct 27 15:01:08 Kong kernel: Call Trace: Oct 27 15:01:08 Kong kernel: <TASK> Oct 27 15:01:08 Kong kernel: dump_stack_lvl+0x44/0x5c Oct 27 15:01:08 Kong kernel: __schedule_bug+0x9a/0xac Oct 27 15:01:08 Kong kernel: __schedule+0x59/0x612 Oct 27 15:01:08 Kong kernel: ? flush_smp_call_function_queue+0x12/0x83 Oct 27 15:01:08 Kong kernel: schedule_idle+0x27/0x3e Oct 27 15:01:08 Kong kernel: cpu_startup_entry+0x1d/0x1f Oct 27 15:01:08 Kong kernel: start_secondary+0x101/0x101 Oct 27 15:01:08 Kong kernel: secondary_startup_64_no_verify+0xce/0xdb Oct 27 15:01:08 Kong kernel: </TASK> Oct 27 16:06:34 Kong smbd[31382]: [2023/10/27 16:06:34.798515, 0] ../../source3/smbd/open.c:3306(smbd_calculate_maximum_allowed_access_fsp) Oct 27 16:06:34 Kong smbd[31382]: smbd_calculate_maximum_allowed_access_fsp: Could not get acl on file home-assistant/www/alerts/driveway.20231023_160000.1168020.3-1.jpg: NT_STATUS_ACCESS_DENIED kong-diagnostics-20231027-2351.zip

I've been having a crash at around 36-48 hours uptime since upgrading to 6.12.4. Upgraded to the latest motherboard BIOS. When I first upgraded to 6.12.x back in August and had this trouble I reverted to 6.11.5 and the problem did go away, now that I have time to work the problem I've upgraded again and am looking for answers. Diagnostics attached. kong-diagnostics-20231017-1023.zip

Had 2 hard lockups last week, but this time I saw an error without the lockup: [104171.459847] mdcmd (79): set md_write_method 1 [104771.511063] mdcmd (80): set md_write_method 0 [108155.198765] mdcmd (81): spindown 6 [108156.829365] mdcmd (82): spindown 5 [111744.657559] mdcmd (83): spindown 2 [113280.017493] mdcmd (84): spindown 4 [115274.774346] mdcmd (85): spindown 3 [120367.842034] mdcmd (86): set md_write_method 1 [129625.634158] kernel tried to execute NX-protected page - exploit attempt? (uid: 0) [129625.634385] BUG: unable to handle page fault for address: ffffc90008097f10 [129625.634606] #PF: supervisor instruction fetch in kernel mode [129625.634824] #PF: error_code(0x0011) - permissions violation [129625.635042] PGD 7fb424067 P4D 7fb424067 PUD 7fb425067 PMD 7f5037067 PTE 800000019ebe4163 [129625.635268] Oops: 0011 [#4] SMP NOPTI [129625.635481] CPU: 11 PID: 16040 Comm: shfs Tainted: G D 5.5.8-Unraid #1 [129625.635705] Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 5220 09/12/2019 [129625.636138] RIP: 0010:0xffffc90008097f10 [129625.636352] Code: ff ff 00 ca cd 4e 81 88 ff ff 00 00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 72 b5 14 81 ff ff ff ff 38 5d 48 00 40 15 00 00 <00> 48 d3 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [129625.637010] RSP: 0018:ffffc9000330bc08 EFLAGS: 00010282 [129625.637227] RAX: ffffc90008097f10 RBX: ffffc90008097f10 RCX: 0000000000000000 [129625.637450] RDX: 0000000000000000 RSI: ffffc90008097d28 RDI: ffff88879c5bb200 [129625.637671] RBP: ffff88879c5bb200 R08: 0000000000000000 R09: ffffc9000330bba8 [129625.637890] R10: 0000000000000068 R11: ffffc9000330bbe0 R12: ffff88877e7a3220 [129625.638107] R13: ffff8887f57bf9c0 R14: ffff88877e7a3250 R15: ffff88877e7a3230 [129625.638327] FS: 000014db1d91c700(0000) GS:ffff8887feac0000(0000) knlGS:0000000000000000 [129625.638548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [129625.638763] CR2: ffffc90008097f10 CR3: 000000079be62000 CR4: 00000000003406e0 [129625.638981] Call Trace: [129625.639198] ? fuse_request_end+0x185/0x19a [129625.639410] ? fuse_dev_do_write+0xa2e/0xa75 [129625.639621] ? fuse_dev_write+0x5b/0x75 [129625.639832] ? do_iter_readv_writev+0xb3/0xf3 [129625.640042] ? do_iter_write+0x7c/0xb8 [129625.640253] ? vfs_writev+0x74/0xb4 [129625.640462] ? __do_sys_newfstat+0x3c/0x5f [129625.640675] ? __fget_light+0x3d/0x47 [129625.640888] ? do_writev+0x79/0xe7 [129625.641102] ? do_syscall_64+0x7a/0x87 [129625.641317] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [129625.641530] Modules linked in: xt_CHECKSUM ipt_REJECT ip6table_mangle ip6table_nat iptable_mangle ip6table_filter ip6_tables vhost_net tun vhost tap veth xt_nat iptable_filter xfs nfsd lockd grace sunrpc md_mod iptable_nat xt_MASQUERADE nf_nat ip_tables wireguard ip6_udp_tunnel udp_tunnel bonding edac_mce_amd kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel igb wmi_bmof mxm_wmi mpt3sas aesni_intel k10temp crypto_simd i2c_piix4 i2c_algo_bit cryptd nvme i2c_core raid_class ahci scsi_transport_sas ccp glue_helper nvme_core libahci wmi button acpi_cpufreq [129625.643307] CR2: ffffc90008097f10 [129625.643960] ---[ end trace 2647bc31763b741b ]--- [129625.644222] RIP: 0010:0xffffc90003b23f08 [129625.644488] Code: Bad RIP value. [129625.644741] RSP: 0018:ffffc90003b53c08 EFLAGS: 00010282 [129625.645000] RAX: ffffc90003b23f08 RBX: ffffc90003b23f08 RCX: 0000000000000000 [129625.645266] RDX: 0000000000000000 RSI: ffffc90003b23d20 RDI: ffff88879c5bb200 [129625.645529] RBP: ffff88879c5bb200 R08: 0000000000000000 R09: ffffc90003b53ba8 [129625.645793] R10: 0000000000000068 R11: ffffc90003b53be0 R12: ffff88841b60fd48 [129625.646060] R13: ffff8887f57bf9c0 R14: ffff88841b60fd78 R15: ffff88841b60fd58 [129625.646325] FS: 000014db1d91c700(0000) GS:ffff8887feac0000(0000) knlGS:0000000000000000 [129625.646594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [129625.646855] CR2: ffffc90003b23ede CR3: 000000079be62000 CR4: 00000000003406e0 [170768.074968] mdcmd (87): set md_write_method 0 researching the error message indicates a possible fuse issue. Chip kong-diagnostics-20200428-2105.zip

any chance we can get the ddclient code updated to current? I'm looking for googledomains support (which I didn't even know existed!!!) https://sourceforge.net/projects/ddclient/ Thanks, Chipper

Would you consider adding my favorite Nerd Tool: multitail I'd be forever grateful. Chipper

@@host implies tcp transport, you may want @host for udp delivery? I too am looking for a standardized way to permanently alter the rsyslog.conf if you have a link I'd appreciate it. Chip