一、overview
目的：Linux系统中的崩溃通常会导致系统停止响应，这时系统会生成一个内核转储文件（也称为 "core dump" 或 "vmcore"）。"linux crash"工具的主要目的是帮助开发人员和系统管理员分析这些转储文件，以找出崩溃的原因。
分析功能："linux crash"工具提供了许多功能，用于在转储文件中执行各种分析。它可以显示CPU寄存器的内容、进程状态、内核模块信息、内存映射、中断状态、系统调用轨迹以及堆栈跟踪等。
调试符号：为了更好地分析转储文件，"linux crash"工具需要与匹配的内核调试符号一起使用。这些符号文件包含了内核的函数和变量名称，这使得"linux crash"能够更好地解释转储文件中的信息。
命令行界面："linux crash"工具是一个命令行实用程序，它需要在终端中运行。用户可以输入不同的命令来获取有关转储文件的不同信息。
调试崩溃：使用"linux crash"工具可以帮助识别可能导致系统崩溃的原因，例如内核错误、内存泄漏、硬件问题等。这对于解决Linux系统崩溃问题非常有用。
二、示例演示
1 生成ramdump文件
每个平台生成ramdump文件的方式不一样，我使用的平台，经过要求的配置后在发生系统crash后，会生成ramdump文件。
[ 178.896278] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000080
[ 178.919532] Mem abort info:
[ 178.919888] ESR = 0x96000044
[ 178.920276] EC = 0x25: DABT (current EL), IL = 32 bits
[ 178.920947] SET = 0, FnV = 0
[ 178.921334] EA = 0, S1PTW = 0
[ 178.921732] Data abort info:
[ 178.922096] ISV = 0, ISS = 0x00000044
[ 178.922581] CM = 0, WnR = 1
[ 178.922958] user pgtable: 4k pages, 48-bit VAs, pgdp=00000001821bb000
[ 178.923785] [0000000000000080] pgd=0000000000000000, p4d=0000000000000000
[ 178.924646] Internal error: Oops: 96000044 [#1] PREEMPT_RT SMP
[ 178.925384] Modules linked in: elpspacc_crypto elpspacc_dev aes_neon_bs aes_neon_blk soc_fb soc_can elpspacc spidev_dummy(O) elpspaccwrapper cfbfillrect bpu_fusa cfbcopyarea elpspacc_stl spidev(O) bpu_cores soc_can_stl crypto_simd algif_rng algif_hash elppdu algif_skcipher algif_aead tee_kpp spi_prot(O) soc_cpudai_jplus soc_rx_offload elppduwrapper aes_arm64 ghash_ce sm2_generic loop ecdh_generic sha256_generic jitterentropy_rng cryptd xts hmac libsha256 aes_generic des_generic md5 ghash_generic crct10dif_ce soc_ddr_stl_test chacha_neon can_raw can_bcm can_gw gcm devfreq_qos sm3_generic ansi_cprng gf128mul soc_dma_jplus bpu_framework_j51 ecc dh_generic soc_idu ccm ecb ctr cpuhp_qos cpufreq_qos drbg sm4_generic af_alg sha3_generic cmac echainiv crypto_user can_dev soc_ynr libdes libchacha libaes soc_i2c soc_vpu soc_spi can soc_jpu soc_stitch pwm_soc_lite5 pwm_soc_lite5_stl soc_isp ipv6 pwm_soc5 pwm_soc5_stl
[ 178.925591] soc_cim soc_gdc soc_pym_jplus soc_vin_vcon soc_sensor soc_deserial soc_camsys soc_vin_node soc_vio_common
[ 178.937594] CPU: 7 PID: 1816 Comm: vio_gw3 Tainted: G O 5.10.59-rt52-g8bad9deb09ec-dirty #2
[ 178.938811] Hardware name: Horizon Robotics Journey 5 DVB (DT)
[ 178.939547] pstate: 60c00005 (nZCv daif +PAN +UAO -TCO BTYPE=--)
[ 178.940306] pc : soc_ynr_fe_work+0x1a8/0x370 [soc_ynr]
[ 178.941024] lr : soc_ynr_fe_work+0x330/0x370 [soc_ynr]
[ 178.941731] sp : ffff8000180e3a30
[ 178.942151] pmr_save: 000000e0
[ 178.942538] x29: ffff8000180e3a40 x28: ffff800009991288
[ 178.943213] x27: ffff80000997c318 x26: ffff000185c5b500
[ 178.943888] x25: ffff00018a970cbc x24: ffff0001894b07b8
[ 178.944562] x23: ffff00018a970080 x22: ffff00018a9758d0
[ 178.945236] x21: ffff00018a970cb8 x20: ffff00018a970080
[ 178.945909] x19: 0000000000000000 x18: ffffffffffc14a87
[ 178.946583] x17: 0000000000000000 x16: 0000000000000000
[ 178.947256] x15: ffff000185c5b928 x14: ffff000185c5b500
[ 178.947929] x13: ffff8000980e3b37 x12: ffff8000180e3b50
[ 178.948603] x11: ffff8000114f8380 x10: ffff80001151b178
[ 178.949277] x9 : 00000000fffffffe x8 : 656d617266206563
[ 178.949951] x7 : 6e65726566657220 x6 : ffff8000180e38a0
[ 178.950623] x5 : 3a656d6172662065 x4 : 636e657265666572
[ 178.951296] x3 : 0000000000000001 x2 : 0000000000000080
[ 178.951970] x1 : ffff8000180e3ab8 x0 : 0000000000000000
[ 178.952645] Call trace:
[ 178.952956] soc_ynr_fe_work+0x1a8/0x370 [soc_ynr]
[ 178.953621] soc_ynr_handle_interrupt+0x2f0/0x318 [soc_ynr]
[ 178.954382] soc_ynr_frame_work+0x210/0x280 [soc_ynr]
[ 178.955078] frame_work_function+0xbc/0x230 [soc_vio_common]
[ 178.955841] kthread_worker_fn+0x94/0x160
[ 178.956357] kthread+0x160/0x188
[ 178.956770] ret_from_fork+0x10/0x18
[ 178.957232] Code: 910223e1 f9405700 a9709444 91020002 (a9081404)
[ 178.958000] Sending NMI from CPU 7 to CPUs 0-6:
[ 178.958578] NMI backtrace for cpu 2
[ 178.959029] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G O 5.10.59-rt52-g8bad9deb09ec-dirty #2
[ 178.960234] Hardware name: Horizon Robotics Journey 5 DVB (DT)
[ 178.960969] pstate: 40c00005 (nZcv daif +PAN +UAO -TCO BTYPE=--)
[ 178.961728] pc : __cpu_do_idle_irqprio+0x30/0x38
[ 178.962321] lr : __cpu_do_idle_irqprio+0x24/0x38
[ 178.962906] sp : ffff8000125bbf20
[ 178.963327] pmr_save: 00000060
[ 178.963713] x29: ffff8000125bbf20 x28: 0000000000000000
[ 178.964388] x27: 0000000000000000 x26: ffff0001801c6a00
[ 178.965062] x25: 0000000000000000 x24: 0000000000000000
[ 178.965737] x23: ffff0001801c6a00 x22: ffff8000113b2578
[ 178.966410] x21: ffff8000114dd710 x20: 0000000000000002
[ 178.967084] x19: ffff8000114dd678 x18: 0000000000000001
[ 178.967758] x17: 0000000000000000 x16: 0000000000000000
[ 178.968433] x15: ffff000180266e28 x14: ffffffffffffffff
[ 178.969107] x13: ffff800012e07028 x12: ffff800012df4080
[ 178.969780] x11: 0000000000000000 x10: 0000000000000950
[ 178.970454] x9 : ffff8000125bbe90 x8 : ffff0001801c73b0
[ 178.971128] x7 : ffff80025db9e000 x6 : 0000000103a275eb
[ 178.971801] x5 : 00ffffffffffffff x4 : 002946ff3f847628
[ 178.972475] x3 : 0000000000010252 x2 : 00000000000000f0
[ 178.973149] x1 : 0000000000000000 x0 : 0000000000000060
[ 178.973823] Call trace:
[ 178.974135] __cpu_do_idle_irqprio+0x30/0x38
[ 178.974678] cpu_do_idle+0x2c/0x38
[ 178.975113] arch_cpu_idle+0xc/0x30
[ 178.975558] default_idle_call+0x24/0x84
[ 178.976059] do_idle+0x23c/0x278
[ 178.976472] cpu_startup_entry+0x20/0x68
[ 178.976970] secondary_start_kernel+0x15c/0x1d0
[ 178.977548] NMI backtrace for cpu 5
[ 178.977995] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G O 5.10.59-rt52-g8bad9deb09ec-dirty #2
[ 178.979201] Hardware name: Horizon Robotics Journey 5 DVB (DT)
[ 178.979936] pstate: 40c00005 (nZcv daif +PAN +UAO -TCO BTYPE=--)
[ 178.980696] pc : __cpu_do_idle_irqprio+0x30/0x38
[ 178.981285] lr : __cpu_do_idle_irqprio+0x24/0x38
[ 178.981872] sp : ffff8000125d3f20
[ 178.982291] pmr_save: 00000060
[ 178.982678] x29: ffff8000125d3f20 x28: 0000000000000000
[ 178.983352] x27: 0000000000000000 x26: ffff000180211a80
[ 178.984026] x25: 0000000000000000 x24: 0000000000000000
[ 178.984700] x23: ffff000180211a80 x22: ffff8000113b2578
[ 178.985373] x21: ffff8000114dd710 x20: 0000000000000005
[ 178.986047] x19: ffff8000114dd678 x18: 0000000000000001
[ 178.986721] x17: 0000000000000000 x16: 0000000000000000
[ 178.987394] x15: ffff0001802d1ea8 x14: ffffffffffffffff
[ 178.988068] x13: ffff800012e67010 x12: ffff800012e5cde8
[ 178.988741] x11: 0000000000000000 x10: 0000000000000950
[ 178.989415] x9 : ffff8000125d3e90 x8 : ffff000180212430
[ 178.990088] x7 : 0000000000000009 x6 : 0000000000000060
[ 178.990761] x5 : ffff00026ef882c0 x4 : ffff80025dbd4000
[ 178.991434] x3 : 0000000000010b5e x2 : 00000000000000f0
[ 178.992107] x1 : 0000000000000000 x0 : 0000000000000060
[ 178.992781] Call trace:
[ 178.993092] __cpu_do_idle_irqprio+0x30/0x38
[ 178.993636] cpu_do_idle+0x2c/0x38
[ 178.994070] arch_cpu_idle+0xc/0x30
[ 178.994515] default_idle_call+0x24/0x84
[ 178.995016] do_idle+0x23c/0x278
[ 178.995427] cpu_startup_entry+0x24/0x68
[ 178.995925] secondary_start_kernel+0x15c/0x1d0
[ 178.996503] NMI backtrace for cpu 4
[ 178.996952] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G O 5.10.59-rt52-g8bad9deb09ec-dirty #2
[ 178.998158] Hardware name: Horizon Robotics Journey 5 DVB (DT)
[ 178.998893] pstate: 40c00005 (nZcv daif +PAN +UAO -TCO BTYPE=--)
[ 178.999652] pc : __cpu_do_idle_irqprio+0x30/0x38
[ 179.000241] lr : __cpu_do_idle_irqprio+0x24/0x38
[ 179.000828] sp : ffff8000125cbf20
[ 179.001247] pmr_save: 00000060
[ 179.001634] x29: ffff8000125cbf20 x28: 0000000000000000
[ 179.002308] x27: 0000000000000000 x26: ffff000180210d40
[ 179.002982] x25: 0000000000000000 x24: 0000000000000000
[ 179.003656] x23: ffff000180210d40 x22: ffff8000113b2578
[ 179.004330] x21: ffff8000114dd710 x20: 0000000000000004
[ 179.005004] x19: ffff8000114dd678 x18: 0000000000000001
[ 179.005678] x17: 0000000000000000 x16: 0000000000000000
[ 179.006352] x15: ffff0001802bb928 x14: ffffffffffffffff
[ 179.007025] x13: ffff800012e47018 x12: ffff800012e42509
[ 179.007699] x11: 0000000000000001 x10: 0000000000000950
[ 179.008372] x9 : ffff8000125cbe90 x8 : ffff0001802116f0
[ 179.009046] x7 : ffff00026ef72c28 x6 : 0000000000000060
[ 179.009719] x5 : ffff80025dbc2000 x4 : ffff00026ef79300
[ 179.010392] x3 : 0000000000032686 x2 : 00000000000000f0
[ 179.011066] x1 : 0000000000000000 x0 : 0000000000000060
[ 179.011739] Call trace:
[ 179.012050] __cpu_do_idle_irqprio+0x30/0x38
[ 179.012593] cpu_do_idle+0x2c/0x38
[ 179.013028] arch_cpu_idle+0xc/0x30
[ 179.013473] default_idle_call+0x24/0x84
[ 179.013973] do_idle+0x23c/0x278
[ 179.014385] cpu_startup_entry+0x20/0x68
[ 179.014884] secondary_start_kernel+0x15c/0x1d0
[ 179.015460] NMI backtrace for cpu 3
[ 179.015908] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G O 5.10.59-rt52-g8bad9deb09ec-dirty #2
[ 179.017114] Hardware name: Horizon Robotics Journey 5 DVB (DT)
[ 179.017849] pstate: 40c00005 (nZcv daif +PAN +UAO -TCO BTYPE=--)
[ 179.018609] pc : __cpu_do_idle_irqprio+0x30/0x38
[ 179.019197] lr : __cpu_do_idle_irqprio+0x24/0x38
[ 179.019782] sp : ffff8000125c3f20
[ 179.020202] pmr_save: 00000060
[ 179.020589] x29: ffff8000125c3f20 x28: 0000000000000000
[ 179.021263] x27: 0000000000000000 x26: ffff000180210000
[ 179.021938] x25: 0000000000000000 x24: 0000000000000000
[ 179.022611] x23: ffff000180210000 x22: ffff8000113b2578
[ 179.023285] x21: ffff8000114dd710 x20: 0000000000000003
[ 179.023958] x19: ffff8000114dd678 x18: 0000000000000001
[ 179.024632] x17: 0000000000000079 x16: 0000000000000000
[ 179.025306] x15: ffff0001802753a8 x14: ffffffffffffffff
[ 179.025980] x13: ffff800012e27020 x12: ffff800012e12967
[ 179.026653] x11: 0000000000000000 x10: 0000000000000950
[ 179.027327] x9 : ffff8000125c3e90 x8 : ffff0001802109b0
[ 179.028001] x7 : 0000000000000009 x6 : 0000000000000060
[ 179.028674] x5 : ffff00026ef642c0 x4 : ffff80025dbb0000
[ 179.029348] x3 : 00000000000168de x2 : 00000000000000f0
[ 179.030022] x1 : 0000000000000000 x0 : 0000000000000060
[ 179.030696] Call trace:
[ 179.031006] __cpu_do_idle_irqprio+0x30/0x38
[ 179.031550] cpu_do_idle+0x2c/0x38
[ 179.031984] arch_cpu_idle+0xc/0x30
[ 179.032429] default_idle_call+0x24/0x84
[ 179.032929] do_idle+0x23c/0x278
[ 179.033341] cpu_startup_entry+0x20/0x68
[ 179.033840] secondary_start_kernel+0x15c/0x1d0
[ 179.034416] NMI backtrace for cpu 1
[ 179.034864] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O 5.10.59-rt52-g8bad9deb09ec-dirty #2
[ 179.036069] Hardware name: Horizon Robotics Journey 5 DVB (DT)
[ 179.036804] pstate: 40c00005 (nZcv daif +PAN +UAO -TCO BTYPE=--)
[ 179.037563] pc : __cpu_do_idle_irqprio+0x30/0x38
[ 179.038152] lr : __cpu_do_idle_irqprio+0x24/0x38
[ 179.038738] sp : ffff8000125b3f20
[ 179.039157] pmr_save: 00000060
[ 179.039543] x29: ffff8000125b3f20 x28: 0000000000000000
[ 179.040218] x27: 0000000000000000 x26: ffff0001801c5cc0
[ 179.040893] x25: 0000000000000000 x24: 0000000000000000
[ 179.041566] x23: ffff0001801c5cc0 x22: ffff8000113b2578
[ 179.042240] x21: ffff8000114dd710 x20: 0000000000000001
[ 179.042913] x19: ffff8000114dd678 x18: 0000000000000001
[ 179.043587] x17: 0000000000000000 x16: 0000000000000000
[ 179.044261] x15: ffff000180261168 x14: ffffffffffffffff
[ 179.044935] x13: ffff800012de7030 x12: ffff800012de31fb
[ 179.045609] x11: 0000000000000001 x10: 0000000000000950
[ 179.046283] x9 : ffff8000125b3e90 x8 : ffff0001801c6670
[ 179.046956] x7 : 0000000000000009 x6 : 0000000000000060
[ 179.047630] x5 : ffff00026ef402c0 x4 : ffff80025db8c000
[ 179.048303] x3 : 0000000000010e82 x2 : 00000000000000f0
[ 179.048976] x1 : 0000000000000000 x0 : 0000000000000060
[ 179.049650] Call trace:
[ 179.049960] __cpu_do_idle_irqprio+0x30/0x38
[ 179.050504] cpu_do_idle+0x2c/0x38
[ 179.050938] arch_cpu_idle+0xc/0x30
[ 179.051383] default_idle_call+0x24/0x84
[ 179.051883] do_idle+0x23c/0x278
[ 179.052295] cpu_startup_entry+0x24/0x68
[ 179.052793] secondary_start_kernel+0x15c/0x1d0
[ 179.053370] NMI backtrace for cpu 6
[ 179.053818] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G O 5.10.59-rt52-g8bad9deb09ec-dirty #2
[ 179.055024] Hardware name: Horizon Robotics Journey 5 DVB (DT)
[ 179.055759] pstate: 40c00005 (nZcv daif +PAN +UAO -TCO BTYPE=--)
[ 179.056520] pc : __cpu_do_idle_irqprio+0x30/0x38
[ 179.057109] lr : __cpu_do_idle_irqprio+0x24/0x38
[ 179.057695] sp : ffff8000125dbf20
[ 179.058115] pmr_save: 00000060
[ 179.058502] x29: ffff8000125dbf20 x28: 0000000000000000
[ 179.059176] x27: 0000000000000000 x26: ffff0001802127c0
[ 179.059851] x25: 0000000000000000 x24: 0000000000000000
[ 179.060524] x23: ffff0001802127c0 x22: ffff8000113b2578
[ 179.061198] x21: ffff8000114dd710 x20: 0000000000000006
[ 179.061871] x19: ffff8000114dd678 x18: 0000000000000001
[ 179.062544] x17: 0000000000000000 x16: 0000000000000000
[ 179.063217] x15: ffff000180320428 x14: ffffffffffffffff
[ 179.063890] x13: ffff800012e87008 x12: ffff800012e725cd
[ 179.064564] x11: 0000000000000000 x10: 0000000000000950
[ 179.065238] x9 : ffff8000125dbe90 x8 : ffff000180213170
[ 179.065912] x7 : 0000000000000009 x6 : 0000000000000060
[ 179.066585] x5 : ffff00026ef9a2c0 x4 : ffff80025dbe6000
[ 179.067258] x3 : 000000000000fa86 x2 : 00000000000000f0
[ 179.067932] x1 : 0000000000000000 x0 : 0000000000000060
[ 179.068605] Call trace:
[ 179.068916] __cpu_do_idle_irqprio+0x30/0x38
[ 179.069459] cpu_do_idle+0x2c/0x38
[ 179.069894] arch_cpu_idle+0xc/0x30
[ 179.070339] default_idle_call+0x24/0x84
[ 179.070838] do_idle+0x23c/0x278
[ 179.071250] cpu_startup_entry+0x20/0x68
[ 179.071748] secondary_start_kernel+0x15c/0x1d0
[ 179.072324] NMI backtrace for cpu 0
[ 179.072771] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 5.10.59-rt52-g8bad9deb09ec-dirty #2
[ 179.073977] Hardware name: Horizon Robotics Journey 5 DVB (DT)
[ 179.074712] pstate: 40c00005 (nZcv daif +PAN +UAO -TCO BTYPE=--)
[ 179.075472] pc : __cpu_do_idle_irqprio+0x30/0x38
[ 179.076060] lr : __cpu_do_idle_irqprio+0x24/0x38
[ 179.076647] sp : ffff8000114d3e90
[ 179.077066] pmr_save: 00000060
[ 179.077453] x29: ffff8000114d3e90 x28: ffff800010c30108
[ 179.078128] x27: 0000000000000200 x26: ffff8000114ecc00
[ 179.078801] x25: 0000000000000000 x24: 0000000000000000
[ 179.079475] x23: ffff8000114ecc00 x22: ffff8000113b2578
[ 179.080149] x21: ffff8000114dd710 x20: 0000000000000000
[ 179.080823] x19: ffff8000114dd678 x18: 0000000000000001
[ 179.081497] x17: 0000000000000000 x16: 0000000000000000
[ 179.082171] x15: ffff0001801c1168 x14: ffffffffffffffff
[ 179.082845] x13: ffff800012dc7038 x12: ffff800012daf2cd
[ 179.083518] x11: 0000000000000004 x10: 0000000000000950
[ 179.084193] x9 : ffff8000114d3e00 x8 : ffff8000114ed5b0
[ 179.084867] x7 : 0000000000000009 x6 : 0000000000000060
[ 179.085540] x5 : ffff00026ef2e2c0 x4 : ffff80025db7a000
[ 179.086213] x3 : 0000000000011c9a x2 : 00000000000000f0
[ 179.086886] x1 : 0000000000000000 x0 : 0000000000000060
[ 179.087560] Call trace:
[ 179.087871] __cpu_do_idle_irqprio+0x30/0x38
[ 179.088414] cpu_do_idle+0x2c/0x38
[ 179.088848] arch_cpu_idle+0xc/0x30
[ 179.089292] default_idle_call+0x24/0x84
[ 179.089793] do_idle+0x23c/0x278
[ 179.090205] cpu_startup_entry+0x24/0x68
[ 179.090703] rest_init+0xd0/0xdc
[ 179.091116] arch_call_rest_init+0xc/0x14
[ 179.091629] start_kernel+0x6e4/0x720
[ 179.092595] ---[ end trace 0000000000000002 ]---
.................................................接着系统开始重启，在uboot阶段开始生成ramdump文件
等待几分钟后，进入系统，查看生成的ramdump文件：
root@:/map/log/ramdump# ls
DDRCS0-1.bin DDRCS0-2.bin DDRCS0-3.bin DDRCS0-4.bin DDRCS0-5.bin DDRCS0-6.bin DDRCS0-7.bin cpu-contexts.bin
root@:/map/log/ramdump# 把这些文件拷贝到uboot系统进行解析。
2 准备解析
将需要的文件拷贝到自己建立的目录。
zhilu.zhang@:~/work/ram-dump$ ls -l
total 8479364
-rwxr-xr-x 1 zhilu.zhang zhilu.zhang 4544 Jul 22 13:14 cpu-contexts.bin #生成的ramdump文件
-rwxrw-r-- 1 zhilu.zhang zhilu.zhang 103530024 Jul 22 13:23 crash #crash软件
-rwxr-xr-x 1 zhilu.zhang zhilu.zhang 2147483648 Jul 22 13:10 DDRCS0-1.bin #生成的ramdump文件
-rwxr-xr-x 1 zhilu.zhang zhilu.zhang 1073741824 Jul 22 13:10 DDRCS0-2.bin #生成的ramdump文件
-rwxr-xr-x 1 zhilu.zhang zhilu.zhang 1073741824 Jul 22 13:11 DDRCS0-3.bin #生成的ramdump文件
-rwxr-xr-x 1 zhilu.zhang zhilu.zhang 1073741824 Jul 22 13:12 DDRCS0-4.bin #生成的ramdump文件
-rwxr-xr-x 1 zhilu.zhang zhilu.zhang 1073741824 Jul 22 13:13 DDRCS0-5.bin #生成的ramdump文件
-rwxr-xr-x 1 zhilu.zhang zhilu.zhang 1073741824 Jul 22 13:13 DDRCS0-6.bin #生成的ramdump文件
-rwxr-xr-x 1 zhilu.zhang zhilu.zhang 795410432 Jul 22 13:14 DDRCS0-7.bin #生成的ramdump文件
-rw-rw-r-- 1 zhilu.zhang zhilu.zhang 260922 Jul 22 13:06 soc_device.ko #将要调试的ko
-rwxrwxr-x 1 zhilu.zhang zhilu.zhang 267479872 Jul 22 13:07 vmlinux #编译系统生成的文件
-rw-rw-r-- 1 zhilu.zhang zhilu.zhang 14245 Jul 22 13:22 vmlinux.lds #系统符号表
zhilu.zhang@:~/work/ram-dump$ 3 开始crash
crash加载ramdump的命令crash ./vmlinux /dev/zero@0x80000000,DDRCS0-1.bin@0x80970000,DDRCS0-2.bin@0x100970000,DDRCS0-3.bin@0x140970000,DDRCS0-4.bin@0x180970000,DDRCS0-5.bin@0x1c0970000,DDRCS0-6.bin@0x200970000,DDRCS0-7.bin@0x240970000 --machdep vabits_actual=48
使用crash 8.0.1以上版本；
0x80000000->0x80970000是secure memory无法使用，这部分内存需要使用/dev/zero填0；
dumpfile和物理地址对应关系在uboot dump的log中显示，如DDRCS0-1.bin@0x80970000；
--machdep vabits_actual=48 TODO；
加载进入：
zhilu.zhang@server:~/work/ram-dump$ ./crash ./vmlinux /dev/zero@0x80000000,DDRCS0-1.bin@0x80970000,DDRCS0-2.bin@0x100970000,DDRCS0-3.bin@0x140970000,DDRCS0-4.bin@0x180970000,DDRCS0-5.bin@0x1c0970000,DDRCS0-6.bin@0x200970000,DDRCS0-7.bin@0x240970000 --machdep vabits_actual=48
crash 8.0.1++
Copyright (C) 2002-2022 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
NOTE: setting vabits_actual to: 48
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=aarch64-elf-linux".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
WARNING: cpu 0: cannot find NT_PRSTATUS note
WARNING: cpu 1: cannot find NT_PRSTATUS note
WARNING: cpu 2: cannot find NT_PRSTATUS note
WARNING: cpu 3: cannot find NT_PRSTATUS note
WARNING: cpu 4: cannot find NT_PRSTATUS note
WARNING: cpu 5: cannot find NT_PRSTATUS note
WARNING: cpu 6: cannot find NT_PRSTATUS note
WARNING: cpu 7: cannot find NT_PRSTATUS note
KERNEL: ./vmlinux [TAINTED]
DUMPFILES: /var/tmp/ramdump_elf_jVAqvf [temporary ELF header]
/dev/zero
DDRCS0-1.bin
DDRCS0-2.bin
DDRCS0-3.bin
DDRCS0-4.bin
DDRCS0-5.bin
DDRCS0-6.bin
DDRCS0-7.bin
CPUS: 8 [OFFLINE: 7]
DATE: Sat Jul 22 13:00:06 CST 2023
UPTIME: 00:02:59
LOAD AVERAGE: 1.08, 0.50, 0.20
TASKS: 339
NODENAME: j5dvb
RELEASE: 5.10.59-rt52-g8bad9deb09ec-dirty
VERSION: #2 SMP PREEMPT_RT Sat Jul 22 12:13:45 CST 2023
MACHINE: aarch64 (unknown Mhz)
MEMORY: 7.8 GB
PANIC: "Unable to handle kernel NULL pointer dereference at virtual address 0000000000000080"
PID: 1816
COMMAND: "vio_gw3"
TASK: ffff000185c5b500 [THREAD_INFO: ffff000185c5b500]
CPU: 7
STATE: TASK_RUNNING (PANIC)
crash>
crash>
crash> 输入dmesg查看log：
crash> dmesg
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x412fd050]
[ 0.000000] Linux version 5.10.59-rt52-g8bad9deb09ec-dirty (root@hz-server-1) (aarch64-linux-gnu-gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #2 SMP PREEMPT_RT Sat Jul 22 12:13:45 CST 2023
[ 0.000000] Machine model: Horizon Robotics Journey 5 DVB
[ 0.000000] earlycon: uart8250 at MMIO32 0x0000000043b90000 (options '921600n8')
[ 0.000000] printk: bootconsole [uart8250] enabled
[ 0.000000] cvsubsys: get_soc_socver socver:2
[ 0.000000] Reserved memory: created CMA memory pool at 0x0000000170000000, size 256 MiB
[ 0.000000] OF: reserved mem: initialized node ion_cma, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created CMA memory pool at 0x00000000a4400000, size 64 MiB
[ 0.000000] OF: reserved mem: initialized node global_cma, compatible id shared-dma-pool
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000020400000-0x00000000ffffffff]
[ 0.000000] DMA32 empty
[ 0.000000] Normal [mem 0x0000000100000000-0x000000026fffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000020400000-0x00000000205fffff]
[ 0.000000] node 0: [mem 0x0000000080000000-0x000000008096ffff]
[ 0.000000] node 0: [mem 0x0000000080970000-0x00000000903fffff]
[ 0.000000] node 0: [mem 0x0000000090400000-0x00000000943fffff]
[ 0.000000] node 0: [mem 0x0000000094400000-0x00000000947fffff]
[ 0.000000] node 0: [mem 0x0000000094800000-0x0000000094dfffff]
[ 0.000000] node 0: [mem 0x0000000094e00000-0x0000000095ffffff]
[ 0.000000] node 0: [mem 0x0000000096000000-0x000000009c3fffff]
[ 0.000000] node 0: [mem 0x000000009c400000-0x00000000a03fffff]
[ 0.000000] node 0: [mem 0x00000000a0400000-0x00000000a43fffff]
[ 0.000000] node 0: [mem 0x00000000a4400000-0x000000026fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000020400000-0x000000026fffffff]
...................................................................................
178.895988] YNR: soc_ynr_video_start 1726 source node, ignored
[ 178.895997] [VPF ops]:[S0][N3][C0][V0] vpf_video_start: done start_cnt 1
[ 178.896010] [VPF ops]:[S0][N3][V8] vpf_prepare_buffer: done
[ 178.896015] [VIO fmgr]:FRM_ID[0308](Free:0; Request:5; Process:0; Complete:0; Used:0)
[ 178.896026] [VIO fmgr]:FRM_ID[0308](Free:0; Request:4; Process:1; Complete:0; Used:0)
[ 178.896034] YNR: [S0][V8]soc_ynr_video_start leader 1
[ 178.896038] [VPF ops]:[S0][N3][C0][V8] vpf_video_start: done start_cnt 2
[ 178.896164] YNR: soc_ynr_frame_work 965 ctx_id = 0
[ 178.896173] [VIO fmgr]:FRM_ID[0300](Free:15; Request:1; Process:0; Complete:0; Used:0)
[ 178.896184] [VIO fmgr]:FRM_ID[0308](Free:0; Request:4; Process:1; Complete:0; Used:0)
[ 178.896192] YNR: soc_ynr_handle_interrupt 2964 flow_id = 0
[ 178.896197] [VIO fmgr]:FRM_ID[0308](Free:0; Request:4; Process:1; Complete:0; Used:0)
[ 178.896205] YNR: soc_ynr_handle_interrupt 3006 FS
[ 178.896209] YNR: soc_ynr_time_out_en 462 Channel 0 is using timeout detection
[ 178.896215] YNR: soc_ynr_time_out_en 479 Timeout detection will be turned on
[ 178.896222] YNR: soc_ynr_pop_compare 565 Failed to get the comparison frame from the stack of channel 0, maybe this is the first frame
[ 178.896228] YNR: soc_ynr_input_compare_frame 765 Ynr's channel 0 read comparison frame failed because the frame is NULL
[ 178.896233] [VIO fmgr]:FRM_ID[0308](Free:0; Request:3; Process:2; Complete:0; Used:0)
[ 178.896241] YNR: soc_ynr_handle_interrupt 2964 flow_id = 0
[ 178.896245] [VIO fmgr]:FRM_ID[0308](Free:0; Request:3; Process:2; Complete:0; Used:0)
[ 178.896252] YNR: soc_ynr_handle_interrupt 3011 FE
[ 178.896260] YNR: soc_ynr_fe_work current frame: V0_640x480_F0_T1690002675890-S1690002006-U669890.yuv
[ 178.896263] YNR: soc_ynr_pop_compare 565 Failed to get the comparison frame from the stack of channel 0, maybe this is the first frame
[ 178.896269] YNR: soc_ynr_fe_work reference frame: V0_640x480 NA
[ 178.896278] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000080
[ 178.919532] Mem abort info:
[ 178.919888] ESR = 0x96000044
[ 178.920276] EC = 0x25: DABT (current EL), IL = 32 bits
[ 178.920947] SET = 0, FnV = 0
[ 178.921334] EA = 0, S1PTW = 0
[ 178.921732] Data abort info:
[ 178.922096] ISV = 0, ISS = 0x00000044
[ 178.922581] CM = 0, WnR = 1
[ 178.922958] user pgtable: 4k pages, 48-bit VAs, pgdp=00000001821bb000
[ 178.923785] [0000000000000080] pgd=0000000000000000, p4d=0000000000000000
[ 178.924646] Internal error: Oops: 96000044 [#1] PREEMPT_RT SMP
[ 178.925384] Modules linked in: elpspacc_crypto elpspacc_dev aes_neon_bs aes_neon_blk soc_fb soc_can elpspacc spidev_dummy(O) elpspaccwrapper cfbfillrect bpu_fusa cfbcopyarea elpspacc_stl spidev(O) bpu_cores soc_can_stl crypto_simd algif_rng algif_hash elppdu algif_skcipher algif_aead tee_kpp spi_prot(O) soc_cpudai_jplus soc_rx_offload elppduwrapper aes_arm64 ghash_ce sm2_generic loop ecdh_generic sha256_generic jitterentropy_rng cryptd xts hmac libsha256 aes_generic des_generic md5 ghash_generic crct10dif_ce soc_ddr_stl_test chacha_neon can_raw can_bcm can_gw gcm devfreq_qos sm3_generic ansi_cprng gf128mul soc_dma_jplus bpu_framework_j51 ecc dh_generic soc_idu ccm ecb ctr cpuhp_qos cpufreq_qos drbg sm4_generic af_alg sha3_generic cmac echainiv crypto_user can_dev soc_ynr libdes libchacha libaes soc_i2c soc_vpu soc_spi can soc_jpu soc_stitch pwm_soc_lite5 pwm_soc_lite5_stl soc_isp ipv6 pwm_soc5 pwm_soc5_stl
[ 178.925591] soc_cim soc_gdc soc_pym_jplus soc_vin_vcon soc_sensor soc_deserial soc_camsys soc_vin_node soc_vio_common
[ 178.937594] CPU: 7 PID: 1816 Comm: vio_gw3 Tainted: G O 5.10.59-rt52-g8bad9deb09ec-dirty #2
[ 178.938811] Hardware name: Horizon Robotics Journey 5 DVB (DT)
[ 178.939547] pstate: 60c00005 (nZCv daif +PAN +UAO -TCO BTYPE=--)
[ 178.940306] pc : soc_ynr_fe_work+0x1a8/0x370 [soc_ynr]
[ 178.941024] lr : soc_ynr_fe_work+0x330/0x370 [soc_ynr]
[ 178.941731] sp : ffff8000180e3a30
[ 178.942151] pmr_save: 000000e0
[ 178.942538] x29: ffff8000180e3a40 x28: ffff800009991288
[ 178.943213] x27: ffff80000997c318 x26: ffff000185c5b500
[ 178.943888] x25: ffff00018a970cbc x24: ffff0001894b07b8
[ 178.944562] x23: ffff00018a970080 x22: ffff00018a9758d0
[ 178.945236] x21: ffff00018a970cb8 x20: ffff00018a970080
[ 178.945909] x19: 0000000000000000 x18: ffffffffffc14a87
[ 178.946583] x17: 0000000000000000 x16: 0000000000000000
[ 178.947256] x15: ffff000185c5b928 x14: ffff000185c5b500
[ 178.947929] x13: ffff8000980e3b37 x12: ffff8000180e3b50
[ 178.948603] x11: ffff8000114f8380 x10: ffff80001151b178
[ 178.949277] x9 : 00000000fffffffe x8 : 656d617266206563
[ 178.949951] x7 : 6e65726566657220 x6 : ffff8000180e38a0
[ 178.950623] x5 : 3a656d6172662065 x4 : 636e657265666572
[ 178.951296] x3 : 0000000000000001 x2 : 0000000000000080
[ 178.951970] x1 : ffff8000180e3ab8 x0 : 0000000000000000
[ 178.952645] Call trace:
[ 178.952956] soc_ynr_fe_work+0x1a8/0x370 [soc_ynr]
[ 178.953621] soc_ynr_handle_interrupt+0x2f0/0x318 [soc_ynr]
[ 178.954382] soc_ynr_frame_work+0x210/0x280 [soc_ynr]
[ 178.955078] frame_work_function+0xbc/0x230 [soc_vio_common]
[ 178.955841] kthread_worker_fn+0x94/0x160
[ 178.956357] kthread+0x160/0x188
[ 178.956770] ret_from_fork+0x10/0x18
[ 178.957232] Code: 910223e1 f9405700 a9709444 91020002 (a9081404)
[ 178.958000] Sending NMI from CPU 7 to CPUs 0-6:
[ 178.958578] NMI backtrace for cpu 2
[ 178.959029] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G O 5.10.59-rt52-g8bad9deb09ec-dirty #2
[ 178.960234] Hardware name: Horizon Robotics Journey 5 DVB (DT)
[ 178.960969] pstate: 40c00005 (nZcv daif +PAN +UAO -TCO BTYPE=--)
[ 178.961728] pc : __cpu_do_idle_irqprio+0x30/0x38
[ 178.962321] lr : __cpu_do_idle_irqprio+0x24/0x38
[ 178.962906] sp : ffff8000125bbf20
[ 178.963327] pmr_save: 00000060
[ 178.963713] x29: ffff8000125bbf20 x28: 0000000000000000
[ 178.964388] x27: 0000000000000000 x26: ffff0001801c6a00
[ 178.965062] x25: 0000000000000000 x24: 0000000000000000
[ 178.965737] x23: ffff0001801c6a00 x22: ffff8000113b2578
[ 178.966410] x21: ffff8000114dd710 x20: 0000000000000002
[ 178.967084] x19: ffff8000114dd678 x18: 0000000000000001
[ 178.967758] x17: 0000000000000000 x16: 0000000000000000
[ 178.968433] x15: ffff000180266e28 x14: ffffffffffffffff
[ 178.969107] x13: ffff800012e07028 x12: ffff800012df4080
[ 178.969780] x11: 0000000000000000 x10: 0000000000000950
[ 178.970454] x9 : ffff8000125bbe90 x8 : ffff0001801c73b0
[ 178.971128] x7 : ffff80025db9e000 x6 : 0000000103a275eb
[ 178.971801] x5 : 00ffffffffffffff x4 : 002946ff3f847628
[ 178.972475] x3 : 0000000000010252 x2 : 00000000000000f0
[ 178.973149] x1 : 0000000000000000 x0 : 0000000000000060
[ 178.973823] Call trace:
[ 178.974135] __cpu_do_idle_irqprio+0x30/0x38
[ 178.974678] cpu_do_idle+0x2c/0x38
[ 178.975113] arch_cpu_idle+0xc/0x30
[ 178.975558] default_idle_call+0x24/0x84
[ 178.976059] do_idle+0x23c/0x278
[ 178.976472] cpu_startup_entry+0x20/0x68
[ 178.976970] secondary_start_kernel+0x15c/0x1d0
[ 178.977548] NMI backtrace for cpu 5
[ 178.977995] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G O 5.10.59-rt52-g8bad9deb09ec-dirty #2
[ 178.979201] Hardware name: Horizon Robotics Journey 5 DVB (DT)
[ 178.979936] pstate: 40c00005 (nZcv daif +PAN +UAO -TCO BTYPE=--)
[ 178.980696] pc : __cpu_do_idle_irqprio+0x30/0x38
[ 178.981285] lr : __cpu_do_idle_irqprio+0x24/0x38
[ 178.981872] sp : ffff8000125d3f20
[ 178.982291] pmr_save: 00000060
[ 178.982678] x29: ffff8000125d3f20 x28: 0000000000000000
[ 178.983352] x27: 0000000000000000 x26: ffff000180211a80
[ 178.984026] x25: 0000000000000000 x24: 0000000000000000
[ 178.984700] x23: ffff000180211a80 x22: ffff8000113b2578
[ 178.985373] x21: ffff8000114dd710 x20: 0000000000000005
[ 178.986047] x19: ffff8000114dd678 x18: 0000000000000001
[ 178.986721] x17: 0000000000000000 x16: 0000000000000000
[ 178.987394] x15: ffff0001802d1ea8 x14: ffffffffffffffff
[ 178.988068] x13: ffff800012e67010 x12: ffff800012e5cde8
[ 178.988741] x11: 0000000000000000 x10: 0000000000000950
[ 178.989415] x9 : ffff8000125d3e90 x8 : ffff000180212430
[ 178.990088] x7 : 0000000000000009 x6 : 0000000000000060
[ 178.990761] x5 : ffff00026ef882c0 x4 : ffff80025dbd4000
[ 178.991434] x3 : 0000000000010b5e x2 : 00000000000000f0
[ 178.992107] x1 : 0000000000000000 x0 : 0000000000000060
[ 178.992781] Call trace:
[ 178.993092] __cpu_do_idle_irqprio+0x30/0x38
[ 178.993636] cpu_do_idle+0x2c/0x38
[ 178.994070] arch_cpu_idle+0xc/0x30
[ 178.994515] default_idle_call+0x24/0x84
[ 178.995016] do_idle+0x23c/0x278
[ 178.995427] cpu_startup_entry+0x24/0x68
[ 178.995925] secondary_start_kernel+0x15c/0x1d0
[ 178.996503] NMI backtrace for cpu 4
[ 178.996952] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G O 5.10.59-rt52-g8bad9deb09ec-dirty #2
[ 178.998158] Hardware name: Horizon Robotics Journey 5 DVB (DT)
[ 178.998893] pstate: 40c00005 (nZcv daif +PAN +UAO -TCO BTYPE=--)
[ 178.999652] pc : __cpu_do_idle_irqprio+0x30/0x38
[ 179.000241] lr : __cpu_do_idle_irqprio+0x24/0x38
[ 179.000828] sp : ffff8000125cbf20
[ 179.001247] pmr_save: 00000060
[ 179.001634] x29: ffff8000125cbf20 x28: 0000000000000000
[ 179.002308] x27: 0000000000000000 x26: ffff000180210d40
[ 179.002982] x25: 0000000000000000 x24: 0000000000000000
[ 179.003656] x23: ffff000180210d40 x22: ffff8000113b2578
[ 179.004330] x21: ffff8000114dd710 x20: 0000000000000004
[ 179.005004] x19: ffff8000114dd678 x18: 0000000000000001
[ 179.005678] x17: 0000000000000000 x16: 0000000000000000
[ 179.006352] x15: ffff0001802bb928 x14: ffffffffffffffff
[ 179.007025] x13: ffff800012e47018 x12: ffff800012e42509
[ 179.007699] x11: 0000000000000001 x10: 0000000000000950
[ 179.008372] x9 : ffff8000125cbe90 x8 : ffff0001802116f0
[ 179.009046] x7 : ffff00026ef72c28 x6 : 0000000000000060
[ 179.009719] x5 : ffff80025dbc2000 x4 : ffff00026ef79300
[ 179.010392] x3 : 0000000000032686 x2 : 00000000000000f0
[ 179.011066] x1 : 0000000000000000 x0 : 0000000000000060
[ 179.011739] Call trace:
[ 179.012050] __cpu_do_idle_irqprio+0x30/0x38
[ 179.012593] cpu_do_idle+0x2c/0x38
[ 179.013028] arch_cpu_idle+0xc/0x30
[ 179.013473] default_idle_call+0x24/0x84
[ 179.013973] do_idle+0x23c/0x278
[ 179.014385] cpu_startup_entry+0x20/0x68
[ 179.014884] secondary_start_kernel+0x15c/0x1d0
[ 179.015460] NMI backtrace for cpu 3
[ 179.015908] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G O 5.10.59-rt52-g8bad9deb09ec-dirty #2
[ 179.017114] Hardware name: Horizon Robotics Journey 5 DVB (DT)
[ 179.017849] pstate: 40c00005 (nZcv daif +PAN +UAO -TCO BTYPE=--)
[ 179.018609] pc : __cpu_do_idle_irqprio+0x30/0x38
[ 179.019197] lr : __cpu_do_idle_irqprio+0x24/0x38
[ 179.019782] sp : ffff8000125c3f20
[ 179.020202] pmr_save: 00000060
[ 179.020589] x29: ffff8000125c3f20 x28: 0000000000000000
[ 179.021263] x27: 0000000000000000 x26: ffff000180210000
[ 179.021938] x25: 0000000000000000 x24: 0000000000000000
[ 179.022611] x23: ffff000180210000 x22: ffff8000113b2578
[ 179.023285] x21: ffff8000114dd710 x20: 0000000000000003
[ 179.023958] x19: ffff8000114dd678 x18: 0000000000000001
[ 179.024632] x17: 0000000000000079 x16: 0000000000000000
[ 179.025306] x15: ffff0001802753a8 x14: ffffffffffffffff
[ 179.025980] x13: ffff800012e27020 x12: ffff800012e12967
[ 179.026653] x11: 0000000000000000 x10: 0000000000000950
[ 179.027327] x9 : ffff8000125c3e90 x8 : ffff0001802109b0
[ 179.028001] x7 : 0000000000000009 x6 : 0000000000000060
[ 179.028674] x5 : ffff00026ef642c0 x4 : ffff80025dbb0000
[ 179.029348] x3 : 00000000000168de x2 : 00000000000000f0
[ 179.030022] x1 : 0000000000000000 x0 : 0000000000000060
[ 179.030696] Call trace:
[ 179.031006] __cpu_do_idle_irqprio+0x30/0x38
[ 179.031550] cpu_do_idle+0x2c/0x38
[ 179.031984] arch_cpu_idle+0xc/0x30
[ 179.032429] default_idle_call+0x24/0x84
[ 179.032929] do_idle+0x23c/0x278
[ 179.033341] cpu_startup_entry+0x20/0x68
[ 179.033840] secondary_start_kernel+0x15c/0x1d0
[ 179.034416] NMI backtrace for cpu 1
[ 179.034864] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O 5.10.59-rt52-g8bad9deb09ec-dirty #2
[ 179.036069] Hardware name: Horizon Robotics Journey 5 DVB (DT)
[ 179.036804] pstate: 40c00005 (nZcv daif +PAN +UAO -TCO BTYPE=--)
[ 179.037563] pc : __cpu_do_idle_irqprio+0x30/0x38
[ 179.038152] lr : __cpu_do_idle_irqprio+0x24/0x38
[ 179.038738] sp : ffff8000125b3f20
[ 179.039157] pmr_save: 00000060
[ 179.039543] x29: ffff8000125b3f20 x28: 0000000000000000
[ 179.040218] x27: 0000000000000000 x26: ffff0001801c5cc0
[ 179.040893] x25: 0000000000000000 x24: 0000000000000000
[ 179.041566] x23: ffff0001801c5cc0 x22: ffff8000113b2578
[ 179.042240] x21: ffff8000114dd710 x20: 0000000000000001
[ 179.042913] x19: ffff8000114dd678 x18: 0000000000000001
[ 179.043587] x17: 0000000000000000 x16: 0000000000000000
[ 179.044261] x15: ffff000180261168 x14: ffffffffffffffff
[ 179.044935] x13: ffff800012de7030 x12: ffff800012de31fb
[ 179.045609] x11: 0000000000000001 x10: 0000000000000950
[ 179.046283] x9 : ffff8000125b3e90 x8 : ffff0001801c6670
[ 179.046956] x7 : 0000000000000009 x6 : 0000000000000060
[ 179.047630] x5 : ffff00026ef402c0 x4 : ffff80025db8c000
[ 179.048303] x3 : 0000000000010e82 x2 : 00000000000000f0
[ 179.048976] x1 : 0000000000000000 x0 : 0000000000000060
[ 179.049650] Call trace:
[ 179.049960] __cpu_do_idle_irqprio+0x30/0x38
[ 179.050504] cpu_do_idle+0x2c/0x38
[ 179.050938] arch_cpu_idle+0xc/0x30
[ 179.051383] default_idle_call+0x24/0x84
[ 179.051883] do_idle+0x23c/0x278
[ 179.052295] cpu_startup_entry+0x24/0x68
[ 179.052793] secondary_start_kernel+0x15c/0x1d0
[ 179.053370] NMI backtrace for cpu 6
[ 179.053818] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G O 5.10.59-rt52-g8bad9deb09ec-dirty #2
[ 179.055024] Hardware name: Horizon Robotics Journey 5 DVB (DT)
[ 179.055759] pstate: 40c00005 (nZcv daif +PAN +UAO -TCO BTYPE=--)
[ 179.056520] pc : __cpu_do_idle_irqprio+0x30/0x38
[ 179.057109] lr : __cpu_do_idle_irqprio+0x24/0x38
[ 179.057695] sp : ffff8000125dbf20
[ 179.058115] pmr_save: 00000060
[ 179.058502] x29: ffff8000125dbf20 x28: 0000000000000000
[ 179.059176] x27: 0000000000000000 x26: ffff0001802127c0
[ 179.059851] x25: 0000000000000000 x24: 0000000000000000
[ 179.060524] x23: ffff0001802127c0 x22: ffff8000113b2578
[ 179.061198] x21: ffff8000114dd710 x20: 0000000000000006
[ 179.061871] x19: ffff8000114dd678 x18: 0000000000000001
[ 179.062544] x17: 0000000000000000 x16: 0000000000000000
[ 179.063217] x15: ffff000180320428 x14: ffffffffffffffff
[ 179.063890] x13: ffff800012e87008 x12: ffff800012e725cd
[ 179.064564] x11: 0000000000000000 x10: 0000000000000950
[ 179.065238] x9 : ffff8000125dbe90 x8 : ffff000180213170
[ 179.065912] x7 : 0000000000000009 x6 : 0000000000000060
[ 179.066585] x5 : ffff00026ef9a2c0 x4 : ffff80025dbe6000
[ 179.067258] x3 : 000000000000fa86 x2 : 00000000000000f0
[ 179.067932] x1 : 0000000000000000 x0 : 0000000000000060
[ 179.068605] Call trace:
[ 179.068916] __cpu_do_idle_irqprio+0x30/0x38
[ 179.069459] cpu_do_idle+0x2c/0x38
[ 179.069894] arch_cpu_idle+0xc/0x30
[ 179.070339] default_idle_call+0x24/0x84
[ 179.070838] do_idle+0x23c/0x278
[ 179.071250] cpu_startup_entry+0x20/0x68
[ 179.071748] secondary_start_kernel+0x15c/0x1d0
[ 179.072324] NMI backtrace for cpu 0
[ 179.072771] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 5.10.59-rt52-g8bad9deb09ec-dirty #2
[ 179.073977] Hardware name: Horizon Robotics Journey 5 DVB (DT)
[ 179.074712] pstate: 40c00005 (nZcv daif +PAN +UAO -TCO BTYPE=--)
[ 179.075472] pc : __cpu_do_idle_irqprio+0x30/0x38
[ 179.076060] lr : __cpu_do_idle_irqprio+0x24/0x38
[ 179.076647] sp : ffff8000114d3e90
[ 179.077066] pmr_save: 00000060
[ 179.077453] x29: ffff8000114d3e90 x28: ffff800010c30108
[ 179.078128] x27: 0000000000000200 x26: ffff8000114ecc00
[ 179.078801] x25: 0000000000000000 x24: 0000000000000000
[ 179.079475] x23: ffff8000114ecc00 x22: ffff8000113b2578
[ 179.080149] x21: ffff8000114dd710 x20: 0000000000000000
[ 179.080823] x19: ffff8000114dd678 x18: 0000000000000001
[ 179.081497] x17: 0000000000000000 x16: 0000000000000000
[ 179.082171] x15: ffff0001801c1168 x14: ffffffffffffffff
[ 179.082845] x13: ffff800012dc7038 x12: ffff800012daf2cd
[ 179.083518] x11: 0000000000000004 x10: 0000000000000950
[ 179.084193] x9 : ffff8000114d3e00 x8 : ffff8000114ed5b0
[ 179.084867] x7 : 0000000000000009 x6 : 0000000000000060
[ 179.085540] x5 : ffff00026ef2e2c0 x4 : ffff80025db7a000
[ 179.086213] x3 : 0000000000011c9a x2 : 00000000000000f0
[ 179.086886] x1 : 0000000000000000 x0 : 0000000000000060
[ 179.087560] Call trace:
[ 179.087871] __cpu_do_idle_irqprio+0x30/0x38
[ 179.088414] cpu_do_idle+0x2c/0x38
[ 179.088848] arch_cpu_idle+0xc/0x30
[ 179.089292] default_idle_call+0x24/0x84
[ 179.089793] do_idle+0x23c/0x278
[ 179.090205] cpu_startup_entry+0x24/0x68
[ 179.090703] rest_init+0xd0/0xdc
[ 179.091116] arch_call_rest_init+0xc/0x14
[ 179.091629] start_kernel+0x6e4/0x720
[ 179.092595] ---[ end trace 0000000000000002 ]---
[ 179.098900] YNR: soc_ynr_video_stop 1806 source node, ignored
[ 179.098908] [VPF ops]:[S0][N3][C0][V0] vpf_video_stop: done start_cnt 1
[ 179.100540] [VPF ops]:[S0][N3] vpf_wait_vnode_shot: cnt 20
[ 179.114191] Kernel panic - not syncing:
[ 179.114681] Oops: Fatal exception
[ 179.114873] SMP: stopping secondary CPUs
[ 179.115795] Kernel Offset: disabled
[ 179.116237] CPU features: 0x0000426,2a00a238
[ 179.116778] Memory Limit: none
[ 179.138001] Rebooting in 1 seconds..加载ko的符合表：
crash>
crash> mod -s soc_device ./soc_device.ko
MODULE NAME BASE SIZE OBJECT FILE
ffff800009996ac0 soc_device ffff800009974000 163840 ./soc_device.ko
crash> 反汇编出错时pc指针所在的位置soc_device_fe_work：
crash> dis -l soc_device_fe_work
0xffff80000997abd8 <soc_device_fe_work>: sub sp, sp, #0x190
0xffff80000997abdc <soc_device_fe_work+4>: stp x29, x30, [sp, #16]
0xffff80000997abe0 <soc_device_fe_work+8>: add x29, sp, #0x10
0xffff80000997abe4 <soc_device_fe_work+12>: stp x19, x20, [sp, #32]
0xffff80000997abe8 <soc_device_fe_work+16>: mov x20, x0
0xffff80000997abec <soc_device_fe_work+20>: stp x21, x22, [sp, #48]
0xffff80000997abf0 <soc_device_fe_work+24>: mov x22, x2
0xffff80000997abf4 <soc_device_fe_work+28>: mov x2, #0xf0 // #240
0xffff80000997abf8 <soc_device_fe_work+32>: stp x23, x24, [sp, #64]
0xffff80000997abfc <soc_device_fe_work+36>: mov x24, x1
0xffff80000997ac00 <soc_device_fe_work+40>: mov w1, #0x0 // #0
0xffff80000997ac04 <soc_device_fe_work+44>: stp x25, x26, [sp, #80]
0xffff80000997ac08 <soc_device_fe_work+48>: mrs x26, sp_el0
0xffff80000997ac0c <soc_device_fe_work+52>: mov w23, w4
0xffff80000997ac10 <soc_device_fe_work+56>: stp x27, x28, [sp, #96]
0xffff80000997ac14 <soc_device_fe_work+60>: add x25, x3, #0x4
0xffff80000997ac18 <soc_device_fe_work+64>: mov x21, x3
0xffff80000997ac1c <soc_device_fe_work+68>: str x24, [sp, #112]
0xffff80000997ac20 <soc_device_fe_work+72>: adrp x28, 0xffff800009991000
0xffff80000997ac24 <soc_device_fe_work+76>: add x28, x28, #0x288
0xffff80000997ac28 <soc_device_fe_work+80>: mov x19, x23
0xffff80000997ac2c <soc_device_fe_work+84>: ldr x0, [x26, #1064]
0xffff80000997ac30 <soc_device_fe_work+88>: str x0, [sp, #392]
0xffff80000997ac34 <soc_device_fe_work+92>: mov x0, #0x0 // #0
0xffff80000997ac38 <soc_device_fe_work+96>: stp xzr, xzr, [sp, #136]
0xffff80000997ac3c <soc_device_fe_work+100>: add x0, sp, #0x98
0xffff80000997ac40 <soc_device_fe_work+104>: bl 0xffff8000103d4180 <__memset>
0xffff80000997ac44 <soc_device_fe_work+108>: mov x0, x25
0xffff80000997ac48 <soc_device_fe_work+112>: bl 0xffff80001082a548 <_raw_spin_lock_irqsave>
0xffff80000997ac4c <soc_device_fe_work+116>: mov w1, #0x2 // #2
0xffff80000997ac50 <soc_device_fe_work+120>: mov x27, x0
0xffff80000997ac54 <soc_device_fe_work+124>: mov x0, x21
0xffff80000997ac58 <soc_device_fe_work+128>: bl 0xffff8000088443b8 <peek_frame_tail>
0xffff80000997ac5c <soc_device_fe_work+132>: mov x1, x27
0xffff80000997ac60 <soc_device_fe_work+136>: mov x24, x0
0xffff80000997ac64 <soc_device_fe_work+140>: mov x0, x25
0xffff80000997ac68 <soc_device_fe_work+144>: bl 0xffff80001082a438 <_raw_spin_unlock_irqrestore>
0xffff80000997ac6c <soc_device_fe_work+148>: add x8, x23, w23, uxtw #2
0xffff80000997ac70 <soc_device_fe_work+152>: mov w2, w23
0xffff80000997ac74 <soc_device_fe_work+156>: ldr w5, [x24, #1864]
0xffff80000997ac78 <soc_device_fe_work+160>: add x0, sp, #0x88
0xffff80000997ac7c <soc_device_fe_work+164>: lsl x8, x8, #4
0xffff80000997ac80 <soc_device_fe_work+168>: adrp x27, 0xffff80000997c000
0xffff80000997ac84 <soc_device_fe_work+172>: sub x8, x8, x23
0xffff80000997ac88 <soc_device_fe_work+176>: add x27, x27, #0x318
0xffff80000997ac8c <soc_device_fe_work+180>: ldr x6, [x24, #1872]
0xffff80000997ac90 <soc_device_fe_work+184>: add x8, x20, x8, lsl #4
0xffff80000997ac94 <soc_device_fe_work+188>: ldr x7, [x24, #1880]
0xffff80000997ac98 <soc_device_fe_work+192>: ldr w3, [x8, #2944]
0xffff80000997ac9c <soc_device_fe_work+196>: ldr w4, [x8, #2948]
0xffff80000997aca0 <soc_device_fe_work+200>: ldr x1, [x24, #1888]
0xffff80000997aca4 <soc_device_fe_work+204>: str x1, [sp]
0xffff80000997aca8 <soc_device_fe_work+208>: adrp x1, 0xffff800009991000
0xffff80000997acac <soc_device_fe_work+212>: add x1, x1, #0x250
0xffff80000997acb0 <soc_device_fe_work+216>: bl 0xffff8000103e97b0 <sprintf>
0xffff80000997acb4 <soc_device_fe_work+220>: add x2, sp, #0x88
0xffff80000997acb8 <soc_device_fe_work+224>: add x1, x27, #0x6c0
0xffff80000997acbc <soc_device_fe_work+228>: mov x0, x28
0xffff80000997acc0 <soc_device_fe_work+232>: bl 0xffff80001080c710 <printk>
0xffff80000997acc4 <soc_device_fe_work+236>: ldr x0, [x24, #168]
0xffff80000997acc8 <soc_device_fe_work+240>: cbz x0, 0xffff80000997ad0c <soc_device_fe_work+308>
0xffff80000997accc <soc_device_fe_work+244>: ldp x2, x3, [sp, #136]
0xffff80000997acd0 <soc_device_fe_work+248>: stp x2, x3, [x0]
0xffff80000997acd4 <soc_device_fe_work+252>: ldp x2, x3, [sp, #152]
0xffff80000997acd8 <soc_device_fe_work+256>: stp x2, x3, [x0, #16]
0xffff80000997acdc <soc_device_fe_work+260>: ldp x2, x3, [sp, #168]
0xffff80000997ace0 <soc_device_fe_work+264>: stp x2, x3, [x0, #32]
0xffff80000997ace4 <soc_device_fe_work+268>: ldp x2, x3, [sp, #184]
0xffff80000997ace8 <soc_device_fe_work+272>: stp x2, x3, [x0, #48]
0xffff80000997acec <soc_device_fe_work+276>: ldp x2, x3, [sp, #200]
0xffff80000997acf0 <soc_device_fe_work+280>: stp x2, x3, [x0, #64]
0xffff80000997acf4 <soc_device_fe_work+284>: ldp x2, x3, [sp, #216]
0xffff80000997acf8 <soc_device_fe_work+288>: stp x2, x3, [x0, #80]
0xffff80000997acfc <soc_device_fe_work+292>: ldp x2, x3, [sp, #232]
0xffff80000997ad00 <soc_device_fe_work+296>: stp x2, x3, [x0, #96]
0xffff80000997ad04 <soc_device_fe_work+300>: ldp x2, x3, [sp, #248]
0xffff80000997ad08 <soc_device_fe_work+304>: stp x2, x3, [x0, #112]
0xffff80000997ad0c <soc_device_fe_work+308>: mov w1, w19
0xffff80000997ad10 <soc_device_fe_work+312>: mov x0, x20
0xffff80000997ad14 <soc_device_fe_work+316>: bl 0xffff800009976448 <soc_device_pop_compare>
0xffff80000997ad18 <soc_device_fe_work+320>: mov x3, x0
0xffff80000997ad1c <soc_device_fe_work+324>: cbz x0, 0xffff80000997aec8 <soc_device_fe_work+752>
0xffff80000997ad20 <soc_device_fe_work+328>: mov x4, #0x4f0 // #1264
0xffff80000997ad24 <soc_device_fe_work+332>: add x0, sp, #0x108
0xffff80000997ad28 <soc_device_fe_work+336>: ldr x1, [x3, #1888]
0xffff80000997ad2c <soc_device_fe_work+340>: madd x4, x23, x4, x20
0xffff80000997ad30 <soc_device_fe_work+344>: str x1, [sp]
0xffff80000997ad34 <soc_device_fe_work+348>: mov x23, x0
0xffff80000997ad38 <soc_device_fe_work+352>: mov w2, w19
0xffff80000997ad3c <soc_device_fe_work+356>: adrp x1, 0xffff800009991000
0xffff80000997ad40 <soc_device_fe_work+360>: ldr w5, [x3, #1864]
0xffff80000997ad44 <soc_device_fe_work+364>: add x1, x1, #0x298
0xffff80000997ad48 <soc_device_fe_work+368>: ldr x6, [x3, #1872]
0xffff80000997ad4c <soc_device_fe_work+372>: ldr x7, [x3, #1880]
0xffff80000997ad50 <soc_device_fe_work+376>: ldr w3, [x4, #2944]
0xffff80000997ad54 <soc_device_fe_work+380>: ldr w4, [x4, #2948]
0xffff80000997ad58 <soc_device_fe_work+384>: bl 0xffff8000103e97b0 <sprintf>
0xffff80000997ad5c <soc_device_fe_work+388>: mov x2, x23
0xffff80000997ad60 <soc_device_fe_work+392>: mov x0, x28
0xffff80000997ad64 <soc_device_fe_work+396>: add x1, x27, #0x6c0
0xffff80000997ad68 <soc_device_fe_work+400>: bl 0xffff80001080c710 <printk>
0xffff80000997ad6c <soc_device_fe_work+404>: add x2, sp, #0x200
0xffff80000997ad70 <soc_device_fe_work+408>: add x1, sp, #0x88
0xffff80000997ad74 <soc_device_fe_work+412>: ldr x0, [x24, #168]
0xffff80000997ad78 <soc_device_fe_work+416>: ldp x4, x5, [x2, #-248]
0xffff80000997ad7c <soc_device_fe_work+420>: add x2, x0, #0x80
0xffff80000997ad80 <soc_device_fe_work+424>: stp x4, x5, [x0, #128]
0xffff80000997ad84 <soc_device_fe_work+428>: mov x0, x25
0xffff80000997ad88 <soc_device_fe_work+432>: ldp x4, x5, [x1, #144]
0xffff80000997ad8c <soc_device_fe_work+436>: stp x4, x5, [x2, #16]
0xffff80000997ad90 <soc_device_fe_work+440>: ldp x4, x5, [x1, #160]
0xffff80000997ad94 <soc_device_fe_work+444>: stp x4, x5, [x2, #32]
0xffff80000997ad98 <soc_device_fe_work+448>: ldp x4, x5, [x1, #176]
0xffff80000997ad9c <soc_device_fe_work+452>: stp x4, x5, [x2, #48]
0xffff80000997ada0 <soc_device_fe_work+456>: ldp x4, x5, [x1, #192]
0xffff80000997ada4 <soc_device_fe_work+460>: stp x4, x5, [x2, #64]
0xffff80000997ada8 <soc_device_fe_work+464>: ldp x4, x5, [x1, #208]
0xffff80000997adac <soc_device_fe_work+468>: stp x4, x5, [x2, #80]
0xffff80000997adb0 <soc_device_fe_work+472>: ldp x4, x5, [x1, #224]
0xffff80000997adb4 <soc_device_fe_work+476>: stp x4, x5, [x2, #96]
0xffff80000997adb8 <soc_device_fe_work+480>: ldp x4, x5, [x1, #240]
0xffff80000997adbc <soc_device_fe_work+484>: stp x4, x5, [x2, #112]
..............................................................................................pc出错处为soc_device_fe_work+0x1a8，0x1a8= 424
[ 178.940306] pc : sco_device_fe_work+0x1a8/0x370 [soc_device]0xffff80000997ad68 <soc_device_fe_work+400>: bl 0xffff80001080c710 <printk>
#将寄存器x2的值设置为栈指针(sp)的值加上0x200
0xffff80000997ad6c <soc_device_fe_work+404>: add x2, sp, #0x200
#将寄存器x1的值设置为栈指针(sp)的值加上0x88
0xffff80000997ad70 <soc_device_fe_work+408>: add x1, sp, #0x88
#将寄存器x0的值设置为存储在地址(x24 + 168)处的数据。x24是另一个寄存器的值
0xffff80000997ad74 <soc_device_fe_work+412>: ldr x0, [x24, #168]
#从地址(x2 - 248)处加载两个数据到寄存器x4和x5。ldp是"Load Pair"的缩写，用于同时加载两个寄存器的值
0xffff80000997ad78 <soc_device_fe_work+416>: ldp x4, x5, [x2, #-248]
#将寄存器x2的值设置为寄存器x0的值加上0x80
0xffff80000997ad7c <soc_device_fe_work+420>: add x2, x0, #0x80
#将寄存器x4和x5的值存储到地址(x0 + 128)处。stp是"Store Pair"的缩写，用于同时存储两个寄存器的值到内存
0xffff80000997ad80 <soc_device_fe_work+424>: stp x4, x5, [x0, #128]计算x4和x5寄存器的值
<soc_device_fe_work+416>: ldp x4, x5, [x2, #-248]得知从地址(x2 - 248)处加载两个数据到寄存器x4和x5。ldp是"Load Pair"的缩写，用于同时加载两个寄存器的值；x2 = 0000000000000000;
x4 = 636e657265666572;
x5 = 3a656d6172662065;
通过addr2line查看出错的位置：
zhilu.zhang@server:~/work/ram-dump$ addr2line.sh soc_device.ko soc_device_fe_work 0x1a5
soc_device_ops.c:2884查看文件，得知此次拷贝出现了空指针导致的。
2884 memcpy(compare_frame->vbuf.group_info.info[0].addr[0] + 128, &slogan[128], 128);现在知道了compare_frame->vbuf.group_info.info[0].addr[0] 是空的。
三、crash指令
https://man7.org/linux/man-pages/man8/crash.8.html
CRASH(8) System Manager's Manual CRASH(8)
NAME top
crash - Analyze Linux crash dump data or a live system
SYNOPSIS top
crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile
form)
crash [OPTION]... [NAMELIST] (live system
form)
DESCRIPTION top
Crash is a tool for interactively analyzing the state of the
Linux system while it is running, or after a kernel crash has
occurred and a core dump has been created by the netdump,
diskdump, LKCD, kdump, xendump kvmdump or VMware facilities. It
is loosely based on the SVR4 UNIX crash command, but has been
significantly enhanced by completely merging it with the gdb(1)
debugger. The marriage of the two effectively combines the
kernel-specific nature of the traditional UNIX crash utility with
the source code level debugging capabilities of gdb(1).
In the dumpfile form, both a NAMELIST and a MEMORY-IMAGE argument
must be entered. In the live system form, the NAMELIST argument
must be entered if the kernel's vmlinux file is not located in a
known location, such as the /usr/lib/debug/lib/modules/<kernel-
version> directory.
The crash utility has also been extended to support the analysis
of dumpfiles generated by a crash of the Xen hypervisor. In that
case, the NAMELIST argument must be that of the xen-syms binary.
Live system analysis is not supported for the Xen hypervisor.
The crash utility command set consists of common kernel core
analysis tools such as kernel stack back traces of all processes,
source code disassembly, formatted kernel structure and variable
displays, virtual memory data, dumps of linked-lists, etc., along
with several commands that delve deeper into specific kernel
subsystems. Appropriate gdb commands may also be entered, which
in turn are passed on to the gdb module for execution. If
desired, commands may be placed in either a $HOME/.crashrc file
and/or in a .crashrc file in the current directory. During
initialization, the commands in $HOME/.crashrc are executed
first, followed by those in the ./.crashrc file.
The crash utility is designed to be independent of Linux version
dependencies. When new kernel source code impacts the correct
functionality of crash and its command set, the utility will be
updated to recognize new kernel code changes, while maintaining
backwards compatibility with earlier releases.
OPTIONS top
NAMELIST
This is a pathname to an uncompressed kernel image (a
vmlinux file), or a Xen hypervisor image (a xen-syms file)
which has been compiled with the "-g" option. If using
the dumpfile form, a vmlinux file may be compressed in
either gzip or bzip2 formats.
MEMORY-IMAGE[@ADDRESS]
A kernel core dump file created by the netdump, diskdump,
LKCD kdump, xendump kvmdump or VMware facilities.
If a MEMORY-IMAGE argument is not entered, the session
will be invoked on the live system, which typically
requires root privileges because of the device file used
to access system RAM. By default, /dev/crash will be used
if it exists. If it does not exist, then /dev/mem will be
used; but if the kernel has been configured with
CONFIG_STRICT_DEVMEM, then /proc/kcore will be used. It
is permissible to explicitly enter /dev/crash, /dev/mem or
/proc/kcore.
An @ADDRESS value must be appended to the MEMORY-IMAGE if
the dumpfile is a raw RAM dumpfile that has no header
information describing the file contents. Multiple
MEMORY-IMAGE@ADDRESS ordered pairs may be entered, with
each dumpfile containing a contiguous block of RAM, where
the ADDRESS value is the physical start address of the
block expressed in hexadecimal. The physical address
value(s) will be used to create a temporary ELF header in
/var/tmp, which will only exist during the crash session.
If a raw RAM dumpile represents a live memory source, such
as that specified by the QEMU mem-path argument of a
memory-backend-file object, then "live:" must be prepended
to the MEMORY-IMAGE name.
As VMware facility, the crash utility is able to process
VMware VM memory dump generated by VM suspend or guest
core dump. In that case, .vmss or .guest file should be
used as a MEMORY-IMAGE and .vmem file must be located in
the same folder.
mapfile
If the NAMELIST file is not the same kernel that is
running (live system form), or the kernel that was running
when the system crashed (dumpfile form), then the
System.map file of the original kernel should be entered
on the command line.
-h [option]
--help [option]
Without an option argument, display a crash usage help
message. If the option argument is a crash command name,
the help page for that command is displayed. If it is the
string "input", a page describing the various crash
command line input options is displayed. If it is the
string "output", a page describing command line output
options is displayed. If it is the string "all", then all
of the possible help messages are displayed. After the
help message is displayed, crash exits.
-s Silently proceed directly to the "crash>" prompt without
displaying any version, GPL, or crash initialization data
during startup, and by default, runtime command output is
not passed to any scrolling command.
-i file
Execute the command(s) contained in file prior to
displaying the "crash>" prompt for interactive user input.
-d num Set the internal debug level. The higher the number, the
more debugging data will be printed when crash initializes
and runs.
-S Use /boot/System.map as the mapfile.
-e vi | emacs
Set the readline(3) command line editing mode to "vi" or
"emacs". The default editing mode is "vi".
-f Force the usage of a compressed vmlinux file if its
original name does not start with "vmlinux".
-k Indicate that the NAMELIST file is an LKCD "Kerntypes"
debuginfo file.
-g [namelist]
Determine if a vmlinux or xen-syms namelist file contains
debugging data.
-t Display the system-crash timestamp and exit.
-L Attempt to lock all of its virtual address space into
memory by calling mlockall(MCL_CURRENT|MCL_FUTURE) during
initialization. If the system call fails, an error
message will be displayed, but the session continues.
-c tty-device
Open the tty-device as the console used for debug
messages.
-p page-size
If a processor's page size cannot be determined by the
dumpfile, and the processor default cannot be used, use
page-size.
-o filename
Only used with the MEMORY-IMAGE@ADDRESS format for raw RAM
dumpfiles, specifies a filename of a new ELF vmcore that
will be created and used as the dumpfile. It will be
saved to allow future use as a standalone vmcore,
replacing the original raw RAM dumpfile.
-m option=value
--machdep option=value
Pass an option and value pair to machine-dependent code.
These architecture-specific option/pairs should only be
required in very rare circumstances:
X86_64:
phys_base=<physical-address>
irq_eframe_link=<value>
irq_stack_gap=<value>
max_physmem_bits=<value>
kernel_image_size=<value>
vm=orig (pre-2.6.11 virtual memory address ranges)
vm=2.6.11 (2.6.11 and later virtual memory address ranges)
vm=xen (Xen kernel virtual memory address ranges)
vm=xen-rhel4 (RHEL4 Xen kernel virtual address ranges)
vm=5level (5-level page tables)
page_offset=<PAGE_OFFSET-value>
PPC64:
vm=orig
vm=2.6.14 (4-level page tables)
IA64:
phys_start=<physical-address>
init_stack_size=<size>
vm=4l (4-level page tables)
ARM:
phys_base=<physical-address>
ARM64:
phys_offset=<physical-address>
kimage_voffset=<kimage_voffset-value>
max_physmem_bits=<value>
vabits_actual=<value>
X86:
page_offset=<CONFIG_PAGE_OFFSET-value>
-x Automatically load extension modules from a particular
directory. If a directory is specified in the
CRASH_EXTENSIONS shell environment variable, then that
directory will be used. Otherwise
/usr/lib64/crash/extensions (64-bit architectures) or
/usr/lib/crash/extensions (32-bit architectures) will be
used; if they do not exist, then the ./extensions
directory will be used.
--active
Track only the active task on each cpu.
--buildinfo
Display the crash binary's build date, the user ID of the
builder, the hostname of the machine where the build was
done, the target architecture, the version number, and the
compiler version.
--memory_module modname
Use the modname as an alternative kernel module to the
crash.ko module that creates the /dev/crash device.
--memory_device device
Use device as an alternative device to the /dev/crash,
/dev/mem or /proc/kcore devices.
--log dumpfile
Dump the contents of the kernel log buffer. A kernel
namelist argument is not necessary, but the dumpfile must
contain the VMCOREINFO data taken from the original
/proc/vmcore ELF header.
--no_kallsyms
Do not use kallsyms-generated symbol information contained
within kernel module object files.
--no_modules
Do not access or display any kernel module related
information.
--no_ikconf
Do not attempt to read configuration data that was built
into kernels configured with CONFIG_IKCONFIG.
--no_data_debug
Do not verify the validity of all structure member offsets
and structure sizes that it uses.
--no_kmem_cache
Do not initialize the kernel's slab cache infrastructure,
and commands that use kmem_cache-related data will not
work.
--no_elf_notes
Do not use the registers from the ELF NT_PRSTATUS notes
saved in a compressed kdump header for backtraces.
--kmem_cache_delay
Delay the initialization of the kernel's slab cache
infrastructure until it is required by a run-time command.
--readnow
Pass this flag to the embedded gdb module, which will
override its two-stage strategy that it uses for reading
symbol tables from the NAMELIST.
--smp Specify that the system being analyzed is an SMP kernel.
-v
--version
Display the version of the crash utility, the version of
the embedded gdb module, GPL information, and copyright
notices.
--cpus number
Specify the number of cpus in the SMP system being
analyzed.
--osrelease dumpfile
Display the OSRELEASE vmcoreinfo string from a kdump
dumpfile header.
--hyper
Force the session to be that of a Xen hypervisor.
--p2m_mfn pfn
When a Xen Hypervisor or its dom0 kernel crashes, the
dumpfile is typically analyzed with either the Xen
hypervisor or the dom0 kernel. It is also possible to
analyze any of the guest domU kernels if the
pfn_to_mfn_list_list pfn value of the guest kernel is
passed on the command line along with its NAMELIST and the
dumpfile.
--xen_phys_start physical-address
Supply the base physical address of the Xen hypervisor's
text and static data for older xendump dumpfiles that did
not pass that information in the dumpfile header.
--zero_excluded
If the makedumpfile(8) facility has filtered a compressed
kdump dumpfile to exclude various types of non-essential
pages, or has marked a compressed or ELF kdump dumpfile as
incomplete due to an ENOSPC or other error during its
creation, any attempt to read missing pages will fail.
With this flag, reads from any of those pages will return
zero-filled memory.
--no_panic
Do not attempt to find the task that was running when the
kernel crashed. Set the initial context to that of the
"swapper" task on cpu 0.
--more Use /bin/more as the command output scroller, overriding
the default of /usr/bin/less and any settings in either
./.crashrc or $HOME/.crashrc.
--less Use /usr/bin/less as the command output scroller,
overriding any settings in either ./.crashrc or
$HOME/.crashrc.
--hex Set the default command output radix to 16, overriding the
default radix of 10, and any radix settings in either
./.crashrc or $HOME/.crashrc.
--dec Set the default command output radix to 10, overriding any
radix settings in either ./.crashrc or $HOME/.crashrc.
This is the default radix setting.
--CRASHPAGER
Use the output paging command defined in the CRASHPAGER
shell environment variable, overriding any settings in
either ./.crashrc or $HOME/.crashrc.
--no_scroll
Do not pass run-time command output to any scrolling
command.
--no_strip
Do not strip cloned kernel text symbol names.
--no_crashrc
Do not execute the commands in either $HOME/.crashrc or
./.crashrc.
--mod directory
When loading the debuginfo data of kernel modules with the
mod -S command, search for their object files in directory
instead of in the standard location.
--src directory
Search for the kernel source code in directory instead of
in the standard location that is compiled into the
debuginfo data.
--kaslr offset|auto
If an x86_64 kernel was configured with
CONFIG_RANDOMIZE_BASE, the offset value is equal to the
difference between the symbol values compiled into the
vmlinux file and their relocated KASLR values. If set to
auto, the KASLR offset value will be automatically
calculated.
--reloc size
When analyzing live x86 kernels that were configured with
a CONFIG_PHYSICAL_START value that is larger than its
CONFIG_PHYSICAL_ALIGN value, then it will be necessary to
enter a relocation size equal to the difference between
the two values.
--hash count
Set the number of internal hash queue heads used for list
gathering and verification. The default count is 32768.
--minimal
Bring up a session that is restricted to the log, dis, rd,
sym, eval, set and exit commands. This option may provide
a way to extract some minimal/quick information from a
corrupted or truncated dumpfile, or in situations where
one of the several kernel subsystem initialization
routines would abort the crash session.
--kvmhost [32|64]
When examining an x86 KVM guest dumpfile, this option
specifies that the KVM host that created the dumpfile was
an x86 (32-bit) or an x86_64 (64-bit) machine, overriding
the automatically determined value.
--kvmio <size>
override the automatically-calculated KVM guest I/O hole
size.
--offline [show|hide]
Show or hide command output that is related to offline
cpus. The default setting is show.
COMMANDS top
Each crash command generally falls into one of the following
categories:
Symbolic display
Displays of kernel text/data, which take full advantage of
the power of gdb to format and display data structures
symbolically.
System state
The majority of crash commands consist of a set of
"kernel-aware" commands, which delve into various kernel
subsystems on a system-wide or per-task basis.
Utility functions
A set of useful helper commands serving various purposes,
some simple, others quite powerful.
Session control
Commands that control the crash session itself.
The following alphabetical list consists of a very simple
overview of each crash command. However, since individual
commands often have several options resulting in significantly
different output, it is suggested that the full description of
each command be viewed by executing crash -h <command>, or during
a crash session by simply entering help command.
* "pointer to" is shorthand for either the struct or union
commands. It displays the contents of a kernel structure
or union.
alias creates a single-word alias for a command.
ascii displays an ascii chart or translates a numeric value into
its ascii components.
bpf provides information on currently-loaded eBPF programs and
maps.
bt displays a task's kernel-stack backtrace. If it is given
the -a option, it displays the stack traces of the active
tasks on all CPUs. It is often used with the foreach
command to display the backtraces of all tasks with one
command.
btop translates a byte value (physical offset) to its page
number.
dev displays data concerning the character and block device
assignments, I/O port usage, I/O memory usage, and PCI
device data.
dis disassembles memory, either entire kernel functions, from
a location for a specified number of instructions, or from
the start of a function up to a specified memory location.
eval evaluates an expression or numeric type and displays the
result in hexadecimal, decimal, octal and binary.
exit causes crash to exit.
extend dynamically loads or unloads crash shared object extension
modules.
files displays information about open files in a context.
foreach
repeats a specified command for the specified (or all)
tasks in the system.
fuser displays the tasks using the specified file or socket.
gdb passes its argument to the embedded gdb module. It is
useful for executing gdb commands that have the same name
as crash commands.
help alone displays the command menu; if followed by a command
name, a full description of a command, its options, and
examples are displayed. Its output is far more complete
and useful than this man page.
ipcs displays data about the System V IPC facilities.
irq displays data concerning interrupt request numbers and
bottom-half interrupt handling.
kmem displays information about the use of kernel memory.
list displays the contents of a linked list.
log displays the kernel log_buf contents in chronological
order.
mach displays data specific to the machine type.
mod displays information about the currently installed kernel
modules, or adds or deletes symbolic or debugging
information about specified kernel modules.
mount displays information about the currently-mounted
filesystems.
net display various network related data.
p passes its arguments to the gdb "print" command for
evaluation and display.
ps displays process status for specified, or all, processes
in the system.
pte translates the hexadecimal contents of a PTE into its
physical page address and page bit settings.
ptob translates a page frame number to its byte value.
ptov translates a hexadecimal physical address into a kernel
virtual address.
q is an alias for the "exit" command.
rd displays the contents of memory, with the output formatted
in several different manners.
repeat repeats a command indefinitely, optionally delaying a
given number of seconds between each command execution.
runq displays the tasks on the run queue.
sbitmapq
dumps the contents of the sbitmap_queue structure and the
used bits in the bitmap. Also, it shows the dump of a
structure array associated with the sbitmap_queue.
search searches a range of user or kernel memory space for given
value.
set either sets a new context, or gets the current context for
display.
sig displays signal-handling data of one or more tasks.
struct displays either a structure definition or the contents of
a kernel structure at a specified address.
swap displays information about each configured swap device.
sym translates a symbol to its virtual address, or a static
kernel virtual address to its symbol -- or to a symbol-
plus-offset value, if appropriate.
sys displays system-specific data.
task displays the contents of a task_struct.
tree displays the contents of a red-black tree or a radix tree.
timer displays the timer queue entries, both old- and new-style,
in chronological order.
union is similar to the struct command, except that it works on
kernel unions.
vm displays basic virtual memory information of a context.
vtop translates a user or kernel virtual address to its
physical address.
waitq walks the wait queue list displaying the tasks which are
blocked on the specified wait queue.
whatis displays the definition of structures, unions, typedefs or
text/data symbols.
wr modifies the contents of memory on a live system. It can
only be used if /dev/mem is the device file being used to
access system RAM, and should obviously be used with great
care.
When crash is invoked with a Xen hypervisor binary as the
NAMELIST, the command set is slightly modified. The *, alias,
ascii, bt, dis, eval, exit, extend, gdb, help, list, log, p, pte,
rd, repeat, search, set, struct, sym, sys, union, whatis, wr and
q commands are the same as above. The following commands are
specific to the Xen hypervisor:
domain displays the contents of the domain structure for
selected, or all, domains.
doms displays domain status for selected, or all, domains.
dumpinfo
displays Xen dump information for selected, or all, cpus.
pcpus displays physical cpu information for selected, or all,
cpus.
vcpus displays vcpu status for selected, or all, vcpus.
FILES top
.crashrc
Initialization commands. The file can be located in the
user's HOME directory and/or the current directory.
Commands found in the .crashrc file in the HOME directory
are executed before those in the current directory's
.crashrc file.
ENVIRONMENT top
EDITOR Command input is read using readline(3). If EDITOR is set
to emacs or vi then suitable keybindings are used. If
EDITOR is not set, then vi is used. This can be
overridden by set vi or set emacs commands located in a
.crashrc file, or by entering -e emacs on the crash
command line.
CRASHPAGER
If CRASHPAGER is set, its value is used as the name of the
program to which command output will be sent. If not,
then command output is sent to /usr/bin/less -E -X by
default.
CRASH_MODULE_PATH
Specifies an alternative directory tree to search for
kernel module object files.
CRASH_EXTENSIONS
Specifies a directory containing extension modules that
will be loaded automatically if the -x command line option
is used.
NOTES top
If crash does not work, look for a newer version: kernel
evolution frequently makes crash updates necessary.
The command set scroll off will cause output to be sent directly
to the terminal rather than through a paging program. This is
useful, for example, if you are running crash in a window of
emacs.
AUTHOR top
Dave Anderson <anderson@redhat.com> wrote crash.
Jay Fenlason <fenlason@redhat.com> and Dave Anderson
<anderson@redhat.com> wrote this man page.
SEE ALSO top
The help command within crash provides more complete and accurate
documentation than this man page.
https://github.com/crash-utility - the home page of the crash
utility.
netdump(8), gdb(1), makedumpfile(8)
COLOPHON top
This page is part of the crash (Linux crash dump analyzer)
project. Information about the project can be found at
⟨http://people.redhat.com/anderson/⟩. If you have a bug report
for this manual page, send it to crash-utility@redhat.com. This
page was obtained from the project's upstream Git repository
⟨https://github.com/crash-utility/crash.git⟩ on 2023-06-23. (At
that time, the date of the most recent commit that was found in
the repository was 2023-06-22.) If you discover any rendering
problems in this HTML version of the page, or you believe there
is a better or more up-to-date source for the page, or you have
corrections or improvements to the information in this COLOPHON
(which is not part of the original manual page), send a mail to
man-pages@man7.org附件
addr2line.sh
#/bin/bash
file=$1
symbol=$2
offset=$3
addr=$(readelf -s $file | grep $symbol | awk -F ' ' '{print $2}')
# echo "addr $addr"
let addr=0x$addr+$offset
addr_16=$(echo "obase=16;$addr" | bc)
# echo "$addr_16"
aarch64-linux-gnu-addr2line -e $file $addr_16 -i