OL 7.5 kernel panic

MatthewLinux

After we updated to Oracle Linux 7.5, the latest kernel would not boot. It panics with this:

kernel panic-not syncing: VFS: unable to mount root fs on unknown block(0,0)

I had to rebuild the initramfs and grub2 config in order to get it to boot properly. Anyone else notice this today?

Tommy.Reynolds-Oracle

It turns out to be a known issue with the upgrade to 7.5, so you have done nothing wrong as far as I can tell.

There is a strong separation between userspace (the usual suspects, libraries, applications, shells, X window, on and on) and the privileged kernel space.  In practical terms this means that most times it does not matter which kernel version you use with which version of userspace, mostly the C standard library.  You can update either independently.

Not this time.

The interface between userspace and the kernel has an incompatible change.  The userspace must be updated first so that when the kernel is updated all the right subroutines and all are ready and waiting in userspace.  There are firmware and microcode which must be updated so the devices operate the way the newly booted kernel expects, else the boot fails.

Since there has not been an RPM version requirement on the kernel and userspace, the updater updates the packages in the wrong order.  The right order is to update userspace first and then update the kernel.  This is why the initrd was not rebuilt.

If you have Oracle Linux support, file a service request (SR) ticket to be notified when a remedy is announced.

Cheers,

Tommy

Tommy.Reynolds-Oracle

Good catch for rebuilding the initrd so the kernel could find its root filesystem. Does your setup differ from that of, say, a desktop computer?

Perchance do have a work/no-work pair of initrd files?  You can take them apart, they are just compressed cpio(1) files last time I looked.

Dude!

I think it would have been good to know what actually caused the issue. Perhaps the boot partition shifted to a new device, or the upgrade ran out of disk space during the rebuild of the boot image. I think the later is somewhat common. How large is your /boot partition?

MatthewLinux

So I just deployed an OL 7 GA machine and upgraded it to OL 7.5. I noticed that after I patched, and BEFORE I rebooted, I checked the grub config:

grep initrd /etc/grub2.cfg

And the latest Kernel was NOT in the list. So, it seems there is something wrong with grub2 after the new kernel is installed. I ran the following and then rebooted and it came up fine.

grub2-mkconfig -o /boot/grub2/grub.cfg

Thanks,

MatthewLinux

/boot is 512MB and is only 28% used

MatthewLinux

Running Oracle Linux as servers on VMware vSphere.

Sergio-Oracle

Do you have the UEK5 preview channel enabled? Can you show output of yum repolist on the machine that failed?

MatthewLinux

No, we're not using the UEK. It's just the ol7 latest channel.

MatthewLinux

I just patched a RHEL server to 7.5 and it did not have the grub2 issue that I'm experiencing on OL 7.5.

Sergio-Oracle

Any chance you can show the before and after grub.cfg and the exact commands you used? Also, is this against ULN or yum.oracle.com?

Tommy.Reynolds-Oracle

It turns out to be a known issue with the upgrade to 7.5, so you have done nothing wrong as far as I can tell.

There is a strong separation between userspace (the usual suspects, libraries, applications, shells, X window, on and on) and the privileged kernel space.  In practical terms this means that most times it does not matter which kernel version you use with which version of userspace, mostly the C standard library.  You can update either independently.

Not this time.

The interface between userspace and the kernel has an incompatible change.  The userspace must be updated first so that when the kernel is updated all the right subroutines and all are ready and waiting in userspace.  There are firmware and microcode which must be updated so the devices operate the way the newly booted kernel expects, else the boot fails.

Since there has not been an RPM version requirement on the kernel and userspace, the updater updates the packages in the wrong order.  The right order is to update userspace first and then update the kernel.  This is why the initrd was not rebuilt.

If you have Oracle Linux support, file a service request (SR) ticket to be notified when a remedy is announced.

Cheers,

Tommy