RAC nodes are up and running, but crsd is down on one of the node
I've just joined the company with 2 x RAC nodes running 11.2.0.3 on Linux.
Everything seems fine, at least there is no complain from the users.
When building auto checking scripts, I notice all srvctl command fails with error "Cannot communicate with crsd" on node 1.
I issue "ps -ef|grep crsd", to my surprise, crsd is not running on node 1!
I review crsd.log & crsdOUT.log, there are lots of error like:
CRSD REBOOT
CRSD exiting: Could not init OCR, code: 26
and crsd had already stopped for 2 weeks on node 1. (Everything is fine on node 2)
I query v$asm_disk from node 1, finding only data disk information, mount_status is "CLOSED" while name and failgroup are "(null)" for all the 3 OCR disks.