Immediate solution was 'crsctl start resource'.
I knew that with oracle restart, if I manually stopped a particular service (such as a db or asm instance), that would be reflected in restart, and it would not attempt to auto restart that particular service until such time as I manually restarted it. But I didn't think stopping the asm instance would cause the cssd service to also be affected.
With Oracle 11.2 some resources have auto start policy set to restore, which means that GI will remember the last state of the resource. If the resource was stopped normally then on the next restart of clusterware it won’t be started. Otherwise if the server crashes or by some reason the OS is rebooted then clusterware will start the database because last state was ONLINE (running).
By default with Oracle 11.2 several important resources come in the profile with attribute AUTO_START=restore, so if Oracle database server is restarted for some reason , it will keep and restore the last state.
The resource ora.cssd has the attribute AUTO_START=never by default and the resouce ora.asm has the attribute AUTO_START=restore by default.
That means which ora.cssd will never start does not matter what was last state of this resource.
So, How and when ora.cssd start?
It's ora.asm which have attribute START_DEPENDENCIES=hard(ora.cssd).
So, if the ora.asm resource don't start at startup (for some reason) the resource ora.cssd will stay down until manually start it or start ora.asm resource.
$ crsctl stat res ora.cssd -p |grep AUTO_START
$ crsctl stat res ora.asm -p |grep AUTO_START
$ crsctl stat res ora.asm -p |grep START_DEPENDENCIES
If you wish you can change it. (Is not supported by Oracle, unless if you have a SR)
$ crsctl modify resource ora.cssd -attr AUTO_START=always
But I don't see any problem doing this.
Why would I get the message:
ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process 6898
in the alert.log and still have all services (css, evm, has) running and available?
We are running Oracle 188.8.131.52.0 on Red Hat Enterprise Linux Server release 5.5 (Tikanga) x86_64