Agent threads consume more than half of nproc; Resource temporarily unavailable
Hi, DBAs
This is a four node cluster on Red Hat; redhat-release-5Server-5.6.0.3.x86_64. It is monitored by 12.1.0.3 agents. Clusterware is 11.2.0.4. User oracle owns all the software: GI, databases and agent. Yesterday, another DBA reported she could not connect to one of the nodes as oracle. The error was "su: cannot set user id: Resource temporarily unavailable". The other three nodes were accessible, but we could not ssh as oracle to the fourth node, either. However, I was able to connect to the node with my personal credentials.
A Community post suggested that a too-low soft value for nproc was responsible. Sure enough, a count of all threads owned by oracle on the node was 2189. This is greater than 2047, the soft value of nproc in /etc/system/limits.conf. This value for nproc is from the installation guide for Grid Infrastructure, the same value is in the Enterprise Manager 12.1.0.3 installation guide.