BCP process and system logs level severity (PROD)

Hi,

We are currently in the process to deploy a new Oracle SBC setup in production with a Splunk server that will help us collect and parse SBC syslog for troubleshooting and monitoring purposes.

We already have the SNMP trap alarming mechanism enable, so we are already getting the essential alarms from the SBC. What we want is to go further and also be able to proactively detect special combinations of system events with Splunk.

We need some advises wrt best practices linked to log level severity, i.e. we are trying to figure out the best log level severity that will not put too much strain on our SBC, but in the same time give us the maximum elementary and pertinent bits of information required to perform a good troubleshooting analysis in real-time (Splunk alerting) and post incident (Splunk backlogs, 5-10 days).