Please be more specific. No one can guess your Linux distribution and version. What are you using? Do you need information about setting up a mail system or how to track failed login attempts? Btw, https://forums.oracle.com/forums/ann.jspa?annID=911
LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 5.3 Beta (Tikanga)
Running an old beta version of RHEL is a bad idea. If you are concerned about security and operation of your OS I suggest to use a more recent release version. You can download, install and use Oracle Linux for free.
Mail processing of failed login attempts is not a good idea and to my knowledge there is no such built-in system setting. I suggest you read the standard documentation or search the Web for information on how to set up a mail system. You will probably need to create a custom script to process failed login attempts.