Any downside to database buffer cache being too large?

Hello,

Have a database server running AIX 6 with 64GB memory - one 11.2.0.2 database.  There has been a lot of discussion in my office about increasing the buffer cache by a couple more GB - just to do it because it is available. The cache analysis graph from EM shows really not too much performance benefit for increasing - line really flattens out at where we are currently at. 

My question is  - is there any downside to having a extra large buffer cache as long as server / other processes don't need it?

Thanks,

Jim