For most CRUD operations that you perform, the JE cache size is important since the internal Btree is cached. See the DbCacheSize utility for how to estimate the optimal size of this cache. For such a large data set, you will need a large JE cache to obtain decent performance.
However, for the count() operation, the JE cache is not used. In this case, the file system cache is more relevant to performance. In your example, it is likely that the file system cache is being filled when you first call count(), and then it is faster the second time you call it. The count() operation is very expensive, since the internal Btree does not maintain a record count. For such a large data set, I don't recommend using it.
Please be sure to post the version of BDB JE you're using when you ask a question.