The Java software generally is free but a hardware server will cost you.
If you have settled for IBM AIX I suggest you contact a sales representative at IBM. They can help you identify your needs. IBM has their own version of Java.
> We want to develop some applications in an IBM AIX environement. Is it sufficient to download a free version of Java EE, for example ? Shall we buy an WebSphere Application Server
That question is WAY too general. You use WebSphere for specific types of applications. And WebSphere is only one type of a specific category of JEE servers.
Based on that you need to do a lot more research before you make any buying decisions.