Does anybody know how to share some DB nodes space between all DB nodes ?
The goal is to allocate unused DB nodes space to a "SAN area" in order to share applications (customer application not Oracle) resources (executables, log, ...) between all nodes. I've read some stuff about iscsi and OpenFiler or zFS but don't know if it fits to our needs ???
The local filesystems on each DB node come from local storage that is only connected to that DB node (it is literally internal to the machine). The only way to "share" that space is by some cross-mounted NFS scheme. I strongly recommend against that approach since it is riddled with HA issues and is likely to cause system hangs if implemented incorrectly. Also, whenever systems are rebooted, the mounts will be stale (at least for a while) and if you're depending on those files, you'll certainly see issues.
If you need a cluster filesystem, I'd suggest evaluating DBFS on Exadata (MOS 1054431.1) or some external NFS server/appliance like the ZFSSA.
DBFS doesn't have those restrictions inherently - those are based on the mount options you use when mounting it on Linux using fuse. If you mount it without the direct_io option, you can execute files from DBFS just fine. See the notes in 1054431.1 related to multiple filesystems as I think I've included some discussion about that in there.
I thought you were looking for a cluster filesystem, so I'm providing you with an option for that.
The only way to (safely, in my opinion) use the local space on DB nodes is for local storage on that DB node. Are you really worried about 300GB? If space is that tight, there are likely other things to consider :)