Oracle SQL Connector for HDFS - Where is the filter applied?

mweedman

I have just managed to get the Oracle SQL Connector for HDFS (YEAH!)
working on a Linux 2VM cluster and was curious where the filter in the where clause is being applied to limit the data.
i.e. is it being pushed clear down into hadoop or is hadoop pulling all the data and then then it's being filtered or does all the data get buffered into the database or what?

Thanks

Mark

User_S7LLG

The filter will be applied in Oracle. OSCH does not push down the filter to Hadoop to execute and MapReduce code. So the predicate would be handled the same way as any other external table or I presume any (non-exadata) table would be.

mweedman

After thinking a little more on this, the more appropriate question might be , "does all the data get loaded into the buffer cache before the filter is applied?"

User_S7LLG

The filter will be applied in Oracle. OSCH does not push down the filter to Hadoop to execute and MapReduce code. So the predicate would be handled the same way as any other external table or I presume any (non-exadata) table would be.

mweedman

Along the same lines of this question, does anybody know a way to connect from Oracle to Hadoop/HDFS in such a way as to push the query filter down into the Hadoop query?

I really don't want to pull ALL my hdfs data into the Oracle buffer cache for processing!

Thanks

Mark