Problem in handling complex CSV file

Anindya Gayen

Hi,

I am facing a strange problem in handling a complex CSV file.

The content of the file is as follows. It is getting executed through an XSD and the target is a table.

First time when the interface is executed ( for e.g session id - 19001) it is getting completed successfully.

Now when making changes in the CSV file suppose adding ( for e.g.-

DEPT,D05,Books,ABC Store

CUST,C01,Sayantan,[email protected]

CUST,C02,Shubham,[email protected])

or deleting ( for e.g.-

CUST,C02,Sarbajit,[email protected])

and running the interface I am not getting the added D05 in the target table nor my C02 data is getting removed i.e.- the updated data in CSV file is not getting fetched and I am getting the same records as i got when i ran interface of session id 19001.

I am not getting why it is happening??

The CSV file used in session id -19001 is:

DEPT,D01,Retail,World Mart

CUST,C01,Anindya,[email protected]

CUST,C02,Rashni,[email protected]

DEPT,D02,Food,Food Bazar

CUST,C01,Abhijit,[email protected]

CUST,C02,Anirban,[email protected]

CUST,C03,Sharmistha,[email protected]

DEPT,D03,Water,SW

CUST,C01,Nirmalya,[email protected]

DEPT,D04,Clothes,City style

CUST,C02,Sarbajit,[email protected]

CUST,C03,Abhishek,[email protected]

Anindya Gayen

Can pls anyone help me with the above problem.

Thanks.

_Phil

I think you need to synchronize your file.

The first time you run the interface, ODI creates the tables based on your XSD and populates them with the data.  If you change the data in the XML file, then you need to synchronize the file for the changes to be picked up.  See B.4.9 SYNCHRONIZE in the XML driver documentation:

Oracle Data Integrator Driver for XML Reference

Anindya Gayen

Hi,

Thanks for the reply.

I have got the solution.

Synchronization is not needed over here.

If we configure the Data Server in the following manner, any update or delete in CSV file will reflect in the target.

jdbc:snps:complexfile?f=C:\cstdb\cstdb.csv&d=C:\cstdb\cstdb.xsd&re=Root&ro=true&dod=yes

Here dod means drop on disc. It automatically drops the schema when we close the JDBC connection.

_Phil

Glad you got sorted.  Yep, DOD will do the trick too.  I personally prefer to maintain the tables post-load and use synchronize, especially if it's a reusable process and I expect the files I'm loading to be in a consistent format.

But as always, whatever works best for your process.

TriTechWorks

Here's what you can do to handle CSV files using HSQL.

Say the CSV file contains order data. Each order record contains data about a particular order like order number, product code, quantity, ordered by, date etc. Now lets take a hypothetical requirement which says, the user needs to know how many orders were ordered per product.

Option 1]  At the basic level, read the order file, parse each line and write the logic to get count of orders per product.

Option 2] Load this CSV file into a database like MySQL and write database queries to get the orders per product.

So where does this leave us? Have we run out of options? I am sure we would have tried & used the above two options . But I wanted a different approach. Following questions were lingering in my mind:

1] Why can’t I write SQL queries against the CSV file itself. After all it’s like any RDBMS table.

2] Why should I load the CSV file into some database before I query it?

3] Why can’t I create a database table & attach the CSV file to it?

Continue Reading Here: Handling CSV Files