Pentaho kettle denormalizer or concat

I have one step that I need to concat x rows into one separated by coma or similar. I have this data on my final step:I need to have the "tarifas" row in the same column with the 5 diferent "tarifas" for the same "REFERENCIA" : example:tarifas:T00:6.9000,T01:7.9000,T02:8.9000,T03:9.9000,T04:10.9000,T05:11.9000I saw denormalizer step but can't get a good result....Read more

geokettle - Filling in running headings in Kettle

I have an XLS file of district-level data, where each district is in a region. But the region column is only filled for the first district in the region. Something like:District Region PopulationFoo Bar 1234 Baz 345 Qux 69Quux Zob 1234 Bax 42and so on. How can I process it so that the empty District fields are filled with the correct District value (for eventual output to a CSV, but I can do that bit) to get:District Region PopulationFoo Bar 1234Foo Baz 345Foo Qux ...Read more

How can I control the commit size while writing to text file using kettle?

I have a oracle table "order" which has a clob column "description" and also primary key is varchar2 "Id". I am reading that column and for every record in the table I am writing to a txt file - order_Id.txt. Table has about 400000000 records and writing to files using text file output is giving outofmemory heap error and gc overhead limit exceeded. How can I process data in chunks ?Tweaked -Xmx settings in spoon.batm/c has 16gb RAM*ADDITIONAL DETAILS *1. These are settings in spoon.bat. I am using kettle PDI 7.0set PENTAHO_DI_JAVA_OPTIONS="-Xm...Read more

kettle - Apache Commons VFS - How to open whatever dir follow by specific dir?

I think this is simple thing but some how I fail with it.I have this :ftp://user:password@host/Folder/Folder/Folder/../${Year}/${Month}/*.csvWhat I'm trying is to open whatever folder name follow by specific folder name. I'm try to access file with the path like that but fail. The fail is in this part /.././../ this part should mean whatever folder, but somehow the path is going back to previous folder. 3 folders are need to open have a fix name so I can't change it.e.g when I use this ftp://user:password@host/Folder/Folder/Folder/../${Year}/${...Read more

CDA kettle over kettleTransFromFile diffren behaviour vs Pentaho Data Integration

I create in pentaho server a form using CDE. This form is a table with some input fields. On button click is generate an array which is send as parameter value. IN db table i have 3 columns: alfa, beta, gamma.//var data = JSON.stringify(array);var data = [ {"alfa":"some txt","beta":"another text","gamma": 23}, {"alfa":"stxt","beta":"anoxt","gamma": 43}]I create a kettle transformation which is run as expected. This 2 rows of array are inserted in database, but when i run same kettle transformation using CDA kettle over kettleTransFromFile in Pe...Read more

Pentaho kettle dataflow

Soure: Machine1Destination: Machine2Pentaho kettle running on Machine3A transformation developed and executed on Machine3 that hits database on Machine1 selects data and inserts into a table into another database on Machine2. Does that data flow through Machine3 or a direct channel is established between Machine1 and Machine2?...Read more

pentaho data integration - PDI - How to implement rijndael CBC encryption in kettle?

I have some csv's containing the sale transactions. The file is come from another department in another state and it's encrypted for security reason. It's encrypted using Rijndael-128 mode CBC encryption.I need to decrypt the file before make a report about it. How do that using pentaho? I know pentaho provide GPG encrypt and decrypt steps but my requirement is use that Rijndael encryption....Read more

inheritance of pentaho log in sub transforms

I have setup to log Pentaho jobs and transformations to a databaseThis works fine provided I define every job and every transformation in its individual log settings dialogue.I see that I can configure the kettle properties file to hold these values.However I can't get this to inherit autoamtically in a transformation when it is called by a job. I assume that if defined in properties it should just inherit and work. Any ideas on what I am missing?Thanks(MS windows env with MS Sql server- we don't have Pentaho enterprise)....Read more

kettle - How to escape question mark (?) character in PDI Pentaho in a query

I'm trying to execute the following query in PDI 7.1 in an Execute SQL Statement step.update table_x set text = '? rows where update using' where 1=1The problem is that the first question mark corresponds to a parameter I get from a previous step but the second one corresponds to the URL so needs to be saved as it is. This makes the job fail with an IndexOutOfBounds exception because kettle use the url question mark as if it were a parameter.I have tried to escape as follows but no results:\???(?)But no one works.At ...Read more