Tuesday, January 29, 2008

Seperate certain data in file to another file.



Let say, i have many raw files, example data files generated from system. Every day the system generates the raw files, with pipe | seperated. At the end of the month i want to extract certain data from each of the files. Another problem arise is, these files contains more 100K of lines.

List of files in my unix server
mixed_raw_data_20080101.csv
mixed_raw_data_20080102.csv
mixed_raw_data_20080103.csv
mixed_raw_data_20080104.csv
mixed_raw_data_20080105.csv
mixed_raw_data_20080106.csv
.....

Content of the file
Hello|amy
Hello|amy
Hello|amy
Hello|amy
Hello|aby
Hello|aby

e.g:
Extract aby from all the listed files. Manually i can do each files like this
more mixed_raw_data_20080101.csv | grep '|aby' > mixed_raw_data_20080101.csv.extract

But what if i have to extract for the whole month which has 30 files....

for f in mixed_raw_data_*.csv; do more $f | grep '|aby' > $f.extract ; done;

No comments: