Comments on: Working with CSV files in Bash http://www.thelinuxblog.com/working-with-csv-files-in-bash/ The Linux Blog, General Linux, Shell Scripts Fri, 17 May 2013 01:16:33 +0000 hourly 1 http://wordpress.org/?v=3.5.1 By: Bogwitch http://www.thelinuxblog.com/working-with-csv-files-in-bash/comment-page-1/#comment-6686 Bogwitch Thu, 01 Mar 2012 23:27:28 +0000 http://www.thelinuxblog.com/working-with-csv-files-in-bash/#comment-6686 My own solution:

I extracted each column using variations of ‘cut’ into separate files. I then used ‘paste’ to create a TAB delimited file from the single column files. It was a long way round and although I could have scripted it, I did it manually so as to check each step did not introduce any errors.
The net result is, I now have a table within my database containing slightly over 75 million records with 100% accuracy!

]]> By: Bogwitch http://www.thelinuxblog.com/working-with-csv-files-in-bash/comment-page-1/#comment-6643 Bogwitch Thu, 23 Feb 2012 18:23:25 +0000 http://www.thelinuxblog.com/working-with-csv-files-in-bash/#comment-6643 Hi,

I have a peculiar requirement which I hope you can help with. I have a few CSV files that I wish to convert to TSV for importing into MySQL.
Each entry consists of a few text entries that are enclosed within quotation marks, some numeric entries are not. Within the text areas there are occasionally commas!

e.g.

“abcde”,”fghij”,”klm,no”,1234,5678,”",”pqrst”

There are an equal number of entries on each line.

I have tried a ‘sed ‘s/”,”/[tab]/g but the commas are left around the numbers, if I tr ‘,’ ‘\t’ the commas in the enclosed text is converted!

Any help, suggestions, etc gratefully received.

Bog

]]> By: TheLinuxBlog.com http://www.thelinuxblog.com/working-with-csv-files-in-bash/comment-page-1/#comment-6421 TheLinuxBlog.com Tue, 20 Dec 2011 19:22:39 +0000 http://www.thelinuxblog.com/working-with-csv-files-in-bash/#comment-6421 Hey @Patrick, make sure there is only one character for the cut -f field. The space is done by using:
-f \
Also make sure -f is only defined once.
Another issue is that sometimes quotes and apostrophes get translated incorrectly, try replacing any of those for the standard issue ones.

Hope this helps!

]]> By: Patrick http://www.thelinuxblog.com/working-with-csv-files-in-bash/comment-page-1/#comment-6420 Patrick Tue, 20 Dec 2011 18:12:55 +0000 http://www.thelinuxblog.com/working-with-csv-files-in-bash/#comment-6420 Hi, i’m afraid your scripts are not working for me. Especially the one fo count the number of elements in the csv file. The error it comes up with is:

“cut: the delimiter must be a single character
Try ‘cut –help’ for more information.”

]]> By: Nikolay http://www.thelinuxblog.com/working-with-csv-files-in-bash/comment-page-1/#comment-3182 Nikolay Tue, 08 Jun 2010 07:05:54 +0000 http://www.thelinuxblog.com/working-with-csv-files-in-bash/#comment-3182 Hallo,
Thanks for the post – it is vetry useful indeed.
However i am strugling with the following – I have
to convert a csv file into ANSII file..
Could you please help me with that – I’d like to point out that I am really inexperienced bash/awk script user, but eager to learn :)

]]> By: TheLinuxBlog.com http://www.thelinuxblog.com/working-with-csv-files-in-bash/comment-page-1/#comment-1878 TheLinuxBlog.com Thu, 12 Feb 2009 14:35:46 +0000 http://www.thelinuxblog.com/working-with-csv-files-in-bash/#comment-1878 @SWATKAT I guess I don’t understand what it is you are exactly trying to do. Split will split the file into the 65,000 lines. Are you trying to add the 65,000 line to the 2nd file if the second field on the 65,000 line is the same as the 2nd field on the 2nd file? I am assuming you would want this at the beginning of the file correct?

Let me know and I’ll try and help.

]]> By: swatkat http://www.thelinuxblog.com/working-with-csv-files-in-bash/comment-page-1/#comment-1877 swatkat Thu, 12 Feb 2009 13:42:50 +0000 http://www.thelinuxblog.com/working-with-csv-files-in-bash/#comment-1877 Hi, i would like to know how to split a large CSV file in chunks of 65000 lines.

split works fine here, but the condition is it should check for the 2nd column.. if the 2nd column has the same value as that of the 65000 – 1th line (2nd column), then this line too should be appended to the 2nd file created.

thx.

]]> By: TheLinuxBlog.com http://www.thelinuxblog.com/working-with-csv-files-in-bash/comment-page-1/#comment-1512 TheLinuxBlog.com Mon, 24 Nov 2008 18:08:50 +0000 http://www.thelinuxblog.com/working-with-csv-files-in-bash/#comment-1512 Hey Jack,

If those records are automatically put out into a file and follow the same format regardless of if the data exists it shouldn’t be too hard.

If there are two line breaks in between each record then you could process the file that way. As there are semicolons and quotes in the file you will have to escape the query.

If you are trying to import 5000 records into mysql you may be better off using a php or perl script to read the file, but it can be done with bash if you HAVE to use it.

]]> By: jack http://www.thelinuxblog.com/working-with-csv-files-in-bash/comment-page-1/#comment-1498 jack Sat, 22 Nov 2008 04:44:58 +0000 http://www.thelinuxblog.com/working-with-csv-files-in-bash/#comment-1498 hi linuxblog, really good intro tutorial for sed on csv, any I’m stuck with some problem on converting text file to csv file, apprieacite if you could help me, i have a file like below (3 records) but with more than 5k records:

Fri Aug 01 00:01:19 2008
NAS-IP-Address = 172.16.160.1
Class = “P1-SHT-AAA01;1217116976;70999″
Calling-Station-Id = “00-1F-FB-00-02-26″
Acct-Status-Type = Stop
Acct-Session-Id = “31908b485a370000″
Framed-IP-Address = 172.16.160.2
WiMAX-BS-ID = 000012A00433
User-Name = “joe”

Fri Aug 01 00:02:02 2008
NAS-IP-Address = 172.16.160.1
Class = “P1-SHT-AAA01;1217116976;71005″
Calling-Station-Id = “00-1F-FB-00-06-61″
Acct-Status-Type = Start
Acct-Session-Id = “31908b485d370000″
Framed-IP-Address = 172.16.160.2
WiMAX-BS-ID = 000012A00413
User-Name = “john”

Fri Aug 01 00:02:02 2008
NAS-IP-Address = 172.16.160.1
Class = “P1-SHT-AAA01;1217116976;71005″
Calling-Station-Id = “00-1F-FB-00-06-61″
Acct-Status-Type = Update
Acct-Session-Id = “31908b485d370000″
Framed-IP-Address = 172.16.160.2
WiMAX-BS-ID = 000012A00413
User-Name = “don”

and I need to make csv file to import to sql db like below:

2008-08-1;172.16.160.1;P1-SHT-AAA01;1217116976;71005;00-1F-FB-00-06-61;Start;31908b485d370000;172.16.160.2;000012A00413;joe

possible to do it in sed ? iam really appreaciete your help, thanks.

]]> By: TheLinuxBlog.com http://www.thelinuxblog.com/working-with-csv-files-in-bash/comment-page-1/#comment-1392 TheLinuxBlog.com Mon, 10 Nov 2008 15:12:20 +0000 http://www.thelinuxblog.com/working-with-csv-files-in-bash/#comment-1392 I appreciate your input @TOM. If there are any topics you or any one else would like me to write on, just let me know!

]]>