Split Multiple Files Using Python

Hello Guys,

 

I found this code really useful and wanted to post it for future reference. At work , at times there is a need to provide data in excel. Unfortunately Microsoft Excel cannot load data which is more than 80K.  In this case scripts like splitting data comes in handy.

The code first creates a function and then its called by passing the file that needs to be split. The code hightlighted in red is all that needs to be edited for your needs .

 

Till next time

Cheers!!

Reference : https://stackoverflow.com/questions/36445193/splitting-one-csv-into-multiple-files-in-python

 

 

Code

import os

def split(filehandler, delimiter=',', row_limit=500000,
 output_name_template='Out_Put_Format_%s.csv', output_path='.', keep_headers=True):
 import csv
 reader = csv.reader(filehandler, delimiter=delimiter)
 current_piece = 1
 current_out_path = os.path.join(
 output_path,
 output_name_template % current_piece
 )
 current_out_writer = csv.writer(open(current_out_path, 'w'), delimiter=delimiter)
 current_limit = row_limit
 if keep_headers:
 headers = next(reader)
 current_out_writer.writerow(headers)
 for i, row in enumerate(reader):
 if i + 1 > current_limit:
 current_piece += 1
 current_limit = row_limit * current_piece
 current_out_path = os.path.join(
 output_path,
 output_name_template % current_piece
 )
 current_out_writer = csv.writer(open(current_out_path, 'w'), delimiter=delimiter)
 if keep_headers:
 current_out_writer.writerow(headers)
 current_out_writer.writerow(row)



split(open('C:\\Users\\Jethin.Abraham\\Documents\\File_to_be_split', 'r'));
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s