Convert A Massive CSV To Many CSVs Using PHP

I’m in the process of working in a CSV file with roughly 26000+ rows and 20 some odd columns. Each row represents an historical purchase order, including information such as date, time, name, address, product name, product sku, etc. My end goal is to import all these transactions into an existing e-commerce system. The trouble is, is that my import tool can only process 1500 rows in one go, or I get connection reset errors from the server.

The solution here is to break up the master CSV file into discreet 1500 row chucks and save in separate files. So rather than hack my way through Excel, copying and pasting 1500 row chunks into new workbooks (ug…) I decided to write me a little PHP script to do the job. It took about 30 minutes. Not only is this a faster way to breakup these transactions, it is completely error free. There’s a high likelihood I would have missed or duplicated some rows having had done this manually.

Current code assumes your master file is in the same directory as the script. Adjust as necessary.

<?php

@ini_set( 'display_errors', 1 );

@ini_set('memory_limit','512M');

echo 'start <br/>';

$file_name_base = 'masterfile';
$master_file = "{$file_name_base}.csv";

$fh = fopen( $master_file, "r" );
	
if( $fh ) { // valid file?

	$i = 0;
	$records_per_file = 1500;
	$header_row = null;

	// loop through csv rows
	while ( ( $row = fgetcsv( $fh, 0, ',' ) ) !== false ) {
	
		if( $i == 0 ) { // first row?
		
			// save column names row
			$header_row = $row;
			
		}

		if( $i % $records_per_file == 0 ) { // time to create a new file?

			// some vars, duh
			$curr_file_name = "{$file_name_base}_{$i}.csv";
			$curr_fh = fopen( $curr_file_name ,"w" );
	
			if( isset( $header_row ) && !empty( $header_row ) && $i > 0 ) { // we cool?
			
				// yes, add header row
				fputcsv( $curr_fh, $header_row );
				
			}
			
			echo "{$i} - create new file: {$curr_file_name}. <br />";
		}
		
		fputcsv( $curr_fh, $row );
	
		$i++;
		
	}		
	
	echo $i . '<br/>';
				        
}

?>