Convert A Massive CSV To Many CSVs Using PHP

One CSV Many CSVs
I’m in the process of working in a CSV file with roughly 26000+ rows and 20 some odd columns. Each row represents an historical purchase order, including information such as date, time, name, address, product name, product sku, etc. My end goal is to import all these transactions into an existing e-commerce system. The trouble is, is that my import tool can only process 1500 rows in one go, or I get connection reset errors from the server.

The solution here is to break up the master CSV file into discreet 1500 row chucks and save in separate files. So rather than hack my way through Excel, copying and pasting 1500 row chunks into new workbooks (ug…) I decided to write me a little PHP script to do the job. It took about 30 minutes. Not only is this a faster way to breakup these transactions, it is completely error free. There’s a high likelihood I would have missed or duplicated some rows having had done this manually.

Current code assumes your master file is in the same directory as the script. Adjust as necessary.


<?php

@ini_set( 'display_errors', 1 );

@ini_set('memory_limit','512M');

echo 'start <br/>';

$file_name_base = 'masterfile';
$master_file = "{$file_name_base}.csv";

$fh = fopen( $master_file, "r" );
	
if( $fh ) { // valid file?

	$i = 0;
	$records_per_file = 1500;
	$header_row = null;

	// loop through csv rows
	while ( ( $row = fgetcsv( $fh, 0, ',' ) ) !== false ) {
	
		if( $i == 0 ) { // first row?
		
			// save column names row
			$header_row = $row;
			
		}

		if( $i % $records_per_file == 0 ) { // time to create a new file?

			// some vars, duh
			$curr_file_name = "{$file_name_base}_{$i}.csv";
			$curr_fh = fopen( $curr_file_name ,"w" );
	
			if( isset( $header_row ) && !empty( $header_row ) && $i > 0 ) { // we cool?
			
				// yes, add header row
				fputcsv( $curr_fh, $header_row );
				
			}
			
			echo "{$i} - create new file: {$curr_file_name}. <br />";
		}
		
		fputcsv( $curr_fh, $row );
	
		$i++;
		
	}		
	
	echo $i . '<br/>';
				        
}

?>

Posted in: Code Samples, Development  |  Tagged with: , ,  |  Leave a comment

Leave a Reply

Most recent work

  • Our Restroom Our Restroom is an (inter)national crowdsourced campaign who’s goal is simple: compel businesses to make their single occupancy restrooms gender neutral. Championed by Kristin Russo, designed by Allison Weiss, built by me on WordPress.
  • How To Win At Feminism The badasses over at Reductress are at it again with their new book How To Win At Feminism.
  • Reductress I crank out on-going web projects for Reductress. Super fun as you can imagine.
  • Everyone Is Gay Everyone Is Gay is an indispensable question and answer resource for the LGBTQ community. Hell, it’s great advice for anyone with a body with an emotion or two. I made the site responsive among other things. Give it a look see.

Need hosting?

  • Media Temple My go to hosting provider. You get rockstar hosting and I make a little cash. No pressure, just clean honest fun.

Past work

  • Christine Chaney Creative The online presence of Seattle artist Christine Chaney. Specializing in architecture, art and apparel.
  • Get Your Shit Together This is a site dedicated to helping you get your shit together before an unexpected tragedy, like the loss of a loved one.
  • IHG Instagram Sweepstakes An awesome Instagram driven sweepstakes site for IHG.
  • Trivial Beersuit A site that provides professional pub trivia to bars and restaurants.
  • Audi YouTube Channel This site was a quick turn-around hair pulling adventure for the Super Bowl: revamping Audi’s YouTube Channel.
  • Teach.org The new Teach.org. Providing tools to help students become teachers.
  • Valhalla DSP A one man Seattle based company, writing pretty sick professional audio plugins. Check ’em out!
  • Microsoft Visual Studio 2012 Launch This sweet new Visual Studio 2012 Launch website sports a mobile first, responsive web design architecture and comes in 14 different languages.
  • Nice Manners Music Nice Manners is an LA based recording studio offering their clients all sorts of digital audio services. This is a demo of an internal genre based music search & player I built for them. I am ecstatic about launching this project—check it out.
  • Holiday Inn Tumblr A quick, fully responsive update to the Holiday Inn Tumblr page.
  • Microsoft's Art of Touch An artful HTML5 Microsoft website. Sadly, this site is no longer live.
  • Microsoft Hardware Healthy Computing Tool A useful little tool to help you find the perfect Microsoft mouse or keyboard.
  • Pella Professional Pella manufactures high quality doors and windows. This is their professional site.
  • Imaging the World A non-profit dedicated to providing ultrasound services in remote areas of the world.
  • Ginny Ruffner: A not so still life A site for a documentary film about Seattle artist Ginny Ruffner.

Blogroll