Read in all csv files from a directory using Python

Read in all csv files from a directory using Python

Thats how Id do it:

import os

directory = os.path.join(c:\,path)
for root,dirs,files in os.walk(directory):
    for file in files:
       if file.endswith(.csv):
           f=open(file, r)
           #  perform calculation
           f.close()

Using pandas and glob as the base packages

import glob
import pandas as pd

glued_data = pd.DataFrame()
for file_name in glob.glob(directoryPath+*.csv):
    x = pd.read_csv(file_name, low_memory=False)
    glued_data = pd.concat([glued_data,x],axis=0)

Read in all csv files from a directory using Python

I think you look for something like this

import glob

for file_name in glob.glob(directoryPath+*.csv):
    x = np.genfromtxt(file_name,delimiter=,)[:,2]
    # do your calculations

Edit

If you want to get all csv files from a folder (including subfolder) you could use subprocess instead of glob (note that this code only works on linux systems)

import subprocess
file_list = subprocess.check_output([find,directoryPath,-name,*.csv]).split(n)[:-1]

for i,file_name in enumerate(file_list):
    x = np.genfromtxt(file_name,delimiter=,)[:,2]
    # do your calculations
    # now you can use i as an index

It first searches the folder and sub-folders for all file_names using the find command from the shell and applies your calculations afterwards.

Leave a Reply

Your email address will not be published.