Ndtyjky

I am using the acs package to download unemployment data for all the metro areas in the US.

From the census website first we download the msa codes, load the .csv, subset all metro areas and then use geo.make to make a new geo set object using acs package:

library(acs)



# read msa codes 

data = read.csv("C:/dir_here/msa_codes.csv", header = TRUE, sep = ",",stringsAsFactors = FALSE)



# subset data for metro only 

data_metro = subset(data, Metropolitan.Micropolitan.Statistical.Area == "Metropolitan Statistical Area")

# Obtain Tracs for all US states (acs package)

all_geo_trac = list()

for (i in 1:nrow(data_metro)) {

  all_geo_trac[[i]] = geo.make(msa=data_metro$CBSA.Code[i])

}

Now we have a list of metro areas we want - next is to grab the census data for each metro area. This is the portion where we iterate through each geo.set and download the corresponding unemployment data from the census website. I wrote a for loop for this however I run out of memory around about 100 iterations. When this portion runs, I can watch the memory usage on task manager slowly increase until maximum. The for loop might be copying on each iteration? Heres the code:

# Obtain unemployment data 

options( warn = -1 ) # warnings off

#options(warn=0) # warnings on

unemployment_2016_out_df = list()

temp = list()

#gcinfo(TRUE)

for (i in 1:length(all_geo_trac)){

  temp[[i]] <- acs.fetch(endyear = 2016, span = 1, geography = all_geo_trac[[i]], table.number = "B23025", col.names = "pretty")

      location = temp[[i]]@geography[1]

  total = temp[[i]]@estimate[,c("Employment Status for the Population 16 Years and Over: Total:")]

  total_unemployed = temp[[i]]@estimate[,c("Employment Status for the Population 16 Years and Over: In labor force: Civilian labor force: Unemployed")]

  #unemployment_2016_out_df[[i]] = data.frame(NAME = location, total_2016 = total, total_unemployed_2016 = total_unemployed)

  unemployment_2016_out_df[[i]] = cbind(location,total,total_unemployed)

  print(unemployment_2016_out_df[[i]])

  cat("iteration",i)

  #gc()

  #Sys.sleep(10)

}

Can anyone see how I can write this differently? The actual temp[[[i]] list I am using to store the dfs on iteration 100 doesnt get past the bytes, same with the output list, unemployment_2016_out_df[[i]] the size doesnt go over 1mb. So those items are not memory intensive which is no surprise there.

My only conclusion is the acs package interfacing with the census api - perhaps the code during the for loop is not releasing the memory on each iteration.

Am I writing the code in a memory efficient way or any other suggestions?