Now that the US Government shutdown is over, it is time to download NOAA weather daily summaries in bulk and store them somewhere safe so that at the next shutdown we do not need to worry. Below is the code to download data for a series of years: NOAA_BulkDownload <- function(Year, Dir){ URL <- paste0("ftp://ftp.ncdc.noaa.gov/pub/data/gsod/",Year,"/gsod_",Year,".tar") download.file(URL, destfile=paste0(Dir,"/gsod_",Year,".tar"), method="auto",mode="wb") if(dir.exists(paste0(Dir,"/NOAA Data"))==FALSE){dir.create(paste0(Dir,"/NOAA Data"))} untar(paste0(Dir,"/gsod_ ..read more

This is another function I wrote to access the MET office API and obtain a 5-day ahead weather forecast: METDataDownload <- function(stationID, product, key){ library("RJSONIO") #Load Library library("plyr") library("dplyr") library("lubridate") connectStr <- paste0("http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/json/",stationID,"?res=",product,"&key=",key) con <- url(connectStr) data.json <- fromJSON(paste(readLines(con), collapse="")) close(con) #Station LocID <- data.json$SiteRep$DV$Location$`i` LocName <- data.json$SiteRep$DV$Location$nam ..read more

This is a very simple function to perform geocoding using the Google Maps API: getGeoCode <- function(gcStr, key) { library("RJSONIO") #Load Library gcStr <- gsub(' ','%20',gcStr) #Encode URL Parameters #Open Connection connectStr <- paste0('https://maps.googleapis.com/maps/api/geocode/json?address=',gcStr, "&key=",key) con <- url(connectStr) data.json <- fromJSON(paste(readLines(con), collapse="")) close(con) #Flatten the received JSON data.json <- unlist(data.json) if(data.json["status"]=="OK") { lat <- data.json["results.geometry.location.lat ..read more

Today I decided to create a new repository on GitHub where I am sharing code to do spreadsheet data manipulation in R. The first version of the repository and R script is available here: SpreadsheetManipulation_inR As an example I am using a csv freely available from the IRS, the US Internal Revenue Service. https://www.irs.gov/statistics/soi-tax-stats-individual-income-tax-statistics-2015-zip-code-data-soi This spreadsheet has around 170'000 rows and 131 columns. Please feel free to request new functions to be added or add functions and code yourself directly on GitHub ..read more

My second Shiny app is dedicated to data visualization. Here users can simply upload any csv or txt file and create several plots: Histograms (with option for faceting) Barchart (with error bars, and option for color with dodging and faceting) BoxPlots (with option for faceting) Scatterplots (with options for color, size and faceting) TimeSeries Error bars in barcharts are computed with the mean_se function in ggplot2, which computes error bars as mean ± standard error. When the color option is set, barcharts are plotted one next to the other for each color (option dodging). For scatterp ..read more

Introduction This is a shiny app to visualize heat maps of Street Crimes across Britain from 2010-12 to 2018-01 and test their spatial pattern. The code for both ui.R and server.R is available from my GitHub at: https://github.com/fveronesi/StreetCrimeUK_Shiny Usage Please be aware that this apps downloads data from my personal Dropbox once it starts and every time the user changes some of the settings. This was the only work-around I could think of to use external data in shinyapps.io for free. However, this also makes the app a bit slow, so please be patient. Users can select a date wi ..read more

This post is more for personal use than anything else. It is just a collection of code and functions to produce some of the most used experimental designs in agriculture and animal science.  I will not go into details about these designs. If you want to know more about what to use in which situation you can find material at the following links: Design of Experiments (Penn State): https://onlinecourses.science.psu.edu/stat503/node/5 Statistical Methods for Bioscience (Wisconsin-Madison): http://www.stat.wisc.edu/courses/st572-larget/Spring2007/ R Packages to create several de ..read more

Power analysis is extremely important in statistics since it allows us to calculate how many chances we have of obtaining realistic results. Sometimes researchers tend to underestimate this aspect and they are just interested in obtaining significant p-values. The problem with this is that a significance level of 0.05 does not necessarily mean that what you are observing is real. In the book "Statistics Done Wrong" by Alex Reinhart (which you can read for free here: https://www.statisticsdonewrong.com/) this problem is discussed with an example where we can clearly see that a signifi ..read more

IntroductionIn the previous post I explored the use of linear model in the forms most commonly used in agricultural research. Clearly, when we are talking about linear models we are implicitly assuming that all relations between the dependent variable y and the predictors x are linear. In fact, in a linear model we could specify different shapes for the relation between y and x, for example by including polynomials (read for example: https://datascienceplus.com/fitting-polynomial-regression-r/). However, we can do that only in cases where we can clearly see a particular shape of the relation ..read more

Assessing the accuracy of our model There are several ways to check the accuracy of our models, some are printed directly in R within the summary output, others are just as easy to calculate with specific functions. Please take a look at my previous post for more info on the code. R Squared This is probably the most commonly used statistics and allows us to understand the percentage of variance in the target variable explained by the model. It can be computed as a ratio of the regression sum of squares and the total sum of squares. This is one of the standard measures of accuracy that R pr ..read more