R web
R Web Applications
See also CRAN Task View: Web Technologies and Services
Rmarkdown: create HTML5 web, slides and more
HTTP protocol
- http://en.wikipedia.org/wiki/File:Http_request_telnet_ubuntu.png
- Query string
- How to capture http header? Use curl -i en.wikipedia.org.
- Web Inspector. Build-in in Chrome. Right click on any page and choose 'Inspect Element'.
- Web server
- Simple TCP/IP web server
- HTTP Made Really Easy
- Illustrated Guide to HTTP
- nweb: a tiny, safe Web server with 200 lines
- Tiny HTTPd
An HTTP server is conceptually simple:
- Open port 80 for listening
- When contact is made, gather a little information (get mainly - you can ignore the rest for now)
- Translate the request into a file request
- Open the file and spit it back at the client
It gets more difficult depending on how much of HTTP you want to support - POST is a little more complicated, scripts, handling multiple requests, etc.
Example in R
> co <- socketConnection(port=8080, server=TRUE, blocking=TRUE) > # Now open a web browser and type http://localhost:8080/index.html > readLines(co,1) [1] "GET /index.html HTTP/1.1" > readLines(co,1) [1] "Host: localhost:8080" > readLines(co,1) [1] "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:23.0) Gecko/20100101 Firefox/23.0" > readLines(co,1) [1] "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" > readLines(co,1) [1] "Accept-Language: en-US,en;q=0.5" > readLines(co,1) [1] "Accept-Encoding: gzip, deflate" > readLines(co,1) [1] "Connection: keep-alive" > readLines(co,1) [1] ""
Example in C (Very simple http server written in C, 187 lines)
Create a simple hello world html page and save it as <index.html> in the current directory (/home/brb/Downloads/)
Launch the server program (assume we have done gcc http_server.c -o http_server)
$ ./http_server -p 50002 Server started at port no. 50002 with root directory as /home/brb/Downloads
Secondly open a browser and type http://localhost:50002/index.html. The server will respond
GET /index.html HTTP/1.1 Host: localhost:50002 User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:23.0) Gecko/20100101 Firefox/23.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Connection: keep-alive file: /home/brb/Downloads/index.html GET /favicon.ico HTTP/1.1 Host: localhost:50002 User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:23.0) Gecko/20100101 Firefox/23.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Connection: keep-alive file: /home/brb/Downloads/favicon.ico GET /favicon.ico HTTP/1.1 Host: localhost:50003 User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:23.0) Gecko/20100101 Firefox/23.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Connection: keep-alive file: /home/brb/Downloads/favicon.ico
The browser will show the page from <index.html> in server.
The only bad thing is the code does not close the port. For example, if I have use Ctrl+C to close the program and try to re-launch with the same port, it will complain socket() or bind(): Address already in use.
Another Example in C (55 lines)
http://mwaidyanatha.blogspot.com/2011/05/writing-simple-web-server-in-c.html
The response is embedded in the C code.
If we test the server program by opening a browser and type "http://localhost:15000/", the server received the follwing 7 lines
GET / HTTP/1.1 Host: localhost:15000 User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:23.0) Gecko/20100101 Firefox/23.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Connection: keep-alive
If we include a non-executable file's name in the url, we will be able to download that file. Try "http://localhost:15000/client.c".
If we use telnet program to test, wee need to type anything we want
$ telnet localhost 15000 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. ThisCanBeAnything <=== This is what I typed in the client and it is also shown on server HTTP/1.1 200 OK <=== From here is what I got from server Content-length: 37Content-Type: text/html HTML_DATA_HERE_AS_YOU_MENTIONED_ABOVE <=== The html tags are not passed from server, interesting! Connection closed by foreign host. $
See also more examples under C page.
Others
- http://rosettacode.org/wiki/Hello_world/ (Different languages)
- http://kperisetla.blogspot.com/2012/07/simple-http-web-server-in-c.html (Windows web server)
- http://css.dzone.com/articles/web-server-c (handling HTTP GET request, handling content types(txt, html, jpg, zip. rar, pdf, php etc.), sending proper HTTP error codes, serving the files from a web root, change in web root in a config file, zero copy optimization using sendfile method and php file handling.)
- https://github.com/gtungatkar/Simple-HTTP-server
- https://github.com/davidmoreno/onion
shiny
See Shiny.
plumber: Turning your R code into a RESTful Web API
1. Too heavy to install 2. Get an error with dependencies when I try it on Ubuntu 16.04 3. check out the servr package
- https://github.com/trestletech/plumber
- Creating a Web Service in R
- https://www.rstudio.com/resources/videos/plumber-turning-your-r-code-into-an-api/
- RStudio 1.2 Preview: Plumber Integration
- Using docker to deploy an R plumber API
Docker
- There are two major Docker images. They include gcc, gfortran, .... So it can be used to install Rcpp package for example.
- Official which supports version tags. The official Docker image's dockerfile still points to rocker/r-base.
- rocker project which only has the latest tag
- Using Docker as a Personal Productivity Tool – Running Command Line Apps Bundled in Docker Containers
- Dockerized RStudio server from Duke University. 110 containers were set up on a cloud server (4 cores, 28GB RAM, 400GB disk). Each container has its own port number. Each student is mapped to a single container. https://github.com/mccahill/docker-rstudio
- RStudio in the cloud with Amazon Lightsail and docker
- Mark McCahill (RStudio + Docker)
- BiocImageBuilder
- Why Use Docker with R? A DevOps Perspective
- Running your R script in Docker. Goal: containerizing an R script to eventually execute it automatically each time the container is started, without any user interaction. An enhanced version of the instruction is at this page.
httpuv and servr
http and WebSocket library.
See also the servr package which can start an HTTP server in R to serve static files, or dynamic documents that can be converted to HTML files (e.g., R Markdown) under a given directory.
Rscript -e "servr::httd()" # default port 4321 # If index.html was found it will be used # Otherwise, it will serve files under the cur dir # Open another terminal Rscript -e "servr::httd('/tmp')" -p4000
beakr
- What is beakr?
- Web Frameworks for R – A Brief Overview and hello world examples.
workflowr
RApache
gWidgetsWWW
- http://www.jstatsoft.org/v49/i10/paper
- gWidgetsWWW2 gWidgetsWWW based on Rook
- Compare shiny with gWidgetsWWW2.rapache
Rook
See Rook.
sumo
Sumo is a fully-functional web application template that exposes an authenticated user's R session within java server pages. See the paper http://journal.r-project.org/archive/2012-1/RJournal_2012-1_Bergsma+Smith.pdf.
Stockplot
FastRWeb
http://cran.r-project.org/web/packages/FastRWeb/index.html
WebDriver
'WebDriver' Client for 'PhantomJS'
https://github.com/rstudio/webdriver
Rwui
CGHWithR and WebDevelopR
CGHwithR is still working with old version of R although it is removed from CRAN. Its successor is WebDevelopR. Its The vignette (year 2013) provides a review of several available methods.
manipulate from RStudio
This is not a web application. But the manipulate package can be used to create interactive plot within R(Studio) environment easily. Its source is available at here.
Mathematica also has manipulate function for plotting; see here.
RCloud
RCloud is an environment for collaboratively creating and sharing data analysis scripts. RCloud lets you mix analysis code in R, HTML5, Markdown, Python, and others. Much like Sage, iPython notebooks and Mathematica, RCloud provides a notebook interface that lets you easily record a session and annotate it with text, equations, and supporting images.
See also the Talk in UseR 2014.
cloudyr and flyio - Input Output Files in R from Cloud or Local
https://blog.socialcops.com/inside-sc/announcements/flyio-r-package-interact-data-cloud/ Announcing flyio, an R Package to Interact with Data in the Cloud]
Dropbox access
rdrop2 package
Javascript
JavaScript for R — ebook https://book.javascript-for-r.com/
Web page scraping
http://www.slideshare.net/schamber/web-data-from-r#btnNext
xml2 package
rvest package depends on xml2.
rvest
On Ubuntu, we need to install two packages first!
sudo apt-get install libcurl4-openssl-dev # OR libcurl4-gnutls-dev sudo apt-get install libxml2-dev
- https://github.com/hadley/rvest
- Visualizing obesity across United States by using data from Wikipedia
- rvest tutorial: scraping the web using R
- https://renkun.me/pipeR-tutorial/Examples/rvest.html
- http://zevross.com/blog/2015/05/19/scrape-website-data-with-the-new-r-package-rvest/
- Google scholar scraping with rvest package
- An update to "An adventure in downloading books"
V8: Embedded JavaScript Engine for R
R⁶ — General (Attys) Distributions: V8, rvest, ggbeeswarm, hrbrthemes and tidyverse packages are used.
pubmed.mineR
Text mining of PubMed Abstracts (http://www.ncbi.nlm.nih.gov/pubmed). The algorithms are designed for two formats (text and XML) from PubMed.
R code for scraping the P-values from pubmed, calculating the Science-wise False Discovery Rate, et al (Jeff Leek)
Get API data
How to get API data with R. See how to write your own R code to pull data from an API using API key authentication.
These R packages import sports, weather, stock data and more
- https://www.computerworld.com/article/3109890/data-analytics/these-r-packages-import-sports-weather-stock-data-and-more.html
- https://github.com/ALShum/rwunderground
- Accessing APIs from R (and a little R programming). A personal key is required. 500 times per day for a free account.
- https://cran.r-project.org/web/packages/rnoaa/index.html. A personal API key (token) is required. 10,000 requests per day
http://ram-n.github.io/weatherData/(not working)
Diving Into Dynamic Website Content with splashr
https://rud.is/b/2017/02/09/diving-into-dynamic-website-content-with-splashr/
Network
netstat
https://cran.r-project.org/web/packages/netstat/index.html
Send email
curl
- curl::send_mail().
- R Internet: Yet Another Way To Send Emails On Windows/Sending Emails with {curl} and Docker
Note
- As we can see in the example snippet, there is a duplicate of sender's and recipient's emails.
- It seems the recipients and sender parts are the real ones
- The sender's email entered in message will not be shown and used at the end
- The recipient's email entered in message will be shown and used at the end
- Question: does sender's email has to be gmail.com?
library(curl) recipients <- "[email protected]" sender <- '[email protected]' # Full email message in RFC2822 format message <- 'From: "R (curl package)" <[email protected]> To: "Roger Recipient" <[email protected]> Subject: Hello R user! Dear R user, I am sending this email using curl.' # Send the email send_mail(sender, recipients, message, smtp_server = 'smtps://smtp.gmail.com', username = 'curlpackage', password = 'qyyjddvphjsrbnlm')
emayili
blastula (RStudio)
mailR
Easiest. Require rJava package (not trivial to install, see rJava). mailR is an interface to Apache Commons Email to send emails from within R. See also send bulk email
Before we use the mailR package, we have followed here to have Allow less secure apps: 'ON' ; or you might get an error Error: EmailException (Java): Sending the email to the following server failed : smtp.gmail.com:465. Once we turn on this option, we may get an email for the notification of this change. Note that the recipient can be other than a gmail.
> send.mail(from = "[email protected]", to = c("[email protected]", "Recipient 2 <[email protected]>"), replyTo = c("Reply to someone else <[email protected]>") subject = "Subject of the email", body = "Body of the email", smtp = list(host.name = "smtp.gmail.com", port = 465, user.name = "gmail_username", passwd = "password", ssl = TRUE), attach.files ="./myattachment.txt", authenticate = TRUE, send = TRUE) [1] "Java-Object{org.apache.commons.mail.SimpleEmail@7791a895}"
- Encrypt the password. First write your password on your environment variable Sys.setenv(GMAIL_PWD = 'mypassword') Then call your env variable in your script passwd = Sys.getenv("GMAIL_PWD").
- MailR SMTP Setup (Gmail, Outlook, Yahoo) | STARTTLS
gmailr
More complicated. gmailr provides access the Google's gmail.com RESTful API. Vignette and an example on here. Note that it does not use a password; it uses a json file for oauth authentication downloaded from https://console.cloud.google.com/. See also https://github.com/jimhester/gmailr/issues/1.
library(gmailr) gmail_auth('mysecret.json', scope = 'compose') test_email <- mime() %>% to("[email protected]") %>% from("[email protected]") %>% subject("This is a subject") %>% html_body("<html><body>I wish this was bold</body></html>") send_message(test_email)
sendmailR
sendmailR provides a simple SMTP client. It is not clear how to use the package (i.e. where to enter the password).
json
jsonlite
- Reading JSON file from web and preparing data for analysis
- xkcd Comics as a Minimal Example for Calling APIs, Downloading Files and Displaying PNG Images with R
rjson
http://heuristically.wordpress.com/2013/05/20/geolocate-ip-addresses-in-r/
RJSONIO
Accessing Bitcoin Data with R
http://blog.revolutionanalytics.com/2015/11/accessing-bitcoin-data-with-r.html
Plot IP on google map
- http://thebiobucket.blogspot.com/2011/12/some-fun-with-googlevis-plotting-blog.html#more (RCurl, RJONIO, plyr, googleVis)
- http://devblog.icans-gmbh.com/using-the-maxmind-geoip-api-with-r/ (RCurl, RJONIO, maps)
- http://cran.r-project.org/web/packages/geoPlot/index.html (geoPlot package (deprecated as 8/12/2013))
- http://archive09.linux.com/feature/135384 (Not R) ApacheMap
- http://batchgeo.com/features/geolocation-ip-lookup/ (Not R) (Enter a spreadsheet of adress, city, zip or a column of IPs and it will show the location on google map)
- http://code.google.com/p/apachegeomap/
The following example is modified from the first of above list.
require(RJSONIO) # fromJSON require(RCurl) # getURL temp = getURL("https://gist.github.com/arraytools/6743826/raw/23c8b0bc4b8f0d1bfe1c2fad985ca2e091aeb916/ip.txt", ssl.verifypeer = FALSE) ip <- read.table(textConnection(temp), as.is=TRUE) names(ip) <- "IP" nr = nrow(ip) Lon <- as.numeric(rep(NA, nr)) Lat <- Lon Coords <- data.frame(Lon, Lat) ip2coordinates <- function(ip) { api <- "http://freegeoip.net/json/" get.ips <- getURL(paste(api, URLencode(ip), sep="")) # result <- ldply(fromJSON(get.ips), data.frame) result <- data.frame(fromJSON(get.ips)) names(result)[1] <- "ip.address" return(result) } for (i in 1:nr){ cat(i, "\n") try( Coords[i, 1:2] <- ip2coordinates(ip$IP[i])[c("longitude", "latitude")] ) } # append to log-file: logfile <- data.frame(ip, Lat = Coords$Lat, Long = Coords$Lon, LatLong = paste(round(Coords$Lat, 1), round(Coords$Lon, 1), sep = ":")) log_gmap <- logfile[!is.na(logfile$Lat), ] require(googleVis) # gvisMap gmap <- gvisMap(log_gmap, "LatLong", options = list(showTip = TRUE, enableScrollWheel = TRUE, mapType = 'hybrid', useMapTypeControl = TRUE, width = 1024, height = 800)) plot(gmap)
The plot.gvis() method in googleVis packages also teaches the startDynamicHelp() function in the tools package, which was used to launch a http server. See Jeffrey Horner's note about deploying Rook App.
GEO (Gene Expression Omnibus)
See this internal link.
Interactive html output
sendplot
RIGHT
The supported plot types include scatterplot, barplot, box plot, line plot and pie plot.
In addition to tooltip boxes, the package can create a table showing all information about selected nodes.
r2d3
r2d3 - R Interface to D3 Visualizations
d3Network
- http://christophergandrud.github.io/d3Network/ (old)
- https://christophergandrud.github.io/networkD3/ (new)
library(d3Network) Source <- c("A", "A", "A", "A", "B", "B", "C", "C", "D") Target <- c("B", "C", "D", "J", "E", "F", "G", "H", "I") NetworkData <- data.frame(Source, Target) d3SimpleNetwork(NetworkData, height = 800, width = 1024, file="tmp.html")
htmlwidgets for R
Embed widgets in R Markdown documents and Shiny web applications.
- Official website http://www.htmlwidgets.org/.
- How to write a useful htmlwidgets in R: tips and walk-through a real example
networkD3
This is a port of Christopher Gandrud's d3Network package to the htmlwidgets framework.
scatterD3
scatterD3 is an HTML R widget for interactive scatter plots visualization. It is based on the htmlwidgets R package and on the d3.js javascript library.
dygraphs
rthreejs - Create interactive 3D scatter plots, network plots, and globes
rayshader: 2D and 3D mapping and data visualization with shades
https://github.com/tylermorganwall/rayshader
On Rstudio server, we need options(rgl.printRglwidget = TRUE) ; see Why is my 3D plot not showing up in R Studio plot viewer?.
d3heatmap
See R
collapsibleTree
- https://github.com/adeelk93/collapsibletree
- https://cran.r-project.org/web/packages/jsTreeR/index.html
svgPanZoom
This 'htmlwidget' provides pan and zoom interactivity to R graphics, including 'base', 'lattice', and 'ggplot2'. The interactivity is provided through the 'svg-pan-zoom.js' library.
DT: An R interface to the DataTables library
reactable: interactive table with rows that expand when clicked
How to create tables in R with expandable rows
getable: creating a 'dynamic' HTML table
Getting Tabular Data Through JavaScript in Compiled R Markdown Documents.
The content stays static while the data could be updated independently without rewriting or recompiling the HTML document. This could be done by utilizing JavaScript’s ability to asynchronously fetch data from the web and generate DOM elements based on these data.
plotly
- Power curves and ggplot2.
- TIME SERIES CHARTS BY THE ECONOMIST IN R USING PLOTLY & FIVE INTERACTIVE R VISUALIZATIONS WITH D3, GGPLOT2, & RSTUDIO
- Filled chord diagram
- DASHBOARDS IN R WITH SHINY & PLOTLY
- Plotly Graphs in Shiny,
- How to plot basic charts with plotly
- How to add Trend Lines in R Using Plotly
- Introduction to Interactive Graphics in R with plotly
- Tooltip
highcharter
https://cran.r-project.org/web/packages/highcharter/ Good for time series plot.
Amazon
Download product information and reviews from Amazon.com
sudo apt-get install libxml2-dev sudo apt-get install libcurl4-openssl-dev
and in R
install.packages("devtools") install.packages("XML") install.packages("pbapply") install.packages("dplyr") devtools::install_github("56north/Rmazon") product_info <- Rmazon::get_product_info("1593273843") reviews <- Rmazon::get_reviews("1593273843") reviews[1,6] # only show partial characters from the 1st review nchar(reviews[1,6]) as.character(reviews[1,6]) # show the complete text from the 1st review reviews <- Rmazon::get_reviews("B07BNGJXGS") # Fetching 30 reviews of 'BOOX Note Ereader,Android 6.0 32 GB 10.3" Dual Touch HD Display' # |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed = 02s reviews # A tibble: 30 x 6 reviewRating reviewDate reviewFormat Verified_Purcha… reviewHeadline <dbl> <chr> <lgl> <lgl> <chr> 1 4 May 23, 2… NA TRUE Good for PDF … 2 3 May 8, 20… NA FALSE The reading s… 3 5 May 17, 2… NA TRUE E-reader and … 4 3 May 24, 2… NA TRUE Good hardware… 5 3 June 21, … NA TRUE Poor QC 6 5 August 5,… NA TRUE Excellent for… 7 5 May 31, 2… NA TRUE Especially li… 8 5 July 4, 2… NA TRUE Android 6 rea… 9 4 July 15, … NA TRUE Remember the … 10 4 June 9, 2… NA TRUE Overall fanta… # ... with 20 more rows, and 1 more variable: reviewText <chr> reviews[1, 6] # 6-th column is the review text
gutenbergr
Feed
feedeR - Feed Reader Package for R
OCR
- Tesseract package: High Quality OCR in R, How to do Optical Character Recognition (OCR) of non-English documents in R using Tesseract?
- https://cran.r-project.org/web/packages/abbyyR/index.html
Wikipedia
WikipediR: R's MediaWiki API client library