Programming with Big Data in R

<h2 id="package-design">Package design</h2>
<p>Programming with pbdR requires usage of various packages developed by pbdR core team. Packages developed are the following.
</p>
<table><tbody><tr><th>General</th><th>I/O</th><th>Computation</th><th>Application</th><th>Profiling</th><th>Client/Server</th></tr><tr><td>pbdDEMO</td><td>pbdNCDF4</td><td>pbdDMAT</td><td>pmclust</td><td>pbdPROF</td><td>pbdZMQ</td></tr><tr><td>pbdMPI</td><td>pbdADIOS</td><td>pbdBASE</td><td>pbdML</td><td>pbdPAPI</td><td>remoter</td></tr><tr><td></td><td></td><td>pbdSLAP</td><td></td><td>hpcvis</td><td>pbdCS</td></tr><tr><td></td><td></td><td>kazaam</td><td></td><td></td><td>pbdRPC</td></tr></tbody></table>

<p>Among these packages, pbdMPI provides wrapper functions to <a href="/facts/Message_Passing_Interface/BEuBgDpm">MPI</a> library, and it also produces a <a href="/facts/Library_(computing)/xRzL3P7q">shared library</a> and a configuration file for MPI environments. All other packages rely on this configuration for installation and library loading that avoids difficulty of library linking and compiling. All other packages can directly use MPI functions easily.
</p>
<ul><li>pbdMPI --- an efficient interface to MPI either <a href="/facts/Open_MPI/3eJUp5JQ">OpenMPI</a> or <a href="/facts/MPICH2/2PsUjMcS">MPICH2</a> with a focus on Single Program/Multiple Data (<a href="/facts/SPMD/SEDIrB0l">SPMD</a>) parallel programming style</li>
<li>pbdSLAP --- bundles scalable dense linear algebra libraries in double precision for R, based on <a href="/facts/ScaLAPACK/LfBYHxLJ">ScaLAPACK</a> version 2.0.2 which includes several scalable linear algebra packages (namely BLACS, <a href="/facts/PBLAS/mdqjxG7u">PBLAS</a>, and <a href="/facts/ScaLAPACK/LfBYHxLJ">ScaLAPACK</a>).</li>
<li>pbdNCDF4 --- interface to Parallel Unidata <a href="/facts/NetCDF/DUwTEW7n">NetCDF</a>4 format data files</li>
<li>pbdBASE --- low-level <a href="/facts/ScaLAPACK/LfBYHxLJ">ScaLAPACK</a> codes and wrappers</li>
<li>pbdDMAT --- distributed matrix classes and computational methods, with a focus on linear algebra and statistics</li>
<li>pbdDEMO --- set of package demonstrations and examples, and this unifying vignette</li>
<li>pmclust --- parallel <a href="/facts/Model-based_clustering/zkVRu6qS">model-based clustering</a> using pbdR</li>
<li>pbdPROF --- profiling package for MPI codes and visualization of parsed stats</li>
<li>pbdZMQ --- interface to <a href="/facts/ZeroMQ/e8R8qTMX">ØMQ</a></li>
<li>remoter --- R client with remote R servers</li>
<li>pbdCS --- pbdR client with remote pbdR servers</li>
<li>pbdRPC --- remote procedure call</li>
<li>kazaam --- very tall and skinny distributed matrices</li>
<li>pbdML --- machine learning toolbox</li></ul>
<p>Among those packages, the pbdDEMO package is a collection of 20+ package demos which offer example uses of the various pbdR packages, and contains a vignette that offers detailed explanations for the demos and provides some mathematical or statistical insight.
</p>
<h2 id="examples">Examples</h2>
<h3>Example 1</h3>
<p>Hello World! Save the following code in a file called "demo.r"
</p>
### Initial MPI
library(pbdMPI, quiet = TRUE)
init()

comm.cat("Hello World!\n")

### Finish
finalize()

<p>and use the command
</p>
mpiexec -np 2 Rscript demo.r

<p>to execute the code where <a href="/facts/R_(programming_language)/LSrkr8K8">Rscript</a> is one of command line executable program.
</p>
<h3>Example 2</h3>
<p>The following example modified from pbdMPI illustrates the basic <a href="/facts/Programming_language_syntax/zM5oInza">syntax of the language</a> of pbdR.
Since pbdR is designed in <a href="/facts/SPMD/SEDIrB0l">SPMD</a>, all the R scripts are stored in files and executed from the command line via mpiexec, mpirun, etc. Save the following code in a file called "demo.r"
</p>
### Initial MPI
library(pbdMPI, quiet = TRUE)
init()
.comm.size <- comm.size()
.comm.rank <- comm.rank()

### Set a vector x on all processors with different values
N <- 5
x <- (1:N) + N * .comm.rank

### All reduce x using summation operation
y <- allreduce(as.integer(x), op = "sum")
comm.print(y)
y <- allreduce(as.double(x), op = "sum")
comm.print(y)

### Finish
finalize()

<p>and use the command
</p>
mpiexec -np 4 Rscript demo.r

<p>to execute the code where <a href="/facts/R_(programming_language)/LSrkr8K8">Rscript</a> is one of command line executable program.
</p>
<h3>Example 3</h3>
<p>The following example modified from pbdDEMO illustrates the basic ddmatrix computation of pbdR which performs <a href="/facts/Singular_value_decomposition/8t3v6wk3">singular value decomposition</a> on a given matrix.
Save the following code in a file called "demo.r"
</p>
# Initialize process grid
library(pbdDMAT, quiet=T)
if(comm.size() != 2)
  comm.stop("Exactly 2 processors are required for this demo.")
init.grid()

# Setup for the remainder
comm.set.seed(diff=TRUE)
M <- N <- 16
BL <- 2 # blocking --- passing single value BL assumes BLxBL blocking
dA <- ddmatrix("rnorm", nrow=M, ncol=N, mean=100, sd=10)

# LA SVD
svd1 <- La.svd(dA)
comm.print(svd1$d)

# Finish
finalize()

<p>and use the command
</p>
mpiexec -np 2 Rscript demo.r

<p>to execute the code where <a href="/facts/R_(programming_language)/LSrkr8K8">Rscript</a> is one of command line executable program.
</p>
<h2 id="further-reading">Further reading</h2>
<ul><li>Raim, A.M. (2013). <a href="https://web.archive.org/web/20140204051402/http://userpages.umbc.edu/~gobbert/papers/pbdRtara2013.pdf"><i>Introduction to distributed computing with pbdR at the UMBC High Performance Computing Facility</i></a> (PDF) (Technical report). UMBC High Performance Computing Facility, University of Maryland, Baltimore County. HPCF-2013-2. Archived from <a href="http://userpages.umbc.edu/~gobbert/papers/pbdRtara2013.pdf">the original</a> (PDF) on 2014-02-04. Retrieved 2013-06-26.</li>
<li>Bachmann, M.G., Dyas, A.D., Kilmer, S.C. and Sass, J. (2013). <a href="https://web.archive.org/web/20140204051351/http://userpages.umbc.edu/~gobbert/papers/REU2013Team1.pdf"><i>Block Cyclic Distribution of Data in pbdR and its Effects on Computational Efficiency</i></a> (PDF) (Technical report). UMBC High Performance Computing Facility, University of Maryland, Baltimore County. HPCF-2013-11. Archived from <a href="http://userpages.umbc.edu/~gobbert/papers/REU2013Team1.pdf">the original</a> (PDF) on 2014-02-04. Retrieved 2014-02-01.{{cite tech report}}:  CS1 maint: multiple names: authors list (link)</li>
<li>Bailey, W.J., Chambless, C.A., Cho, B.M. and Smith, J.D. (2013). <a href="https://web.archive.org/web/20140204055902/http://userpages.umbc.edu/~gobbert/papers/REU2013Team2.pdf"><i>Identifying Nonlinear Correlations in High Dimensional Data with Application to Protein Molecular Dynamics Simulations</i></a> (PDF) (Technical report). UMBC High Performance Computing Facility, University of Maryland, Baltimore County. HPCF-2013-12. Archived from <a href="http://userpages.umbc.edu/~gobbert/papers/REU2013Team2.pdf">the original</a> (PDF) on 2014-02-04. Retrieved 2014-02-01.{{cite tech report}}:  CS1 maint: multiple names: authors list (link)</li>
<li><a href="/facts/Dirk_Eddelbuettel/rrKp5wRI">Dirk Eddelbuettel</a> (13 November 2022). <a href="https://cran.r-project.org/web/views/HighPerformanceComputing.html">"High-Performance and Parallel Computing with R"</a>.</li>
<li><a href="http://www.r-bloggers.com/r-at-12000-cores/">"R at 12,000 Cores"</a>.This article was read 22,584 times in 2012 since it posted on October 16, 2012, and ranked number 3<a class="footnote-ref" id="fnref:7" href="#fn:7"><sup>7</sup></a></li>
<li>Google Summer of Code - R 2013. <a href="https://archive.today/20130629095333/http://rwiki.sciviews.org/doku.php?id=developers:projects:gsoc2013:mpiprofiler">"Profiling Tools for Parallel Computing with R"</a>. Archived from <a href="http://rwiki.sciviews.org/doku.php?id=developers:projects:gsoc2013:mpiprofiler">the original</a> on 2013-06-29. {{cite web}}: |author= has generic name (help)CS1 maint: numeric names: authors list (link)</li>
<li>Wush Wu (2014). <a href="http://rpubs.com/wush978/pbdMPI-linux-pilot">"在雲端運算環境使用R和MPI"</a>.{{cite web}}:  CS1 maint: numeric names: authors list (link)</li>
<li>Wush Wu (2013). <a href="https://www.youtube.com/watch?v=m1vtPESsFqM">"快速在AWS建立R和pbdMPI的使用環境"</a>. <i><a href="/facts/YouTube/db276jst">YouTube</a></i>.{{cite web}}:  CS1 maint: numeric names: authors list (link)</li></ul>

<h2 id="external-links">External links</h2>
<ul><li><a href="http://www.r-pbd.org">Official website</a></li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1"><p>Ostrouchov, G., Chen, W.-C., Schmidt, D., Patel, P. (2012). "Programming with Big Data in R".{{cite web}}:  CS1 maint: multiple names: authors list (link) <a href="http://r-pbd.org" target="_blank">http://r-pbd.org</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li>
<li id="fn:2"><p>Chen, W.-C. & Ostrouchov, G. (2011). "HPSC -- High Performance Statistical Computing for Data Intensive Research". Archived from the original on 2013-07-19. Retrieved 2013-06-25. <a href="https://web.archive.org/web/20130719020318/http://thirteen-01.stat.iastate.edu/snoweye/hpsc/" target="_blank">https://web.archive.org/web/20130719020318/http://thirteen-01.stat.iastate.edu/snoweye/hpsc/</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li>
<li id="fn:3"><p>"Basic Tutorials for R to Start Analyzing Data". 3 November 2022. <a href="https://learnshareit.com/tutorials-for-r/" target="_blank">https://learnshareit.com/tutorials-for-r/</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li>
<li id="fn:4"><p>Yu, H. (2002). "Rmpi: Parallel Statistical Computing in R". R News. <a href="https://cran.r-project.org/package=Rmpi" target="_blank">https://cran.r-project.org/package=Rmpi</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li>
<li id="fn:5"><p>Yu, H. (2002). "Rmpi: Parallel Statistical Computing in R". R News. <a href="https://cran.r-project.org/package=Rmpi" target="_blank">https://cran.r-project.org/package=Rmpi</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li>
<li id="fn:6"><p>Mike Houston. "Folding@Home - GPGPU". Retrieved 2007-10-04. <a href="http://graphics.stanford.edu/~mhouston/" target="_blank">http://graphics.stanford.edu/~mhouston/</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li>
<li id="fn:7"><p>"100 most read R posts in 2012 (stats from R-bloggers) – big data, visualization, data manipulation, and other languages". <a href="http://www.r-bloggers.com/100-most-read-r-posts-for-2012-stats-from-r-bloggers-big-data-visualization-data-manipulation-and-other-languages/" target="_blank">http://www.r-bloggers.com/100-most-read-r-posts-for-2012-stats-from-r-bloggers-big-data-visualization-data-manipulation-and-other-languages/</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li>
</ol>

Programming with Big Data in R open-in-new

Programming with Big Data in R