Is there a way to multi-thread R?

#1

Hi,

I am doing comparisons to see how R runs in Alces Flight compared to my local computer. I ran a simulation in R directly on a compute node and as an interactive job. Both times, it took R longer to process the simulation than my local computer. I tried increasing the size of my compute nodes, but it went from 50% CPU utilization to 6% with the increase. I have read that R is not multi-threaded and that this is probably why the job is not being processed efficiently. Can Alces Flight multi-thread R to make it run jobs more efficiently?

Note: I have tried Hyperthreading Control (http://docs.alces-flight.com/en/stable/customisation/hyperthreading-control.html), but just received errors when running the exact commands found in the documentation.

Thanks in advance for your help.

#2

Does anyone have any ideas on how to do this?

#3

Hi Courtney,

Thanks for your interest in Alces Flight -

While it’s possible to run multi-threaded workflows using R, it’s necessary to prepare your code appropriately, and use some suitable multi-threaded libraries for what you want to do. By default, R code runs as a single-thread.

There are a number of multi-threaded libraries which you could look at using - a good overview on parallel programming with R can be found at the R-project.org website at the following URL:

https://cran.r-project.org/web/views/HighPerformanceComputing.html

As R is primarily single-threaded, most users should find that modern versions of R available via Flight will deliver close to the maximum performance available from your cloud platform. For a single job, it’s possible that a desktop processor with a faster CPU could perform faster than a cloud computing instance, which tend to be hosted by server-grade CPUs instead. However - if you’re running lots of different R jobs at the same time, Flight can help you coordinate many different jobs across different nodes to give you more overall throughput than a desktop machine can provide.

Hope that helps -

#4

That helps a lot, thank you.

#5

What linear algebra libraries does Flight use with R? On our local HPC system, we’ve recently switched to using a build of R that was compiled with Intel compilers and MKL. http://rse.shef.ac.uk/blog/intel-R-iceberg/

#6

Our R packages don’t currently use an accelerated linear algebra library. I’ll escalate looking into adding support for these to the engineering team.

It’d also be great to hear about the build steps involved for different linear algebra build configurations, and we would be pleased to accept pull requests to add ATLAS, OpenBLAS, MKL etc. support for R to the Gridware project!

#7

Sounds like fun :slight_smile:

I’m new to the community (have yet to actually use Alces flight for example!) so could I ask 2 stupid newbie questions please:-

  • Where is the GitHub page for the project you speak of?
  • Is the MKL itself already part of the project?
#8

Great, welcome to Alces flight!

Why not give it a go? Flight has features that you should find useful when optimising R, all gridware can be re-compiled on the fly so you could modify the upstream metadata in order to suit your requirements.

In answer to your question the upstream gridware project internal to the Alces Flight marketplace product is hosted here:

Intel MKL is scheduled in a future release but feel free if you’d like to contribute in the meantime