Difference between running TensorFlow on singularity or conda?

Hi,

I’ve been experimenting with running TensaFlow on OpenFlight. Normally I’m using pip to install so the Conda method described in the tutorials is most familiar. I’ve heard that singularity can be better performance - should I be using it?

Kaya

Hi Kaya,

From a look at timing the example Tensorflow job in both ecosystems, I get the following results (testing on a cloud machine with 2 cores and 4GB of RAM)

# Singularity
<singularity> [centos@openflight (myresearchenv) ~]$ time singularity exec docker://tensorflow/tensorflow:1.15.0 python ./models/tutorials/image/mnist/convolutional.py
INFO:    Convert SIF file to sandbox...
WARNING: underlay of /etc/localtime required more than 50 (91) bind mounts
**<snip>**
Step 8500 (epoch 9.89), 124.7 ms
Minibatch loss: 1.607, learning rate: 0.006302
Minibatch error: 0.0%
Validation error: 0.9%
Test error: 0.8%
INFO:    Cleaning up image...

real	18m55.011s
user	36m5.320s
sys	0m15.821s
# Conda
(tensorflow) <conda> [centos@openflight (myresearchenv) ~]$ time python ./models/tutorials/image/mnist/convolutional.py
WARNING:tensorflow:From ./models/tutorials/image/mnist/convolutional.py:340: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.
**<snip>**
Step 8500 (epoch 9.89), 125.7 ms
Minibatch loss: 1.603, learning rate: 0.006302
Minibatch error: 0.0%
Validation error: 0.9%
Test error: 0.8%

real	18m7.655s
user	35m25.991s
sys	0m9.789s

It looks to me like the difference in performance is negligible with Singularity having a slightly longer startup time due to launching the container that runs Tensorflow.

Hope this helps.

Stu

Thanks I will stick with conda because it seems much easier!