Use openflight on microsoft azure

Hello!

I recently attended a openflight bootcamp at Liverpool Uni, I’d like to do some TensaFlow work across multiple GPU on my azure account, when I start an instance I can’t find any openflight images - can any one help me with how do I get an image to be able to use openflight?

Thankyou in advance

Kaya

Hi Kaya,

Welcome to the OpenFlight community forums!

OpenFlight doesn’t provide images for it’s environment as the tools are conveniently available through RPMs. All you need to do is launch instances with a basic CentOS 7 image (for example, the image “CentOS 7 Minimal” by Tunnelbiz should do the trick) and then to install the OpenFlight user tools as described in the usage knowledgebase.

Once you have the tools installed you can start with your Tensorflow workflow, a couple of example workflows are available through the learn knowledgebase.

If you’re looking to share storage or data between the nodes in your research environment then I’d suggest taking a look at setting up the research environment with the OpenFlight Ansible playbook.

Hope this helps.

Cheers,

Stu

thanks i have started that instance. I think your document are broken maybe because I needed to add “sudo” in front of some commands even to make them seen to work.

I still don’t get “flight” commands after those I get
[kaya@kayasingle ~]$ flight start

-bash: flight: command not found

Can you help?

Hi Stu.

I fixed this, flight works now! Maybe something runs by cron in the background because the flight command just appeared after I restarted cron and a did full reboot.

I can get some singularity commands to run but when following the TensorFlow example I get this->
<singularity> [kaya@kayasingle ~]$ singularity exec docker://tensorflow/tensorflow:1.15.0 python ./models/tutorials/image/mnist/convolutional.py INFO: Converting OCI blobs to SIF format INFO: Starting build... Getting image source signatures Copying blob sha256:22e816666fd6516bccd19765947232debc14a5baf2418b2202fd67b3807b6b91 25.45 MiB / 25.45 MiB [====================================================] 2s Copying blob sha256:079b6d2a1e53c648abc48222c63809de745146c2ee8322a1b9e93703318290d6 34.54 KiB / 34.54 KiB [====================================================] 0s Copying blob sha256:11048ebae90883c19c9b20f003d5dd2f5bbf5b48556dabf06c8ea5c871c8debe 849 B / 849 B [============================================================] 0s Copying blob sha256:c58094023a2e61ef9388e283026c5d6a4b6ff6d10d4f626e866d38f061e79bb9 162 B / 162 B [============================================================] 0s Copying blob sha256:fb153ade6d147fb3ecf01f9cc24b489e684885946290c54503fa9667e0b587ac 25.05 MiB / 25.05 MiB [====================================================] 2s Copying blob sha256:1acae33444ea5e4d7d5d9d14d0927070c9debb0f57736009e94424487969781b 119.68 MiB / 119.68 MiB [=================================================] 13s Copying blob sha256:b5faf7a0deec9e293ebd6cb83f31911842a1cd5ba62dc14407b4b0e793277b8a 3.22 MiB / 3.22 MiB [======================================================] 0s Copying blob sha256:1723b9e0e1fbf4e2e93c58bbfa84a6dcbb2cea1fd78a55546fe0c6523bf6a077 170 B / 170 B [============================================================] 0s Copying blob sha256:001607d3f37b0493933ef3f153a63401ccff8bd767fa8e0b625f4523bc7b12f9 845.00 MiB / 845.00 MiB [================================================] 1m2s Copying blob sha256:1b4d997478fe8ecace99aa2c66a757f21d6cccbf689af0e89d00ba4fa4690645 1.01 KiB / 1.01 KiB [======================================================] 0s Copying blob sha256:f4e7edc97ff49dfbd8204e0cae29e4786f79c4719eeecb6d81497a5894f686c4 1.01 KiB / 1.01 KiB [======================================================] 0s Copying config sha256:eb3bff22ae06d82cee447090d5fc7d33aacf9e80ffac234b7450e09c76f1b325 5.04 KiB / 5.04 KiB [======================================================] 0s Writing manifest to image destination Storing signatures INFO: Creating SIF file... INFO: Build complete: /home/kaya/.local/share/flight/env/singularity+default/cache/oci-tmp/347a9da65b6eaf4d233d3b639f345a055e6f03c4535e92d60dc30c7ab55023f9/tensorflow_1.15.0.sif INFO: Convert SIF file to sandbox... WARNING: underlay of /etc/localtime required more than 50 (91) bind mounts python: can't open file './models/tutorials/image/mnist/convolutional.py': [Errno 2] No such file or directory INFO: Cleaning up image... <singularity> [kaya@kayasingle ~]$

I’ve already tried sudo “echo 100 > /proc/sys/user/max_user_namespaces” from the flight env info singularity command but that hasn’t helped.
Any ideas?

Hi Kaya,

A restart of the terminal session is usually needed to expose the flight command as it is setup through a profile script, I’ll add a note of that to the documentation - thanks for spotting this!

It looks like the Tensorflow binary is unable to find the example model, judging by the following line in your output:

python: can't open file './models/tutorials/image/mnist/convolutional.py': [Errno 2] No such file or directory

From further investigation it looks like the Tensorflow models repository has been evolving alongside Tensorflow releases. With the example in our documentation using an older version of Tensorflow (1.15.0) and the models repo having modified directory structure for the 2.0+ releases, we’ll need to specifically checkout a compatible release branch of the models repository as follows:

cd ~/models/
git checkout v1.13.0

After performing the above, the tutorials directory will now be present in the git repo. If you cd ~ and run the singularity command again then you should find that the job begins to run.

Let me know how that goes!

Cheers,

Stu

Thankyou very much that works!
I didnt see the python error, but I thought the warning about /etc/localtime was the error.

Do you know of any free images I can use with flight? I think the tunnelbiz image is quite expensive in relation to the included ubuntu images. If I search for “centos” there seem to be quite a few free ones, is the one you suggested better?

1 Like

Glad to hear that sorted it! That localtime error was quite the deception, no worries.

I hadn’t notice that one had extra costs, I’ve had a quick look at the marketplace. Sadly it looks like a lot of the images are “free trials” so would come with extra costs. However, the “Node.js 12 with CentOS 7.6” image by Cognosys Inc. may be a good choice.

Optionally, you can upload a clean CentOS image that is available in the openflight image repo, a guide to uploading one of these images to Azure is available on the openflight website.

Cheers,

Stu