cuDNN in CNTK

Coordinator
Dec 16, 2015 at 5:50 PM
Hi CNTK fans,
We have recently added NVIDIA cuDNN v4 support to CNTK. cuDNN library provides highly optimized primitives that allow very efficient and fast training of convolutional neural nets so if you are using CNNs we strongly recommend enabling cuDNN in CNTK.
CNTK supports most of the features of cuDNN including recent v4 features like batch normalization.

cuDNN is an optional dependency, that is, if you don’t use CNNs you don’t have to change or install anything. And even if you are using CNNs but, for some reason, don’t want to install cuDNN then everything should work as before.

To compile CNTK with cuDNN follow these steps:

On Linux:
• Download Linux version of cuDNN v4 from NVIDIA website: http://developer.download.nvidia.com/compute/redist/cudnn/v4/cudnn-7.0-linux-x64-v4.0-rc.tgz
• Unpack the archive to a folder. We recommend using the following folder by default: /usr/local/cudnn-4.0
o Make sure the folder structure is preserved when unpacking the archive.
• Update LD_LIBRARY_PATH: export LD_LIBRARY_PATH=/usr/local/cudnn-4.0:$LD_LIBRARY_PATH
• Run ./configure script in CNTK source directory. Verify that cuDNN path was set correctly by the script by looking at Config.make file which should have CUDNN_PATH set to a proper directory.
o The script will look for /usr/local/cudnn-4.0 by default. If you installed cuDNN to a different directory then run the script with --with-cudnn option:
./configure --with-cudnn=<path_to_cudnn4>
• Build the code: make -j all

On Windows:
• Download Windows version of cuDNN v4 from NVIDIA website: https://developer.nvidia.com/cudnn
o The site requires registration which is free.
• Unzip the file to a folder, e.g. c:\NVIDIA\cudnn-4.0
• Set environment variable CUDNN_PATH to the cuDNN cuda directory, for example: CUDNN_PATH=C:\NVIDIA\cudnn-4.0\cuda
o You can set it from the command line: setx CUDNN_PATH C:\NVIDIA\cudnn-4.0\cuda
• Restart Visual Studio and rebuild the CNTK solution.

You can find samples of convolutional networks in ExampleSetups\Image folder.
You can also run benchmarks from https://github.com/Alexey-Kamenev/Benchmarks/tree/master/CNTK

Let me know if you have any questions or problems building and running the code.
Thanks.

Alexey