Some implementation/design suggestions

Jun 9, 2015 at 10:44 PM
Edited Jun 9, 2015 at 10:57 PM
  1. PreCompute nodes:
    It's annoying that when trainning different model, CNTK have to go over all data to calculate InvStdOfFeatures, MeanOfFeatures and Prior.
    These should be calculated separately and stored as an input for CNTK training.
  2. Event if there is no PreCompute nodes, CNTK seems to go over all data to find out how many data records. (Does this step really necessary?)
  3. BinaryReader/Writer:
    I follower the example in Configuration Files to Cache data to bin file. However, I figured some errors:
    3.1
    # wsize - inital size of the file in MB
    # if calculated size would be bigger, that is used instead
    wsize=256
    the comment is wrong, the size should be large enough otherwise a error will occur as :
    "Error attemping to create BinaryReader
    Unable to Open/Create file ***********.bin, error 20"
    3.2
    even if using cache, there throws a "EXCEPTION occurred: bad allocation"
    I am not sure if this is memory problem for CPU or GPU allocation error. (traceLevel = 1)
    Suggestion: for EXCEPTIONs please add GPU exceptions or CPU...
  4. Computation Node:
    Some common nodes that maybe popular: Split (feat => feat1, feat2) and Concatenate (feat1, feat2 => feat)
Overall, I like using CNTK and I will start read the code to contribute some code.

Best,
Wenyi
Jun 9, 2015 at 11:07 PM
Hi Wenyi,

Thanks for your suggestions. Here are some answers.

1.PreCompute nodes:
If you don't need these values, you don't need to add them in your model. If you do and want to compute them once, you can also load them from exiting models using MEL before training. In addition, by default it will go through the whole data but you want to compute them from a subset of your data you can also do so by specifying it in the SGD block.

2.Event if there is no PreCompute nodes, CNTK seems to go over all data to find out how many data records. (Does this step really necessary?)
Which reader are you using? different readers may behave differently. For example, if you are using UCIFastReader, it needs to go through it only if you didn't tell the size of each epoch (e.g., epoch=0) or you want it to do randomization on-the-fly.

3.BinaryReader/Writer:
I am forwarding it to the author of the BinaryReader/Writer.

4.Computation Node:
Some common nodes that maybe popular: Split (feat => feat1, feat2) and Concatenate (feat1, feat2 => feat)
We already have the RowSlice node to do split you want. We will add the RowStack node for the concatenation operation.

5.Can CNTK work with multiple GPUs(2) in a same machine?
Yes, if you turn on the MPI mode it currently supports model averaging based multi-GPU operation.
Jun 10, 2015 at 1:00 AM
Edited Jun 10, 2015 at 1:04 AM
Hi Dongyu,

Thanks for the quick reply!
  1. I will try out the MEL.
  2. Yes I was using UCIFastReader. I didn't tell the size of epoch, so I think that's the problem (but does it really matter to let the model know how many data will be used? ).
  3. Thanks!
  4. I didn't find RowStack node in NDL documentation, nor in the code ... maybe the master branch needs update :)
  5. Thanks, I found out in the document that setting deviceID to all will enable the CNTK to use all GPUs.
    However, I am not sure how to turn on MPI mode in VS2013 (does it get automatically turned on? )
Thanks,
Wenyi



Best,
Wenyi
Jun 10, 2015 at 1:27 AM
2.Yes I was using UCIFastReader. I didn't tell the size of epoch, so I think that's the problem (but does it really matter to let the model know how many data will be used? ).
Yes, this is needed to decide when to save a check point file. Of course you may modify UCIFastReader to make it smarter.

3.Thanks!

4.I didn't find RowStack node in NDL documentation, nor in the code ... maybe the master branch needs update :)
As I mentioned in the previous response, we WILL implement RowStack.

5.Thanks, I found out in the document that setting deviceID to all will enable the CNTK to use all GPUs.
However, I am not sure how to turn on MPI mode in VS2013 (does it get automatically turned on? )
You need to check the CNTKTook checked into the source code tree.
Jun 10, 2015 at 7:29 AM
with regard to the BinaryReader/Writer, here is the response from the author:

"the comment is incorrect, the wsize likely has to be at least as large as the actual file will end up being in current code. It can be bigger, and the file will shrink to the actual size when closed. It was coded this way because it’s much easier to have a given memory mapped file size that won’t change as you write the file, and shrink it down after."

so we should change the comment and make the size larger.