Restarting CN

Jan 19, 2015 at 4:36 PM
I would like a way to graph the progress of training, for example by plotting the prediction error for each epoch.

One approach would be to run a single epoch (or a small number of epochs) but without the overhead of writing a complete set of output data. Then the prediction error for this run could be obtained from the log file and plotted. Repeat until satisfied with the and then invoke an output operation. How could this be done, or is there a simpler way?
 Richard
Coordinator
Jan 19, 2015 at 4:53 PM
You can turn off majority of logging info by setting trace level to 0. If that is still not enough you just need to "grep Finished" on the log file to show only epoch result.
Jan 19, 2015 at 5:11 PM
Yes, that works fine after training is finished for a specified maxEpochs. But what I want is to be able to create a plot (in a Matlab GIU figure) that tracks the progress as training is happening. And then I want to be able to stop training by hand if I decide it is already OK.

Richard

At 09:54 AM 1/19/2015, you wrote:

From: dongyu
You can turn off majority of logging info by setting trace level to 0. If that is still not enough you just need to "grep Finished" on the log file to show only epoch result.

Read the full discussion online.

To add a post to this discussion, reply to this email ([email removed])

To start a new discussion for this project, email [email removed]

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe on CodePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at CodePlex.com
Coordinator
Jan 19, 2015 at 5:15 PM
well, you just write a simple mat program to grep the log file periodically (e.g., after several minutes depends on the size of the training set and model) and plot the result.
Jan 19, 2015 at 5:46 PM
That would work but I would prefer to see the latest error figure as soon as each epoch was finished.

And if I do what you propose, then when I am satisfied, how do I stop any further epochs of training, and invoke output processing (action=write)?

Richard


At 10:15 AM 1/19/2015, you wrote:

From: dongyu
well, you just write a simple mat program to grep the log file periodically (e.g., after several minutes depends on the size of the training set and model) and plot the result.

Read the full discussion online.

To add a post to this discussion, reply to this email ([email removed])

To start a new discussion for this project, email [email removed]

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe on CodePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at CodePlex.com
Coordinator
Jan 19, 2015 at 6:42 PM
There is a script solution and an integrated solution.

In the script solution, you write a script to run CNTK with train command and followed by running CNTK with the next command. you lunch the monitor program written in matlab from a different command window. you kill the train command when you decide to do so (can be just simple kill or implementing a mechanism to check user inputs which may make program slower). The next command then will run.

In the integrated solution you need to add plumbing code into CNTK in a different thread and call matlab code after each epoch.

The first solution does not require CNTK code change. The second one requires changes. We welcome if you can add these changes to CNTK. If you do so, I suggest you just to add an option for the CNTK to launch external shell command after each epoch from a different thread so that it won't break the main thread.
Jan 19, 2015 at 7:39 PM
Thanks,

I will immediately try your script solution, and may work on the integrated solution, which is more desirable but it will take longer.

Richard

PS: Is there a plan to implement using multiple GPU's?



At 11:42 AM 1/19/2015, you wrote:

From: dongyu
There is a script solution and an integrated solution.

In the script solution, you write a script to run CNTK with train command and followed by running CNTK with the next command. you lunch the monitor program written in matlab from a different command window. you kill the train command when you decide to do so (can be just simple kill or implementing a mechanism to check user inputs which may make program slower). The next command then will run.

In the integrated solution you need to add plumbing code into CNTK in a different thread and call matlab code after each epoch.

The first solution does not require CNTK code change. The second one requires changes. We welcome if you can add these changes to CNTK. If you do so, I suggest you just to add an option for the CNTK to launch external shell command after each epoch from a different thread so that it won't break the main thread.

Read the full discussion online.

To add a post to this discussion, reply to this email ([email removed])

To start a new discussion for this project, email [email removed]

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe on CodePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at CodePlex.com