The following advice is targeted at beginners to neural networks, and is based on my experience giving advice to neural net newcomers in industry and at Stanford. Neural nets are fundamentally harder to debug than most programs, because most neural net bugs don't result in type errors or runtime errors. They just cause poor convergence. Especially when you're new, this can be very frustrating! But an experienced neural net trainer will be able to systematically overcome the difficulty in spite of the ubiquitous and seemingly ambiguous error message: Performance Error: your neural net did not train well. To the uninitiated, the message is daunting. But to the experienced, this is a great error. It means the boilerplate coding is out of the way, and it's time to dig in!
By far the most common first question I get from students is, "Why am I
getting NaNs." Occasionally, this has a complicated answer. But most often,
the NaNs come in the first 100 iterations, and the answer is simple: your
learning rate is too high. When the learning rate is very high, you will get
NaNs in the first 100 iterations of training. Try reducing the learning rate
by a factor of 3 until you no longer get NaNs in the first 100 iterations. As
soon as this works, you'll have a pretty good learning rate to get started
with. In my experience, the best heavily validated learning rates are 1-10x
below the range where you get NaNs.
If you are getting NaNs beyond the first 100 iterations, there are 2 further common causes. 1) If you are using RNNs, make sure that you are using "gradient clipping", which caps the global L2 norm of the gradients. RNNs tend to produce gradients early in training where 10% or fewer of the batches have learning spikes, where the gradient magnitude is very high. Without clipping, these spikes can cause NaNs. 2) If you have written any custom layers yourself, there is a good chance your own custom layer is causing the problems in a division by zero scenario. Another notoriously NaN producing layer is the softmax layer. The softmax computation involves an exp(x) term in both the numerator and denominator, which can divide Inf by Inf and produce NaNs. Make sure you are using a stabilized softmax implementation.
Once you stop getting NaNs, you are often rewarded with a neural net that runs
smoothly for many thousand iterations, but never reduces the training loss
after the initial fidgeting of the first few hundred iterations. When you're
first constructing your code base, waiting for more than 2000 iterations is
rarely the answer. This is not because all networks can start learning in
under 2000 iterations. Rather, the chance you've introduced a bug when coding
up a network from scratch is so high that you'll want to go into a special
early debugging mode before waiting on high iteration counts. The name of the
game here is to reduce the scope of the problem over and over again until you
have a network that trains in less than 2000 iterations. Fortunately, there
are always 2 good dimensions to reduce complexity.
1) Reduce the size of the training set to 10 instances. Working neural nets can usually overfit to 10 instances within just a few hundred iterations. Many coding bugs will prevent this from happening. If you're network is not able to overfit to 10 instances of the training set, make sure your data and labels are hooked up correctly. Try reducing the batch size to 1 to check for batch computation errors. Add print statements throughout the code to make sure things look like you expect. Usually, you'll be able to find these bugs through sheer brute force. Once you can train on 10 instances, try training on 100. If this works okay, but not great, you're ready for the next step.
2) Solve the simplest version of the problem that you're interested in. If you're translating sentences, try to build a language model for the target language first. Once that works, try to predict the first word of the translation given only the first 3 words of the source. If you're trying to detect objects in images, try classifying the number of objects in each image before training a regression network. There is a trade-off between getting a good sub-problem you're sure the network can solve, and spending the least amount of time plumbing the code to hook up the appropriate data. Creativity will help here.
The trick to scaling up a neural net for a new idea is to slowly relax the simplifications made in the above two steps. This is a form of coordinate ascent, and it works great. First, you show that the neural net can at least memorize a few examples. Then you show that it's able to really generalize to the validation set on a dumbed down version of the problem. You slowly up the difficulty while making steady progress. It's not as fun as hotshotting it the first time Karpathy style, but at least it works. At some point, you'll find the problem is difficult enough that it can no longer be learned in 2000 iterations. That's great! But it should rarely take more than 10 times the iterations of the previous complexity level of the problem. If you're finding that to be the case, try to search for an intermediate level of complexity.
Now that your networks is learning things, you're probably in pretty good shape. But you may find that your network is just not capable of solving the most difficult versions of your problem. Hyperparameter tuning will be key here. Some people who just download a CNN package and ran it on their dataset will tell you hyperparameter tuning didn't make a difference. Realize that they're solving an existing problem with an existing architecture. If you're solving a new problem that demands a new architecture, hyperparameter tuning to get within the ballpark of a good setting is a must. You're best bet is to read a hyperparameter tutorial for your specific problem, but I'll list a few basic ideas here for completeness.
Debugging neural nets can be more laborious than traditional programs because almost all errors get projected onto the single dimension of overall network performance. Nonetheless, binary search is still your friend. By alternately 1) changing the difficulty of your problem, and 2) using a small number of training examples, you can quickly work through the initial bugs. Hyperparameter tuning and long periods of diligent waiting will get you the rest of the way.