Loss function giving nan in pytorch

In pytorch, I have a loss function of 1/x plus a few other terms. The last layer of my neural net is a sigmoid, so the values will be between 0 and 1.

Some value fed to 1/x must get really small at some point because my loss has become this:

loss: 11.047459  [729600/235474375]
loss: 9.348356  [731200/235474375]
loss: 7.184393  [732800/235474375]
loss: 8.699876  [734400/235474375]
loss: 7.178806  [736000/235474375]
loss: 8.090066  [737600/235474375]
loss: 12.415799  [739200/235474375]
loss: 10.422441  [740800/235474375]
loss: 8.335846  [742400/235474375]
loss:     nan  [744000/235474375]
loss:     nan  [745600/235474375]
loss:     nan  [747200/235474375]
loss:     nan  [748800/235474375]
loss:     nan  [750400/235474375]

I'm wondering if there's any way to "rewind" if nan is hit or define the loss function so that it's never hit? Thanks!

Clip your loss to fall within a reasonable range to prevent gradient explosion (i.e. continually climbing out of the local neighborhood as in BrockBrown's answer.)

I'd recommend something like this:

 epsilon = 1e-01
 loss = 1/(x + epsilon) # bounded maximum value of 10

or:

 eta = 5 # maximum loss value
 loss = torch.clamp(loss,max = eta)

As a third option, you can simply check for nan values and not backpropagate in these cases. This won't help with stability, it will just remove outlier loss values, but you still may not get good convergence properties depending on the nature of when these values occur.

 if (loss.isnan.sum() == 0): # no nan values 
      loss.backward()
      optimizer.step()

I'd also note that if your loss is 1/x, and x is bounded between 0 and 1, you can't ever achieve a loss less than 1. Generally your loss should be 0 for a perfect output. Combining this fact with the previous ideas:

loss = 1/(epsilon + x) - (1/(1+epsilon))

When x = 1, loss approaches 0 , and when x = 0, loss is approximately 1/epsilon. You can adjust epsilon to attenuate your loss within a desired range for stability.

Your loss is jumping all over the place instead of steadily decreasing. Have you tried decreasing your learning rate? It looks like it's jumping across the minimum, bouncing back and forth. This can happen if the learning rate is too high.

To answer your question about rewinding, ideally you shouldn't have to rewind, the loss should be steadily decreasing. Also you may want to look into learning rate schedulers.