As dg says, let the blade do the cutting and don't try to get back on the line by pushing sideways on the wood. It took me a while to learn that I have to turn the wood to get back onto the line, I can't push the wood sideways. Now, every so often, I stop moving the wood into the blade and almost let go of the wood. If I've been pushing sideways on the wood, the tension of the blade will move the wood sideways. Then I know I have been cutting at an angle.
Keep practicing until you can follow a circle pattern and the cutout will come out of the wood upwards or downwards.
Jim