My suggestion, or, "The way I do it".......
I usually DRILL the inside circle to get a perfectly round hole. I use a forester drill bit in the drill press to get a nice, smooth hole with as little tear-out as possible. If I don't have one of the EXACT size, I drill using the next size smaller and file or sand the inside to the final dimension.
As for the outer circle, I use the scroll saw to cut just a bit OUTSIDE the line and then sand down to the final dimension.
Of course, it depends on what you are cutting the circle for and if it is a complete ring or maybe has something protruding to the inside or outside of it.
As for using JUST a scroll saw and, depending on the diameter of the circle, I cut just off of the line so that it can be sanded or otherwise worked down to final dimension and/or appear to be perfectly round. Just go slow, take your time, breathe, realize that NOTHING IS PERFICT, it just LOOKS that way!
At any rate, accept the fact that is going to take some sanding and hand work and it may not be exact but, it will LOOK GOOD to everyone else.
Rog