Question Consistency Issues

First off I have a g420 with HF-500 spindle. Running Windows 10 and using Fusion 360 for cam. I've had this machine for approx 3 months. Most of what I cut is 3mm acrylic sheets, delrin, and ultem.

The issue that I am having is lets say I'm cutting a simple rectangle 70mm x 38mm. Using the contour cut with no other adjustments my box will come out to 70.66 x 38. 38. From that point I'll spend some time fine tuning to get my box into a tolerance of +/- .05mm and then things fall apart. I have now successfully cut a box that I'm happy with and for the most part I can continue to cut those box successfully but if I shut down my machine and then reopen that program the box suddenly now starts cutting 69.45 x 37.33. Basically I have to start from scratch on getting my tolerances back to an acceptable range. Every day the first few hours of the day is devoted to readjusting my cam just so I can get started on my projects. Also I've noticed that when setting up for a prototype (single object) the settings don't translate over to more of a small production run (6 objects cut from the same material sheet). At this point it is pointless for me to do a prototype since I'll be starting from scratch once I try to cut several and when I re-open that file at a later date. The way I have been adjusting my file is thru the "stock to leave" option in Fusion. Sometimes I'll have to change the model because Fusion doesn't seem to let me adjust x and y axis separately. This is going to be a nightmare if I'm doing a project a little more complicated than a simple box shape.

I've asked around for advice and everyone seems to think I have backlash problem. I have run a basic test using the pen and drawing circles, adjusting the backlash in UCCNC until the circles look to be ok. By the way, it didn't make any difference as far consistency but did seem to fix how far off the tolerance were. I'm looking into getting a full set of anti-backlash nuts and a decent dial probe so I can try to get that under control but will it fix the issue? I know I'll have to get them eventually but is this where I should focus my efforts?

Any suggestions would be greatly appreciated. Also worth a mention, I'm very new to operating a cnc machine and still figuring things out. Lots of trial and error, youtube videos, stalking facebook communities and forums for answers.