I think it's because your measurements are all based on a falsely-introduced orienting of the plate to some external edge datums. The print only shows a reference to datum A, so whichever hole you call the zero, the next hole to be measured is just a straight-line connection, not both X and Y.
In the second scenario where hole #3 is the origin, there is no way all of the other three holes would have deviation in both the X and Y directions, because you can only build the X-Y system as you progress around the 4 holes (actually just the first two to be measured would be enough to square up the X-Y system). There is no perpendicular and parallel until after the second hole is measured for location.
For instance, if I start with #3 and then measure down to #4 as my next hole, the Actual Location would be 0 in the X direction and -21.0238 in the Y direction. (This comes from the -1 and -21, but using the Pythagorean Theorem to make a straight line between those two centers.) Thus, there is no deviation in two directions because it's just a straight shot center-to-center.
I haven't figured out all the numbers to really see if it all works out, but that is where I believe the issue is.
John-Paul Belanger
Certified Sr. GD&T Professional
Geometric Learning Systems