Looking at your post again the Class CC fuse that you are using is not designed use in other then small motors. If you motor is a 20 HP 575-volt motor the FLA would be 22 amperes. If it is a 25 HP 575-volt motor the FLA would be 27 amperes. I would suggest a 30 to 35 ampere or a 35 to 45 ampere...
If you use 40 ampere RK-5 fuses does not necessarily mean you need to increase you wire size. Code requires a minimum of 125% ampacity for wire size, but you may put fuses sized up to 175% FLA. This is for short current and locked rotor protection. You still need to have overload protection...
In the case of 1 phase going to "ground" in an ungrounded system current would not flow, you would not trip a breaker or blow a fuse.
If 1 phase gets unintentionally grounded what you would have then in effect is a grounded system. The only time that you would have real current flow is when...
