"...MV system with isolated neutral, the relay has 50N/51N protection set. It senses three phase currents and vectoral sums them, and it is set the %10 of the nominal load to trip. Would this be enough for ground fault protection ?"
You mean you measure the residual current with "Holmgreen connection?" (3 CTs in parallel). Forget it. You can trip by zero sequence current only by perfectly unbalanced load! Remember that a protection CT has an error of 1% at it's rated current... 1% in phaseA, -0.5% in phase B, 0.2% in phase C... TRIP!
If possible (cable), measure the current with a thoroidal transformer, which directly gives a secondary current proportional to the zero sequence magnetic flux!