I am given the task of fault studies of hydro power plant and to provide relay settings for each relay of generator and generator-transformer.
Is there any estblished procedure like calculating relay settings?
Try one or more of the references in faq238-1287.
The Buff Book is a good starting point.
Seriously, this question indicates you are probably not qualified to perform this task. This is a very complex subject. If you are trying to learn and working under an engineer experienced with this, that is fine. Otherwise, get some help.