If you want to calculate the gear deflections, the
FEA or BEM are the correct methods to use. The AGMA,
ISO, DIN only provide the methods to calculate the
stresses for the involute gears. For the non-involute
gears, one will use these numerical methods to calculate
the stresses too.
In my opinion, the most difficult part of the gear FEA is
to obtain the geometry equations for various parts of
the gear surfaces, which is actually not related to
the FEA. Once you are confident in the geometry equations
of the gears, the gear FEA is not much different from
the general contact program FEA.
But be aware of the fact that someone may question
your results. For example, I've obtained very different
FEA results of the gears from the other two FEA analysts
involved in the same projects.