If I understand your question correctly you want to know how to calculate the amount and location of balancing mass needed on the flywheel to offset imbalance in given rotating/reciprocating assembly.
The textbook we used in my uni was Kinematics and Dynamics of Machines, by Martin.
I don't believe the theory has changed appreciably in the past 20 years.
Unfortunately it is an old book and hard to find to buy. Luckily my local university library had a copy when I needed it. With a bit of hunting a second hand copy may be found.
It is a simple book, but explains the fundamentals of engine balance (funny that!) for a wide variety of configurations pretty well for a beginner.
There are many books that cover balancing engines, but the standard reference is "The Internal Combustion Engine In Theory and Practice" by CF Taylor. Any of the editions will do.
Most of the other balancing texts I've seen have been abridged versions of Taylor.
Norton: Machine Design - Covers the basics on how much inertia you need to hold rpm withina given range,
Norton: design of machinery (sequel to the first book) Gives much more detailed info on where to put it.
The first book was the capstone for my mechanical engineering bachelors and the second is for your masters.
You don't mention the reason for your request but another source of information is to contact the manufacturers of engine balance systems and ask them if they can recommend some documentation.
Any textbook in in library that has a title like "Dynamics of Machinery" is likely to cover engine balance. The older the book, the better. The technology goes back to the early 1900s, maybe earlier if you consider steam engines and locomotives (note the big counterweights on steam locomotive drive wheels).