Please provide a bit more information if possible. . . .
You mention you're reading L1/L2 voltages of 120/122 volts unloaded (I assume unloaded - verify).
When loaded, you are getting voltages of ???/124. I suspect the ??? side is seeing a voltage dip, please verify.
If ??? is seeing a voltage dip, it's possible you have a bad neutral-ground bond.
If the ??? side is instead rising in voltage, something very strange is going on -- it would sound almost like a leading power factor situation, which would be unusual in a residential installation.
Either way, please be more specific regarding voltage and current readings on each line, and provide some additional detail regarding the appliances you're turning on. Also please verify this is a 120/240v, 3ph. 4w+gnd system.