No, I don't think you can discount fastener loosening, because measuring residual torque with torque wrench (re-torque measurements) can be a very misleading test. It is very operator dependant and can lead to higher that actual results because the peak measured is due to overcoming static...
If you are getting slip immediately at start-up it is likely to be torque-rate dependant (effect of pully inertia causes slipping at lower loads than iniitally predicted). If slippage occationally happens over time it is likely a case of loosening. Is the pully located by a precision fit over...