This is an interesting question. The best answer, I think, is that it depends on the relay.
Old electro-mechanical instantaneous units respond to asymmetrical current, so to be safe, you need to coordinate these based on the peak asymmetrical rms current that can occur. The ratio of symmetrical to asymmetrical current will be function of the X/R ratio of your system.
For new digital relays, it depends on the relay. For GE/Multilin relays (SR and UR), my understanding, based on conversations with their tech support engineers, is that these relays use software filtering. The dc component (and harmonics) are filtered out and the Overcurrent units respond only to fundamental symmetrical current, rms. I believe the harmonic filtering is user-selectable in the UR relays, but dc offset is always filtered.
Schweitzer (SEL) relays, in the past, have applied hardware harmonic filtering, but NOT dc offset filtering, so these relays do respond to the peak asymmetrical (rms) current. I have seen some promotional literature from SEL regarding new software filtering algorithm, but haven't read it yet.
If in doubt, the best assumption is to assume you need to coordinate at the peak asymmetrical current. I would probably use rms, but some low voltage trip units do respond to peak current. Best bet is to check with the relay manufacturer.
You should also allow a little safety factor for CT error and saturation. In the old days, we assumed 1.6 x symmetrical current (rms) plus a 10% safety factor. DC offset filtering can cut this down quite a bit.