There is no workable theory for vortex tube design. It's still an unsolved problem in that sense. The 50K temperature difference mentioned by Hilsch (below) and Herrada et al. (below) may be a limit. The following references might be of some help:
Singh, P. K., Tathgir, R. G., Gangacharyulu, D., & Grewal, G. S., 2004, An Experimental Performance Evaluation of Vortex Tube. IE India Journal 84, pp. 149-153.
Ahlborn, B. & Gordon, J. M., 2000, "The vortex tube as a classic thermodynamic refrigeration cycle." Journal of Appied Physics 88, pp. 3645-3653.
Herrada, M. A., Pérez-Saborid, M., & Barrero, A., 1999, "Thermal separation in near-axis boundary layers with intense swirl." Physics of Fluids vol. 11, pp. 3678-3687.
Fröhlingsdorf, W. & Unger, H., 1999, "Numerical investigation of the compressible flow and the energy separation in the Ranque-Hilsch vortex tube." International Journal of Heat and Mass Transfer vol. 42, pp. 415-422.
Ahlborn, B., Keller, J. U., & Rebhan, E., 1998, "The Heat Pump in a Vortex Tube." Journal of Non-Equilibrium Thermodynamics vol. 23, pp. 159-165.
Ahlborn, B. & Groves, S., 1997, "Secondary flow in a vortex tube." Fluid Dynamics Research vol. 21, pp. 73-86.
Kurosaka, M., Chu, J. Q., & Goodman, J. R., 1982, "Ranque-Hilsch Effect Revisited: Temperature Separation Traced to Orderly Spinning Waves or 'Vortex Whistle'." AIAA Paper No. 82-0952, 13 pp.
Chanaud, R. C., 1963, "Experiments Concerning the Vortex Whistle." The Journal of the Acoustical Society of America vol. 35, pp. 953-960.
Vonegut, B., 1954, "A Vortex Whistle." The Journal of the Acoustical Society of America vol. 26, pp. 18-20.
Hilsch, R., 1947, "The Use of the Expansion of Gases in a Centrifugal Field as Cooling Process." The Review of Scientific Instruments vol. 18, pp. 108-113.