ChemE2912:
I've had the opportunity of employing, designing and operating both cooling and heating coils in industrial applications in the shape of Archimedes Spirals (pan cakes), single helix, and "trombone" type. As EGT01 suggests, I often resorted to Kern's classical "Process Heat Transfer". However, you are in for some frustration if you are going to dedicate yourself to finding film coefficients. I am going to assume that by "the heat transfer coefficient on the oil side" you mean the Overall Heat Transfer Coefficient, "U". If you are trying to calculate individual film coefficents, you are going to waste a lot of time and generate a lot of grey hairs - in my opinion.
First, you are confronted with a situation (to quote Kern, himself) where there is dearth of data in the literature on the transfer of heat to helical coils by free convection. That was in 1950 and, I'm afraid, still the situation from a practical standpoint. The effects of free convection in a tank are varied and difficult to predict due to many factors and the conditions for sediment, fouling, property changes, and proper condensate drainage in the coil are factors that can change the film one way or another. It is all theory when you finally obtain an equation. Poor Don Kern and his buddy McAdams did the best they could, but they never presented any hard, workable equation. All they were able to come up with were "suggestions" and correlations of offerings from others that they neither backed up or recommended for design use. They (like authors or today) never present a relationship or equation that they can firmly recommend or back up. But that's not their fault. The subject is so complex and lacking in research results that it can only be dealt with theoretically by them. The answer to designing a practical heating coil subject to natural convection is found in empirical data: a proven "U". The problem then becomes one of finding as similar an application as yours and using that "U". That's why Kern published his listing of approximate "Us" on page 840. There, you will find that for steam heating, the U can vary from 5 to 100 Btu/hr-ft2-oF. But be aware that the convection effects are not defined. So, in the end, you wind up using a conservative "U".
One important point to bear in mind: plan on having a mechanical engineer working with you to ensure that you can design and build a coil that produces positive condensate drainage to the outside steam trap without covering too much coil inside surface up with liquid. You want to have as much inside area available for steam condensation as you possibly can. Additionally, any convection that you can create or stimulate on the oil side will be of some help. I wish I could help you with more information; it's not a pretty picture to try to illustrate with powerful mathematical expressions.
Art Montemayor
Spring, TX