The formulas you've pasted across from API 12K are indeed heat flux expressions, so the values you quoted earlier should be Btu/hr/ft2 and not Btu/hr/ft2/degF.
Okay.
The first of these expressions, a modification of the Stefan Boltzmann eqn, is identical to equation 5-155 in Perry Chem Eng Handbook 7th ed, with the appropriate correction factor (1+ε1)/2 included to account for the gray firetube inside wall. While the second is the familiar Dittus-Boelter eqn. Presumably you've used correct values for each of these.
Calculations for surface temp involve iterations, you should manually check / verify some of these values, which you can do using first principles also. The overall flux values indicate it would be better to use a slightly larger surface area to keep overall flux value below 12000 btu/hr/ft2. What does API 12K or Campbell say the max skin or film temp should be for this application ?
Its been many years since I worked on these heaters, and the largest was 84inch (I vaguely recall) for NIOC, which is much smaller than this unit of 180inch dia.
Both these expressions require a value for Tcs, which can only be derived by several iterations for each section - have you verified at least a few of these values ?
Found the 2009 edn of API 12K on the internet. It states the additional upper cross sectional heat flux to be 15000 btu/hr/in2 - have you verified this for the hottest first pass at least ?
Pls note that water to be used for these heaters should be demin water, with corrosion inhibitor added for assurance.