For a 4 story building? An hour seems like too long of a solution time. When investigating this sort of thing I usually do the following:
1) Try to determine which load combinations are taking so long to solve. If your analysis program has a solution status window that tells you what it's doing then this should be easy.
2) Try running the model without P-Delta to see how long it solves.
3) Use the alternate approach with DAM where you don't iterate the flexural stiffness adjustments. See if this makes a major difference. Or, just turn off DA method temporarily (while using P-Delta) and see how long it takes it to solve.
4) Look at other potential issues for iteration or long solution times. Tension only or compression only members. Link elements and such.
What I'm trying to do with the above is determine the source of the long solution times. So, that I can take a closer look at the cause and determine if it is legit or if there is a modeling problem.