Actually, thinking more in depth about it, if the soil force is higher than the seismic force for the levels below the grade, you will still need to ensure:
1) the floor is a diaphragm and the loads can be transferred to the soil on the "far" side of the building, and be resisted by the soil...
I would say that your base is the bottom floor of your structure. Normally the ground and the bottom/first floor are the same, so this doesn't come up.
When doing your analysis, you'll need to analyze the building as a 5 story building for seismic, and only a 2 story building for wind...