The recommended curve number for impervious surfaces is 98, unless you're in an area of high temperatures/low humidity, where you can go as low as 95. The curve number for gravel depends on your soil type. I would interpolate between the appropriate CN for gravel and paved road, depending on the extent of the breaking in the pavement.
To be conservative, I'd probably use 98 (assume no breaks). The curve numbers for gravel for types A, B, C & D soil, respectively, are: 76, 85, 89 & 91. I've never come across a type A soil, and most clay soils are type C. Building rubble would definitely be type D, so perhaps a number in the region of 92-96 would be appropriate.
Depending on the area of broken paving, you may find that there is little difference in the amount of runoff between using 98 and interpolating a CN.
