It is like investigating a crime - use all the evidence available. If you drill through a layer of grass and black topsoil, most likely everything above that is fill.
Natural soils at shallow depth may have roots in place, worm holes, root holes, or thin stratifications etc that tend to get smashed shut or disrupted during placement of fill.
Loess and some alluvial soils may have root holes that tend to be predominately vertical. In fill, the root holes may be closed up or going in random directions.
Poorly compacted fill may have visible clods. Compacted fill may have sheepsfoot roller impressions you can find.
Fill often has a mixed-up quality, especially random mixtures of color, whereas natural deposits have a more uniform appearance.
Someone will laugh, but I find that when you dig into some natural soils, say silty clays, there is a "crunchy" feel and sound that doesn't occur in fill. Think of biting a Kit-Kat bar as opposed to a Milky Way. I attribute it to a weak cementation that develops with age.
Finding a gum wrapper or pop can helps too.
In short, you are right, it is not always easy, and what helps in some soils doesn't help in others. It is an art worth developing, however.