Put another way, the +5V 'line' has finite impedance, as does the ground return to the power supply. The 4.7uF (other values would do) serves as a local supply.
A local supply is needed because logic chips draw large currents during transitions between states, and are sensitive to the supply voltage. Without these 'bypass caps' on each logic chip, the transient current spikes from one chip would pull down the line voltage (and/or pull up the ground voltage) and interfere with the operation of other chips.
If you build a circuit board without the bypass caps, it will probably mostly work, but it will exhibit strange and logically inexplicable behaviors that you will need a very fast oscilloscope to even capture.
If you look at precision analog circuits or op-amp circuits, you may find 4.7uF caps at the downstream end of low value resistors in the supply, effecting RC low-pass filters. Same idea, better filtering.
The 4.7uF is not entirely arbitrary. It's a large value that can be had in a tiny package. You can compute its cutoff frequency by estimating the resistance of the supply path and counting that as a resistor, again in an RC circuit.
For sensitive circuits where high frequencies are present, you may find several bypass caps of widely different values, parallelled right across the chip, to deal with transients of different speed. At GHz speeds, 4.7uF caps don't really behave like caps, so you need a few pF to bypass the really fast transients around them.
To not quite answer your last question, "what will work" depends on what you are trying to do, and in what environment you are trying to do it. But in general, if a circuit doesn't seem to be working like you think it should, and you didn't bother to bypass it, you should start debugging by doing it.
Mike Halloran
Pembroke Pines, FL, USA