I am not much of a math person, but if we look at the how the brain is wired, we see that the dendrites (the inputs) of a neuron are hundreds of micrometers in length, and the axons (the outputs) are millimeters, and very rarely can stretch to tens of centimeters. So they can sample only a tiny amount of internal state, and affect a much larger, but usually still small output.

In math terms, this means a layer of a network can be represented with a block matrix in the whole 'layer' matrix, which I think means its sparse as you said.

As I said, my math knowledge is rusty, but I remember that a lot of matrix optimization techniques center around decomposing large matrices into these smaller blocks, which are then evaluated, and the output is combined in a final pass. Which leads to a huge reduction on parameter numbers and the time it takes to evaluate the result