Knowledge is physically or materially based because our senses including the mind are materially based. Reality is material based.
I deal with the material reality, or I should only deal with the material reality.
The abstract, conceptual, non-material “realities” by definition have no effects upon the material reality. Therefore we should be concerned with them.
We should only concern ourselves with the material which can perform abstraction, conceptualisation, and think about non-material realities, and not concern ourselves with abstractions, concepts and non-real “things”.