matter-of-fact adj.

c. 1200, materie, "the subject of a mental act or a course of thought, speech, or expression," from Anglo-French matere, Old French matere "subject, theme, topic; substance, content; character, education" (12c., Modern French matière) and directly from Latin materia "substance from which something is made," also "hard inner wood of a tree." According to de Vaan and Watkins, this is from mater "origin, source, mother" (see mother (n.1)). The sense developed and expanded in Latin in philosophy by influence of Greek hylē (see hylo-) "wood, firewood," in a general sense "material," used by Aristotle for "matter" in the philosophical sense. 

The Latin word also is the source of Spanish, Portuguese, and Italian materia, Dutch, German, and Danish materie, vernacular Spanish madera, Portuguese madeira "wood" (compare Madeira). The Middle English word also sometimes was used specifically as "piece of wood."

From c. 1200 as "a subject of a literary work, content of what is written, main theme;" sense of "narrative, tale, story" is from c. 1300. Meaning "physical substance generally" is from mid-14c.; that of "substance of which some specific object is or may be composed" is attested from late 14c. Meaning "piece of business, affair, activity, situation; subject of debate or controversy, question under discussion" is from late 14c. In law, "something which is to be tried or proved," 1530s.

Matter of course "something expected" attested from 1739 (adjectival phrase matter-of-course "proceeding as a natural consequence" is by 1840). For that matter "as far as that goes, as far as that is concerned" is attested from 1670s. What is the matter "what concerns (someone), what is the cause of the difficulty" is attested from mid-15c., from matter in the sense of "circumstance or condition as affecting persons and things." To make no matter to "be no difference to" also is mid-15c., with matter in the meaning "importance, consequence."

1530s, "action, a thing performed, anything done, a deed," good or evil but in 16c.-17c. commonly "evil deed, crime;" from Latin factum "an event, occurrence, deed, achievement," in Medieval Latin also "state, condition, circumstance" (source also of Old French fait, Spanish hecho, Italian fatto), etymologically "a thing done," noun use of neuter of factus, past participle of facere "to do" (from PIE root *dhe- "to set, put").

An earlier adaptation of the Old French word that also became feat. The older senses are mostly obsolete but somewhat preserved in such phrases as after the fact, originally legal, "after the crime." Also compare matter-of-fact.

The modern, empirical, sense of "thing known to be true, a real state of things, what has really occurred or is actually the case," as distinguished from statement or belief, is from 1630s, from the notion of "something that has actually occurred." The particular concept of the scientific, empirical fact ("a truth known by observation or authentic testimony") emerged in English 1660s, via Hooke, Boyle, etc., in The Royal Society, as part of the creation of the modern vocabulary of knowledge (along with theory, hypothesis, etc.); in early 18c. it was associated with the philosophical writings of Hume. Middle English thus lacked the noun and the idea of it; the closest expression being perhaps thing proved (c.1500).

Hence facts "real state of things;" in fact "in reality" (1707). By 1729, fact was being used of "something presented as a fact but which might be or is false."

By fact is also often meant a true statement, a truth, or truth in general ; but this seems to be a mere inexactness of language .... Fact, as being special, is sometimes opposed to truth, as being universal ; and in such cases there is an implication that facts are minute matters ascertained by research, and often inferior in their importance for the formation of general opinions, or for the general description of phenomena, to other matters which are of familiar experience. [Century Dictionary]

Facts of life is by 1854 as "the stark realities of existence;" by 1913 it had also acquired a more specific sense of "knowledge of human sexual functions." The alliterative pairing of facts and figures for "precise information" is by 1727.

Facts and Figures are the most stubborn Evidences; they neither yield to the most persuasive Eloquence, nor bend to the most imperious Authority. [Abel Boyer, "The Political State of Great Britain," 1727]

Facts are stubborn things is in an 1822 translation of the popular novel "Gil Blas" (earlier English translations have facts speak for themselves).