1797, contraction of and per se and, meaning "(the character) '&' by itself is 'and' " (a hybrid phrase, partly in Latin, partly in English). An earlier form of it was colloquial ampassy (1706). The distinction is to avoid confusion with & in such formations as &c., a once-common way of writing etc. (the et in et cetera is Latin for "and"). The letters a, I, and o also formerly (15c.-16c.) sometimes were written a per se, etc., especially when standing alone as words.
The symbol is based on the Latin word et "and," and comes from an old Roman system of shorthand signs (ligatures) attested in Pompeiian graffiti, and not (as sometimes stated) from the Tironian Notes, a different system of shorthand, probably invented by Cicero's companion Marcus Tullius Tiro. It used a different symbol, something like a reversed capital gamma, to indicate et. This Tironian symbol was maintained by some medieval scribes, including Anglo-Saxon chroniclers, in whose works a symbol resembling a numeral 7 indicates the word and.
In old schoolbooks the ampersand was printed at the end of the alphabet and thus by 1880s the word ampersand had acquired a slang sense of "posterior, rear end, hindquarters."