1670s, probably from lug (n.) in some obscure sense; perhaps so called from the "ear" of sail formed by the oblique hang of the yard from the mast.
Entries linking to lug-sail
lug n.
a broad-meaning word used of things that move slowly or with difficulty, "of obscure etymology" [OED]. From 1620s as "handle of a pitcher," this sense probably from Scottish lugge "earflap of a cap; ear" (late 15c. and according to OED still the common word for "ear" in 19c. Scotland), which is probably from Scandinavian (compare Swedish lugg "forelock," Norwegian lugg "tuft of hair") and influenced by the verb. The connecting notion is "something that can be gripped and pulled."
Applied 19c. to mechanical objects that can be grabbed or gripped. Meaning "stupid fellow" is from 1924; that of "lout, sponger" is 1931, American English. Compare lug-nut (1869), nut closed at one end as a cap.
lugger n.
"small two- or three-masted fishing or coasting boat" (also favored by smugglers), always with lug-sails, 1757, from lug-sail. Or else [OED] from Dutch logger, which is perhaps from Middle Dutch loggen "to fish with a dragnet."