Joseph Ellison was born in Manhattan in 1948, where he first fell in love with music, especially the diverse sounds of Hank Williams Sr, Thelonious Monk, and The Maguire Sisters. His Texas born father instilled a strong sense of morality in him while moving him around the nation.