Pastore was born to an Italian-American family in the Bronx, New York City, and grew up in New Rochelle, New York. Following his graduation from high school, he enlisted as a sailor in the United States Navy and then attended Pace University for three years, before eventually going into the acting industry after befriending Matt Dillon and Kevin Dillon. On June 3, 2015, during an appearance on Good Day New York, Pastore said he was in the club business for close to 30 years, and got into acting in his forties.