Horny Pony was released as the b-side of the single Gett Off, the first single from Prince's thirteenth album Diamonds And Pearls and the first single to be credited to Prince and the New Power Generation. It was also included as the b-side of the following single, Cream. In 2001, it was included, with a computerized spoken intro (stating that the song is about dancing, and does not endorse extra-marital sex), as part of NPG Music Club Edition # 7.
Initial tracking took place on 11 September, 1990 at Paisley Park Studios, Chanhassen, MN, USA (during the same session as Fancy Dancer, Jughead, The Flow and a reworked version of Something Funky (This House Comes)) within hours of returning to the USA from the Nude Tour in Japan. The track was included as the ninth track on a December, 1990, configuration of the Diamonds And Pearls album, and was included as the seventh track on a 8 May, 1991 configuration. Also in May, 1991, it was included as the first track on an acetate of a Gett Off EP, which was aborted soon after compilation. Horny Pony was replaced on the Diamonds And Pearls album by Gett Off in June, 1991, but the track is still mentioned in the rap portion of Push (along with other songs on the album), and is printed on the corresponding tracklist on the packaging's back cover (quoted from Push).