Originally titled "Brown-Skinned Girl," Van renamed the song, but later said "That was just a mistake…it just slipped my mind [that] I changed the title." The Independent's Paul Bignell suggests the song is about an interracial relationship, and speculates that the title was changed to make it more palatable to radio ...

The song is a folk rock ballad that variously takes the form of a first-person lament as well as a third-person sketch of a boxer. The lyrics are largely autobiographical and partially inspired by the Bible and were written during a time when Simon felt he was being unfairly criticized.