Joseph B. Brown was born in Washington, D.C.
, and later moved to south-central Los Angeles. He earned his law degree from UCLA, supporting himself by working as a substitute teacher. In 1973, Brown moved to Memphis, Tennessee, where he began his legal career with legal services and later joined the Equal Employment Opportunity Commission (EEOC).
He became the city's first African American prosecutor and subsequently served as director of the Public Defender's office.