We consider a pair of identical theta neurons in the active regime, each coupled to the other via a delayed Dirac delta function. The network can support periodic solutions and we concentrate on solutions for which the neurons are half a period out of phase with one another, and also solutions for which the neurons are perfectly synchronous. The dynamics are analytically solvable, so we can derive explicit expressions for the existence and stability of both types of solutions. We find two branches of solutions, connected by symmetry-broken solutions which arise when the period of a solution as a function of delay is at a maximum or a minimum.