Ridership on public transit routes has been declining over the past several years. One key reason for the decline is poor service reliability. The main approach to deal with unreliable service on high frequency transit routes is to control headways. The methodological approach used in this research for headway control is Reinforcement Learning (RL). RL agents learn by interaction to determine the best actions for every state in a state space. This research applies two Q-learning agents to control streetcar headways, in both directions, of a high frequency streetcar route. The results are tested using the microscopic traffic simulation software Paramics. Results were a reduction in very large and very small headway deviations and a reduction in the average headway deviation. It is recommended that states with few visits be merged for faster convergence and the G/C ratio be included in the reward for more efficient use of green time.