Reward Programming: Optimizing RL Efficiency and Safety

How do we design rewards that guide reinforcement-learning agents toward the behavior we actually intend? This course examines reward design as a programming and specification problem in reinforcement learning. Classical reinforcement learning usually assumes that the objective is given as a scalar reward function. In practice, however, many real tasks involve goals, constraints, temporal order, safety requirements, recurrence, partial observability, hierarchy, other agents, and long-run behavioral expectations that are difficult to express through one-step rewards alone. Poorly designed rewards can lead to reward hacking, specification gaming, and policies that optimize the written objective while missing the designer’s intent. The course introduces reward programming as a structured approach to specifying, shaping, inferring, monitoring, and auditing what an agent should learn. Learners study temporal logic, automata, product MDPs, and reward machines as tools for representing objectives that depend on history, progress, safety, and long-run behavior. They also study reward shaping, inverse reinforcement learning, preference-based feedback, and automata-learning approaches for inferring or improving reward mechanisms. The course then examines richer modeling abstractions for reward programming, including partially observable Markov decision processes, memory and beliefs, hierarchical and recursive tasks, multi- agent settings, and continuous-time systems. The final module studies safety, shielding, constrained RL, auditing, stress testing, and a reward-engineering workflow for connecting designer intent to specifications, reward mechanisms, learning, safety layers, and revision. By the end of the course, learners will be able to design and infer structured reward mechanisms, evaluate whether they align with intended behavior, and reason about their implications for safety, transparency, and reliability. This course can be taken for academic credit as part of