This book is devoted to a unified treatment of Competitive Markov Decision Processes. It examines these processes from the standpoints of modeling and of optimization, providing newcomers to the field with an accessible account of algorithms, theory, and applications, while also supplying specialists with a comprehensive survey of recent developments. The treatment is self-contained, requiring only some knowledge of linear algebra and real analysis. Topics covered include: Mathematical programming: Markov decision processes (the non-competitive case), and stochastic games via mathematical programming.- Existence, structure and applications: Summable stochastic games, average-reward stochastic games and applications and special classes of stochastic games.- Appendices on: matrix games, bimatrix games and nonlinear programming; a theorem of Hardy and Littlewood; Markov chains; and complex varieties and the limit discount equation.