Review of Probability Short History Review of Basic Probability Definitions Some Common Probability Distributions Properties of a Probability Distribution Properties of the Expected Value Expected Value of a Random Variable with Common Distributions Generating Functions Moment Generating Functions Exercises Discrete-Time, Finite-State Markov Chains Introduction Notation Transition Matrices Directed Graphs: Examples of Markov Chains Random Walk with Reflecting Boundaries Gambler'e(tm)s Ruin Ehrenfest Model Central Problem of Markov Chains Condition to Ensure a Unique Equilibrium State Finding the Equilibrium State Transient and Recurrent States Indicator Functions Perron-Frobenius Theorem Absorbing Markov Chains Mean First Passage Time Mean Recurrence Time and the Equilibrium State Fundamental Matrix for Regular Markov Chains Dividing a Markov Chain into Equivalence Classes Periodic Markov Chains Reducible Markov Chains Summary Exercises Discrete-Time, Infinite-State Markov Chains Renewal Processes Delayed Renewal Processes Equilibrium State for Countable Markov Chains Physical Interpretation of the Equilibrium State Null Recurrent versus Positive Recurrent States Difference Equations Branching Processes Random Walk in Exercises Exponential Distribution and Poisson Process Continuous Random Variables Cumulative Distribution Function (Continuous Case) Exponential Distribution o(h) Functions Exponential Distribution as a Model for Arrivals Memoryless Random Variables Poisson Process Poisson Processes with Occurrences of Two Types Exercises Continuous-Time Markov Chains Introduction Generators of Continuous Markov Chains: The Kolmogorov Forward and Backward Equations Connection Between the Steady State of a Continuous Markov Chain and the Steady State of the Embedded Matrix Explosions Birth and Birth-Death Processes Birth and Death Processes Queuing Models Detailed Balance Equations Exercises Reversible Markov Chains Random Walks on Weighted Graphs Discrete-Time Birth-Death Process as a Reversible Markov Chain Continuous-Time Reversible Markov Chains Exercises Bibliography.