Markov Chains
A Markov chain is a stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event. Markov chains are used in various fields including queueing theory, statistics, and machine learning.
P(Xn+1 = x | Xn = xn, ..., X1 = x1) = P(Xn+1 = x | Xn = xn)
Where:
- Transition Matrix: Square matrix where each element Pij represents the probability of moving from state i to state j
- Steady-State Probabilities: Long-term probabilities of being in each state
Results
n-Step Transition Probabilities
Steady-State Probabilities
Practical Examples
Example 1: Weather Model
States: Sunny, Cloudy, Rainy
Transition Matrix:
| Sunny | Cloudy | Rainy | |
|---|---|---|---|
| Sunny | 0.6 | 0.3 | 0.1 |
| Cloudy | 0.4 | 0.4 | 0.2 |
| Rainy | 0.2 | 0.3 | 0.5 |
Steady-State: Sunny (42.9%), Cloudy (33.3%), Rainy (23.8%)
Example 2: Machine States
States: Working, Degraded, Failed
Transition Matrix:
| Working | Degraded | Failed | |
|---|---|---|---|
| Working | 0.9 | 0.1 | 0.0 |
| Degraded | 0.2 | 0.7 | 0.1 |
| Failed | 0.0 | 0.0 | 1.0 |
Failed is an absorbing state. Mean time to failure: 23.3 steps
Markov Chain Applications
| Application Area | Use Case |
|---|---|
| Queueing Theory | Modeling system states in service systems |
| Reliability Engineering | Predicting equipment failure states |
| Finance | Credit rating transitions |
| Health Care | Disease progression modeling |
| Manufacturing | Production system state analysis |