An evolutionarily stable strategy (ESS) is a strategy such that, if all the members of a population adopt it, then no mutant strategy could invade the population under the influence of natural selection. Suppose that there are two types of strategies in the population, A and B. Let E(B, A) denote the payoff strategy B receives in interacting with strategy A. The strategy A is evolutionarily stable if either
1. E(A, A) > E(B, A), or
2. E(A, A) = E(B, A) and E(A, B) > E(B, B)
is true for all B. There is also an alternative definition of ESS, which places a different emphasis on the role of the Nash equilibrium concept in the ESS concept. The strategy A is evolutionarily stable if both
1. E(A, A) ≥ E(B, A), and
2. E(A, B) > E(B, B)
for all B. The concept of ESS considers those situations when a single mutant invades an infinite population of homogeneous strategies. It is not concerned with the structure of the population, the selection scheme, and other parameters of evolutionary dynamics. If an ESS exists for an evolutionary game, the evolutionary process is, in theory, likely to converge to the state where ESSs are common. However, if no ESS exists, it is difficult to forecast the result of the evolution. This definition is so strict that no known strategy is evolutionarily stable in infinite length or indefinite length IPD.
Nowak and Sigmund (1992) considered the size of the cluster that is needed for an invader to invade a finite population of a particular strategy. The minimal cluster size for one strategy to invade another can be treated as an invasion barrier. If the invasion barrier for a strategy is low, it means that a small cluster of invaders can successfully invade and it is difficult to maintain a homogeneous population. On the contrary, if the invasion barrier for a strategy is high, successful invasion requires a large cluster of invaders. Therefore, different strategies can be compared by means of their invasion barriers. Strategies with a higher invasion barrier are evolutionarily stronger than those with a lower invasion barrier. By means of replacing the quantity of each strategy in the population with the ratio of the quantity to the size of population, an invasion barrier can be used in evolutionary IPDs with both finite and infinite populations.
Consider a population consisting of two types of strategies, A and B. pAand pBare the frequencies of A and B in the population respectively. The IPD between two strategies is denoted by the payoff matrix:
| A | B | |
| A | a | b |
| B | c | d |
Then the fitness of two strategies, EA and EB , can be expressed as:
EA = apA+ bpB
EB = cpA+ dpB
The condition for A to invade B is EA > EB, or (a – c)pA > (d – b)pB. On the other hand, the condition for B to invade A is (a – c)pA < (d – b)pB. Therefore, (a – c)pA= (d – b)pBis the transition point for the evolution. Without loss of generality, suppose that a > c and let λ = pA/ pB. The invasion barrier can be expressed as
λ0 = (d-b) / (a-c)
In the case of λ = λ0 , two strategies will coexist and maintain their current frequencies in the population.λ > λ0 and λ < λ0 indicate contrary directions of the evolution. In one case, A tends to become dominant and B tends to die out; in the other case, Adies out and B becomes dominant.