Also known as: Mechanics obfuscation
Designing rules systems so that players and game masters have to spend significant time in order to learn how to optimize/minmax characters and how to counter them. The resultant System Mastery tends to lead to an unwillingness to learn new games due to the time already invested in learning the existing game.
The simplest way of doing this is by introducing new rules (most easily by way of new powers) that it becomes difficult for anyone to see every use case or catch any flawed mechanics. If powers are explicitly allowed to be combined that will add further complexity.
A flaw and feature of Dungeons & Dragons 3rd edition and its clone, Pathfinder.
See also: Ivory Tower Game Design