There are really two kinds of rules: Terminal rules and intermediary rules. Terminal rules adjudicate something. Intermediary rules modify or impact another rule. A rule for controlling a runaway carriage-and-four is a terminal rule. A rule that increases your chance to hit for standing on higher ground is an intermediary rule. Terminal rules interact with other rules (the rules for being run over by a carriage, for instance) but they don't change how those rules work all the time. Optional terminal rules tend to replace the original. That's not so bad, but the reason the original is the original, and the optional rule is optional is that the designers thought original rule was better. Intermediary optional rules are worse. They modify the system. But they're also not usually play-tested.
If you have N binary (using or not using) optional rules, there are 2^N different combinations of those rules that can be applied -- 2^N versions of the game. If you have X hours of playtesting available, you have X/2^N hours per combination of optional rules for testing them out.
But that's silly. First of all, nobody would actually test all the possible combinations. The chances that two will interact very badly are pretty high, but so are the chances that a game designer or playtester reading the rules over would notice it without actual playtesting.
The designers should try several one-shot playtests with some of the optional rules, focusing on the ones that they feel will be most commonly used, in combinations they think will be commonly used together. They should also tap one or two groups to run whole adventure with all the optional rules. That way they can catch the worst bad interactions. They should also run several groups through the same adventure, each with a different optional rule turned on to see how each affected the game in isolation. That's still a lot more playtest load than you'd need for just a core game or a game with one or two optional rules.
131,000 D&D Next'sMike Mearls has listed 17 optional rules in Legends & Lore. He's not leading the development of the next edition of D&D. He's leading the development of the next one hundred thirty one thousand editions of D&D. To be honest, some of these rules he suggests are not compatible with each other, so maybe it's more like ten thousand editions.
The problem Mike has is that he's either decided -- or been ordered -- to write a GURPS of D&D: This is supposed to be a universal D&D that satisfies all the players at least some of the time. He doesn't realize that every edition of D&D so far has been that universal game. Let's look at AD&D 2nd edition.
AD&D Second Edition sucked. It was the dark ages of D&D, where players, fed up with a system that sucked, moved on to other games. No longer dominant in a world with Shadowrun, Vampire, Werewolf, GURPS, and Champions, AD&D languished and faded in the 1990s. But it's also the edition known for the most diversity in how troupes played it.
What caused this versatility?
Well it sure wasn't the system -- not even with Player's Option. Some might say it was how bad the system was that made everyone play it differently, since to make it function you had to interpret things anyway. Those people are wrong.
It was the modules and settings. Do you want to play an epic game where gods are overthrown and continents are destroyed? A plucky heroic game? A street-level punk game in a doughnut-shaped city at the nexus of the planes? Star Trek D&D in space? Far East political intrigue? Outlaw revolutionaries in an impoverished desert world? Horror?
If your group liked spying, intrigue, crime, mysteries, dungeons, wars, high magic, low magic, tactical combat, or theater of the mind, there was a setting or module for you. The only thing good about 2nd edition was all the modules and settings.
Know what else happened in the 1990s? The World of Darkness happened. The core system was built for brooding self-destructive personal horror in Vampire, but it was adapted for violent dark superhero Werewolf, X-Files wizardpunk Mage, goofy Changeling, and more other settings than I can recall.
Ingredients for Success: A system that functions, a setting that adapts it to work slightly differently.
The smoother and more well-balanced the system, the better the play experience is going to be; naturally. The better the core play experience is, the better your modular versions are going to be.
Say it with SettingSo my advice to Mearls is to package the optional rules in specific campaign settings, playtest them later, and release them later. Focus on a single core system that does the core D&D stuff well -- exploring dangerous locations and fighting monsters. If you want rules for "speeding up battles that involve lots of monsters and the characters" write them into the new Thunderspire Labyrinth module series, and put the rules for "mass combat between armies" into D&D Next's Birthright DM's Guide. If you want to include "rules for horror and sanity" put them in the D&D Next Ravenloft Boxed Set. Add the rules that give "mechanical weight to character motivation" into the Planescape Next boxed set, because Planescape, with its factions and focus on alignment and intrigues, just cries out for that sort of mechanic.
Oh and this idea about an optional rule "treating the DM and players as co-authors of a narrative with a specific focus" wasn't even new in 1990. Folks have been using D&D that way for ages. But again, if you want to make specific optional rules for it, put it in a module or setting.