There are now a great many games that attempt, in some way, to model madness and morality, with greater or lesser degrees of success. I'm not aiming to lay down some expert opinion here; these are just a few thoughts on some of these systems.
D&D had its clear-cut axis of good vs evil; characters were good, neutral or evil. In AD&D this was expanded into the (in)famous binary alignment system, with every character's alignment lying on two axes: Good-Neutral-Evil, with good defined as heroic, charitable and giving, and evil as villainous, grasping and dominating; and Chaotic-Neutral-Lawful, defining your attitude to law, order and rules in general. This wasn't a terrible system, although I suspect it led to a massive glut in the Chaotic-Good bracket, since that way you could feel good about yourself and still show a middle finger to the man and embark on epic debauches between adventures, so why not go that route (unless you wanted to be one of the character classes that required you to be Lawful or Neutral in some way).
Call of Cthulhu mixed things up a bit by making morality a player choice and focusing on Sanity as a core mechanic and an ever-dwindling resource. It was a focus of the game and it has always affected the way that players approach the game, possibly too much so when parties of CoC investigators habitually burn books and turn away from the adventure because they know it to be the best survival strategy. On the up side, it means that even in the event of an unlikely triumph, there is a cost to everything.
White Wolf's World of Darkness has a morality system and a madness system, but links them (except in its latest iteration, God Machine, which recognises that linking moral decline to madness was a somewhat Victorian idea, held over from the old WoD, in which the only mechanical morality system was the decaying humanity of vampire characters. Now, some people criticise the absence of a morality mechanic for other characters, but in the setting it made sense. Vampires were unique in that they were not fully functional moral agents. Werewolves and Mages could choose to be arseholes or heroes or anything in between; vampires had to actively strive to be anything other than complete arseholes, such being the nature of their curse.
In the new WoD setting, everyone has a morality stat, although what it is and how it works varies without real consistency. It is clear, for example, why Mages need to hold to the path of Wisdom (of not abusing their power for personal advancement), but not why common human Morality ceases to have any impact on them. I've not played God Machine, but their system - which separates morality and madness entirely - looks better.
Then there is Unknown Armies, which has a system I really like, although it is not without its flaws. The system breaks your mental stability into five categories: Violence, Unnatural, Self, Helplessness and Isolation, representing key stimuli for madness. When confronted with violence, supernatural gribblies, something which makes you question your own self-image, an inability to act effectively or soul-crushing loneliness, you make a roll and if you fail, slip a little further into insanity.
What I like about UA, however, is that if you pass, you become a little more detached; Hardened, in the game's parlance. On the plus side, you can ignore a certain level of that stimuli now; on the minus side, you're a little less human in your interactions. What I like is that whatever happens, these things always change the character.
My final look, and the one that started this line of thought, is Trail of Cthulhu, which again differentiates more, offering both Sanity and Stability, the former standing for your connection to fundamentally human concepts and constructs of moral and right (or indeed wrong) behaviour, and the latter for your ability to act in a functional and directed manner. This is a useful divide in Mythos games, as it allows particularly for cultists - or investigators - to be clinically insane, morally bankrupt, and yet highly functional. Coming from the Gumshoe system, stability is also impacted by violence, so that the pragmatic solution of just shooting that cultist may still have an impact, although if you pass the roll you can shrug it off.
Perhaps the most interesting thing about the Gumshoe mechanic is that you can voluntarily spend Stability to improve your Stability roll, gambling a smaller loss against a greater, effectively accepting a mad justification to save you from worse.
I'm not sure which system I like best, but it definitely comes down to Trail/Gumshoe and UA. WoD's is the weakest, although God Machine's variation is much stronger. The fantasy games don't have a very good morality system at all, but then again, they don't entirely need it. Fantasy heroes are often not entirely moral agents either, instead being part of a more mythic narrative.
