Ideas for various NPC behaviors, compiled from r.g.r.d.
Raise an alarm/Notify superior officers entrench (create favored terrain) set traps involving other monsters as the mechanism or bait play powerful monsters against each other hoping to loot corpses run’n’gun tactics hack’n’back tactics coerce other monsters to act via blackmail or domination spell argue with each other fight each other show off in foolhardy ways visit battle sites or barber shops looking for a bit of his hair (to bind the head on that arrow of Mike-the-Fighter slaying…) seeking vengeance for killed family/community Hold the @’s family or familiar hostage. play stupid drinking games in monster taverns brew strong drink/cultivate strong plants (and get really upset when the @ destroys all those mushrooms of hallucination, weeds of mellowness, or potions of booze…) Make friends with an @ who gives them booze Play BDSM games with each other Torture as spectator sport (could also be a religious rite) Trigger a “recall item” spell to conjure an item they own out of the @’s hands.
Seek hero to fix unpleasant situation. Seek hero to deliver message. Seek hero to hand over an item.
“crab” movement: fast sideways, slow up/down. 
”turtle” – can retreat into a high-defense “shell” mode. 
”swarmer” – monsters follow just outside of the player’s light radius 
until there’s a small gang, then swarm in. 
”slug” – leaves a slime trail when frightened, which slows the player 
down. 
”nest-breeder” – Two can combine to make a nest, which acts as a 
Gauntlet-like monster generator until bashed. 
”chess moves” – e.g. straight/diagonal limitations or things like the 
knight-jump. 
”territorial” – tends to patrol its “territory” and bully monsters 
within it. 
”charmer” – will try to charm nearby monsters and amass an army. sphynx – asks a riddle, and attacks if the wrong answer is given lurker – like the D&D monster: hidden floor/wall/chest variant of mimic/ambusher doppelganger – touch attack to take on the target’s basic stats stegosaurus – “porcupine” type attack if hit from behind (or use turn direction if implemented) “Mario” turtle – next attack while in “shell” mode sends it careening (into other monsters?) wuss – if encountered alone, flees and returns with next monster it meets mephit – can emerge from torches, braziers, etc. or escape back into them water nymph – same, but with fountains/pools alchemist – gathers potions and recombines them, explodes occasionally
“grid-bug crab” – have movement costs for the monster be greater in the y direction than the x direction. “realistic crab” – have movement costs for the monster be greater in a direction parallel to its target than in directions perpendicular to its target. (If your roguelike implements facing and turning costs, obviously use those instead.) “porcupine” – can retreat into a zero-movement mode; melee attacks on the monster reactively damage the attacker while in this mode. “Grue-like” – this monster hates light. If the player has no light source, it closes in on the player to attack. If the player has a light source, it flees away from the player (at a faster speed) until it is outside of the light radius. A smart player might be able to trap the monster in a corner and kill it while it is helpless. May be made more general by having the substance (blood, poison, confusion potion, etc) and condition (IS_FRIGHTENED, !IS_HUNGRY, IS_LIT, etc) given as parameters. “nest-maker” – like “nest-breeder”, but only needs one to make a nest (a fertile ant queen, for instance). “fusing” – like “nest-breeder”, but the resulting creature is mobile. “evolving” – like “nest-maker”, but the resulting creature is mobile.
I think the most basic monster behavior that changes the rules of 
game when that monster appears on the level is collecting items. There 
are three variants: item-eater, which destroys the items that it picks up, 
single-item-carrier, which picks up a single item and drops it on death 
(or when it finds a better item), and item-collector, which collects many 
items and drops them all when dead. You can add variety by making the 
monsters only target some kinds of items (food, gold, magical items). Examples: 
 rust monster that eats all metal weapons and armor it finds 
 leprechaun that picks up all gold it finds 
 gelatinous cube/amoeba that swallows all items and expells them upon 
 death 
 ants that pick up single items and carry them through all the dungeon A variation on the topic is a monster that becomes somehow different when 
it finds specific item (and maybe reverts back when the item is 
destroyed). A goblin might turn into a goblin warrior after finding 
a weapon – and it doesn’t have to be simulated by it actually equipping 
the weapon. The item might also make the monster weaker: in Rogue orc 
becomes immobile after finding gold, for example. Another common theme is special reaction to certain attacks. This gets 
interesting if you allow the player character to throw potions at 
monsters. For example, as someone mentioned on #rgrd recently, silencing 
should work as blinding on bats and bat-like creatures, while blinding or 
invisibility might not affect them at all. Generally speaking, status-altering attacks are a source of great variety 
in the game, as they let monsters interact with each other through the 
player character. The status may include poisoning, blinding, immobilizing 
(lasting until the monster is defeated or as paralysis for a certain 
number of turns), teleporting, confusing, disarming, weakening, etc. all 
of which will change the relative strengths of other monsters around.
Hard-coded riddles will get old pretty quickly. You might want to generate quiz questions based on qualities randomized each game, so the answer to “Is goblin blood green or blue?” or “Is the Throne Room of the Minotaur Kings found in the Gnomish Mines or the Kobold Warrens?” would change from game to game, rewarding players for exploring and paying attention. (I think that the possible answers (maybe 2 to 4?) should be presented in the question, because then even if the player hasn’t explored enough to know the answer, they might learn something new from the encounter, for example, that the Throne Room of the Minotaur Kings will be in either the Gnomish Mines or the Kobold Warrens.) Related: “The Bridgekeeper” – Asks a question (or three); a dangerous trap is triggered if the wrong answer is given. “transmuter” – collects metal items. When it collects enough, it destroys those items and creates a random new item composed of a more valuable metal; the total value of the new item should be less than the total value of all the items destroyed. “The Armorsmith” – collects and converts non-armor metal items into random armors made from the same metal “The Weaponsmith” – collects and converts non-weapon metal items into random weapons made from the same metal
If they are non-hostile, you can buy the newly forged items from them; if they are hostile, they may use the forged items against you, or give them to other hostile monsters to use against you. (Even if they are non-hostile, they might sell these items to monsters that are hostile to you, so even if you are a paladin you might decide to buy that Vile Unholy Greataxe of Paladin Butchering +9 in order to keep it out of the wrong hands.)
There is another variation possible for some more magical/special creatures. Not just item damage/destruction, but outright item changing. Azure Dreams had a monster that appeared on only one dungeon level 
that was effectively a living magical bag. It attacked like other 
monsters, but if you threw an item at it, that item would disappear 
and it would throw a random item to you in return. This behavior 
was meant to be abused to make building a winnable character easier. 
Indeed, you could go as far as to make one a pet, and thus have 
item transformation available on every level in trade for not being 
able to take a more effective attack animal in that spot.
While not necessarily pure random transformation, you could have 
monsters that pick up one item and transform it into a different 
(but not necessarily inferior) item, even perhaps dropping said 
new item back to the ground. Maybe a rock eater would turn metal 
objects into small piles of the related metal. It eats a gold 
crown and leaves a few gold nugget "crumbs" on the ground (or 
simplify and just leave "coins".)
A fire elemental might burn up some objects on the ground, but 
maybe others could be transformed. To steal and slightly modify 
a couple of ideas from Tobal 2’s quest mode, bad meat might be 
safely cooked if a flame creature briefly walks over it, but 
cooked meat will be burnt to a crisp and destroyed. Potions 
might be transformed as well. In Tobal 2’s Quest mode, holding 
a potion inside a fire would cause its color to lighten and 
thus transform it into a new potion type. Expose it too long 
though and it would evaporate, and thus be destroyed. (Tobal 
randomly reassigned potion effects on every dungeon entrance and 
town visit, so this was sometimes quite useful and sometimes 
quite useless. The mechanics of the game also made it take 
serious effort to abuse, though it was likely expected for 
serious players to take as much advantage of it as they could.)
A gremlin or certain demons might curse objects, while a 
non-violent angel might bless them. Temporarily or permanently, 
depending on what it means to your game system.
Tinkerer/Craftsman – collects broken objects and fixes them (might need some raw materials in addition to the object) Another item-related behavior would be collectors with short attention spans. Such monsters would pick up objects freely, but discard them when they lose interest. Losing interest could be after performing an action, or it could be upon finding a new object. For example, the craftsman above might lose interest in an object once he has fixed it, simply discarding it onto the ground. On the other hand, a giant crow might pick up one object only to discard it the moment that it finds a shinier object. You could make a difficult or dangerous to kill enemy such a collector, so that if it picks up an object that the player wants, the player’s best method of dealing with it would be to drop something else in its path that would command its interest.
Facehugger: rushes to you to infect you with something. 
Screecher: mostly harmless, makes a huge noise if it’s hurt, 
attracting more monsters. Can be deadly in the same room as several 
other monsters, since it’ll mostly shut down your aoe (if your game 
implements any). 
eggs: monster-on-a-timer, there might or might not be a mother nearby. 
unseen: your average invisible monster, turns visible when it attacks 
you, and stays visible for a turn after that. 
guerilla fighter: ranged attacker, plays hit and run. Problematic if 
your PCs are mostly melee fighters.
I’m surprised that no one has mentioned the obvious “ambusher” yet (monster hides/pretends to be something else, then springs a surprise attack on the player when he steps on it/tries to manipulate it/ whatever). Potentially interesting variant: the monster pretends to be an NPC or other harmless creature. For a more complex interaction between monsters, consider the cultivator: 
a more intelligent monster that “plants” less intelligent sessile 
monsters either around its lair/treasure stash for protection, or in a 
location where it can drive or lure the player into them. A variation on 
that would be a monster that uses traps for similar purposes. Nothing 
like chasing after a monster that moves in 2-square jumps every second 
turn, and being dropped into a pit trap in a square that it skipped 
over… Timothy Pruett
Typical – Attack percieved enemies with percieved strength lower than or equal to own. When wounded to a certain point, attempt retreat. If retreat is impossible, attempt to attack with full force (berzerk). Scavenger – Avoid enemies with percieved strength within certain percent of own, but follow within certain distance. Attack enemies if they are weakened. If valuables/food is found, take it. If wounded lightly, retreat. Coward – Run from any enemy, no matter what. Only fight back when cornered. Patrol – Patrol specified area, until enemy presence is detected. When enemy is detected, attack. Other than the patroling part, the rest of this module is identical to the Typical module. Special – Specially scripted AI for unique enemies, bosses, and townsfolk. Auric
Insane – Always attack the enemy, no matter how strong it is, and fight until death. (Standard hack-n-slash game AI) Coward 2 – Hide from the enemy, and later attack from the enemy’s blind side. (D&D thief) Long-range – Attempt to keep the enemy from getting close, while attacking with ranged attacks. Trapper – lays a trap and waits for the enemy to walk into it, only entering actual combat if the enemy isn’t killed by the trap. Experienced – fights with good tactics: if wounded to a certain point, will use healing magic/potions on self; fights cautiously but not cowardly, if outmatched, attempts to bargain its way out; etc.
Daredevil – Defends if attacked or allies attacked, but only initiates combat if forces are “close enough” to equal. Represents groups that fight to prove courage; suicidal attacks are still stupid, but an attack with a strong advantage shows no heroism, so is pointless. Greedy – Largely ignores target strength. Attacks if target appears wealthy (including obvious magic goodies, like flaming sword). Fanatic – Ignores others unless they break a religious taboo (so each Fanatic type needs AI modified for different taboos), then makes berserk attacks.
Looks like as good a place as any for my “Stupid NPC Tricks” list:
I think monster-monster interaction is definitely a good idea. It is not easy to get right though; on a large map it could leave parts of the level in a ‘strange’ state. OTOH, if properly balanced, this could be part of the fun - maybe even worth running the level without the PC for some turns immediately after it is created, to allow some interesting situations to develop. The Aging Minotaur
…For instance, the game features a kind of giant moth (“night swarmer”) that is never hostile to anything, instead moving completely at random and performing an attack on whatever comes in its way. Crossing a cave with these beasts is a trying task, but you can also lead hostile monsters into such a flock and retreat to a safe distance. There are also groups of adventurers, who are neutral to the pc, but will attack certain monster types. OTOH, these bands tend to clog passages the pc needs to get through, or use area attacks that can harm the player if s/he gets in the way (or even their comrades, causing in-fighting, which is amusing in itself, and tends to leave some fair equipment lying around when the dust has settled). In a game where the player can only move orthogonally, there are a few monsters that move along the diagonals. Snakes became an interesting addition: they’re only allowed to move diagonally, wants to attack everything, and retreats from enemies standing orthogonally adjacent to them. This makes them difficult to hit, but usually easy to avoid. If you are being chased by some brutish melee monster, however, a snake pit can turn out to be either a useful dungeon feature or a deadly trap.
…
The idea now is to generate random cultures for two species at the beginning of the game. One is loosely based on Swift’s lilliputs, and one on Breughel’s dancing skeletons. The customs will be “molecular” kits that can be patched onto the data objects used to create these critters and their environs. A lot will be cosmetic, or have only minor effects. Details pertaining to archetecture and inventory lists is one example. In one game, a species might live in small, connected rooms and favor books for treasure, in another they may prefer to build houses in larger caves and keep mummies in crypts[1]. Other quirks will set or rearrange switches in the state machine of species/individuals. Here you’ve got local and global taboos (“don’t eat in the temple”, “don’t harm
Try taking some cues from games of a different genre, as some games have AI with interesting behavior.
The Descent series had several fine examples of AI, such as the Thief Bot and several sneaky-stalker bots. The Thief Bot did the usual stealing of player weapons and running (typically during fights), but out of sight it would also horde the level items in a “den” it selects somewhere in the level. The stalker bots, just stayed at a safe distance while peeking around corners making sounds alerting its presence and changing its viewing location (out of player sight). Whenever a player tried paying any attention to it, such as looking at it, it would shyly hide behind the corner again and if the player tried getting close or chase it, it would run away to stalk again. It would do this until the oppertunity arises to rush the player in a berserk swarm of missiles. I’ve read a player’s story about him trying to do paranoid sweeps of levels in order to kill these things as they would usually be the last monsters standing, taking a rather long time to find and kill.