As it turns out, it's a little more complicated than it first seems.
One problem I was having was the "creature mask;" the color the monster changes when certain spells hit them (ie, blue for Ice, orange for Fire). Changing the spell animation color didn't change this mask.
As it turns out, I was wrong about the meaning of one of the bytes. The third byte isn't actually the animation shape. The third byte is actually an animation routine. This byte (and the high bit from the second byte, which as of yet I've never seen set) is passed to a variable ($F586) that in turn performs a lookup on a table at D9/7D42. This table is a series of two-byte value indicating the address of the animation routine to perform. These addresses all appear to be in the upper half of the D9 bank.
(Aside: they should be very easy to re-direct to expand)
The animation routines vary widely. I'm still decoding them, but the basic format is a three-byte "header" whose purpose I have not discovered yet. After the header are the instructions. The end of the routine is given by a value of #FF. This routine is read at (roughly) C1/C48A.
The instructions have two types. Anything less than #80 is a simple frame call. Anything with the bit 7 set (ie, above #7F) is a special instruction, given by the remaining bits. Special instructions often take parameters, which are bytes that follow the instruction. The number can vary, and is determined by the special instruction.
An example of the very simple code for Fire (at D9/8F5C):
00 20 00 80 0A 10 11 12 13 80 00 FF
The first three bytes are a header, whose function I have not determined.
The next byte is 80, which indicates a special function. In this case, #80 indicates to use a creature mask of the color given in the next byte. That makes this instruction 80 0A, and sets the creature mask to orange. Changing the parameter on #80 does change the mask color; more on this below.
Following that are individual frames to play: 10, 11, 12, 13.
Next, we have another call to #80, this time with the parameter 00. This indicates to clear the creature mask.
Finally, FF indicates the routine is over.