OK, so as I've discussed previously, there is a table of five-byte entries at 7CE00 (FCE00 in LoROM) that controls the details for monsters' special sizes (when size byte is set as 80 thru BF).
Here's what I know for sure:
Byte 0:
Upper four bits: x-position
Lower four bits: y-position
Byte 1:
Lower six bits: pointer to actual rectangular size (in size table at 6FF5E)
Byte 2: Palette
Byte 3:
Lower four bits: transition style (as detailed by avalanche
here)
Upper two bits: indicates which monster type in battle is affected by a transition command performed by that monster in battle
Byte 4: Tile arrangement pointer
So, the remaining mysteries here are...
- the upper two bits of byte 1
- bits 4 and 5 of byte 3
I don't really care much about those bits in byte 1. So far I haven't seen anything that references them.
What I am trying to figure out is the specific use of those two mystery bits in byte 3.
They do something, and it's graphical. That much I know. Near as I can tell, it has something to do with how the game treats the monster's rectangular size. That is, it doesn't affect the actual size itself, but it affects the position of the finger cursor when pointing to the monster, and it affects which tiles blink when the monster attacks or has magic cast on it.
In battles that feature multiple special-sized monsters, the different monsters have different values, but different forms of the same monster have the same value. So, none of the Octomamm forms have any of those two bits set; but Sandy, Cindy and Mindy are 00, 10 and 20, as are Baigan, Left Arm and Right Arm.
Finding out the exact nature of these two bits has become kind of important to me. I'm up to the Watery Pass in my hack, and I'm reimagining Octomamm in a way that really makes use of hacking the battle system. Basically, he's going to have six independent arms - three that use magic and three that use physical attacks - and they will constantly shuffle between seven different positions throughout the battle. The problem with all this is that, based on a lack of understanding of those two bits, their attack flash (and getting attacked by magic) sometimes does not make the entire tentacle flash. It's just an aesthetic thing, but it's really bugging me.
I doubt this is a thing that can be solved with just experimentation, though I'd welcome anyone who wants to try. It probably will require somebody watching a live disassembly of a read on that byte. I've tried that, but I haven't been able to figure out what I need to know.
Anyway, if anyone reads this and has any ideas or motivation to help - thanks!