So... here's the basic formula... this is not perfect or totally accurate but if you have better ideas... please share.
Basic Attack Formula:
Basic Damage = (Attacker's Attack Power * rand(1, 1.25) - Target's Defense Power)
If Attacker uses Aim, set Attacker's Accuracy = 99 (although it is probably set to 255 internally).
If Attacker is Blind, set Attacker's Accuracy = 0.
If Attacker is in back row w/o back row compatible weapon, Accuracy = Accuracy * .5 (both monsters and characters)
If Target is in back row and Attacker is not using back row compatible weapon, Accuracy = Accuracy *.5 (both monsters and characters)
Accuracy Multiplier = (Attacker's Attack Multiplier * Accuracy - Target's Defense Multiplier * Target's Evasion)
If Accuracy Multiplier < 0, Skip All Steps and Miss!
If Accuracy Multiplier < 1, Test Chance To Hit (Attacker's Attack Multiplier * Accuracy = probability to hit)
On success: Accuracy Multiplier = Attacker's Attack Multiplier * .5, Otherwise, Miss and skip all other steps.
(for low accuracy weapons like the Lilith's Rod, this is applicable)
Note: For monsters, evasion is still zero AFAIK, with possible exceptions to the rule (Cagnazzo and Barbariccia and possibly other monsters)
Raw Damage = Basic Damage * Accuracy Multiplier
Then factor in all damage modifiers:
If Attacker is Mini or Toad, Damage = 1, Skip All Other Steps
If Target is Mini or Toad, Raw Damage = Raw Damage * 2
(actually, I think the defense holdover of dropping it to zero might still happen, but I barely got to test it)
If Attacker is Cursed, Raw Damage = Raw Damage * .5
If Target is Cursed, Raw Damage = Raw Damage * 2
(For cursed testing, I'm not actually sure this is correct, it could be applied earlier).
If Target has matching racial weakness to equipped weapon, Raw Damage = Raw Damage * 1.5 (for characters vs monsters, only one racial match will apply)
If Target has matching racial resistance thru armor, Raw Damage = Raw Damage * .5 (for monsters vs characters, only one racial match will apply)
If Target has matching elemental weakness to equipped weapon, Raw Damage = Raw Damage * 2 (for characters vs monsters generally, only one elemental match will apply)
If Target has matching elemental resistance to equipped weapon, Raw Damage = Raw Damage * .5 (for both characters and monsters, only one elemental match will apply)
Note: Elemental weakness still takes precedence to elemental resistance when it comes to weapons.
If Attacker is in back row w/o back row compatible weapon, Raw Damage = Raw Damage * .5 (both monsters and characters)
If Target is in back row and Attacker is not using back row compatible weapon, Raw Damage = Raw Damage *.5 (both monsters and characters)
(yes, lacking back row compatible weaponry hurts you double).
If Target has Protect applied, Raw Damage = Raw Damage * (1 - .125 * X)
X is the # of times Protect is applied, and is capped @ 4.
If Attacker dishes a critical hit (< 5% probability I think), Raw Damage = Raw Damage * 1.5
Note: It is possible to dish a critical hit, and totally miss. Critical hits will stack with Tenketsu, Focus, Jump, Human Kite, Big Throw, Aim (don't bother expecting it)
Special command multipliers:
Focus = Raw Damage * 2
Tenketsu = Raw Damage * 1.25
Jump = Raw Damage * 2
Human Kite = Raw Damage * 2
For magic, it's pretty much the same.
I believe the only rule that's different is the following:
Meteor cast by the enemy must apply the split damage penalty of 5 regardless of the # of targets on the field (both alive or dead).
Apply 1.25 to the Modified Calculated Damage (add or subtract from the Raw Damage, depending on the phase of the moon)
There are also special band rules:
1) In attempting to calculate the damage, you must break the attack into its bare components, ignoring ANY racial or elemental bonuses that the weapons provide. Do not factor in critical hits.
2) Most attack/physical bands will automatically apply a 50% damage bonus... so apply 1.5 to damage to the sum of the individual attacks.
3) Many bands will apply an elemental aspect to the attack, so they are generally factored in only there. Apply double damage (*2) to the damage for elemental weakness and/or half damage (*.5) to the damage to elemental weakness. Attacks with multiple elementals will stack the elemental modifiers (so, don't use your elemental bands on the Blue Dragon!). The special case of Cecil+Cid's Machine Break will try to deal bonus damage (like, maybe 2 times), then adding probably a 1.5 machine bonus. I'll have to make sure this is happening.
4) For those bands that don't quite seem to fit anywhere, you try to base it on the commands they occur with... White Magic will imply Spirit whereas Black Magic will imply Intelligence. If the spell effect you see is Holy-like, it probably is.. just like Flare. Healing will generally be SPR based. You should be able to tell who does what in any band.
5) The moon phase can affect spells more dramatically... like Infinity. Just cancel each offending bonus counters out (White Magic+, Black Magic-)
6) As mentioned before, Jump/Human Kite is INSANE with the damage bonus from the moon... but it's equally crippling.
7) Row does affect most physical based bands. Equipping back row compatible weapons or simply not targeting from/to the back row helps.
Damage will still round to 9999 unless provided with the Limit Ring or the natural limit break it provides.
If the attack is weapon based, apply all status effects from the weapon onto the enemy assuming no status resistance (if it has a compatible resistance, it will fail outright).
Hint: Status effects generally don't work on bosses...
I have a headache just typing this up. I need the sleep badly. Comment as you wish.
Added back row stuff, added Protect, etc etc
There probably should be an accompanying spell damage algo, but the simplicity of it is that the damage range is significantly lower (25% variance) but added variance to spells that are normally fixed.
Added more back row stuff, and one more golden rule about bands.
