Monsters are invincible/Algorithm question???

You have the right idea, but as you said, $00 is the BRK (break) instruction, so it's not surprising that your game crashes. What you want is the NOP (no operation) instruction, which is the $EA opcode.
Replace the ASL at C2/2B79 with NOP and the (Spell Power * 4) portion will be (Spell Power * 2 instead).

You have the right idea about the division part as well, but I'm confused by what you wrote, as the vanilla code is already set to loop the division function 5 times (divide by 32) by the LDA #$04 at C2/2B8C. If you set it to loop 6 times you'll get DIV 64, ie. change that LDA #$04 to LDA #$05.

Explanation: the function call at C2/2B90 calls a function at C2/0DD1 that divides a 32-bit number $E8 by 2^A+1. It does this by bit shifting/rotating the bytes and then looping A times.

The operand length of an instruction is determined by the register length flag. That is, if A is 8-bit, you can't supply an LDA with a 16-bit operand; only the first byte will be interpreted as an operand. The second byte you supply will then be interpreted as an operator. In your case, you put $00 there so it's another BRK instruction.

EDIT: The offsets in this post are all unheadered, by the way. It appears your ROM has a header so you'll have to add $200 to them.

---

By the way, I think B-Run included a fixed version of the 24-bit monster HP patch in his monster AI upgrade patch, which can be found in this forum.