A Guide to Ability Learning

So I have been at this for a week, and I.....have no idea what I'm doing.  I appreciate your help on this, but I don't think I made much leeway on this.

I actually did not know of a NOP command.  Honestly I tend to glance over words like BEQ or JSR in the coding.  Anyway I was trying to do research on NOP commands and from my observation the byte is $EA.  So I started by trying to change the first byte in every code of the Mog Dances, since commands strings I saw started with that byte.

Code:

C2/5EE5: EA 0A 30 LDA $300A
C2/5EE8: EA 16 BMI $5F00
C2/5EEA: EA E2 11 LDX $11E2 (get combat background)
C2/5EED: EA 5B 8E ED LDA $ED8E5B,X (get corresponding dance #)

And....I broke he game.  I tried adding the byte to a couple of the strings, and it either broke the game or defaulted to the "Wind Song" learned.  I tried to replace the codes with FF to all of them and it resulted in him learning ALL the songs.  So clearly I'm not understanding these command prompts like I thought I did.

Ooch, if you do this, you're going to have a bad time.
NOP, or "No Operation", is a one-byte command.  That means the processor reads it as 'hey, take a breath and then read the next byte'.  However, you didn't NOP the rest of the former command, so it reads 0A as a new command instead of what it used to be... the little end of an address.
That means it's interpreted as ASL, "Arithmetic Shift Left".  The value of A is multiplied by 2.  ASL is too a one-byte command, so the processor does this command and moves to the next byte, assuming it's a new command (instead of what it used to be... the big end of an address.)
But the next byte is 30, which the processor reads as BMI, a conditional local jump "Branch if Minus".  It checks A for the minus flag (that is, is the high-bit set, or is A greater than or equal to $80?), and branches if so.  The distance it branches is determined by the value of the next byte, which is EA because you were trying to NOP the next line.
...The result?  A gets arbitrarily checked if it is minus.  If it is, the code jumps back -$12 bytes, likely ending up in the middle of OTHER code.  The sequence of logic has no way to get back on-sequence, so the game unravels like a ratty sweater.

If you're going to NOP a line, you need to replace each byte of that line with an EA.  For instance, if you're removing LDA $300A, you need to replace all three bytes with EA.

EA EA EA

I want to reply to your other stuff too, but this is too important to be ninja'd.  I'll edit in more as soon as I can.

Of course even if I did manage it correctly, I was unable to figure out how to change the dances to level up. The first problem is adding a new line of code to this

Code:

Who learns what at level up
C0/A182: C900 CMP #$00 (is character Terra? note there's no need for this CMP here)
C0/A184: F010 BEQ $A196 (branch if so)
C0/A186: C906 CMP #$06 (is character Celes?)
C0/A188: F02E BEQ $A1B8 (branch if so)
C0/A18A: C902 CMP #$02 (is character Cyan?)
C0/A18C: F04C BEQ $A1DA (branch if so)
C0/A18E: C905 CMP #$05 (is character Sabin?)
C0/A190: D003 BNE $A195 (branch if not)
C0/A192: 4C01A2 JMP $A201
C0/A195: 60 RTS

I thought that maybe I could try to divert one of these branches to some free space then bounce back to this code afterwards so I can add an extra line of code for Mog, though I wasn't sure if I could do that properly. I found using a Hex calculator IF you take the last byte and add it to the Next offset, gives you the correct branch, (IE: C0/A184: F010 BEQ $A196 [Take 10 from F010 and add to A186, which is the code after this and it works) However with Sabin's code it's not a branch, it's a jump, and not only do I not know how those work, I don't know where Sabin's leveling code is since his codes says "Branch if NOT" Of course there's also the problem of the recruitment strings.

Code:

Who learns what at recruitment
C2/61B9: C9 00 CMP #$00 (is it character #0, Terra?)
C2/61C0: C9 06 CMP #$06 (is it character #6, Celes?)
C2/61C7: C9 02 CMP #$02 (if it character #2, Cyan?)
C2/61E3: C9 05 CMP #$05 (is it character #5, Sabin?)

and I have no idea what CMP commands do, let alone how to make room to add and extra line of code for Mog. So by now I'm just lost.

CMP is 'Compare A', and is extremely important. It checks if A has a certain value, by subtracting the number that follows from it (without updating it) and setting the carry accordingly. For instance, if the current value in A is 7, and you CMP 6, then the result of the subtraction is 1 and the carry is not set (because 7 doesn't have to 'borrow' a number to subtract 6). If you took A is 7 and you CMP 8, then the result of the subtraction is -1 ($FF) and the carry IS set, as 7 was too small to subtract 8.

Next, we need to discuss BNE and BEQ. BNE is "Branch if not equal". It checks if the result of a CMP is anything BUT 0. If it is, then it skips a number of bytes equal to the number that follows (forward if less than $80, backward if more than $80... to find out how many bytes backward, subtract the hex value from $100). BEQ is "Branch if equal", and does the same thing but only branches if the result of the CMP is 0. In the context of level learning, this method is used to check the identity of the character who is gaining the level.

So, we need to insert a redirection to some freespace somewhere in this comparison list that will let us check if the character is Mog and handle him accordingly. Here's a trick I learned from Hatzen's patches: if you need to jump to new code or a subroutine and all the commands are two bytes long, the easiest way to insert something is to make the jump you use use 4 bytes total. That means it should be long-addressed.

So, what we're going to do is overwrite Terra's check with a jump to freespace.

Code:

C2/61B9: C9 00        CMP #$00       (is it character #0, Terra?)
C2/61BB: F0 3F        BEQ $61FC      (if yes, branch to see if she learns any spells)

replace with

C2/61B9: 5C Za Yb Xc      JML $XcYbZa

JML is "Jump to address, long". It goes to the exact address listed in the processor, in this case whatever freespace you have to work with. The address is listed little-endian, meaning the largest number of the address is the rightmost byte. For instance, if this were to say 5C 42 63 C3, it would make the processor jump to $C36342.

Code:

{Your freespace here.}
XC/YBZA:
C9 0A         CMP #$0A  (Mog is character #$0A, so we check if it's him)
D0 04         BNE #$04  (If it's not Mog, skip the next four bytes, effectively skipping the next line)
5C TT RR QQ   JML $QQRRTT  (Jump long to more freespace after all this stuff)
C9 00         CMP #$00  (Now we need to recreate the code we replaced, so we check for Terra)
F0 04         BEQ #$04  (If it is Terra, we branch to her natural learning, skipping the next line)
5C BD 61 C2   JML $C261BD (If it isn't Terra or Mog, go back to checking the characters, by jumping to the Celes check)
5C FC 61 C2   JML $C261FC (This goes to Terra learning magic at level-up)

Determining what goes at TT/RRQQ is going to be dependent on what freespace you're using. The above check takes 20 bytes, and from my estimate setting up a dance learning routine will take at least 47 bytes, so we need a space that has 67 or more free bytes to work with (four blank lines in HxD, with three extra on the front or back).