Streamlined Battling? (2x rewards, 1/2 encounters)

What causes the overflow error? When the amount of AP exceeds 255 or something like that?

I think this is right. Like in the above example, 13AP x 20 (learning rate) in one battle will give you 260 AP, so it will give you only 4AP (260 - 256). You just gotta make sure it does not exceed 255 each battle. The other solution is putting the AP on two bytes instead of one but this is a ASM hack.

Edit: On one byte with the learning rate of 20, the most AP you ca give is 12.