Stay away from the $ amount and focus on the number of incidents. Example: Man presents a check for $275,000 that is a counterfeit. By policy, a front-line teller will never make a decision on this check. Man, presents a check for $1,990. Most likely, the same teller is relied upon to use his/her good judgment in taking this check.
If we reward by $ amount, we reward a lot for catching something easy like an unusually large check, while the $1,990 check seems more normal and is more difficult to catch, but we reward a smaller amount for that.
We reward a point for each incident, regardless of $ amount. We've had one for almost $500,000 and we've had several in the $400 range. If you catch it and save a loss, you earn a point. If you cash it and cause a loss, you LOSE a point (they must also be held somewhat accountable for mistakes...can't solely be positive). Set a point total at which a teller earns a bonus (we set ours at 10, initially, but nobody was getting there, so we changed it to 6). We pay a $100 bonus for every 6 points a teller has.
One branch manager has 21 points (working on his 4th bonus) while others have just a few points. This also rewards longevity, rather than simply paying a bonus each and every time a teller catches something.
Also, we are a retail branch bank. Scammers go from branch to branch. To encourage inter-branch communication, I also award a point if someone warns of a possible situation and that warning is partly responsible for stopping a scammer from hitting the next branch.
I do spend a LOT of time in monitoring this program, but I also spend a LOT of time training our tellers to prevent fraud. The result, after 14 months, is a lower fraud-loss total and fewer cases for me to try to prosecute.
Basically, my tagline is my motto when it comes to doing my job.