I cannot figure out any exact rule for it's radius calculation. It seems like mobs just randomly reset aggro somewhere between 40-120 yards distance.
I'm just wondering if it's bug or it should be like this. However it's very annoying either if i want to escape from mobs and they don't reset or if i want to pull more mobs and they reset nearly at my frostbolt range so i need to run back and re-pull them.