After observing intermittent problematic POP connector received email bounces I dug a little deeper and discovered a KB that covers this issue in great detail. This is linked below.
For my particular discovered problematic SBS 2003 R2 environment, Exchange had been re-installed. Apparently this contributed to the condition and following the directions towards the end of the KB worked for me. This was specifically to re-register the POP3 Connector event sink in IIS. This involved running:
%programfiles%\microsoft windows small business server\networking\pop3\imbreg.exe
Following the subsequent directions I verified it by creating and running the described script. So far so good.
How to troubleshoot the POP3 Connector in Windows Small Business Server 2003