While there are risks associated with any kind of external access, the benefits of a properly implemented system to allow external access to email may often outweigh those risks. The key is proper implementation and security precautions to surround it.
The greatest risk would be for the login method to be compromised and allow an outsider to access a user's mailbox. This could compromise confidential customer information. Depending on how external access is implemented it may also allow access to the entire network. Another risk is that the email messages may be intercepted if they are not properly encrypted between the mail server and the client. There are additional risks as well.
Take an in depth look at why you think you need external access (Do you have users who work outside the bank who would benefit from having access to email externally or do they just think it would be nice to have?). Then you need to examine the necessary changes to your security policy, additional risk assessment requirements, and audit procedures that would go along with this system. Depending on the size of your organization and how many people will access email externally, auditing the system alone may be a full time job.
A user name - password implementation is not nearly secure enough. You will need to install a public key infrastructure at the very least. You'll also need to contract with an expert to set it up. A system that is secure enough to allow access while outweighing the associated risks will take a considerable amount of effort to implement and may easily cost more than you feel the access is worth.
My comments and opinions do not represent legal advice nor the opinions of my employer.