At this point, your new public key should be on your EC2 instance in the authorized_keys file, and all you have to do is remove the old one.