The reason was that there where no cleanup after a failed 'change master'. Fixed by doing a cleanup of created relay log files in remove_master_info()
