The Solution:
First, I make confirm that key-based authentication is working as I was able to log in with the ssh key provided by the AWS console. After that, I created a new key with type ecdsa. Open a terminal and generate a new key-pair: Once the key is generated, add the public key to remote servers ~/.ssh/authorized_key file. Now, I can successfully connect to the remote server with FileZilla over SFTP.