It seems like the problem in this application is that attention itself. Makes me wonder if using a transformer for transcription is the correct architecture.