The bigger problem are delays in workflow and in getting feedback. It’s also useful to realize that you are task-switching not multi-tasking.
The primary causes of task switching are interruptions and not having the capabilities you need. Either of these require you to switch to another task.
The switch itself is not costly unless it takes time to reframe where you were or if the delay between when you stopped and restarted the work creates additional work (e.g., a requirement aging). The greater the time until you switch back, the greater the unplanned work. This extra work increases the chance of more delays and interruptions.
We can reduce delays by attending to flow, the amount of work in process, and how teams are organized. This requires looking at the flow of work directly. We will be much more effective if we look at the cause, not the symptom.
Agile can be improved by incorporating this Flow-thinking into it instead of requiring people to figure it out on their own. I suggest integrating it into any approach you adopt from the start.