There's definitely more appetite for senior individual contributor roles, and some people really just aren't people-managers, so a lot of organisations share your views and are developing alternative pathways.
Personally, I've made it to mid-senior level without ever having formally line-managed anyone. I recognise this as a gap in my skills - my path, from now on, might be limited to roles where I don't have line reports, and given that most organisations are roughly pyramid-shaped, means I have very few options ahead of me unless I'm ready to take on that responsibility whenever it arises.
Besides, it has been thrust on me from time to time, whether or not I sign off that person's leave form: onboarding an intern, project work, having an associate that helps me but reports to someone else - these sorts of 'dotted line' relationships happen all the time, so we need to get good at it.
The factor you mention is time, but your maths don't math - you would be doing several of these at the same time, every time - problem-solving AND decision-making; having meetings AND communicating.
The hard part is the relationships, and yes that takes time and focus, but you're not going to get out of that, whether you sign off their leave form or not.
BTW, this might help you. I haven't bought it yet myself, but I plan to, and it comes highly recommended.
