^ that's what I was thinking, but the polythene membrane is often missing on a 15+y.o. car particularly on the driver's door without being a huge annoyance. It's almost as common that some previous hasn't done a good job of putting the door card back on and, in your case, may have "that'll do" refitted the door release handle. A bit of take off and refit might solve the problem and won't take all afternoon.
If it's the actual arm-rest bit, that shouldn't have been removed but there are several tabs that are - best description I can do - melted in place so it stays firm but could have been snapped by rough handling over the years. Any obvious looseness that you can pinpoint?