Can We Stick to the Subject?

This week's assignment was to pay attention to how people listen to me. Also, to understand the devices I use to get people to pay attention.
Much harder than to pay attention to how I listen. I gave it a good try.
Here's what I found. Since I am a person who does best when conversations are kept on topic, short and to the point, I find that I tend to do that with others. In meetings, I resist attempts to "open up" the discussion to topics not on the agenda. As soon as I get an opportunity, I try to summarize what I have heard others say and then move the conversation back to the topic we started out to discuss.
I find I do this in personal conversations as well as business conversations. I am not good with "changing the subject." I want things in focused, manageable portions.
Does this work?
In business, it does, generally, as long as the meeting started out with an agenda or topic. They understand the statement that we need to get back to the topic we planned to discuss. I have found little resistance, although sometimes people struggle to get in a final word about the "side" topic before they agree to move back to the main topic.
In personal conversations, it's more difficult to do. People want to be heard, and many people tend to ramble. Since I find I have little patience for this, I am sure that I offend people by displaying that in various ways (losing focus, looking or sounding bored, looking for opportunities to end the conversation).
Empathetic listening was not one of my strong points, so I guess it makes sense that I don't particularly look to strike empathy when I am speaking to others. I want others to pay attention to me on the topic I started out to discuss.
Do I need to work on my ability to tolerate "chit-chat"?