Can cats understand what people say? Why can't we understand them?

My cat Sammy has only said the occasional "meow". I was a little upset after a while that he couldn't respond to me in English. I usually tell him things and he walks away, and its really mean and ignorant of him to do that. Why can't he understand me? And why cant I understand his dialect?