Told my boss that I don't care?

Yes that is rude. He was simply telling you about a fun experience he had. If you didn’t care you could’ve told him politely that you’re in a rush or something 

I'm surprised you still have a job actually. 

No you weren't being too mean. You were being honest. You do not care and you are not obliged to act like you care just to not hurt his little feelings. He is a grown up adult who is capable of managing his feelings. He is not a baby who you need to take care of. Some people will like you for your direct honesty and others won't. You are not for everyone to like.

I would have tactfully told him that I was glad he had a good trip and that perhaps he could tell me later as I had to meet someone in 5 minutes.

 Your boss was wasting your time detailing his vacation zx

 . . . . . . . . . .

 , , , , , , , , , ,

All what your boss did was to describe the trip he went with his family .That is because he has enjoyed the trip and he simply want share the pleasure with you .You should have told him how nice glad to hear .Instead  you came out with a crude remark " I don't care ." Now your boss is having a very poor opinion about you .You should understand that you must create a friendly atmosphere in your working place. Yes you were little too mean. Cultivate some good manners .That will take you a long way.

He might have a form of autism perhaps you should go to a psychologist or psychiatrist who specializes in determining mental illness

You don't have to care about his stupid vacation, and i have no idea why he had to go on and on about it.  I suppose if you wanted to know about his vacation, you'd have asked.  But you didnt.

You apparently don't want this job.  Boss will probably find a reason to fire you for your smart mouth.  Civility in the office is necessary.  There may be consequences for what you did.

Yeah totally. Be nice to people