Why doesn't "my" cat like me anymore?

Well, i kinda "adopted" this little male (not fixed) cat. It's a neighbours, but my neighbours only feed it and stuff, they don't really care much about it other than that. so one day i saw it and it decided it liked me, rubbed its head and everything on me and loves my pets and would follow me around. and that continued for about 2 days. today its the third, the first time i came back from school and he was like usual just a little less affectionate, but still pretty nice, i spent about 10 mins i guess, then i headed home for lunch. the second time i came, i thought i would be nice and give him a feather toy to play with. he kinda plucked out one of the feathers and i kept it cos i didnt want him to swallow anything. and i brought my fraidy little sister too, (9) she's scared of cats, so she stood in the corner. and then the cat decided to try to sniff at my sister or investigate her. that sent my sister running and left the cat pretty suprised. then it came to me and went meow and i just petted him again. then i left, i came back the third time, and this time it only closed its eye at my pets and it wouldn't even let me touch it body now, its would lick it constantly, all it did was lick most of the time i was there during the third time. and just as i was about to leave, his ears were turned back a lot. i knew it meant he was annoyed so i left him. why is he like that? why isn't he affectionate anymore? is it cos my sister startled him? oh and by the way, he has this cat friend too, they can be found together a lot. i'm not sure of the cat friend's gender though. and what should i do to fix it?

on the second day it even showed its belly to me, which is a sign of trust right? so why is he ignoring me now?why isn't he affectionate like last time? should i ignore him for a period of time?