Selfish or cautios?

This post is completely off topic, but recently I had an encounter that made me lose sleep.
Last week, after waiting for 15 minutes in the bus station I decided to walk home, happy that the week-end was there and trying to beat the cold when a girl, crying asked:
"Can you help me?" and I stopped. "I'll try!" thinking that she might need directions or something. And then she starts crying saying she has a baby at home and has no baby food, and that she has no job and only 8 grades education. As soon as I saw what she meant, I said: "I'm sorry I don't have my wallet with me", to which she replied: "I understand!" And she resumed crying.
Of course I was lying...I did have my wallet with me, but in there was only a $20 bill. And in that second, I didn't want to give it away. But about 2 minutes later I felt so bad that I stopped and had one of those internal fights like you see in the movies...seriously considering to go back and give her my $20. But for some reason, in my head came this little Mexican lady that sells coffee at the entrance of the hospital and she's there every day, with her oxygen tank. Or the guy in shipping who told me that works 2 shifts back to back just because he's not getting retirement. Or the old dog walker lady I saw one day with 3 little dogs on a freezing drizzle doing her job. The more I thought, the more I didn't feel that my decision was bad. They are there, probably not with a higher education either, but they found jobs and make the best out of their situation.
Well, moving from a small town, where we knew the 3 homeless people with their sad stories, and they didn't beg. We knew the lady with a death wish that was riding her bike in wrong direction, and the bearded guy who collected cans in the shopping cart. But they never asked for anything. Actually my boyfriend wanted to fix his bike but they were just shadows not wanting to inconvenience in any way. Well, Chicago is not like that, they're in your face asking, sometimes for strange amounts (one guy needed the other day 27 cents). And they're everywhere, I learned the "don't look them in their eyes" walk, hopping not to be asked.
And so many times I feel bad, sometimes I give them 1 dollar or how much change I have, but that is a very selective process (once we've seen this girl sitting at a corner of a street with a cat in her lap...I'm not sure if the girl or the cat impressed us the most but I gave her $2).
So, was I mean? selfish for not helping the lady the other night?
Every day I thank God and my lucky stars for the comfortable life I have, and I know there are many sad stories out there that I can't help in any way, but for this one I felt bad.