Should I just give up on him?

I would kind of wait and see what happens. If he's really into you, he'll get in touch with you. But try not to worry about it too much, and just live your life. You'll find someone who really appreciates you, and makes the time to talk to you.