Only buy tools that you can use a lot. Yes that baby feet punch is adorable but how many times will you use it? When I stared the only punch I had was the corner adorner by Creative Memories. This is one of the few tools you can use on almost every page without them looking the same.
You'll need a good paper trimmer. I started out with the mini one from Creative Memories but then upgraded to a 12x12 style. I wore out the big one and replaced it with a rotary trimmer. Before you freak out at the price, I chop up a LOT of paper and chipboard. I needed the refill blade style. The one I had before that was around $10-$15 at Office Max (their brand) and would have lasted a lifetime for the average scrapper. You need one with a ruler grid and a sharp blade. You don't need anything fancy. I lasted with my little one for a few years before I got the 12x12.
Buy paper in packs to save money or wait for Jo Anne's 6/96 cents sale on open stock. Then when you do need a special paper from your local scrap store (LSS) you can splurge on that special sheet.
Alphas can get expensive! I always looked for deals on alpha stickers before I got my Cricut. I also bought vellum title packs by DCWV to save on alphas.
Think outside the box! Just because something is sold for one thing, doesn't mean you can't use it for something else. You don't need expensive ribbon just for scrapbooking. Buy the spools for 50 cents in the basic colors. Use your scissors to chop up stickers and other embellies to make just what you need.
Save all your scraps. You never know when you might need that itty bitty piece of paper. Pattern on one side? Turn it over and use the white side.
Adhesive is a must, obviously. You can use a glue stick, but I find that it doesn't hold photos well. It is, however, great for adhering paper to paper. I use a tape runner for holding photos. Just a 1/4 inch or so on each corner is all you need. I use the Creative Memories tape runner but you can find them at Jo Anne's, Micheals and the dollar store. A little really goes a long way and I think this is the cheapest option.
Buy kits. Kits are wonderful because they coordinate and are almost always cheaper than buying individually. What you have left over can be used on another project. Target has a lot of kits available lately.
I hope this helps you get started scrapping. I'd love some more tips in the comments if you can think of something I've forgotten.