There are certain materials that can only be found underground and are crucial to gameplay.
Coal is an important material used to make torches and to smelt/cook items. You have to mine it in order not to run out of torches while cave exploring. One stack (64 lumps) will be an ample supply, but in practice, you can start with less and mine more as you go. Long-term, it's worth accumulating a stack or few, as once you get seriously into mining, you'll be using a lot. Coal is not strictly necessary since you can make charcoal from wood, but it is common, and you have plenty of other things to do with wood.
Iron is probably the most important and versatile resource you can find underground. You can use it to craft good quality tools, armor and a plethora of other things. Gather as much of it as possible, but it needs a stone pickaxe or better, so be careful not to use a wooden pick on it.
Once you have quite a bit of iron ore, make a base with a furnace to smelt it into iron ingots, that you can then craft. First of all craft a sword and a bucket and fill it with water, then craft iron armor, after that craft iron pickaxe, optionally an iron shovel and lastly some spare buckets to hold a second water source or stray lava.
Gold, Lapis Lazuli, and Redstone are more specialized ores—gold and redstone need an iron pick, and lapis benefits from one. You don't really need them much at the beginning, so you don't have to mine them at first. Also, they are only found in deeper parts of the caves.
Once you do have a little gold and redstone dust, probably the first things you'll want to use them for are a compass (iron+redstone), a clock (gold+redstone), and some golden apples (gold+apples). Once you find some sugar cane, you can add a map (paper+compass) to keep track of your explorations and help avoid getting lost.
Diamond is a very rare mineral, sought after the most, due to the fact that it can be crafted into very durable and quick tools. They can be found at the very bottom of the world, more commonly 5 to 12 levels above bedrock. Once you hit bedrock (unbreakable blocks letting off floating gray particles) climb back up about 10 levels and mine around for a while before slowly making your way down one level at a time. At levels 5 and below, you'll be working around bedrock, so it's better to keep above that. A good way to check what level you are currently on is to press F3 on your keyboard and read the Y-axis value.
Emeralds are normally gained (and used) by trading with villagers, but scattered blocks of the ore can also be found beneath "Extreme Hills" biomes. Emerald ore is found at the same depths as gold ore. It is not used for tools, or anything else besides trading.