There are two types of caves that can be found in ARK: Progression Caves and Resources Caves, the latter being exclusively found underwater on The Island. Some are easy to explore, while others are extremely difficult to explore. If you survive the dangers of the caves, however, they each contain a variety of rewards.
Note that the Hard Caves spawn creatures at extremely high levels (often exceeding 250), and that these creatures are untameable. Creatures within the Caverns of Lost Faith are also untameable, but spawn at standard levels.
Note that all creatures inside these caves are high-leveled, untameable variants, similar to the creatures found within The Island's Swamp and Snow caves.
Resource Caves
There are currently 12 Resource Caves on The Island, and are all found underwater. These caves can often be difficult to find, but often contain a dense collection of Silica Pearls and Oil, although the latter of which provide less oil than nodes found elsewhere in the game. The only indication of the position from the surface is a small flow of bubbles, appearing as a thin white line. On lower graphics settings, a grey box can be seen on the ocean floor from the surface around the cave's entrance.
The entrances to the resource caves also appears to be a spawn location for pairs of Plesiosaurs, who tend to linger in the area, as well as singular a Mosasaurus, which in turn tend to linger near to the Plesiosaurs.
Hide or Fur armor is helpful for many, but not all caves.
Grappling hooks are useful for avoiding creatures, getting a good vantage point, and to not have to bring foundations and ceilings to bridge gaps.
Underwater Caves
Lazarus Chowder will decrease the needed Oxygen while underwater and allow you to dive for longer times, making it easier to find the entrance.
Riding a Megalodon can help against other Sharks and allows you a very quick movement in the Water. They are also a good way to spot the bubbles from the surface. In an earlier version, there seemed to be a problem with taking your Shark into the Cave. However, Megalodons can now enter the underwater caves without issue .
If you have a Tribemate with a Pteranodon or Argentavis, he or she can transport you directly to the location of the cave and drop you down, avoiding most of the swim, and with that, most of the Megalodons on the way to the cave. Once finished the Pteranodon or Argentavis can pick you up again from the water surface. This is also a very resource friendly method, as it allows you to go down with minimum armor and weapons, as fighting can be reduced to a minimum. That way, the loss of dying is less of a setback. This method does, however, require you to have found the entrance beforehand.
Riding a Sarco allows you to use a single mount as he can move quick in both water and ground. Moreover, it's faster than a Megalodon, does not require stamina in water and can fit in most underwater caves, making it safer while gathering resources.
As of patch 246, Eurypterids can spawn in groups (3-5 at once) in even small underwater caves. Be prepared to deal with their stings!
Notes/Trivia
There is a large flat faced rock that looks like it could be a cave entrance on the South haven island cliff face. Could be another cave in the future.
Building a base in an underwater cave will stop resource spawning in that cave.
This is especially detrimental when trying to get to one of the places that spawns a loot crate, which is along a long, thin corridor that is littered with spiders, wolves, sabertooths, and Yetis (possibly one of the most concentrated areas of aggressive creatures in the game).
These nodes can block movement, bullets, arrows, etc. but can be helpful as it blocks some creatures against walls, while other nodes can act as walls, allowing you to be unseen as long as it is between you and a creature.