I always go around the area to find lootable things. From time to time I'd steal expensive items, that and consumable items (they're really handy when your character is new). From there though, I go by the story.
I have a ton of mods installed on my game, so it makes sure that the game never gets too boring. In general, I try to complete most of the side quests first rather than the opposite. I try to make sure that I do not miss out on anything and get all the goodies I can to use in the actual main story. Of course, with all the mods, there's even more ground that needs to be covered now.