It seems like I should use them everywhere. Is there a good rule of thumb?
You can probably use semantic elements in place of some of the div tags.
Use the semantically correct tag whenever possible, but if there is no tag that fits what you need or you need the tag purely for styling then use a <div>.
You can learn a lot about the <div> tag by finding and clicking into it on this page.
Take particular note of the usage note and example section.