Scraping Tables

Hi All

The help on here has been great already and I’m really enjoying learning how to code.

I am trying to scrape a table from a website in the aim to pull the data into a CSV file. I can collect the table headers however only 1 row of data is being provided back even though I am using the find all functionality.

Can anybody provide me with some advice how I can obtain all of the rows of data from all pages please?

import requests
from bs4 import BeautifulSoup
import lxml


html_content = requests.get(url).text

soup = BeautifulSoup(html_content, "lxml")

gdp_table = soup.find("table", attrs={"class": "browse-table"})
gdp_table_headings = gdp_table.find_all("tr")
gdp_table_data = gdp_table.find_all("tr")

headings = []
for th in gdp_table_headings[0].find_all("th"):

data = []
for td in gdp_table_data[1].find_all("td"):


I’m not sure it is the cause, but it would be cleaner to find the th in the thead and the td in the tbody, something like

gdp_table_headings = gdb_table.find("thead")
headings = []
for th in gdp_table_headings.find_all("th"):

gdp_table_data = gdb_table.find("tbody")

(If I remember correctly, find_all does not limit itself to direct children, so you don’t the array indexing gdp_table_headings[0] anyway)