Python Web Scrapping Dilemma?

I am trying to get all the results it shows in the shell to go into my text file but it will only put the last one found. I’m sure its a simple but I have tried several ways and can’t seem to figure out how to get all the results into my text file!

res = requests.get('https://www.infosecblog.org//').text

soup = bs4.BeautifulSoup(res , 'lxml')

writer = ('headline', 'summary', 'url')

for article in soup.find_all('article'):
    headline = article.h1.a.text
    print(headline)

    summary = article.find_all('div', class_='entry-content')
    for s in summary:
        print(summary)
# Grabbing an anchor tag using dictionary formating
    url = article.find('a')['href']
    print(url)

    print()

with open('scrape.txt', 'w') as file:
    csv_writer = csv.writer(file)
    csv_writer.writerow([headline])
    csv_writer.writerow([summary])
    csv_writer.writerow([url])

Hi, it seems you’re using Python, try to change the open function mode to “append” by changing the line
with open('scrape.txt', 'w') as file:
to
with open('scrape.txt', 'a') as file:

More info here
https://www.programiz.com/python-programming/methods/built-in/open
or here
https://www.w3schools.com/python/ref_func_open.asp

Yes, it is python I fixed my topic header so others will know. Unfortunately, I have tried this and get the same result. I think I may have to setup a larger loop to encapsulate it so that it will continue to run and hopefully write the data every time it runs.

Have you tried indenting this part, so it lands inside the first for loop

with open('scrape.txt', 'w') as file:
    csv_writer = csv.writer(file)
    csv_writer.writerow([headline])
    csv_writer.writerow([summary])
    csv_writer.writerow([url])

I’m not 100% sure that it solves it but I played with it here

What do you think?

It still works but not for the writer portion its giving me this error: Traceback (most recent call last):

File “C:\Users\ajohn\Documents\scrapping.py”, line 46, in
csv_writer.writerow([summary])
File “C:\Users\ajohn\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py”, line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: ‘charmap’ codec can’t encode character ‘\U0001f609’ in position 659: character maps to

Great!

About that new problem, see if this guides you in the right direction

@ajj71310 Did this solve the issue? :slightly_smiling_face: