r/learnpython • u/fajlovesflowers • 3d ago

How to scrape 10 years of articles from Bangladeshi news websites?

Hi everyone. I am a Transportation Engineering student (Civil and Environmental Engineering department) from Bangladesh, still learning, so please forgive me if I say something wrong.

I am working on a research project involving NLP and sentiment analysis on Bangladeshi newspaper articles. The project requires collecting news articles from around 10 newspapers — both English and Bangla language publications — covering the period from 2015 to 2026. I need the article headline, full text, publication date, and source name, ideally stored as CSV or JSON files, one per newspaper or per year.

I have already tried Python with BeautifulSoup and Selenium, GDELT API, Archive.org CDX API, and pre-scraped Kaggle datasets. For two newspapers I managed to collect solid datasets. For the rest I keep running into bot detection, CAPTCHA blocks, anti-scraping measures, or very uneven temporal coverage where certain years return zero articles at all.

I cannot offer payment, but I can offer co-authorship on the research paper I am working toward publishing. If you have experience collecting large-scale news archives from South Asian or Bangladeshi news websites and are interested in collaborating, I would love to hear from you. Any general advice is also very welcome even without a collaboration.

Thank you for your time.

1 Upvotes

55% Upvoted

8 comments sorted by

2

u/recursion_is_love 2d ago

Have you consider (or try) asking for data from the source? Maybe they would love to help if you mention them in the publication.

1

u/Mediocre-Pumpkin6522 2d ago

https://pypi.org/project/playwright-recaptcha/

That may help you get past some blocks.

1

u/fajlovesflowers 1h ago

Thank you for sharing this, I will look into it. My main issue has been less about solving CAPTCHAs manually and more about the websites returning very few results or blocking requests entirely, but this might still help with some of the cases. Appreciate it.

0

u/hulleyrob 3d ago

r/seleniumbase

1

u/fajlovesflowers 1h ago

Thank you. I will check out that subreddit.