Debugging Metacritic

Metacritic, whyyyy? In this entry I'm going to talk about 2 major issues that came up while scraping data from metacritic.com, which is now the most important data source for my Luther Project. The 2 major issues involved TERRIBLE pagination issues and annoying redirects. Let's start with the pagination…

Scraped Knees, Logging Helps

Scraping Problems BeautifulSoup for Python is a great tool for scraping but that doesn't necessarily mean everything is easy. Scraping is all about the unknown. And one of the golden rules is that you can't trust the page you're trying to scrape. You can't trust that it's telling you the…