Expanding Storage On EC2

While streaming hundreds of thousands of tweets from my EC2 cloud machine, I realized that my tiny 8gb EC2 instance was starting to run out of memory. So I went to my AWS dashboard to increase the memory, thinking that it would just take a couple clicks. Little did I…

Keeping Things Running on EC2

One of the joys of cloud computing is not having to keep your laptop/computer on 24/7. I remember back in the old old days, downloading at less than 1kb/s and having to leave the computer on overnight to download anything more than a few megabytes in size!…

Project Fletcher: Twitter Streaming Counties

Given a longitude and latitude point, how do you determine what county it's in. This is the problem I have finally solved. First of all, it seems really simple, and there are definitely simple ways to do it, but given the large volume of data I needed to process, I…

Moving Blog from Github to Digital Ocean

I once again moved my blog. I first moved from using Jekyll to using Ghost with a tool called Buster. I've documented this process already here. While this was great, it also had several different issues that I had to work around. Static pages were a pain in the butt…

Twitter Stream from US Only

I won't lie, Twitter's streaming API is pretty amazing. I still find it amazing that companies like twitter give out so much much information for free. The streaming API returns a fraction of all current tweets an allows you to filter based on many different factors. One of the factors…

Buster Redirects and Static Pages

Buster (the tool I use to generate static pages from my ghost blog that runs on a node.js server so that I can then push it to Github pages where it's hosted for free) has certain issues when it comes to single page generation because of the way it…

Prism Syntax Highlighting

Adding prism.css/prism.js to Ghost Blogs So if you are a frequenter of this blog, you may have noticed that the aesthetics of the code on this blog has changed. It's not quite as ugly as this anymore: for element in my_list: do_something(element) But, all…

Logging Like a Professional Lumberjack

Basic Logging Tutorial for Python A while back, I wrote an entry where part of it describes a how to on logging in Python. If that was a super basic tutorial on logging, this one kicks it up a notch as a regular basic tutorial! What is it? I guess…

[post for post in posts if post == this]

Comprehending List Comprehensions List comprehensions are pretty cool. They make a lot of sense and are really well designed in Python. While working on some challenges today at Metis, I was able to shorten a lot of my code while using this technique. List comprehensions with Python are only now…

Easy, Peezy, Google Shee[t]zy

Making Google Sheets Look Good! Simple Google Sheets Aesthetic Tutorial I've seen so many horrible looking Google Sheets files that I wanted to write this tutorial to give 4 simple tips to make any sheets file look a million times better (only a little bit of exaggeration). It might even…

Essential (and free!) Mac Apps/Tools

My Favorite Mac Tools I've been using macs since I was a senior in high school, so for over a dozen years ago! But I didn't really get "good" at using them till I became a coder a few years ago. But I've found that there are definitely tools that…

Debugging Metacritic

Metacritic, whyyyy? In this entry I'm going to talk about 2 major issues that came up while scraping data from metacritic.com, which is now the most important data source for my Luther Project. The 2 major issues involved TERRIBLE pagination issues and annoying redirects. Let's start with the pagination…

How Many Data Scientists Does It Take to Make a Cup of Coffee?

Making Slow Drip Coffee One of the great parts of being in this program is the amazing group of people I get to be a part of. And while some of them may be really amazing in one area, that excellence usually transfers to other hobbies that they have. One…

Hulk, Bash!

Bash Profiles One of the cool things we've had an opportunity to do at Metis was to play with our .bash_profile files. I've edited mine before but never really knew exactly what it did. This time around, I wanted to make sure that I try some more complex things…

Scraped Knees, Logging Helps

Scraping Problems BeautifulSoup for Python is a great tool for scraping but that doesn't necessarily mean everything is easy. Scraping is all about the unknown. And one of the golden rules is that you can't trust the page you're trying to scrape. You can't trust that it's telling you the…

In a Pickle

Getting Out of a Pickle with Pickle The Problem One of the problems with writing functions for scraping data is that in the process while testing you may have to do multiple calls of the scraping program. And when you are sometimes making hundreds, if not thousands, or even far…

Just Scraping By

BeautifulSoup Today, we got into a python tool called BeautifulSoup (v.4) that is used to scrape data from websites. This wasn't my first time scraping since I have some experience using differnt PHP tools, especially SimpleHTMLDom. But, after using BeautifulSoup just for a few minutes, I quickly realized how…

I See Ghost(s), Goodbye Jeykll

Pushing Ghost to Free Github Hosting So after blogging the first week of camp on Jekyll which is a great tool for blogging, I decided to move my blog over to Ghost. I've heard a lot about Ghost as the node.js built, strictly-blogging platform, and have always wanted to…