How to scrape Reddit

In [1]:

from urllib2 import urlopen
from urlparse import urljoin
from BeautifulSoup import BeautifulSoup
#BeautifulSoup is a 3rd party library
#install via command line "pip install bs4"

In [2]:

url = 'http://www.reddit.com/'
html = urlopen(url)
soup = BeautifulSoup(html)

In [3]:

posts = soup('a',{'class':'title may-blank '})
#get the links with the class "title may-blank "
#which is how reddit defines posts

In [4]:

for post in posts:
    print post.contents[0]
    #print the post's title
    
    if post['href'][:4] =='http':
        print post['href']
    else:
        print urljoin(url,post['href'])
    #print the url.  
    #if the url is a relative url,
    #print the absolute url.    

What's the most effective bit of advertising you've ever seen?
http://www.reddit.com/r/AskReddit/comments/2pkzgz/whats_the_most_effective_bit_of_advertising_youve/
TIFU by not doing anything as an undergraduate to get references as I apply for graduate school.
http://www.reddit.com/r/tifu/comments/2pkz5o/tifu_by_not_doing_anything_as_an_undergraduate_to/
Veteran Crossing Guard Gets a New Ride
http://www.wfaa.com/story/features/2014/12/15/beloved-crossing-guard-gets-awesome-new-ride/20459907/
My 8 month old puppy saw his first snow ever and I'm pretty sure he's going to have a wonderful time here in Finland.
http://i.imgur.com/AsWZCuY.jpg
[WP] During an exploration project, a group finds an abandoned Santa's Workshop at the North Pole.
http://www.reddit.com/r/WritingPrompts/comments/2pkz1n/wp_during_an_exploration_project_a_group_finds_an/
Litchis from Reunion Island .
http://imgur.com/gallery/8wzc2
A crazy photo of the campus of Ohio University.
http://i.imgur.com/hbMGt7P.jpg
TIL Jeremy Clarkson was given the first ever Paddington Bear toy when his parents Shirley and Eddie Clarkson created it in 1972
http://en.m.wikipedia.org/wiki/Paddington_Bear
Afghans fighting in Syria alongside regime
http://www.dailystar.com.lb/News/Middle-East/2014/Dec-17/281400-afghans-fighting-in-syria-alongside-regime-activists.ashx
Anyone know this song?
http://www.reddit.com/r/Music/comments/2pkz78/anyone_know_this_song/
My best friend and I have still not seen Frozen...
http://imgur.com/UxY5hpY
Curiosity Rover Drills Into Mars Rock, Finds Water
http://news.yahoo.com/curiosity-rover-drills-mars-rock-finds-water-122321635.html
What a view to wake up to...
http://i.imgur.com/qukCSon.jpg
Russia invites North Korea leader, Kim Jong Un, for visit
http://www.reuters.com/article/2014/12/17/us-northkorea-russia-idUSKBN0JV09E20141217
Be prepared! The Christmas-noobs are coming...
http://i.imgur.com/cuJgWXd.jpg
I love sleep because it's like a time machine to breakfast.
http://www.reddit.com/r/Showerthoughts/comments/2pjxxk/i_love_sleep_because_its_like_a_time_machine_to/
The Bend in Oregon [1998x1331] OC
http://imgur.com/uyu4s3S
Garbage men of Reddit, what's the most illegal, strange or valuable thing you have seen while gathering people's trash?
http://www.reddit.com/r/AskReddit/comments/2pjsf9/garbage_men_of_reddit_whats_the_most_illegal/
This colored quarter I got back in my change today.
http://i.imgur.com/3sGrG0z.jpg
TIL that Weird Al's parents both died on the same day of carbon monoxide poisoning. Only hours later, he performed a concert and started by saying "since my music had helped many of my fans through tough times, maybe it would work for me as well."
http://en.wikipedia.org/wiki/%22Weird_Al%22_Yankovic#Refused_parodies
Beetle suplexes the fuck out of a scorpion
http://i.imgur.com/txgYoV4.gif
My boss brought her new puppy in to work today!
http://i.imgur.com/wt8m45J.jpg
Going full retard [0:08]
http://youtu.be/jSeuDhtCBwY
"Copper kills everything": A Copper Bedrail Could Cut Back On Infections For Hospital Patients
http://www.npr.org/blogs/goatsandsoda/2014/12/15/369931598/a-copper-bedrail-could-cut-back-on-infections-for-hospital-patients
Police officer won't be charged with the 4-5 pounds of marijuana found in his home.
http://www.contracostatimes.com/breaking-news/ci_27148829/richmond-officer-found-marijuana-home-wont-face-charges
I asked JK Rowling about Jewish Wizards on Twitter, She responded.
https://twitter.com/jk_rowling/status/544946669448867841
LPT: If you are leaving your house unattended for a couple days, shut off water main.
http://www.reddit.com/r/LifeProTips/comments/2pjrl4/lpt_if_you_are_leaving_your_house_unattended_for/
Humans May Be Causing the Sixth Great Extinction in Half a Billion Years
https://news.vice.com/article/humans-may-be-causing-the-sixth-great-extinction-in-half-a-billion-years?utm_source=vicenewsaunztwitter
2014: The Year In Tiny Desk Concerts. I could watch these all day.
http://www.npr.org/2014/12/16/371184887/2014-the-year-in-tiny-desk-concerts?utm_source=facebook.com&utm_medium=social&utm_campaign=npr&utm_term=nprnews&utm_content=2042
[Image]The realist adjusts his sail.
http://imgur.com/PK1GSIm
"Blackfish" - Killer whales, an animals of great intelligence, yet some do not realize what goes on backstage in Seaworld. Warning to those haven't seen this- it can be disturbing. (2013) - [83:06]
http://www.dailymotion.com/video/x1iepl4_blackfish-find-out-what-really-happens-at-seaworld_animals
My dad with his dog, Shadow. 1983
http://imgur.com/8sXbSFV
PsBattle: Anna Kendrick holding a sheet of paper
http://i.imgur.com/ewJPmME.jpg
Cold and damp? That's stewin' weather! So I made Irish stew with garlicky mashed potatoes.
http://i.imgur.com/0Iey5nn.jpg
Seth Rogen and James Franco Cancel All Media Appearances for ‘The Interview’
http://variety.com/2014/film/news/seth-rogen-and-james-franco-cancel-all-media-appearances-for-the-interview-1201380917/

In [4]: