How I learned to Code

February 08, 2017

    

5 min read

  
print('Hello Web Scrapers')

My road into programming has been like most of my life. Full of sudden changes. I'm bound to say that it's because I thrive in a fast-paced environment and I like to move fast and break things. Facts however don't care about our intentions, the truth is that I get both bored and excited very easily.

First real encounter with programming came with an MITx course which taught the introduction to Computer Science using Python. Awesome course and learned tons doing it. However I still didn't know how to set up a web server nor mobile app.

Starting Web Development

It all began with a Gem, a Ruby colored gem riding the Rails to API heaven. My puns suck and I'm sorry you have to read them. After some months of learning Ruby on Rails I went to my local Django meetup and decided to take it up.

Django was cool. The truth is I hated the regExp routes and it all seemed like Rails with a litle more overhead. Community kept me through though. I also learned web scrapping with Python and fell in love. There's a special kind of love of seeing your program interact with a website...and work!

First Job

This is actually a fun story. I was making good progress on Ruby on Rails and knew my way around bootstrap enough to make a decent. For decent I meant of course usable as my design skills sucked even more back then.

I applied to the local companies in Barranquilla and didn't hear back even though I had found a typo in one company's application.

I was longing for cash. Any cash.

I sold my bicycle for living funds. For living I mean eating out, transportation and buying beer. Then when funds started running out I did the thing I've always dreaded. Joined a call center selling stuff.

On my second day of training at the call center. I went out with a friend through CouchSurfing. We were having a beer and a hamburguer at a bar, speaking english as my Swedish is as good as my Chinese.

We sit on a table next to some guys and they engaged us asking us where we were from. I started talking to the guy next to me and when he asked what I did I told him I was working at a call center and that I did websites on the site, hoping for a potential client and my break in the industry.

He managed a transportation services outsourcing company and they were starting their IT team doing some real development. He asked me

Are you good? Do you want to work with us?

I said yes to both.

The CallCenter Dilema

Bear in mind that I had told my parents that I was on the call center track to make the big bucks selling overpriced energy plans to people on the East Coast.

The training drills were fun as I had a good practice of my english prononciation. But scamming people was really not my jam even though they wanted us to believe people were saving money thanks to us.

I talked it out with my parents and they decided that even if the developer job was very likely that I should at least finish the call center training week.

While I was walking the next day to the bus stop. I had to make a quick decision about continuing my training because I didn't want to take a long, hot, bus ride (20 minutes but ok). After walking for more than two minutes I was standing just on the edge of the bus stop.

I wasn't going to the call center.

I went instead to a Café. Juan Valdez on 82 and 52 in Barranquilla. Calm, cool place that supplied me with air conditioning, wifi, and a nice assortment of breads and coffe's. I always asked for the Irish Capuccino anyway, not like I care about diversity.

So on. That same day I made a resume which wasn't tailored for medieval workplaces call centers.

I sent my CV that very same day. It took about 3 hours of good pounding to get it decent. And it was horrible.

Inverview Time

Okay so this is day 4 of call center training or Thursday. I arrive at the Cafe in the morning and get called in to interview on the afternoon.

Turns out they had quite a laugh about my resume. I found out years later that it was because I included my volunteer for the FIFA Under 20 World Cup in Colombia...only the job description was a little off. One of my responsibilities included taking food confiscated at the entrance. Guess humor is not a good fit for the resume world just yet.

My story kept straight. I arrived after 3 at home and went out to the cafe at 7:30 in the morning. It was still going to the job training, just a different job, which I had not been ofered yet.

Post-interview: Web scraping

After the inverview I was told I should send the resume over the internet and do some english tests that were required. My past job was english-spanish interpreter so english was breeze.

However I found something interesting when uploading my resume. It uploaded it as a post on php site. And it was public. I could see every person who had applied to the company for the last 3 months.

Hacker me decided it was too good of an opportunity to pass up. Not for the data of course, I don't care. I wanted to show them how this could be explited.

Enter script scraper.

import requests
from lxml import html

#Parameters for logging in
session_requests = requests.session()
login_url = "INSERT_URL"
payload = {
    "username": "mateomgj",
    "password": "SECRET_MY_FRIEND",
}

#Login
result = session_requests.post(login_url,
                               data=payload,
                               headers= dict(referer=login_url))
#Scraping prep
scrape_url="http://WEBSITE_TO_SCRATE.com/THIS_IS_AN_EXAMPLE/blog/index.php?courseid=0&blogpage=LOOK_FOR_PAGE"
target_names=['john', 'fitzgerald', 'kennedy', 'lyndon', 'baines', 'johnson'] #Example names
resume_container = []

#Range depends on the number of pages
for i in range(23):
    query = session_requests.get(scrape_url+str(i))
    tree = html.fromstring(query.content)
    for name in target_names:
        if name in tree.text_content().lower():
            print("Found "+str(name)+" at "+str(i))
            #Get all resumes links
            resume_page = tree.xpath(".//div[@class='attachments']//a/@href") #XPath may vary
            resume_container.append(resume_page)

#This will write all resume's on a page containing a giving target, not only the targets.
#For getting only the target. Iterate further into Resume
target = open('results.html', 'w')
for page in resume_container:
    for resume in page:
        target.write(resume+"\n")


target.close()

In a nutshell. With this script I could type target names and it would tell me on which page those names appeared and a link to their resume.

I edited it out and added names of US presidents in the 60's. I was reading a LBJ book back then so go figure.

The Wait

Two weeks had gone by and I was going insane. Not a no yet, but still not a yes. The worst feeling, just when I could grasp the job by its hands yet it was fleeting.

I got the job in the end and ended up being one of the greatest learning experiences of my life. They let me make a lot of architectural decisions about the code and my input was very well regarded.