2011

This post is a follow on from my previous post about the oAuth protocol in general in Ruby. Here I detail how to let a user authenticate themselves via Twitter to your web app, request a list of people they’re following (a GET operation) and how to follow a specific user (a POST operation). As this is a follow-on from an earlier post, the following functions used here can be seen there:

• params(consumer_key)
• generate_nonce(size)
• signature_base_string(method, uri, params)
• url_encode(string)
• sign(key, base_string)
• header(params)
• request_data(header, base_uri, method, post_data=nil)

The Log-in Process

When you sign up for an API key from Twitter, Yelp etc. you’ll be given a Consumer Key and a Consumer Secret. You may also be given an Access Token and an Access Token secret, but that’s for you logging into your own account. If that’s all you wish to do, you can skip this section.

So, if you want to let a user log in to their Twitter (or whatever) account via your site, you need to get an access token. The process for this is as follows:

1. You request a unique Request Token from Twitter for your ‘Log-in with Twitter’ button
2. You use this request token, as well as where on your site you want the user to be re-directed back to after they’re authenticated, to build the URL the button points to
3. They click the ‘Log-in with Twitter’ button on your site
4. The user is brought to Twitter where they enter their username and password
5. Twitter re-directs them back to your site, to the URL you gave them in step 2
6. Back at your site, you’ll now have an oAuth verifier
7. This can be used to get the user’s Twitter user info, which has been authenticated by Twitter

Step 1: Getting an oAuth request token for your current session

Twitter’s URL for getting a request token from is https://api.twitter.com/oauth/request_token. The request for the token contains a number of parameters, which are then combined with the URL you’re going to be sending the data to and the method of your request (i.e. GET, POST etc.) to generate what’s called a base signature. At this point you don’t have an access token, so your signing key is simply your consumer secret followed by an ‘&‘. Using the functions I’ve mentioned earlier, this can be done as follows:

Steps 2-5: Authenticating with Twitter

Once you have your request/access token, the URL to direct your user to is simply:

You can use standard <%= link_to 'Login', @login_url %> to generate a HTML anchor in your ERB template or whatever you choose. The user will then be directed to Twitter, where they enter their log-in details and get directed back to your callback URL.

Steps 6-7: Getting the user’s info

When the user is directed back to your site, you’ll be sent an oauth_verifier. This can be obtained via Rails in your callback URL’s corresponding controller, via params[:oauth_verifier]. You need to use this to request their user info, as a final check before they’re fully logged in. This results in a new auth token (or what Twitter calls an access token) and auth token secret, which should replace your previous stored values. It is assumed the code below is stored in a model class, where you need to pass the auth verifier from your controller.

data will now contain an array with things such as screen_name, user_id etc. of the user who just logged in. It’ll also contain a new oauth_token and oauth_token_secret, which should be saved as they’ll be used again.

Now you have a fully validated user, who has authenticated you to access information on Twitter via their Twitter account. So now, let’s access some of that info.

Getting the people a user is following (a GET request)

The process for all GET requests is pretty similar and roughly follows what we’ve done before. We have our array of standard parameters. To this, each of the GET parameters are passed. We use our access token and consumer key & secret to generate our oAuth signature, make the request and parse the response.

Following a specific user (a POST request)

The process for a POST is pretty similar, the only difference being how you handle the parameters to the request. In the example below, I’m assuming you have the user ID of the person you want to follow and that it’s stored in a variable called followee_user_id.

So, assuming that was successful, the user should now be following the person with user ID followee_user_id.

Conclusion

Hopefully this will fill in some of the gaps in Twitter’s documentation. When coding this I found plenty of instances where the documentation would say something like “now sign the key”, without actually telling you how to sign it! Very confusing indeed.

Disclaimer: I obviously wrote all this in proper Ruby on Rails classes and controllers, but have extracted the code out here to be presented in modular form. Thus, what’s here is not fully tested, or even elegant, but there should be enough to figure out what you need to do.

Read more...

Recently I decided to figure out what the oAuth 1.0 protocol was all about and try to implement it in Ruby, as part of a way a) to practice by Ruby on Rails, b) have a look at the Twitter API and c) use both to get an understanding of how websites let you log in/comment via your Facebook/Twitter/Google etc. account, for potential use in future web projects. Sure there’s an oAuth gem out there, and a Twitter gem and probably a generic login gem (or if not, there’s an idea!) but I thought I’d get more out of the process by coding everything from scratch. So, first up is a generic overview of the oAuth protocol.

Each request will have a method (i.e. GET, POST etc.), a base URL to handle the request at the source site (Twitter, Yelp etc.) and a certain set of parameters. Every request I’ve dealt with has had the same 5 parameters, along with various other ones specific to the request you’re making. So, in Ruby, I’d have something like:

Next, you’ll need to add in any extra parameters to your params hash, e.g. your access token if you have it, and then combine all the above to generate a base string:

Next up, you need to generate a signing key, which is a combination of your consumer secret and your access token for the current session, if you have one at this stage (you may not, if the user still hasn’t logged in yet: in that case, a blank string will suffice). With this signing key, you sign your signature base string to get your oauth signature:

At this point, you’ve all your info nicely encoded in the oauth_signature using your private consumer secret. So, in a kind of public/private key partnership, you need to give the service your public consumer key, so it can validate the encoding of the oauth_signature at the destination:

So, you’re nearly ready to make your oAuth request. One final thing: all these parameters need to go into the Authorization line in your HTTP header, which is simply a matter of generating another string, as well as indicating you’re using oAuth:

So, to make your HTTP request, I wrote a generic function (request_data) that will do either a GET or a POST, make the request and return the response body:

And there you go. You should have a response from the API in whatever format you requested, be it JSON, XML or whatever. In my next post, I’ll explain how to use the above specifically for the Twitter API.

Read more...

Last month I resigned my post from daft.ie and am about to take up a development position with dotMobi, a leading mobile Internet services company. I had done interviews with several places, while reading plenty of articles on various other companies’ hiring practices. At the end of these couple of months, I came to the conclusion that in all my professional career, Daft had the best hiring process for a developer that I had ever both experienced and read about.

The process

Initial interview

First up, as with all jobs, you submit your resume/CV. This is then screened over by the development manager, who invites adequate candidates to a first-stage interview. This interview consists of him and another developer, basically going through the CV with the candidate, asking some general programming related questions, all just to try and get a better overview of the person, ensuring they pass the “no jerks” policy.

Programming Assignment

This is the main beauty of their hiring process, and is discussed in more details below. Candidates are given an assignment to work on in their own time at home, typically over a weekend. The assignment asks the user build a simple web form, with one text input to take free-text search strings that a user might enter when searching for property. They even give sample queries, such as “2 beds for sale in Dublin”. This string is to be submitted to a PHP script where all the necessary data is to be parsed from the string. Once parsed, a call is to be made to the Daft database via their API in order to get a list of results back. This result set is then displayed to the user. Finally, they need to write-up their code, explaining what’s going on in simple English.

Second Interview

Assuming you do a good job in the assignment, you’re then invited back for a more formal interview with senior management (company of about 60 people), to discuss career aspirations, salary requirements etc.

Why I think this is great

Skills it shows

As mentioned earlier, the key part of the process is the programming assignment. First up, the initial entry form and data submission. Here the user can show their knowledge of web security, CSS, (X)HTML standards and more. While none of these are actual requirements, it’s an easy place to show existing, fundamental web knowledge.

Once the data’s at the server, cleaned and verified, relevant data blocks (e.g. area, for sale/to let, number of bedrooms etc.) need to be parsed from the string. There are a number of different ways to do this and gives the developer plenty of scope for flair. A certain amount of analysis is required here also, mapping the domain of real estate to known keywords and looking for these in the string.

You also need to try and spot any relevant areas (e.g. County Dublin, Galway City etc.) the user may have entered. Without going into too much detail (in case potential candidates are reading this), there’s a key optimisation when doing this via the API and it’s a great way to spot those who have been programming seriously for a few years. Even if a programmer doesn’t implement this optimisation, they should still be aware of the problem it solves; this awareness can come across in their second interview, which is also accepted.

Once you know what the user is searching for, you then need to build a call to Daft’s SOAP API. This is good because it usually requires the candidate install PHP’s SOAP interface. While not overly complicated, it’s not trivial either and shows that the candidate is able to play around on a UNIX machine. Using the API also illustrates a candidates ability to follow documentation and quickly get to grips with a new system. Finally, by building a query from your parsed data that can be understood by the API, the candidate again has a chance to show some good, tidy code.

After pulling data from the API, the candidate again has a chance to show off XHTML/CSS skills to display the results set. Further ‘enhancements’ could be shown by doing all this over AJAX, without a page reload.

Finally, the candidate needs to document their code. This will illustrate communication skills, their attention to detail, writing skills (grammar, spelling) and more. Everyone in the software industry knows that communication is a large part of software development and it’s important to have good communicators on your team. Good code followed by an average write-up can often be less preferable than average code (which is generally easier to improve on) with a good write-up.

Why it’s better than others

While reading about other people’s interview experiences, one thing that stood out as annoying most developers is white-board coding, i.e. where you’re required to write some code on the fly, either on a white board or piece of paper. This a completely unrealistic situation and is asking a programmer to do something way outside their normal environment. Similar to this would be writing a program in the employer’s office in the space of an hour to 90 minutes; again this is wholly unrealistic as programmers are used to their own environment, have their own tools etc. The beauty of the Daft assignment is that it’s too tricky to do in less than two hours, plus they let you do it at home on your own machine. Yes, this does allow for potential plagiarism, but once you’re offered a job, you’re put on 6 months initial probation, so if you don’t code your assignment it’ll soon become obvious and they’ll have scope to not make you permanent.

Another great thing about it is that there’s no generic analytical questions such as “Why are man-holes round?” or “How many ties were sold in New York City last year?”. The value in asking these questions is surely waning these days, as they are easy enough to prepare for and people have learnt to expect them. Good programmers need to be good at analysis, but enough information about a candidates skills can be gleaned from how they approach the assignment.

Finally, there’s no over-bloated 7 round interview process. From the 2 interviews a candidate does, coupled with the assignment, it’s very easy to see what a person is like and that they’ve a decent level of programming competency.

One flaw

Not everything’s perfect and after chatting to my manager about the interview process, in preparation for this blog post, he pointed out that it’s poor at highlighting any “rockstar” programmers, as well as any existing systems administration skills. So, if you have 2 people competing against each other, one a decent programmer, the other amazing, it can be hard to differentiate between the 2; what’ll happen more often than not is that the ‘nicer’ one will be offered the job first.

Conclusion

From the above, it should be easy to see that current interview processes are far too complicated. 2 interviews will always be the least that’s required, but when you throw in a slightly complicated programming assignment, there shouldn’t be need for much more. The assignment’s a good way to test programming competency while the interviews are good to test a person’s likeability and career aspirations.

Read more...

Whenever anything generates a MySQL error at work, the whole technical team gets an email about it, with full debug info. We recently got one for an area that I look after, so it was up to me to investigate. The title of the error was a slightly cryptic

Error: Duplicate entry ‘0’ for key 1

Clearly this had something to do with the primary key, which was a simple unique integer ID. As this is a pretty large table, I had a feeling that we had reached the upper limit of what could be stored for the field type (MySQL’s MEDIUMINT, signed). Looking at the maximum value for the ID, I saw it was 8388607; according to this table on the MySQL website, this value is the maximum that can be stored in that type of field, so this was clearly the problem. (N.B. an auto-incrementing field shouldn’t ever be defined as signed, as you’ll never go into the negative indices, but that was done before I came along!)

The solution? Surely a simple readjusting of the key to be unsigned, or to further future proofing, changing to the larger INT… That’s what I thought and quickly (in test first, of course!) did a

This should then change the maximum value allowed in the ID field to 16777215. So, to test this, I went to insert a row, specifying NULL for the primary key value. However, I still got the same error. Doing a DESC on the table told me that the ID field had been changed to MEDIUMINT UNSIGNED correctly, so that wasn’t the issue. After further research I determined that what happend was that MySQL’s internal counter for that auto increment field was still set to 0, due to the rollover caused after reaching the maximum integer value. To overcome this, you need to point the internal counter back to where it should be, i.e. the current maximum value of your ID field, as follows:

So, you set it to a value that’s one greater than the current maximum ID.

Read more...

UPDATE: I’ve recently been asked by entertainment.ie to stop scraping thier licensed data, to which I’ve duly agreed. Thus, the app is no longer live. However, the lessons learned below are still valid. END

Haven’t posted for a while, mainly because I’ve been busy trying to teach myself Ruby on Rails, so haven’t created anything new in my spare time. Have come up with a few interesting fixes/ways to do things in relation to Facebook code in work though, so will hopefully do a post on those in the near future.
In the meantime, I wrote a very small little web application last week using the pretty cool jQuery Mobile framework. The app, called What’s on now (link removed, see UPDATE above), is simply a list of what’s on now and next on Ireland’s 17 basic cable channels. The data is pulled from a similar page on entertainment.ie, with the unnecessary channels filtered out. At the moment the app is pretty simple, but I plan to add to it over time (e.g. to add an option for the non-basic channels), updating this article as I go.

I did this mainly to have a quick go with jQuery Mobile, to see what it could do. I could’ve used PHP and built a mobile-sized HTML page, but it’s always good to try new things! Most of the development was pretty straight forward; however, because the data is retrieved dynamically every time the page is loaded, there’s a couple of tricks you need to apply to get the jQuery Mobile framework to do it’s magic.

The main page you’re greeted with is simply a glorified un-ordered list of programmes, with separator list items to distinguish the channels. I’m not going to go into the details of how you need to structure a page (see links at the end of this post) but here’s a snippet of the HTML:

When the page loads, ul#channels is empty. The data is called via a jQuery GET, which gets over cross-domain restrictions by using YUI, thanks to the Cross-Domain AJAX mod. The relevant data is filtered out and formatted and each of the li‘s are built and inserted into #channels. At this point, if you look at the page in your browser, it’ll still look like an ordinary list – you need to tell jQuery to work it’s magic on the dynamically created data. In this instance it’s done as follows:

Once I had my list of programmes, I thought I may as well add the info for each program, seeing as I already had the data at my disposal. The route I decided to go down here was to create new ‘page’ div‘s for each program, each one having it’s own ID, then link to each of these pages from the li‘s. Again, the scope of building one of these pages is beyond this blog post and well documented elsewhere, but here’s a quick sample:

This is simply added to the body using $('body').append(page); (where the variable page is a string of HTML such as the above). So, again here you need to tell jQuery Mobile that you’ve added a new page, so it can do it’s magic. This is achieved by the one simple line:

Hopefully this post will clear up a couple of things for anyone using jQuery Mobile with dynamically generated data. As I promised, here are some links of articles that helped me get a better understanding of the framework:

• http://msdn.microsoft.com/en-US/scriptjunkie/hh144955.aspx
• http://jquerymobile.com/demos/1.0a4.1/
• http://www.elated.com/articles/jquery-mobile-what-can-it-do-for-you/

Full implementation code (UPDATE 2)

I was requested by @danny to post the full source code, seeing as the app is actually no longer available, so I’ve decided to put most of it here. I excluded some of the scrape JS (as indicated in the code comments) to prevent the app being re-used somewhere else.

So, first up, is the initial HTML page, with header and footer blocks, an empty content block and the JS and CSS includes:

Next up is the javascript file, whatson.js in the above. Don’t forget, the $.ajax call has been overwritten by the Cross-Domain AJAX plug-in I mentioned earlier. Addtionally, I’ve used a few functions from php.js. to replicate this functionality in JS, namely str_replace, in_array and trim. I’ve excluded them here but they can be found in the php.js website.

Also, just to re-clarify, the page i was scraping had a list of channels and programs that were on now (prog1) and next (prog2).

I realise this code could be alot cleaner, but the app was still in it’s early stages before I was asked to take it down, thus I haven’t spent time tidying it up. Hopefully there’s enough here to figure out how to do what you need!