Speeding up “Feed Them Social” for District 101

The District 101 Toastmasters website uses the Feed Them Social plugin to add our Facebook and Twitter feeds to our homepage.

It works well, but it can be slow – if it needed to go to Facebook to update its data, it could take as much as 10 seconds to build the page; even if data was in the plugin’s cache, it took a couple of seconds to build the HTML. You can see the shortcodes I had on the home page below.

I thought about writing my own program to mine the Facebook Graph and the Twitter Feed. It would let me produce exactly what I needed, but it would be a perpetual maintenance headache (and I don’t plan to be Webmaster forever). I needed a better answer.

And then it struck me – if I could figure out a way to use the plugin as part of a batch process and save the HTML, I could get the plugin’s processing time out of the critical path, and I wouldn’t have to keep up with changes to the Facebook and Twitter APIs. Here’s what I did:

• Created a special version of the home page, with the calls to the plugin in the proper place in the page layout (just in case it mattered), but with none of the other features of the home page.
• Surrounded the calls to the plugin on the special page with flag lines so I could easily pull out the HTML generated by the plugin, using a simple Python program.
• Made the special page password-protected in WordPress to keep it out of the visible menu structure shown to site visitors.
• Wrote a script to be called periodically by cron that:
  • used curl to fetch the special page
  • ran the Python program to extract the HTML from Feed Them Social
  • saved the result as a file
• Replaced the calls to the plugin on the homepage with a couple of lines of PHP to include the saved file as part of the homepage

The result: the homepage loads at least 3.5 seconds faster and looks the same.

Here’s the relevant part of the special page (the calls to FTS are the same as those I used to have on the homepage).

Start Flag
[fts_facebook id=d101tm posts_displayed=page_only type=page]
[fts_twitter twitter_name=d101tm]
End Flag

Here’s the shell script:

# Update the cached static file for social media from Feed Them Social
mydir=$PWD
secret=password for the WordPress page
cd ~/files/social
# The next operation will take a while, so we write to a temporary file
outfile=newfts$$.html
curl -sL "http://d101tm.org/wp-login.php?action=postpass" -d "post_password=$secret" -e "http://d101tm.org/path to the special page/" -b /dev/null | $mydir/updatefts.py > $outfile
# If all went well, we can replace the real file
if [ -n $outfile ]; then
    mv $outfile fts.html
else
    echo $outfile is zero length!
    exit 1
fi

And here’s the Python program:

#!/usr/bin/env python
""" Extract the HTML generated by Feed Them Social from stdin, write to stdout"""

import sys
startflag = 'Something unlikely to appear on the page'
endflag = 'Something else unlikely to appear on the page'
havestart = False
haveend = False

def findendflag(s):
    """ Returns a tuple: (any data before the endflag,
                          whether the endflag was found) """
    if endflag in s:
        return (s.split(endflag,1)[0], True)
    else:
        return (s, False)

for l in sys.stdin.readlines():
    if not havestart:
        res = l.split(startflag,1)
        if len(res) > 1:
            havestart = True
            (l, haveend) = findendflag(res[1])
            sys.stdout.write(l)
    elif not haveend:
        (l, haveend) = findendflag(l)
        sys.stdout.write(l)