Paul Swartz’s PCF Devblog

I hacked together a quick feature that I’ve been wanting for a while: RSS feeds for the guide. Right now, there are three of them:

https://www.miroguide.com/feeds/features/
https://www.miroguide.com/feeds/new/
]https://www.miroguide.com/feeds/categories/[category name

Let me know what you think!

Last week, I spent most of my time working with Dean on a redesign of the channel details page. We have some new features in Miro (like the ability to download invididual videos) that aren’t being used by the Guide, so we’re trying to work them in.

I spent some time working on making subscriptions faster. Previously, the subscription-hit page added a row directly into the subscription table, which was getting slow sometimes (1,201,922 subscriptions total). Now, they go into a holding table which is dumped into to main table when the popularity statistics are updated (every 5 minutes). From a quick test, it seems much much faster, and my timing statistics back that up.

Matt moved the Guide over to the new server so that Trac and Videobomb won’t interfere with it, so I did a little bit of work figuring out what that needed, and setting things up once it moved over. It certainly feels much faster.

This week I’ve been doing some more work on making the Guide faster. With the new co-branded versions being released in the near future and an API in the not-as-near future, we think that the Guide could be getting a lot more traffic and so it’ll need to be faster. Right now I’m working on putting a lot of the non-frequently-changed data into memcached.

I’m taking today-Friday off, but then I’ll be finishing up the caching stuff and moving back to the details page redesign.

• Wednesday: fixed some Unicode errors on the search and profile pages. Also, finished up the new caching for the channel details page and the featured queue by adding tests.
• Thursday: Unicode errors on the tags page, and some changes to the landing page
• Friday-Monday: Adult flagging (see this post). Also, some preparation for Miro 1.0: a static page in case the Guide got overloaded.

Next week: a redesign of the channel details page and a Data API.

We just rolled out a couple new features onto the Guide.

1. Featured Queue: This is mostly relevant for moderators. Previously, to feature a channel on the front page, moderators had to manually unfeature an older channel before featuring a new one. Now, when a channel is featured, it goes into a queue with all the other channels featured by moderators. Then, once a day, the oldest featured channel is removed from the front page (but still visible on the Featured page) and replaced by an entry from the queue. It picks the channel round-robin style so that all the moderators get channels on the front page.
2. Adult Flagging: This is an oft-requested featured. Now moderators will be able to flag channels as Adult and they’ll be filtered (by default) from all views. If you want to see them, then you can visit the Adult category and agree to see them.

Matt moved the database onto its own server, which along with some new caching code has sped the site up a lot. Hopefully it’ll be able to handle the spike when Miro 1.0 comes out.

One thing that I’ve been thinking a lot about (esp. now that Miro is nearing 1.0) is an API for the Guide. I’d love for the Guide to be more directly useful from Miro (think rating channels, getting recommendations, etc), but also more useful for other projects. After all, the Guide is a great resource for Internet TV channels, regardless of the software ones uses.
So I’m posing a question to the developers/users out there: what would you like to see in a Guide API? What would you like to be able to do/access? And how would you like to be able to access it?

On Wednesday, I spent the day finishing up the Landing page. It didn’t end up quite like Dean and I wanted due to some issues with the video, but I think it’s still good for the new users to get an introduction to Miro.

Thursday and Friday were spent working on the new featured channel queue. It’s not live yet, but it should make the featured channel section more useful for everyone. Now when a super-moderator features a channel, it goes into a queue. The featured section on the front page is updated once per day, pulling a channel from the queue round-robin style so that everyone gets a chance to see their channel on the front page.

This week I’ve been working on making the Guide faster in preparation for the eventual 1.0 release of Miro. The database has been getting more and more use and so I’m trying to add more caching to keep the guide going fast.

That’s what I’ll be working on for the rest of this week, along with some visual tweaks. If I’ve got time, I’ll be working on #9161, adding a category for Adult and a category for Sexuality channels.

I’ve been reading some of the architecture articles on High Scalability (YouTube, Twitter, Amazon, Flickr) and while the descriptions of hardware aren’t particularly useful to me (Miro Guide is one, about to be two, servers; not that big), but a lot of the tips and lessons learned I think may be applicable.

• Keep it Simple: I was a victim of this earlier, trying to make a system that was too complicated.
• Don’t make the database the central bottleneck: We’re running into this problem now, where sometimes a simple insert will time out because the database is being hit so much.
• Denormalize: I haven’t done this, nor do I have plans to in the near future, but it’s a least something to think about.
• Avoid complex joins: Nor this, but it may be becoming a problem. The next item may help with this
• Cache everything: We don’t do this very well, and it’s one of my goals. We just added a Squid server, so giving some hints to that about what’s changed will be good. Also, using memcached a bit more.
• Make your website an open service by creating an API. After the 1.0 release, this is one of the big features I want to work on. I think it’ll make Miro Guide a lot more useful both inside and outside of Miro and allow some refactoring of the actual guide,
• Measure, measure, measure: I starting doing this a couple weeks ago. Now I get an daily e-mail saying what parts of the Guide are running slowly. It’s both a good way to see what needs work, but also how well the caching is working.
• Abstraction: again, something that I’d like to do in conjunction with the API.

This wasn’t a particularly productive week for me. Mostly I was working on the landing page, fixing some small issues and doing a new layout for the channel subscription section. I did a lot of work trying to make the test cases for guide work again, and they’re almost there. I’ve just got a couple hard-to-diagnose bugs. I also added a couple new tests for the newly-approved channel queuing feature. I’ve also made a little progress on the featured queue, but mostly that’s waiting on the test cases passing again so I can add tests for that before I start in.
Today I was working on an extension to the Guide’s error reporting mechanism. Currently, errors are logged to a file which I check periodically, but mostly I’m notified by Dean who’s notified by users. Now those errors are mailed to me, along with a bit of context so that Dean doesn’t have to play messenger.
Next week, I’ll be working on the landing page, a quick redesign of the error page to make it friendlier, the tests, and then the featured channel queue.

Most of what I’ve done this week was for the new landing page. A quick summary: it’s a three-step page: 1) A video introducing the user to Miro 2) A way to get some channels they may enjoy 3) some FAQs. Embedding the video was difficult, but I think I got it working on Windows and Mac. I don’t think it’ll work for GTK, so I’m not trying anymore.
The FAQ section is working perfectly, and I’ll be finishing up with the subscription section this week.

I also got two smaller features onto the Guide. First, there’s a new notification section at the top of pages. This will let us announce things in an unobtrusive way, and also remind users of things they’re supposed to do. Second, there’s a new queuing feature for the new channels. Instead of being dropped onto the front page and new channels pages all at once when moderators approve them, they’re added one per hour. This gives them more time on the front page and therefore more exposure.

This last week was less productive than I would have liked. I’ve been fighting off a cold, for the most part unsuccessfully. However, I have managed to get some work done on the Guide. (Features may not be on https://www.miroguide.com/ yet, but they will be soon)

• Supermoderators can now send e-mails to the owners of featured channels through the guide
• Statistics for channel owners: on the channels by user page, each channel has some simple statistics on how many people have subscribed today/this month and what rank that channel is at.
• Fixed a bug with the rating stars flashing the non-jQuery version, and a bug on Safari where the stars would be all yellow even without a rating

This week, I’ll be working on:

• Queuing for newly approved channels, so that they don’t appear in the guide all at once
• A system for displaying notifications at the top of the page, like the message that are currently there for announcing ratings and reminding users to approve their accounts
• Some small frontend speedups (enabling gzip compression, CSS sprites)
• The big feature: recommendations based on ratings.