Last week, I spent most of my time working with Dean on a redesign of the channel details page. We have some new features in Miro (like the ability to download invididual videos) that aren’t being used by the Guide, so we’re trying to work them in.
I spent some time working on making subscriptions faster. Previously, the subscription-hit page added a row directly into the subscription table, which was getting slow sometimes (1,201,922 subscriptions total). Now, they go into a holding table which is dumped into to main table when the popularity statistics are updated (every 5 minutes). From a quick test, it seems much much faster, and my timing statistics back that up.
Matt moved the Guide over to the new server so that Trac and Videobomb won’t interfere with it, so I did a little bit of work figuring out what that needed, and setting things up once it moved over. It certainly feels much faster.
This week I’ve been doing some more work on making the Guide faster. With the new co-branded versions being released in the near future and an API in the not-as-near future, we think that the Guide could be getting a lot more traffic and so it’ll need to be faster. Right now I’m working on putting a lot of the non-frequently-changed data into memcached.
I’m taking today-Friday off, but then I’ll be finishing up the caching stuff and moving back to the details page redesign.
- Wednesday: fixed some Unicode errors on the search and profile pages. Also, finished up the new caching for the channel details page and the featured queue by adding tests.
- Thursday: Unicode errors on the tags page, and some changes to the landing page
- Friday-Monday: Adult flagging (see this post). Also, some preparation for Miro 1.0: a static page in case the Guide got overloaded.
Next week: a redesign of the channel details page and a Data API.
We just rolled out a couple new features onto the Guide.
- Featured Queue: This is mostly relevant for moderators. Previously, to feature a channel on the front page, moderators had to manually unfeature an older channel before featuring a new one. Now, when a channel is featured, it goes into a queue with all the other channels featured by moderators. Then, once a day, the oldest featured channel is removed from the front page (but still visible on the Featured page) and replaced by an entry from the queue. It picks the channel round-robin style so that all the moderators get channels on the front page.
- Adult Flagging: This is an oft-requested featured. Now moderators will be able to flag channels as Adult and they’ll be filtered (by default) from all views. If you want to see them, then you can visit the Adult category and agree to see them.
Matt moved the database onto its own server, which along with some new caching code has sped the site up a lot. Hopefully it’ll be able to handle the spike when Miro 1.0 comes out.
One thing that I’ve been thinking a lot about (esp. now that Miro is nearing 1.0) is an API for the Guide. I’d love for the Guide to be more directly useful from Miro (think rating channels, getting recommendations, etc), but also more useful for other projects. After all, the Guide is a great resource for Internet TV channels, regardless of the software ones uses.
So I’m posing a question to the developers/users out there: what would you like to see in a Guide API? What would you like to be able to do/access? And how would you like to be able to access it?
On Wednesday, I spent the day finishing up the Landing page. It didn’t end up quite like Dean and I wanted due to some issues with the video, but I think it’s still good for the new users to get an introduction to Miro.
Thursday and Friday were spent working on the new featured channel queue. It’s not live yet, but it should make the featured channel section more useful for everyone. Now when a super-moderator features a channel, it goes into a queue. The featured section on the front page is updated once per day, pulling a channel from the queue round-robin style so that everyone gets a chance to see their channel on the front page.
This week I’ve been working on making the Guide faster in preparation for the eventual 1.0 release of Miro. The database has been getting more and more use and so I’m trying to add more caching to keep the guide going fast.
That’s what I’ll be working on for the rest of this week, along with some visual tweaks. If I’ve got time, I’ll be working on #9161, adding a category for Adult and a category for Sexuality channels.
I’ve been reading some of the architecture articles on High Scalability (YouTube, Twitter, Amazon, Flickr) and while the descriptions of hardware aren’t particularly useful to me (Miro Guide is one, about to be two, servers; not that big), but a lot of the tips and lessons learned I think may be applicable.
- Keep it Simple: I was a victim of this earlier, trying to make a system that was too complicated.
- Don’t make the database the central bottleneck: We’re running into this problem now, where sometimes a simple insert will time out because the database is being hit so much.
- Denormalize: I haven’t done this, nor do I have plans to in the near future, but it’s a least something to think about.
- Avoid complex joins: Nor this, but it may be becoming a problem. The next item may help with this
- Cache everything: We don’t do this very well, and it’s one of my goals. We just added a Squid server, so giving some hints to that about what’s changed will be good. Also, using memcached a bit more.
- Make your website an open service by creating an API. After the 1.0 release, this is one of the big features I want to work on. I think it’ll make Miro Guide a lot more useful both inside and outside of Miro and allow some refactoring of the actual guide,
- Measure, measure, measure: I starting doing this a couple weeks ago. Now I get an daily e-mail saying what parts of the Guide are running slowly. It’s both a good way to see what needs work, but also how well the caching is working.
- Abstraction: again, something that I’d like to do in conjunction with the API.
