database related modules

Modules in the miro package namespace located in portable/ in the source tarball that contain storage or database related code.

Miro’s database is stored using sqlite in a file named sqlitedb.

Prior to Miro 2.5, the database consisted of two tables: dtv_variables and dtv_objects. The dtv_objects table had an id field and an object field that was a binary blob holding binary Python pickle objects.

As of Miro 2.5, the database is now a relational database with no Python pickles in it.

Note

We encourage outside tools to access and query this data.

The tables should be pretty self-explanatory. If you are confused about the data in the tables, look at the schema.py module which holds schema information.

As of Miro 2.5, this is the only way to access metadata stored in Miro.

storedatabase - crux of the database storage code

miro.storedatabase – Handle database storage.

This module does the reading/writing of our database to/from disk. It works with the schema module to validate the data that we read/write and with the upgradedatabase module to upgrade old database storages.

Datastorage is handled through SQLite. Each DDBObject class is stored in a separate table. Each attribute for that class is saved using a separate column.

Most columns are stored using SQLite datatypes (INTEGER, REAL, TEXT, DATETIME, etc.). However some of our python values, don’t have an equivalent (lists, dicts and timedelta objects). For those, we store the python representation of the object. This makes the column look similar to a JSON value, although not quite the same. The hope is that it will be human readable. We use the type pythonrepr to label these columns.

class miro.storedatabase.LiveStorage(path=None, object_schemas=None, schema_version=None)

Handles the storage of DDBObjects.

This class does basically two things:

1. Loads the initial object list (and runs database upgrades)
2. Handles updating the database based on changes to DDBObjects.

get_obj_by_id(id)

Get a particular DDBObject.

This will throw a KeyError if id is not in the database, or if the object for id has not been loaded yet.

remove_obj(obj)

Remove a DDBObject from disk.

update_obj(obj)

Update a DDBObject on disk.

upgrade_database()

Run any database upgrades that haven’t been run.

class miro.storedatabase.TimeModuleShadow

In Python 2.6, time.struct_time is a named tuple and evals poorly, so we have struct_time_shadow which takes the arguments that struct_time should have and returns a 9-tuple

exception miro.storedatabase.UpgradeDiskSpaceError

While upgrading the database, we ran out of disk space.

schema - Holds Schema objects

miro.schema – The schema module is responsible for defining what data in the database gets stored on disk.

The goals of this modules are:

• Clearly defining which data from DDBObjects gets stored and which doesn’t.
• Validating that all data we write can be read back in
• Making upgrades of the database schema as easy as possible

Module-level variables:

• object_schemas – Schemas to use with the current database.
• VERSION – Current schema version. If you change the schema you must bump this number and add a function in the databaseupgrade module.

Go to the bottom of this file for the current database schema.

class miro.schema.MultiClassObjectSchema

ObjectSchema where rows will be restored to different python classes.

Instead of the klass attribute, MultiClassObjectSchema should define 2 class methods: ddb_object_classes() returns the list of all classes that this schema works with and get_ddb_class() which inputs a dictionary containing db data and return which class we should use to restore that row.

class miro.schema.ObjectSchema

The schema to save/restore an object with. Object schema isn’t a SchemaItem, it’s the schema for an entire object.

Member variables:

• klass – the python class that this schema is for
• table_name – SQL table name to store the class in
• fields – list of (name, SchemaItem) pairs. One item for each attribute that should be stored to disk.

class miro.schema.SchemaBinary(noneOk=False)

Defines the SchemaBinary type for blobs.

class miro.schema.SchemaBool(noneOk=False)

Defines the SchemaBool type.

class miro.schema.SchemaDateTime(noneOk=False)

Defines the SchemaDateTime type.

class miro.schema.SchemaDict(keySchema, valueSchema, noneOk=False)

Special case of SchemaReprContainer that stores a simple dict

All keys and values must have the same type.

class miro.schema.SchemaFilename(noneOk=False)

Defines the SchemaFilename type.

class miro.schema.SchemaFloat(noneOk=False)

Defines the SchemaFloat type.

class miro.schema.SchemaInt(noneOk=False)

Defines the SchemaInt type.

class miro.schema.SchemaItem(noneOk=False)

SchemaItem represents a single attribute that gets stored on disk.

SchemaItem is an abstract class. Subclasses of SchemaItem such as SchemaAttr, SchemaList are used in actual object schemas.

Member variables:

• noneOk – specifies if None is a valid value for this attribute

validate(data)

Validate that data is a valid value for this SchemaItem.

validate is “dumb” when it comes to container types like SchemaList, etc. It only checks that the container is the right type, not its children. This isn’t a problem because saveObject() calls validate() recursively on all the data it saves, therefore validate doesn’t have to recursively validate things.

validateType(data, correctType)

Helper function that many subclasses use

class miro.schema.SchemaList(childSchema, noneOk=False)

Special case of SchemaReprContainer that stores a simple list

All values in the list must have the same type.

class miro.schema.SchemaObject(klass, noneOk=False)

SchemaObject type.

class miro.schema.SchemaReprContainer(noneOk=False)

SchemaItem saved using repr() to save nested lists, dicts and tuples that store simple types. The look is similar to JSON, but supports a couple different things, for example unicode and str values are distinct. The types that we support are bool, int, long, float, unicode, None and datetime. Dictionary keys can also be byte strings (AKA str types)

class miro.schema.SchemaSimpleItem(noneOk=False)

Base class for SchemaItems for simple python types.

class miro.schema.SchemaStatusContainer(noneOk=False)

Version of SchemaReprContainer that stores the status dict for RemoteDownloaders. It allows some values to be byte strings rather than unicode objects.

class miro.schema.SchemaString(noneOk=False)

Defines the SchemaString type.

class miro.schema.SchemaTimeDelta(noneOk=False)

Defines the SchemaTimeDelta type.

class miro.schema.SchemaURL(noneOk=False)

Defines the SchemaURL type.

exception miro.schema.ValidationError

Error thrown when we try to save invalid data.

exception miro.schema.ValidationWarning

Warning issued when we try to restore invalid data.

item - Item

miro.item – Holds Item class and related things.

class miro.item.FeedParserValues(entry)

Helper class to get values from feedparser entries

FeedParserValues objects inspect the FeedParserDict for the entry attribute for various attributes using in Item (entry_title, rss_id, url, etc...).

class miro.item.FileItem(*args, **kwargs)

An Item that exists as a local file

get_channel_category()

Get the category to use for the channel template.

This method is similar to get_state(), but has some subtle differences. get_state() is used by the download-item template and is usually more useful to determine what’s actually happening with an item. get_channel_category() is used by by the channel template to figure out which heading to put an item under.

• downloading and not-downloaded are grouped together as not-downloaded
• Items are always new if their feed hasn’t been marked as viewed after the item’s pub date. This is so that when a user gets a list of items and starts downloading them, the list doesn’t reorder itself.
• Child items match their parents for expiring, where in get_state, they always act as not expiring.

class miro.item.Item(*args, **kwargs)

An item corresponds to a single entry in a feed. It has a single url associated with it.

delete_files()

Stops downloading the item.

download(autodl=False)

Starts downloading the item.

download_progress()

Returns the download progress in absolute percentage [0.0 - 100.0].

find_new_children()

If this feed is a container item, walk through its directory and find any new children. Returns True if it found childern and ran signal_change().

fix_incorrect_torrent_subdir()

Up to revision 6257, torrent downloads were incorrectly being created in an extra subdirectory. This method “migrates” those torrent downloads to the correct layout.

from: /path/to/movies/foobar.mp4/foobar.mp4 to: /path/to/movies/foobar.mp4

getUandA()

Get whether this item is new, or newly-downloaded, or neither.

get_auto_downloaded()

Returns true iff item was auto downloaded.

get_duration_value()

Returns the length of the video in seconds.

get_expiration_time()

Returns the time when this item should expire.

Returns a datetime.datetime object, or None if it doesn’t expire.

get_feed()

Returns the feed this item came from.

get_link()

Returns the URL of the webpage associated with the item.

get_payment_link()

Returns the URL of the payment page associated with the item.

get_pub_date_parsed()

Returns the published date of the item as a datetime object.

get_release_date_obj()

Returns the date this video was released or when it was published.

get_seen()

Returns true iff video has been seen.

Note the difference between “viewed” and “seen”.

get_size_for_display()

Returns the size of the item to be displayed.

get_state()

Get the state of this item. The state will be on of the following:

• new – User has never seen this item
• not-downloaded – User has seen the item, but not downloaded it
• downloading – Item is currently downloading
• newly-downloaded – Item has been downoladed, but not played
• expiring – Item has been played and is set to expire
• saved – Item has been played and has been saved
• expired – Item has expired.

Uses caching to prevent recalculating state over and over

get_viewed()

Returns True iff this item has never been viewed in the interface.

Note the difference between “viewed” and seen.

get_watched_time()

Returns the most recent watched time of this item or any of its child items.

Returns a datetime.datetime instance or None if the item and none of its children have been watched.

has_original_title()

Returns True if this is the original title and False if the user has retitled the item.

has_shareable_url()

Does this item have a URL that the user can share with others?

This returns True when the item has a non-file URL.

is_external()

Returns True iff this item was not downloaded from a Democracy channel.

is_playable()

Is this a playable item?

is_uploading()

Returns true if this item is currently uploading. This only happens for torrents.

is_uploading_paused()

Returns true if this item is uploading but paused. This only happens for torrents.

looks_like_torrent()

Returns true if we think this item is a torrent. (For items that haven’t been downloaded this uses the file extension which isn’t totally reliable).

mark_item_seen(markOtherItems=True)

Marks the item as seen.

on_download_finished()

Called when the download for this item finishes.

revert_title()

Reverts the item title back to the data we got from RSS or the url.

setFeed(feed_id)

Moves this item to another feed.

split_item()

returns True if it ran signal_change()

torrent_seeding_status()

Get the torrent seeding status for this torrent.

Possible values:

None - Not part of a downloaded torrent ‘seeding’ - Part of a torrent that we’re seeding ‘stopped’ - Part of a torrent that we’ve stopped seeding

update(entry)

Updates an item with new data

entry - dict containing the new data

classmethod update_folder_trackers()

Update each view tracker that care’s about the item’s folder (both playlist and channel folders).

feed - Feed

miro.feed – Holds Feed class and related things.

FIXME - talk about Feed architecture here

class miro.feed.DirectoryFeedImpl(*args, **kwargs)

A feed of all of the Movies we find in the movie folder that don’t belong to a “real” feed. If the user changes her movies folder, this feed will continue to remember movies in the old folder.

expire_items()

Directory Items shouldn’t automatically expire

class miro.feed.Feed(*args, **kwargs)

This class is a magic class that can become any type of feed it wants

It works by passing on attributes to the actual feed.

static bulk_set_folders(folder_assignments)

Set the folders for multiple feeds at once.

This method is optimized to be a bit faster than calling set_folder() for each individual folder.

expire_items()

Returns marks expired items as expired

getExpireDays()

Returns the number of days until a video expires

getExpireHours()

Returns the number of hours until a video expires

getMaxFallBehind()

Returns”unlimited” or the maximum number of items this feed can fall behind

getNewItems()

Returns the number of new items with the feed

get_expiration_time()

Returns the total absolute expiration time in hours. WARNING: ‘system’ and ‘never’ expiration types return 0

get_max_new()

Returns “unlimited” or the maximum number of items this feed wants

get_max_old_items()

Returns the number of items to remember past the current contents of the feed. If self.maxOldItems is None, then this returns “system” indicating that the caller should look up the default in prefs.MAX_OLD_ITEMS_DEFAULT.

get_viewed()

Returns true iff this feed has been looked at

icon_changed()

See item.get_thumbnail to figure out which items to send signals for.

isAutoDownloadable()

Returns true iff item is autodownloadable

isVisible()

Returns true iff feed should be visible

mark_as_viewed()

Sets the last time the feed was viewed to now

num_available()

Returns string with number of available videos in feed

num_downloaded()

Returns the number of downloaded items in the feed.

num_downloading()

Returns the number of downloading items in the feed.

num_unwatched()

Returns string with number of unwatched videos in feed

remove(moveItemsTo=None)

Remove the feed. If moveItemsTo is None (the default), the items in this feed will be removed too. If moveItemsTo is given, the items in this feed will be moved to that feed.

setExpiration(type_, time_)

Sets the expiration attributes. Valid types are u’system’, u’feed’ and u’never’.

Expiration time is in hour(s).

set_max_new(max_new)

Sets the maxNew attributes. -1 means unlimited.

class miro.feed.FeedImpl(*args, **kwargs)

Actual implementation of a basic feed.

clean_old_items()

Called to remove old items which are no longer in the feed.

Items that are currently in the feed should always be kept.

hasLibrary()

Returns true iff this feed has a library

isLoading()

Returns true iff the feed is loading. Only makes sense in the context of UniversalFeeds

onRemove()

Called when the feed uses this FeedImpl is removed from the DB. subclasses can perform cleanup here.

setUpdateFrequency(frequency)

Sets the update frequency (in minutes). A frequency of -1 means that auto-update is disabled.

update()

Subclasses should override this

class miro.feed.HTMLFeedURLParser

Grabs the feed link from the given webpage

class miro.feed.HTMLLinkGrabber

Parse HTML document and grab all of the links and their title

class miro.feed.ManualFeedImpl(*args, **kwargs)

Downloaded Videos/Torrents that have been added using by the user opening them with democracy.

class miro.feed.RSSFeedImplBase(*args, **kwargs)

Base class from which RSSFeedImpl and RSSMultiFeedImpl derive.

createItemsForParsed(parsed)

Update the feed using parsed XML passed in

truncateOldItems(old_items)

Truncate items so that the number of items in this feed doesn’t exceed self.get_max_old_items()

old_items should be an iterable that contains items that aren’t in the feed anymore.

Items are only truncated if they don’t exist in the feed anymore, and if the user hasn’t downloaded them.

update_finished(old_items)

Called by subclasses to finish the update.

class miro.feed.ScraperFeedImpl(*args, **kwargs)

A feed based on un unformatted HTML or pre-enclosure RSS

getHTML(urlList, depth=0, linkNumber=0, top=False)

Grabs HTML at the given URL, then processes it

saveCacheHistory()

This puts all of the caching information in tempHistory into the linkHistory. This should be called at the end of an updated so that the next time we update we don’t unnecessarily follow old links

scrapeHTMLLinks(html, baseurl, setTitle=False, charset=None)

Given a string containing an HTML file, return a dictionary of links to titles and thumbnails

setup_restored()

Called by pickle during deserialization

class miro.feed.SearchFeedImpl(*args, **kwargs)

Search and Search Results feeds

class miro.feed.SingleFeedImpl(*args, **kwargs)

Single Video that is playing that has been added by the user opening them with democracy.

class miro.feed.ThrottledUpdateFeedImpl(*args, **kwargs)

Feed Impl that uses the feedupdate module to schedule it’s updates. Only a limited number of ThrottledUpdateFeedImpl objects will be updating at any given time.

miro.feed.add_feed_from_file(fn)

Adds a new feed using USM

miro.feed.add_feed_from_web_page(url)

Adds a new feed based on a link tag in a web page

miro.feed.validate_feed_url(url)

URL validitation and normalization

folder - Folders

miro.folder – Holds Folder class and related things.

class miro.folder.FolderBase(*args, **kwargs)

Base class for ChannelFolder and Playlist folder classes.

getChildrenView()

Return the children of this folder.

class miro.folder.PlaylistFolderItemMap(*args, **kwargs)

Single row in the map that associates playlist folders with their child items.

frontendstate - Frontend state

miro.frontendstate – Objects that store data for the frontends.

guide - Guide and sites

miro.guide – Holds ChannelGuide class and related things.

ChannelGuide is the class used for storing guides and sites.

class miro.guide.GuideHTMLParser(url)

Grabs the feed link from the given webpage

playlist - Playlists

miro.playlist – Miro playlist support.

class miro.playlist.PlaylistItemMap(*args, **kwargs)

Single row in the map that associates playlists with their child items.

class miro.playlist.PlaylistMixin

Class that handles basic playlist functionality. PlaylistMixin is used by both SavedPlaylist and folder.PlaylistFolder.

add_id(item_id)

Add a new item to end of the playlist.

add_item(item)

Add an item to the end of the playlist

remove_id(item_id, signal_change=True)

Remove an item from the playlist.

remove_item(item, signal_change=True)

remove an item from the playlist

reorder(new_order)

reorder items in the playlist. new_order should contain a list of ids one for each item in the playlist.

class miro.playlist.SavedPlaylist(*args, **kwargs)

An ordered list of videos that the user has saved.

This class is called SavedPlaylist to distinguish it from app.Playlist, which is a temporary playlist that holds the videos we’re playing right now.

MapClass

alias of PlaylistItemMap

tabs - tabs, tab order and friends

miro.tabs – Holds the TabOrder DDBObject.

class miro.tabs.TabOrder(*args, **kwargs)

TabOrder objects keep track of the order of the tabs. Miro creates 2 of these, one to track channels/channel folders and another to track playlists/playlist folders.

classmethod audio_feed_order()

View of audio feeds based on order.

get_all_tabs()

Get all the tabs in this tab ordering (in order), regardless if they are visible in the tab list or not.

move_tabs_after(anchor_id, tab_ids)

Move a sequence of tabs so that they are after another tab.

classmethod playlist_order()

View of playlists based on order.

reorder(newOrder)

Saves the new tab order.

restore_tab_list()

Restores the tablist.

setup_new(type)

Construct a TabOrder. type should be either channel or playlist.

classmethod site_order()

View of sites based on order.

classmethod video_feed_order()

View of feeds based on order.

classmethod view_for_type(type)

View based on tab type.

theme - ThemeHistory

miro.theme – Holds the ThemeHistory object.

class miro.theme.ThemeHistory(*args, **kwargs)

DDBObject that keeps track of the themes used in regards to setting up new themes and changing themes.