I wonder whether there is a data format for archiving messages from twitter and similar microblogging sites, including metadata. The Library of Congress gets all messages from Twitter but I found no information how they store them and how they would make them available for researches. There are several tools to backup your Tweets, in most cases you don't get a download including all data. One can also access Twitter API but then everyone stores his archive in different formats so there is no easy way to exchange and combine archived messages.
On the other hand there is interest in analyzing microblogging and tweets are analyzed by companies and researchers. But are there open and standard tools to collect, archive, and analyze tweets? And which collections of archived microblogs exists in addition to the LoC? I don't want to propagate "MARC for Twitter" but at least something more precise than "just use CSV/JSON". Relying on the custom format of one particular provider (Twitter) at one particular point in time does not look like a reliable solution for long term archiving.
P.S: Ed Summers gave a brief overview of the format used by LoC for archiving Twitter. There are some open questions how do use the format for other services (e.g. Google+ and for particular selections of messages (somehow documented in
bag-info.txt). I'd like to see tools to create your own archives in the same format - for instance all tweets by some user accounts or all tweets with a given hashtag like #VenusTransit in a given time - and tools to read and analyze these archive files. There are several closed web applications to do so but they don't provide import/export in such a defined standard format, do they?