Our archives has been offered over 15TB of digital photography, which we are motivated to take. Here are some of the pertinent details:
- The files consistently include two versions of every image:
- A RAW file in Nikon's proprietary NEF format
- A high-quality JPG, which include some minor editing by the photographers (contrast, color, etc., adjustments). A typical JPG that I was sent as a sample is 8MB.
- The images are of interest for their content, not their artistic merit. These are the event photographs, official portraits, campus shots, etc. that are familiar to any institutional archives. Important elements of our institutional history, but generally not of artistic or scientific interest.
- We anticipate that this is just the first of what will be an ongoing series of accessions; we will probably be adding over a TB per year with the current formats.
- The metadata for these images is very, very scant. They are organized by shoot, and we may have some basic date and event information, but image-level metadata is limited to what was automatically created by the camera. So there will be, for example, a lot of photographs of people at events, but no identification of who the people are.
- Our library is in the process of building a (probably) Hydra-based Fedora repository that will be used to store these and other born-digital materials. We're thinking this would be a good project to build some of the ingest and preservation functionality around.
So here's a set of questions I'm pondering, and to which I welcome discussion:
- Do we want the NEF files? Heresy, I know, but a question I need to ask since based on some preliminary reading I've done on the subject, it looks safe to estimate that the NEF files make up about 2/3 of the storage space. What would be the potential loss to the archives and to the historical record if we only preserved the JPGs?
- If we take the NEF files, do we convert them immediately to DNG, TIFF or another format?
- What about some kind of heuristically-guided, content-based weeding? In other words, we keep both versions of every image we're keeping, but we are selective about which images we keep. At 15 TB, we would have to develop some really big sorting buckets, but we could make decisions based on event type, keeping only a limited number of shots from portrait sittings, etc. The department is very eager for us to take these and is willing to follow our guidance on how to proceed, and also has student labor to throw at a project if needed.
- 15 TB would be the largest set of materials we've accessioned, by a very long shot. For those of you who work in this scale routinely, what computational or other challenges will this volume pose that we probably haven't encountered before?
I'd appreciate SOPs for libraries/collection managers in this type of situation. Anecdata backed up by canonical sources is acceptable.