Managing and Storing Your Digital Image Files
by Dr. Robert Berdan
January 27, 2011 (Updated)
This may not be the sexiest topic in digital photography, but it may be one of the most important ones you read!
Screen shot of Picasa 3 a free image database program from Google lets you find images quickly on your hard drive
Digital photography has changed the way we shoot, and the way we store and manage images. What you may not realize is that most forms of digital media are not very reliable, at least not for long term storage, so if your digital images are valuable to you and you want to be able to view them in the future I think you will find some of the techniques and tools I use useful. I am not suggesting that these techniques are the only ones or the best ones - simply the ones that work for me. At the end of the article there is a list of links to additional resources including recovery software and how National Geographic backs up their image files. What prompted me to write this article is that I have just finished making 2 back ups of a 2 Terra Byte (TB) hard drive that contained about 500,000 images that I took over the last decade. Backing up the 2 TB drive took me two days. Then I needed to rebuild the image database using Picasa so I could find the images quickly for sales purpose. If you shoot a lot of digital files you will want to think carefully about how you organize your images, the file formats you keep, the type of media you store your images on and how many backups you should have. Here are a few suggestions:
1) Never store your image files on just your computer's main or boot hard drive
If you only store your important data on your laptop or main computer's hard drive you are guaranteed to lose those files due to a computer virus, or some other hardware problem in the future. I have 12 computers in my studio and it seems that one of them breaks down due to a power surge, virus, or other software conflict almost every other month. I run a gamut of operating systems from Windows XP, various versions of Vista and Windows 7 and all of them are unstable in my opinion. I wish I could say Macs were a lot more stable, but I have used them in the past and the little bomb symbol came up just about as often as things lock up on my PCs - in any case when you lose files its usually because of a hardware failure and most often its the hard drive. The logical solution seems to be to store your data on an external hard drive so if your computer fails you can connect the drive to another computer. Another tip is if you don't need access to the drive while you surf the web turn the external drive off - its less prone to attack. Since it's just a matter of time before a hard drives fails (depends on how much you use it, quality etc) but generally speaking in my experience hard drives seem to be reliable for about 3-7 years of regular use. While many factors influence the lifetime of a hard drive the point I am trying to make is don't relay on a single drive to store your valuable digital assets - use 2 or 3 drives. There are various hard drive configurations called RAID where if one disk fails the data can be recovered, but it comes down to having your data on multiple drives no matter how you want to look at it. In an ideal situation you should have the cloned hard drives stored in different locations.
Array of three external hard drives of various sizes - be sure to back up your image files on at least 2 external hard drives
2) What about Virtual drives are they a good solution to store your digital files?
A virtual drive refers to a place on the Internet where you can upload and store you files. They may be useful for certain types of files like JPGs, but they are not currently practical for up loading several Giga Bytes of RAW files as it would take hours using current Internet technology. So while you might have large amounts of space available it simply takes to long to upload RAW files. Furthermore, I have no idea how secure those spaces are and if they are also backed up - servers do get hacked and they do fail. This option might be a good one in the future for storing RAW image files when our internet connections are capable of transferring data at Gigabytes\sec - I think I would give it a few more years.
3) What about storing your digital files on a CD, DVD or Tape?
I started backing up my files on DVDs, but within a year I had several hundred and it became very difficult to find the files I needed quickly so it's not a practical solution for a professional photographer or anyone that shoots a lot. Furthermore it's unclear how long CD's and DVD's will be reliable. A music CD that was pressed may be stable for hundred years or more, but most of use CDs and DVDS that are burned onto photosensitive disks using a laser light and the stability of these types of disks depends on the dyes and materials used in their construction. I think it is a good idea to store you best images on some CDs or DVDs, but it is not a practical solution for those that shoot a lot. Magnetic tape has been used to back up data since 1953 and according to research studies it appears to be stable for between 15-30 years under normal storage conditions and it is one of the strategies used by National Geographic. The main problem with backing up data to Tape is that it is linear and you can't find specific data quickly. There is also the relatively high cost of tape drives and cost of individual tapes which you will need quite a few of - see links below for curent tape backup costs from Hewlett Packard.
This photo shows CDs and DVDS used to backup digital image files and represents about 3 years worth of shooting.
While the disks might be good for 20 years or more I am not able to find specific images quickly.
One of the things I do when storing images on a CD or DVD is to make a cover using Adobe Photoshop and I include a contact sheet that I can view quickly to see what images are on the CD as shown above. This of course takes time to do.
4) What about storing files on USB sticks or Compact Flash Drives?
With current technology it is possible to read\write to a compact flash card or USB stick theoretically about a million times. This technology is still changing quickly but for the time being USB sticks and Compact Flash cards are simply not big enough and too expensive to archive large amounts of digital data. Smaller 1-8 GB USB drives have been pretty reliable in my experience though I have not had much luck with larger 32 GB USB drives. Flash drive technology is changing quickly so this technology may be viable in the future for archiving lots of images. SanDisk recently announced a 256 GB compact flash card designed primarily for digital video at a cost of about $1400 - I will wait a year or two until the card drops to under a few hundred dollars, but again too expensive compared to hard drive technology for storing images. Bottom line - compact flash cards and USB sticks aren't an economical method for storing lots of digital image files (RAW) for long periods of time, but they can be used to archive your best images, web pages and music files.
5) What should you do if your hard drive or compact flash card is corrupted and you can't read the files?
First never format your compact flash card or hard drive if you receive an error message indicating the drive is unreadable - if you do you may never be able to recover the data. If a compact flash card becomes corrupted sometimes the data can be rescued. I have tried the rescue disks that come with my compact flash cards and they are useless in my experience. What I found that does work is a software program called Stellar Phoneix. The company that makes this software specializes in software recovery and my first experience was after one of my external hard drives failed and I did not have a back up of some of my best images of lightening. I downloaded the software for free and ran an analysis, it showed me the lost files and asked if I wanted to recover them - then it asked me for payment ($150). I paid the fee and proceeded to recovered my image files. Lesson learned - always make two or more copies of your important digital files. Recently the company has created custom software to recover windows data, Mac data and photo recovery. The photo recovery software is now only $10 and the windows recovery software is $49.00. I have not used the new software as my original software purchased a few years ago still works fine.
The Lacie portable drive and the San Disk Extreme Pro card above both had corrupted boot sectors and the Stellar Phoneix
software I purchased was able to recover the data in both cases.
The Stellar Phoneix recovery software has not been able to recover every damaged hard drive I have tried, but it has worked on most of the compact flash cards and hard drives. If this software does not work I usually recommend to my friends that they seek out the service of professional data recovery experts, but they are not cheap so you need to decide how valuable the data is before you contact them.
6) How can you reduce the number of images you need to sort through - edit them before or after you download them, and don't keep all your image files.
If you only take your camera out to shoot once every few months then go ahead download everything onto your hard drive. But if you shoot a lot or even over several years you will begin to accumulate a lot of "garbage" files that you have to sort through. Initially I used to download all my files because I thought it would take a long time to fill up a 1-2 Terabyte drive - not anymore. The more files you store, the more time it will take to sort through them to find the good ones. Now I try to edit out the real bad ones by deleting them off my camera before I even begin to download.
Downloading and editing image files after a day of shooting is always a good idea as it will reveal if you have any
great shots or whether or not there might be any problems with your technique or camera gear.
7) What is the best file format to store your images in? Depends on who you ask!
Most professional photographers I know shoot RAW files because they offer the best quality and the most flexibility, but they are also proprietary and different for most camera brands and RAW files from the same company can be significantly different on different camera models. This prompted Adobe to recommend a common RAW file format called DNG or digital negative format. Adobe Photoshop can convert any RAW file into DNG, however only a few camera manufacturers have started to use this file type. Nikon and Canon so far has refused to use this format because they feel it's too early in the development of RAW files to support only one. Currently Adobe has to update the RAW converter used in Lightroom and Photoshop every few months in order to keep up with the newer camera models, but it also gives Adobe an excuse to force their customers to upgrade their software as they only update the RAW converter for a particular version until they begin to sell a newer version.
According to Adobe: the main benefits of using the DNG RAW file format for photographers are:
- DNG format helps promote archival confidence, since digital-imaging software solutions will be able to open raw files more easily in the future.
- A single raw processing solution enables a more efficient workflow when handling raw files from multiple camera models and manufacturers.
- A publicly documented and readily available specification can be easily adopted by camera manufacturers and updated to accommodate technology changes.
Here are what your options are if you shoot RAW files.
- Keep using proprietary RAW files and if you are using Canon and Nikon there is a pretty good chance they will still be around in 20 years or more - at least I would think so, but I also thought Kodak and Polaroid would be around longer as well.
- You can covert your RAW files to DNG in Adobe's RAW converter
- You can simply edit and process your RAW files and store them as .TIF files. If you only shoot .JPG files you should convert your best ones to .TIF format. JPG files degrade every time you save them. However always keep your RAW files.
- Hold on to your old versions of Photoshop if they can edit the RAW files you should be able to do so in the future if you keep an older computer around. The use of RAW files on Digital cameras is only about 10 years old, my first camera that supported RAW was the Nikon D1 (2.7 Megapixel camera).
Never throw your RAW files away even if you processed your images. The RAW converter software is getting better every year and has already undergone vast improvements. For instance if you had a RAW file you took several years ago at high ISO speed its now possible to reduce the graininess of the image using the latest Adobe Photoshop RAW converter. Your RAW files allow you to process your images differently and the images hold more information then any other file format.
If you are still shooting on film your images may out last the many digital file formats, but then you lose out on many of the advantages that digital cameras offer. I believe there will be new storage solutions within the next decade that will solve the problem of keeping large quantities of digital data in a secure stable format for long periods of time (decades or centuries) - it's critical for our information based society that we find new and better storage solutions.
Shooting on Slide or Negative film provides a stable method of preserving images at least for one's life time if stored properly..
Currently I do not convert my Nikon or Canon RAW files to DNG - it just takes too much time for me to use on a regular basis and I believe Nikon and Canon will still be around in the next 20-40 years. I also process my very best images into .TIF files and store them in a folder called Best images for any particular year (e.g. Bestimages_2010).
8) Cataloging and finding your Image files requires an image viewer and database
Once you save your files onto an external hard drive you need a system to organize and find your images quickly. I usually transfer my digital images in folders from my compact flash cards and leave the camera name associated with the folder but add the date to them. So essentially I catalog my folders by camera type and date.
101ND700-Aug08_10 (this is format I use - however it doesn't tell me what its in the folder)
This translates to folder with pictures taken with a Nikon D700 camera August 8, 2010.
I can sort the images or folders by date if I need to. Once the images are copied to my hard drive I start up a free Image database program available from Google called Picasa 3 - this software supports movies and most major file formats including RAW files and is one of the best free programs available on the Internet. The software automatically searches your hard drive and creates thumbnails of your images (all major file formats including RAW) and movies. You can also restrict the program to search only certain folders, or hard drives by selecting Tools>Folder manager and set which hard drives or folders you want the software to scan. The program does a lot more, but the best way to find out is to simply download it and play with it.
Picasa 3.8 - on the left are some basic image editing tools. To find an image you can simply right click on a thumbnail and select either Locate File or Open in Photoshop - it's fast and convenient and makes finding specific images a breeze.
When you click on a thumbnail in Picasa you can see an enlargement and zoom into the image (see above). The software includes many other extras including some basic image editing, makes screen savers, movies etc. Some photographers like to use Adobe Lightroom or other image database software (e.g. Apple Aperture 3), but it's hard to beat free and frankly Picasa is the best image database program I have encountered. Picasa is available for both PC and Mac. You can add keywords to specific images, search automatically for pictures with faces, movies or images you have assigned stars or keywords. Keep in mind if your computer crashes you won't lose any of the images if they are stored on an external hard drive, but you will lose any keywords you created and you need to rebuild the database. Assigning keywords can help you find certain images quickly, but if you spend a lot of time doing this and your hard drives crashes you will lose them so I only add keywords to "special" images. Rebuilding the database or thumbnails can take a 24-48 hours for a 2 TB hard drive full of images (connect your hard drive using USB 3.0 or E-SATA connections they are currently the fastest methods of transferring data to an external hard drive). To increase performance I use a RAID 2TB drive in my main computer to store my images and access them using Picasa (but I also have two backups of the images on external drives).
9) What is the best way to store digital files for the future?
I wish I could give you one solid answer but I can't. CDs and DVDs were supposed to be good for between 10-100 years, but they are not big enough to store large numbers of RAW files. Compact flash cards are still too expensive to be used as storage and no-one really knows for how long the digital data is stable for on these devices. So your best solution today is to store you valuable images and other digital data on two or three external hard drives. When newer technology comes along we will likely have to transfer the files over to these devices.
In spite of sophisticated mathematical analysis of the mean time to failure for hard drives no one knows for sure how long one will last as it depends on many variables (See Google study of Hard drive failure trends below). In my 20 years of experience with computers it appears that hard drives seem to have a lifetime of between 2-7 years before they fail. The size of hard drives has also increased exponentially and as the data density increases it seems likely that larger hard drives will fail sooner then smaller hard drives.
Recently Solid State hard drives are being sold and placed inside laptop computers. These drives have no moving part and work similar to compact flash and SD cards. I have read estimates that they can read\write between 1 to 1.5 million times before failure and other sources suggest 100,000 to 200,000 write cycles before failure What this translates into everyday use is still to be seen. The main benefit of solid state drives is there they have no moving parts and they transfer data more quickly (up to 80 MB\sec). At the moment they are more expensive (128 GB Drive $500, 240GB drive $649) than drives with rotating disks (e.g. 2 TB is under $150 at this time). If they prove to be more reliable then rotating hard drives - they may make the older spinning drives obsolete, but it's too early to tell.
Simply put, the data integrity of hard drives has failed to keep pace with their growing capacity. Large disk populations are vulnerable to random uncorrectable data losses.
Jan 1, 2011 12:00 PM, By Zsolt Kerekes - The future of data storage (see link below)
A Russian company, Storelab, released a long-term study that compared hard drives from different vendors that were sent in to have data recovered. They report that while hard drives from some manufacturers last 2.5 years on average others from another manufacturer lasted only 1.5 years on average (not very long). The majority of hard drives sold worldwide are produced by six companies: Fujitsu\Toshiba, Hitachi, Samsung, Seagate and Western Digital. Seagate had the highest failure rate at 56%, Western Digital 19.5%, Fujitsu\Toshiba 10%, Samsung 7%, Hitachi 5%. Those made by Hitachi seem to be the most reliable. (Note when you buy a hard drive e.g. Lacie - mine at least were made by Samsung). As for Western Digital's failures, 59% of the company's examined hard drives had a capacity of up to 500 GB, and their average lifespan was 3.5 years. The remaining 41% exceeded 500 GB. Due to their construction and additional platters, these larger models are less durable, exhibiting an average lifespan of only 1.5 years. See full review at Tom's Hardware.com which summarizes Hard drive reliability study in Russia.
For now at least it seems that using multiple hard drives is the most economical method of archiving your digital image files and it's also a good idea to back up some of your best images onto DVD's or CD's. Tape might be the most reliable method, but with tape back up systems starting around $500 and tapes running at $50-$70 each this isn't the most affordable solution for the average photographer or artist. The most important point to take away with you is that all hard drives will fail - it is not a matter of if, but when and if you have valuable data (photographs, movies etc) you better have a backup plan.
Links and Additional Resources
[ Top ]