Most of us store photos and videos in consumer cloud services. While convenient, this comes with hidden costs and risks. This article explores why taking control of your digital memories matters and how I transitioned to holding my own data.

Background

I’m a millenial nearly in my forties. Growing up, my parents made sure to share with me the many picture books of my family. I now own many of these and expect to gain more as I grow older. These are great and definitely something I appreciate. I came to the realization about three years ago that I don’t have that same thing for my children. Instead, our family photos are all digital. From the time my wife and I got married, everything has been digital. I’m sure this is the same for many of us at this point in life. I think the digital storage of these assets is great, especially from an organization standpoint. However, many have ceded control of those assets without realizing that, not only have you given great power over them, there are costs that come along with this strategy.

Consumer Cloud Services

It seems quite convenient that various services offer to backup your photos and videos for you. I mean, why not? My original strategy for backing up my mobile phone pictures and videos was an SD card. Once that fell out of favor, it seemed like a consumer cloud service was the only choice to make sure none of my precious memories were lost if my phone got run over by a truck or dropped into a well. If you get a new iPhone, iCloud is there automatically backing up all your content for you. If you get a new Android, Google Photos does roughly the same the first time you open it. There’s so much convenience! How could there be a problem with this?

Initial Problems

If you use one of these services for any amount of time, the first problem you will likely run into is that you’re running short on space. 50GB of free storage space goes pretty quick when you’re taking high-quality photos and videos regularly (does anyone really need to see that strange bird again?). So you’re faced with a decision in most providers about whether or not you want to pay for more space. My family initially hit this with Google Photos quite a while ago. You get a choice: get more space or, perhaps worse, lower the quality of the photos that are stored to save space. I regret that I chose the latter option and now have about a year’s worth of photos that are definitely not the fidelity of the others. After that, we made the decision to go with Google Workspaces (or whatever it’s called now) and we instantly gained significantly more space and our problem was solved for the time being.

More recently, my family made the switch to iPhone devices and we hit the same problem on multiple devices. It seemed natural: buy the 2TB upgrade for iCloud and we can all share it - and so we did. We didn’t have to sacrifice any quality, thankfully, but we committed more of our digital footprint into the cloud. Now I am also paying an extra $10 a month for that privilege as well. What happens when that becomes $20 a month because they no longer offer the same package you had originally (yes, I’m sour at Verizon)?

Long-Term Issues

It’s all well and good to have your data stored somewhere convenient like iCloud as long as everything being done with that data is something you like. However, that could change at any given point and you don’t really get to control that. Now, many people would conjure up ideas that they suddenly start using all your photo data to train AI models or they begin using the likeness of your family for their own purposes and without asking your permission (I’m sure we all have a family member that participated in the Facebook scare several years ago). But the more realistic scenario for me is that the data no longer works the way that I want it to work. We hit this problem on my wife’s laptop with iCloud.

My wife is a content creator so her photos and videos are abundant and perhaps a bit more vital to her day-to-day than just pictures of our vacations. We worked out a plan when she started using iCloud to do her photo backups that she would use Photos on her MacBook to then pull that content into her video editor for use. This worked for a while but we suddenly started having problems where the Photos in iCloud would become unavailable. We found a resolution in rebooting her machine but that only works some of the time. The frustration has grown quite intense. She didn’t ask for this and, to be frank, nobody at Apple even cares - but that’s the reality of what we have.

Data Extraction

Google was the first consumer cloud provider that I’d run across that touted their Take Out service - get all your data from Google at any time! How glorious! I control all my data! Except the reality is a bit different in my own experience. Yes, Google allows you to extract your data but there are some greater considerations to make. In what format? And how am I going to get it? We turn back to my wife for a moment as she represents a techier-than-usual consumer. When we moved to iCloud a few years ago (and because of an unfortunate incident), she wanted all of her photos out of Google Photos. No problem - Take Out to the rescue, right? Unfortunately, not so simply. The website was frustrating beyond belief to use with slow downloads and constantly logging her out (invalidating the downloads!). It took her several days to download her Google Photo data with much frustration.

In a more recent situation, I wanted all of my own data out of Google Photos. I also used Take Out and, though I didn’t experience any problems with the downloads, the photos come out without any metadata that I could tell. Without metadata, photos lose timestamps and locations, making organization and memories harder to track. I ended up having to use GooglePhotosTakeoutHelper to reassociate the metadata with my photos. I still ended up with many that are incorrect (because I know when they were taken) but it fixed the broad majority. That process was not quick and was also frustrating.

Finally, we wanted to pull all of my wife’s photos and videos out of iCloud (the video production frustrations continued). The Apple Privacy website was also full of the same shenanigans we’d seen with Google Take Out the first time. The website downloads are slow and prone to error and you regularly get logged out of the site (within maybe an hour). With 33 files to download at >10GB each, the process still continues. There are some files that actually just won’t download properly and I’ve had to result to workarounds to get that data (curl, if you’re curious).

Holding Your Own Data

In the sections above, I’ve detailed just some of the issues I’ve personally seen with consumer cloud services. These problems have ultimately driven us to try to hold all of our own data (at least with regards to pictures and videos). That’s not actually all that difficult of a task in theory but you do need to prepare to meet with some realities that I’ll cover in the sections below. If you’d like to go through a detailed understanding of my technical setup, see my post here.

Location

We had to decide where we wanted to actually hold our own data. Did I want to hold it in one of many data storage providers out there including some of the public cloud providers? That’s certainly an option but that comes with some logistics considerations and an ongoing monthly cost. I don’t oppose storing my data in a public cloud provider because, unlike a consumer cloud offering, these are business-focused with apparent costs and the ability to do things like control your own encryption keys.

Ultimately and against my commitment not to host anything at my own home, I decided that I would buy and maintain the equipment here at my house. I don’t like the idea of having to host things at my house because it makes my job just a little harder in just trying to exist. My home internet can’t merely “be out” - now there are things beyond complaining family members that might need that access. I determined this was worth it. I ended-up creating a space in my master bedroom that safely holds a small amount of tech gear for this purpose (I use a sound machine so it masks the tech sounds!).

Additionally, storage devices like a NAS (typically used for this) need to be on redundant, protected power. No technical devices seem to take well to brownouts or short losses of power (except those made for it) but spinning disk systems seem to be some of the worst in terms of tolerance. I’m not at the point where I want to afford SSD versions of the storage sizes I need for a NAS so putting it on an uninterrupted power supply (UPS) is critical.

Access

Consumer cloud providers have one major advantage beyond the convenience of the service: their apps also make things pretty convenient too. Google Photos or Apple Photos do a great job of providing a really good end-user experience on mobile devices. You can go back in time and see thumbnails of all your content and open it up at any point - almost magically. As someone who tried to make that a reality using NextCloud and every image library known to man, I can tell you that it doesn’t just work that well when you roll-your-own. But if I’m going to host my own, I don’t want it to be any less-convenient.

Perhaps I’ve become spoiled by these apps but I really like that user experience. And more than that, my family has come to expect that user experience. I needed to consider their access patterns in this plan to make sure that I was providing a very similar experience.

For that reason, I decided to go with my NAS manufacturer’s solution (Synology Photos if you must know). I’d heard great things about this solution and, in my own testing, it worked very well. There was one problem in that I’m beyond the days of exposing any external ports for devices within my home to the Internet. So how would my family (and myself) reach this when we’re out-and-about? I decided to put Tailscale on everyone’s phones. Now, we can seamlessly access all our data regardless of where we are. It’s not quite as fast as a cloud service (we’re going over consumer broadband upload, of course), but it seems to be pretty good overall.

Backup

It isn’t enough just to hold all of your own data in a single place. For us, this is every picture of our kids that we own, every video of pets long past, and key moments in our lives (like the 2017 solar eclipse). It would be irresponsible to not ensure that we have proper backups of this data. Those backups come in many different forms and there are several options.

I was, once again, faced with the option of doing backups with a public cloud provider or a storage service. And, once again, I was faced with the ongoing monthly costs of that data residing elsewhere. Worse yet, public cloud providers will ding you with the real cost of your data when you go to extract it if you, unfortunately, need it. Egress costs, or the costs associated with data leaving their network, are a real thing.

Ultimately, I chose two external USB drives that attach to my NAS. One drive will stay attached to my NAS for a weekly backup of data. I’m using my NAS manufacturer’s solution (Synology HyperVault) once again primarily to encrypt and manage the backups. This might offend some that are totally dedicated to openness in holding your data (because, of course, Synology could make changes). However, it’s worth the risk given that Synology is regularly used in business contexts and is unlikely to make breaking changes in that way (lest they alienate a large swath of their business). The other drive goes to another location away from my home in case of critical loss. Monthly, the drives get swapped around and the process repeats.

Ongoing Costs

Finally, let’s not forget that, if you choose to hold your own data on your own hardware like I’ve done, there are ongoing costs that you have to face with this too. They work a bit differently than consumer cloud providers and, I’d argue, they are probably lower - but still present. You’ll need to account for failures in your storage devices (whether the chassis or the individual drives) but also the expansion of your data. What used to be 2TB of data for my family is slowly growing (especially now that I have a child contributing her data to the pile). My NAS is in a redundant configuration to hopefully help me bide time when the drive failures come and I use limits on storage to help me monitor the impending doom of more data.

Final Word

Overall, I’m happy (right now) with my choice to hold my own data. I’m glad that I experienced some of the pitfalls I mentioned here early enough to make a decision to change. The more data we accumulate, the harder it is to move it around to the most desirable location. Hopefully this information is helpful to others in the same boat.

I would like to end with some encouragement about data maintenance no matter where you store your assets. Some of you that are my age may be at the point where you are receiving old family items (not just photos!) and you aren’t even sure what to do with it. Maybe it’s a pile of boxes in a storage facility or possibly even a whole house that you must unravel. At least when it comes to my digital assets, I don’t want that for my kids. I’m taking time now to try to clean up and organize my data getting rid of the things I don’t need. I encourage you to do the same so that, one day, when someone looks back on that information, they get what you were after in preserving it this whole time.