Possibly interesting factoid: Drobo S and APFS File System

[rbpeirce]

My Drobo was formatted for the previous file system and I could use the Drobo Dashboard capacity button to monitor use. A couple of days ago I reformatted it to APFS to take advantage of its Volumes capability.

Imagine my surprise in checking capacity to find 88% of the capacity was used. It said 9.6Tb was used when I had only put about 5Tb on the drive. It finally dawned on me that the capability of Volumes to change size as needed meant that as far as Mac OS was concerned all the available space was being used.

Disk Utility helped but not completely. Disk Utility shows how much space each Volume is actually using, which you can easily use Get Info for if just checking one Volume, but it also says the free space includes the reserved space Drobo uses for its recovery capability. Fortunately, you can used Disk Utility to make a quick estimate of how much of the total capacity Drobo Dashboard Capacity shows is actually being used.

The current Drobo Dashboard can't really deal with APFS in a meaningful way, so this is a work-around until it can, if that ever happens.

 

[MacInWin]

It finally dawned on me that the capability of Volumes to change size as needed meant that as far as Mac OS was concerned all the available space was being used.

I have no idea what that sentence means, sorry. Volumes share whatever space is in the Container that holds the volumes. So, ever Volume will report the same free space because it is reporting the free space in the Container. Yes, whatever is any given Volume is "all" that is in the Volume, but free space is dependent on the Container, not the Volume.

For example, if you have a Container with 1TB of space, with two Volumes, one with 200GB and one with 300GB consumed, each of them will report that it has 500GB free space because that is how much is left in the Container.

I have no idea how Drobo works, nor how it uses space. I am surprised you could even format it as APFS, as most NAS drives have their own format.

 

[Raz0rEdge]

I have no idea how Drobo works, nor how it uses space. I am surprised you could even format it as APFS, as most NAS drives have their own format.

That's my confusion as well. My Synology NAS and most NAS' out there are Linux based and use something like EXT4 for the data while exposing it over the network through AFP/SMB/etc.

You should ONLY be formatted external drives that will only be used with macOS as APFS.

 

[rbpeirce]

I have no idea what that sentence means, sorry. Volumes share whatever space is in the Container that holds the volumes. So, ever Volume will report the same free space because it is reporting the free space in the Container. Yes, whatever is any given Volume is "all" that is in the Volume, but free space is dependent on the Container, not the Volume.

Exactly. The container is a physical partition and there can be more than one on a drive. I'm talking about how Drobo Desktop sees them. The container/partition may actually be empty but Drobo sees an APFS partition as full. On the previous format it showed actual usage. Volumes are incidental to this except they share the capacity of the container/partition.

That's my confusion as well. My Synology NAS and most NAS' out there are Linux based and use something like EXT4 for the data while exposing it over the network through AFP/SMB/etc.

You should ONLY be formatted external drives that will only be used with macOS as APFS.

This isn't a network drive. It is attached. Many Mac users have used them for years. It was previously formatted as what Apple calls Mac OS Extended, at least that's how it looked to the OS. I have no idea how it was physically formatted. I'm willing to try something to see if it works. In this case it did, or at least, that's how it looks to the OS.

Maybe the underlying drive is still formatted something else but I can't see it. It looks like an APFS drive to the computer.

 

[rbpeirce]

I seem to have confused some folks. This post was really only meant for people using the Drobo Dashboard who are contemplating formatting their Drobo to APFS. All I'm trying to say is the Capacity button isn't very helpful and you have to rely on Get Info and Disk Utility. I thought this was pretty interesting in view of the fact that it seemed to work correctly under HFS+.

 

[pm-r]

All I'm trying to say is the Capacity button isn't very helpful and you have to rely on Get Info and Disk Utility. I thought this was pretty interesting in view of the fact that it seemed to work correctly under HFS+.

You might be interested in Adam Engst's recent TidBITS article:
Auditing Free Drive Space: Where Have All the Gigabytes Gone?

Auditing Free Drive Space: Where Have All the Gigabytes Gone? - TidBITS

Curious about how Time Machine snapshots can supposedly prevent the space occupied by deleted files from being recovered right away, Adam Engst ran some tests and came away more confused than when he started.

tidbits.com

It seems that Apple's APFS is suffering from what he refers to as the APFS Uncertainty Principle.
Some might find the article rather interesting.
Some might be interested to know how much data actually is on their APFS formatted Drive. Good luck it seems.

Maybe some actually know how it's all supposed to work.





- Patrick
=======

 

[rbpeirce]

You might be interested in Adam Engst's recent TidBITS article:
Auditing Free Drive Space: Where Have All the Gigabytes Gone?


- Patrick
=======

Just read it. It deals mostly with the Macintosh HD, snapshots and Time Machine, which is a whole other bag of worms. I find Get Info and Disk Utility adequate for getting around the Drobo Desktop Capacity issue, which is my primary concern.

When I used HFS+ my backups were all to sparsebundles. With APFS I can backup to individual volumes. I don't know if they take up more or less space than a sparsebundle, but that doesn't matter as long as I have enough space. What I do like is I don't have to mount them to see what is in them and they are pretty much instantly accessible. Also, although I've had very little trouble with them I've read sparsebundles can be unreliable.

 

[MacInWin]

APFS is still a mystery. Apple has not released any significant details on the way APFS actually works, so the fact that a third party like Drobo cannot handle it is not surprising. As as far as the Container being shown as full, that is how APFS works. A Container is NOT a partition, those are two separate, and different, things. If DROBO is treating a Container like a Partition, that can lead to all sorts of strange goings on.

Even Disk Utility reports the space in a Container as completely used. Here is a screenshot of Disk Utility in three ways:

First, the hardware

The entire drive is consumed, according to that image. The storage has ONE Container, taking the full drive.

Now, here is a picture of the Container

Now you can see the Container is less than half used, with 579.19GB FREE.

Here is the Macintosh HD volume:

Still showing 579.19 GB free space.

And finally, Macintosh HD - Data:

Also the same space free.

So, what does all that mean? Well, it means that to the system, the drive is entirely consumed and dedicated to Container disk3, but Container disk3 is less than half used.

You might be interested in Adam Engst's recent TidBITS article:
Auditing Free Drive Space: Where Have All the Gigabytes Gone?

Auditing Free Drive Space: Where Have All the Gigabytes Gone? - TidBITS

Curious about how Time Machine snapshots can supposedly prevent the space occupied by deleted files from being recovered right away, Adam Engst ran some tests and came away more confused than when he started.

tidbits.com

It seems that Apple's APFS is suffering from what he refers to as the APFS Uncertainty Principle.
Some might find the article rather interesting.
Some might be interested to know how much data actually is on their APFS formatted Drive. Good luck it seems.

Maybe some actually know how it's all supposed to work.





- Patrick
=======

Patrick, he got some things wrong. Here is a better article from Howard Oakley:

Free space on an APFS volume is an illusion

You’re in control of a traditional file system, but with snapshots, clones, sparse files and other volumes sharing free space, APFS isn’t as simple.

eclecticlight.co

and these two:

Inside APFS: from containers to clones

How APFS containers and volumes work. What hard links, clones and sparse files are, and when they break down.

eclecticlight.co

Last Week on My Mac: How APFS trims a disk image to size

How APFS can trim free space in a read/write disk image and transform it into a sparse file, so it uses storage more efficiently.

eclecticlight.co

Bottom line: APFS is significantly different from HFS+, and what we thought we used to know about HFS+ doesn't really relate to APFS much at all, if any.

 

[pm-r]

Bottom line: APFS is significantly different from HFS+, and what we thought we used to know about HFS+ doesn't really relate to APFS much at all, if any.

That's certainly a big understatement Jake, and thanks for the links, but I think it will be a long time before I try using any APFS formatted Drive again.
A recent update attempt involved it and it screwed up my Boot Drive and a cloned backup as well as another backup partition.
So much for attempting to upgrade from Mavericks to High Sierra. I'm still recovering... Weeks Later... And ended up using OS X 10.11.6: El Capitan.

Interesting images you attached... What a confusing mess trying to understand what one has on one's Drive...
Welcome to Apple's new SSD Drive structure, I guess.



- Patrick
=======

 

[MacInWin]

That's certainly a big understatement Jake, and thanks for the links, but I think it will be a long time before I try using any APFS formatted Drive again.
A recent update attempt involved it and it screwed up my Boot Drive and a cloned backup as well as another backup partition.
So much for attempting to upgrade from Mavericks to High Sierra. I'm still recovering... Weeks Later... And ended up using OS X 10.11.6: El Capitan.

Interesting images you attached... What a confusing mess trying to understand what one has on one's Drive...
Welcome to Apple's new SSD Drive structure, I guess.



- Patrick
=======

Patrick, it's not really confusing, it's just very different. I remember when cars first had automatic transmissions that some folks insisted on going through the gears anyway, because that was "how things worked." Nowadays, everybody knows you just put it in Drive and move along.

HFS+ was a more primitive manager of drive space where you had to consider partitions and sizing them properly to make sure you had enough room before you even wrote the first byte to it. Now APFS just handles things, so you create the Container, Volumes get made and you just move along. And I haven't even mentioned that files are, by default, stored as sparse files in APFS, which saves a ton of actual storage for some uses, but not in all cases. Howard gave an example where he created a multi-GB file, left it empty and it shrank to under 80K actual size, despite being more than a GB virtually. That can be confusing if you then delete the file, expecting to save Gigabytes and only see 80KB actually made free. Or think you can copy an 80KB file to a thumb drive with a different format and suddenly find multiple GB being consumed because that format doesn't support the sparse function.

But tech moves on, we learned to drive automatics, we can learn APFS.

 

[rbpeirce]

A Container is NOT a partition, those are two separate, and different, things. If DROBO is treating a Container like a Partition, that can lead to all sorts of strange goings on.

This is more a matter of usage terminology. If you split an attached drive and it is in the HFS+ format, you get partitions. If it is in the APFS format you get containers. You can name a partition but not a container, or at least, I haven't figured out how. From a usage standpoint they are almost the same thing except the continuer will have mountable volumes while the partition won't.

That's why I like APFS. With HFS+ I had to back up to numerous sparsebundles. With APFS I backup to numerous Volumes. It is a heck of a lot easier to manipulate and access.

 

[pm-r]

But tech moves on, we learned to drive automatics, we can learn APFS.

And now one can even buy a car with an automatic transmission that allows or almost insists the driver uses the stick-shift to "change gears" to make it seem even sportier like the good old days... ;-) ... but certainly quite a bit more expensive...

And no need for any double declutching for any that still recall having to use that technique, or even a session using heel-and-toe shifting... ahh.. the memories... and the four-wheel drift around the corners???... All left as only memory these days... And I think the cops prefer it that way...



- Patrick
=======

 

[rbpeirce]

and the four-wheel drift around the corners???

I think that went away with bias ply tires. You get a little slip today but nothing like 40-50 years ago.

 

[MacInWin]

This is more a matter of usage terminology. If you split an attached drive and it is in the HFS+ format, you get partitions. If it is in the APFS format you get containers. You can name a partition but not a container, or at least, I haven't figured out how. From a usage standpoint they are almost the same thing except the continuer will have mountable volumes while the partition won't.

That's why I like APFS. With HFS+ I had to back up to numerous sparsebundles. With APFS I backup to numerous Volumes. It is a heck of a lot easier to manipulate and access.

No, they are not the same. A partition is a hard-defined space on the drive that is then mounted as it exists. That is why you can name a partition; it will be mounted with that name. But it will be fixed in size at whatever was defined when the partition was created. It is difficult to change the size of a portion once created. Not impossible, sometimes, but difficult. And you can have multiple Partitions on a drive formatted APFS, if you wanted to. Each Partition would be fixed in size and formatted however you want, even with one as HFS+ and one as APFS. Any partition with HFS+ would mount with whatever name is is given.

But, a Container is not named because you do not mount a Container, you mount the Volume(s) inside a Container. The Container is fixed in size, but the Volumes in the Container are variable, with all of the space available in the Container available to all of the Volumes in the Container. Container and Partition aren't even close to the same thing. And thinking that way is an excellent way to get in serious trouble. We have had folks come here who did not understand the difference between HFS+ and APFS who ended up with multiple boot Volumes in the same Container who then had multiple Volumes with the same name that led to all sorts of confusion because they thought they had lost files when their actions created new Data Volumes that didn't have what had been in the original Data Volume that was still there, just not mounted. One user even had multiple Data volumes showing up on the Desktop with different things in them! All because they didn't understand how APFS handles Containers and Volumes. We have to think differently with APFS.

You are missing the point that an APFS volume is already a sparse format. You don't back up to a sparsebundle, TM used sparsebundle format to reduce network traffic when it backed up to a network drive. You, as the user, point to the drive you want as backup and TM makes the determination whether it's a normal backup (not sparse) or a sparsebundle backup. With APFS, even the internal "drive" is now using sparse files, where it can, and the snapshot process uses a sparse function to compress out unused space, but you still don't back up to a sparsebundle, you back up to a mounted drive, or to a network drive. Again, TM makes the choice whether to use smarsebundle or not..

It's more than just terminology. We have to learn to think about storage differently. I've been trying to stop talking about my internal storage on my M1 as a "drive' or "SSD" because it isn't either, it is STORAGE, and it is managed differently than the old technology.

 

[rbpeirce]

No, they are not the same. A partition is a hard-defined space on the drive that is then mounted as it exists. That is why you can name a partition; it will be mounted with that name. But it will be fixed in size at whatever was defined when the partition was created. It is difficult to change the size of a portion once created. Not impossible, sometimes, but difficult. And you can have multiple Partitions on a drive formatted APFS, if you wanted to. Each Partition would be fixed in size and formatted however you want, even with one as HFS+ and one as APFS. Any partition with HFS+ would mount with whatever name is is given.

As I said, this is a usage thing not a strict definition. For example, I have an externally mounted drive that happens to be formatted APFS. It has one 1TB container I use for Time Machine backups which, has a single volume called Time Machine, and I just use the other container for anything I want to put into it. If this was formatted HPFS+ under Monterey for example I could have a 1Tb partition for Time Machine and it would be called Time Machine. Both could be mounted and would look the same to the user. That's all I meant.

As near as I can tell, APFS just adds another layer to get to the same thing. I have no problem with that and appreciate the added capabilities.

BTW, before everybody jumps on me I know Time Machine is supposed to be on a dedicated drive but I've been doing this for years without any problems.

 

[MacInWin]

No need for two containers, but do whatever you want. You would have more flexibility with one Container for the whole drive, with multiple Volumes for whatever you want.

As for TM, you may find it takes over the drive and blocks access to the rest. As I said, what you have been doing for years no longer works the same way. So, if you want to do what you are doing, I would suggest a second drive to duplicate as a backup whatever is in Track Data - TM and disk9 for when TM takes over the whole thing. That way you won't lose it.

 

[rbpeirce]

No need for two containers, but do whatever you want. You would have more flexibility with one Container for the whole drive, with multiple Volumes for whatever you want.

As for TM, you may find it takes over the drive and blocks access to the rest. As I said, what you have been doing for years no longer works the same way. So, if you want to do what you are doing, I would suggest a second drive to duplicate as a backup whatever is in Track Data - TM and disk9 for when TM takes over the whole thing. That way you won't lose it.

As I understand it TM will use all the space allocated to it which means it will continue to expand until the container is full. That's why I set up two containers. As I understand it they have hard limits and won't expand. If I am wrong about that I have multiple backups of everything except TM and I have two of those running so I should be ok.

 

[MacInWin]

The new TM goes beyond that. It "takes over" the "drive" and will block read/write to the "drive" by anything else, including the user. In the past, what you are doing was workable. Today, not so much. Going forward, I anticipate Apple will be getting even more aggressive at protecting the TM backups as part of their emphasis on security.

As I said, what used to be good practice and worked, may not be so now. What I see coming is the use of smaller dedicated "drives" for backup, whether SSD or rotating, with nothing but the backups on the drive. So, no need for an 8TB backup, just a drive with the standard 2X of what is in the full backup. And with the way APFS manages things, just one partition, one container per hardware, with multiple Volumes in the Container for maximum efficiency at using the hardware.

Tech moves on.

 

[rbpeirce]

The new TM goes beyond that. It "takes over" the "drive" and will block read/write to the "drive" by anything else, including the user. In the past, what you are doing was workable. Today, not so much. Going forward, I anticipate Apple will be getting even more aggressive at protecting the TM backups as part of their emphasis on security.

Thanks. I'll have to watch for that. It might be best to delete any backups older than I need, say three to six months, but I'd like to see what happens when the larger one reaches capacity first. With two TM backups and 2-3 backups of everything else I ought to be okay.

BTW, I think this discussion should be on a different thread. Is there any way to move it? It has nothing to do with Drobo anymore.

 

[MacInWin]

BTW, I think this discussion should be on a different thread. Is there any way to move it? It has nothing to do with Drobo anymore.

Nah, threads wander all over the place here. This one is about read to wind down, I think.