Discussion:
Finding space occupied by files in Tandem
(too old to reply)
Shrikant Mitakari
2021-10-29 13:21:23 UTC
Permalink
Hi All,

I wanted to check the space occupied by files (and how much is available) in each volume and subovolumes on a node. How can we do that? Which command and utility will be useful.

Thank you !
Keith Dick
2021-10-29 23:54:48 UTC
Permalink
Post by Shrikant Mitakari
Hi All,
I wanted to check the space occupied by files (and how much is available) in each volume and subovolumes on a node. How can we do that? Which command and utility will be useful.
Thank you !
DSAP provides a number of ways or reporting. You can get per-file, per subvolume, per user, per volume, by choosing the appropriate options.
It is described in NTL pretty extensively in the Guardian Disk and Tape Utilities Reference Manual.
Good luck,
Randall
Randall is correct that DSAP probably is the program you should use.

Please note that available space is only tracked at the volume level. The idea of asking how much space is available in a subvolume is wrong, so do not expect to find anything in DSAP that would report about that.

DSAP can report how much space is used by all of the files in a subvolume, and how much of that space could be released by deallocating extents of files in the subvolume that have not yet been used, but no space that is not currently part of a file is attached to a subvolume.
Randall
2021-10-30 21:48:32 UTC
Permalink
Post by Keith Dick
Post by Shrikant Mitakari
Hi All,
I wanted to check the space occupied by files (and how much is available) in each volume and subovolumes on a node. How can we do that? Which command and utility will be useful.
Thank you !
DSAP provides a number of ways or reporting. You can get per-file, per subvolume, per user, per volume, by choosing the appropriate options.
It is described in NTL pretty extensively in the Guardian Disk and Tape Utilities Reference Manual.
Good luck,
Randall
Randall is correct that DSAP probably is the program you should use.
Please note that available space is only tracked at the volume level. The idea of asking how much space is available in a subvolume is wrong, so do not expect to find anything in DSAP that would report about that.
Good point. There is no concept of space allocation limits by subvolume. Subvolumes themselves to not take up any space themselves. Even directories in OSS and Linux are mostly virtual, except for minimal space that is used to define the directory's inode - unless they are mount points.
Post by Keith Dick
DSAP can report how much space is used by all of the files in a subvolume, and how much of that space could be released by deallocating extents of files in the subvolume that have not yet been used, but no space that is not currently part of a file is attached to a subvolume.
Don't forget that NONE of this applies directly on vNonStop where drives are virtualized off of underlying disks. SMF has its own stuff too, but we're getting way off topic now.

This is what I meant, yes. Thanks for clarifying.
Keith Dick
2021-10-31 01:22:03 UTC
Permalink
Post by Randall
Post by Keith Dick
Post by Shrikant Mitakari
Hi All,
I wanted to check the space occupied by files (and how much is available) in each volume and subovolumes on a node. How can we do that? Which command and utility will be useful.
Thank you !
DSAP provides a number of ways or reporting. You can get per-file, per subvolume, per user, per volume, by choosing the appropriate options.
It is described in NTL pretty extensively in the Guardian Disk and Tape Utilities Reference Manual.
Good luck,
Randall
Randall is correct that DSAP probably is the program you should use.
Please note that available space is only tracked at the volume level. The idea of asking how much space is available in a subvolume is wrong, so do not expect to find anything in DSAP that would report about that.
Good point. There is no concept of space allocation limits by subvolume. Subvolumes themselves to not take up any space themselves. Even directories in OSS and Linux are mostly virtual, except for minimal space that is used to define the directory's inode - unless they are mount points.
Post by Keith Dick
DSAP can report how much space is used by all of the files in a subvolume, and how much of that space could be released by deallocating extents of files in the subvolume that have not yet been used, but no space that is not currently part of a file is attached to a subvolume.
Don't forget that NONE of this applies directly on vNonStop where drives are virtualized off of underlying disks. SMF has its own stuff too, but we're getting way off topic now.
This is what I meant, yes. Thanks for clarifying.
Your comment that none of this applies to virtual NonStop systems makes me curious. I don't have any experience with virtual NonStop systems. Do you mean that DSAP cannot be used on virtual NonStop systems, or that none of the part of DSAP that reports on free space works on virtual NonStop systems, or some third thing?
Gustavo Martinez
2021-11-01 19:00:36 UTC
Permalink
Post by Randall
Post by Keith Dick
Post by Shrikant Mitakari
Hi All,
I wanted to check the space occupied by files (and how much is available) in each volume and subovolumes on a node. How can we do that? Which command and utility will be useful.
Thank you !
DSAP provides a number of ways or reporting. You can get per-file, per subvolume, per user, per volume, by choosing the appropriate options.
It is described in NTL pretty extensively in the Guardian Disk and Tape Utilities Reference Manual.
Good luck,
Randall
Randall is correct that DSAP probably is the program you should use.
Please note that available space is only tracked at the volume level. The idea of asking how much space is available in a subvolume is wrong, so do not expect to find anything in DSAP that would report about that.
Good point. There is no concept of space allocation limits by subvolume. Subvolumes themselves to not take up any space themselves. Even directories in OSS and Linux are mostly virtual, except for minimal space that is used to define the directory's inode - unless they are mount points.
Post by Keith Dick
DSAP can report how much space is used by all of the files in a subvolume, and how much of that space could be released by deallocating extents of files in the subvolume that have not yet been used, but no space that is not currently part of a file is attached to a subvolume.
Don't forget that NONE of this applies directly on vNonStop where drives are virtualized off of underlying disks. SMF has its own stuff too, but we're getting way off topic now.
This is what I meant, yes. Thanks for clarifying.
Your comment that none of this applies to virtual NonStop systems makes me curious. I don't have any experience with virtual NonStop systems. Do you mean that DSAP cannot be used on virtual NonStop systems, or that none of the part of DSAP that reports on free space works on virtual NonStop systems, or some third thing?
DSAP will report the amount of space used on the virtualized the disk, like $DATA01. However, you cannot tell, relative to the underlying hardware, how much space is used in reality. Docker is a bit different because root has the ability to look inside the Docker container to find out what is really going on. Fragmentation on the underlying hardware is invisible to DSAP, so the real use may be masked - unless the partition is managed as a contiguous block by the virtualization OS. I suspect it mostly is, but from experience, this is not an absolute. Speaking from a bit of frustration on our main CI/CD box, the 5 Ubuntu VMs share space out of the root partition of the Gentoo OS, so I really cannot tell where a particular file is located or how fragmented it might be, from the point of view of the VM itself. I have to drop into Gentoo to look at the allocations. Sure, this is different than the vNonStop recipe, but I have not yet seen the details and do not know whether this will
change in the future. Basically, in vNonStop-land, you follow the VM rules, so the traditional NonStop view of disk subsystems is not the real picture of what is going on on the hardware. That was really my point.
I don't understand most of what you wrote, probably because I have very little experience with using any kind of virtual machines.
Let me ask this: Outside of DSAP, what effect does this have on other software that runs on the NonStop system? For example, I have some vague recollection that the sort package chooses where to put its scratch files by looking at the free space available on the different disk volumes. I think some of the SQL query executor does something similar. Do those pieces of software see enough information about free space so they make good choices?
The OP was asking about space usage, seemingly with concern a out files and discs that are reaching capacity.
The lack of insight into the underlying fragementation of the files themselves is not unique to vNonStop, it also applies to usage of a SAN to host the volumes.
I think that DSAP is the right tool for the OP based on his original question (see above).
Bill
Hi all,

You can also check Myinfo from https://www.greenhouse.de/freeware/guardian-freeware/

Regards,
Gustavo.
j-ma...@pacbell.net
2021-11-02 20:54:11 UTC
Permalink
When you virtualize disk space, you usually relinquish the one-to-one relationship between the disk space that the guest operating system (i.e. NonStop) thinks it has access to and the amount of physical storage that sits behind the virtual disk.

I think we are getting beyond the realm of specific NonStop implementation into the general area of storage virtualization. The NonStop documentation is assuming that you are already familiar with these computer science concepts. Perhaps it would help you to start with some research into the general topic. The following Wiki articles might be a good place to start:

https://en.wikipedia.org/wiki/Virtual_disk_and_virtual_drive
https://en.wikipedia.org/wiki/Storage_virtualization
https://en.wikipedia.org/wiki/Software-defined_storage

Jon Marcus
Bill Honaker
2021-11-02 21:50:25 UTC
Permalink
Post by Gustavo Martinez
Post by Randall
Post by Keith Dick
Post by Shrikant Mitakari
Hi All,
I wanted to check the space occupied by files (and how much is available) in each volume and subovolumes on a node. How can we do that? Which command and utility will be useful.
Thank you !
DSAP provides a number of ways or reporting. You can get per-file, per subvolume, per user, per volume, by choosing the appropriate options.
It is described in NTL pretty extensively in the Guardian Disk and Tape Utilities Reference Manual.
Good luck,
Randall
Randall is correct that DSAP probably is the program you should use.
Please note that available space is only tracked at the volume level. The idea of asking how much space is available in a subvolume is wrong, so do not expect to find anything in DSAP that would report about that.
Good point. There is no concept of space allocation limits by subvolume. Subvolumes themselves to not take up any space themselves. Even directories in OSS and Linux are mostly virtual, except for minimal space that is used to define the directory's inode - unless they are mount points.
Post by Keith Dick
DSAP can report how much space is used by all of the files in a subvolume, and how much of that space could be released by deallocating extents of files in the subvolume that have not yet been used, but no space that is not currently part of a file is attached to a subvolume.
Don't forget that NONE of this applies directly on vNonStop where drives are virtualized off of underlying disks. SMF has its own stuff too, but we're getting way off topic now.
This is what I meant, yes. Thanks for clarifying.
Your comment that none of this applies to virtual NonStop systems makes me curious. I don't have any experience with virtual NonStop systems. Do you mean that DSAP cannot be used on virtual NonStop systems, or that none of the part of DSAP that reports on free space works on virtual NonStop systems, or some third thing?
DSAP will report the amount of space used on the virtualized the disk, like $DATA01. However, you cannot tell, relative to the underlying hardware, how much space is used in reality. Docker is a bit different because root has the ability to look inside the Docker container to find out what is really going on. Fragmentation on the underlying hardware is invisible to DSAP, so the real use may be masked - unless the partition is managed as a contiguous block by the virtualization OS. I suspect it mostly is, but from experience, this is not an absolute. Speaking from a bit of frustration on our main CI/CD box, the 5 Ubuntu VMs share space out of the root partition of the Gentoo OS, so I really cannot tell where a particular file is located or how fragmented it might be, from the point of view of the VM itself. I have to drop into Gentoo to look at the allocations. Sure, this is different than the vNonStop recipe, but I have not yet seen the details and do not know whether this
will
Post by Gustavo Martinez
change in the future. Basically, in vNonStop-land, you follow the VM rules, so the traditional NonStop view of disk subsystems is not the real picture of what is going on on the hardware. That was really my point.
I don't understand most of what you wrote, probably because I have very little experience with using any kind of virtual machines.
Let me ask this: Outside of DSAP, what effect does this have on other software that runs on the NonStop system? For example, I have some vague recollection that the sort package chooses where to put its scratch files by looking at the free space available on the different disk volumes. I think some of the SQL query executor does something similar. Do those pieces of software see enough information about free space so they make good choices?
The OP was asking about space usage, seemingly with concern a out files and discs that are reaching capacity.
The lack of insight into the underlying fragementation of the files themselves is not unique to vNonStop, it also applies to usage of a SAN to host the volumes.
I think that DSAP is the right tool for the OP based on his original question (see above).
Bill
Hi all,
You can also check Myinfo from https://www.greenhouse.de/freeware/guardian-freeware/
Regards,
Gustavo.
Well, Bill, the OP did ask about space available, too, so I'm not sure why you seem to be ruling out discussion of that.
It is interesting that you mention that some of the same questions about managing free space apply to using a SAN for the disk volumes of a NonStop system. I have never heard anything about special considerations for disk volumes on SANs, probably because I have never been responsible for managing a NonStop system that used SANs for the disks.
Do you know of a manual that explains the differences between managing disk space on volumes of a NonStop system that are traditional individual physical disks vs. managing space on volumes that are on a SAN or on a virtrual NonStop system? If so, I'd appreciate it if you could tell me which manual contains such explanations. If you don't know of a manual that explains the differences, could you give a very short explanation of what differences there are? (Just in a very few words if you don't have time to write a thorough explanation.)
Keith,

The storage space used by the files, and overall on a per volume basis, are all correct from DSAP no matter what.

The area that is different deals with extents and fragmentation.
It still reports it correctly, but since the underlying data is completely virtualized (both in XP and in vNS), it's much less useful.

My meaning was that, if using SAS disks, the Disk Process's belief of where the data lies is straightforward.
If using Solid State drives, it's less so.
But for either virtualized solution, there is no real correlation with what is assumed by the disk process and the underyling reality.

I hope that is clearer.
Bill
Randall
2021-11-03 18:52:33 UTC
Permalink
Post by Bill Honaker
But for either virtualized solution, there is no real correlation with what is assumed by the disk process and the underyling reality.
That was actually my point (but you said it way better than I did). I was not sure what the OP really wanted to know - although the DSAP answer still is the most relevant to NonStop allocation even if it is sometimes not connected to the underlying reality.

I guess it is top of mind because I am trying to work out my own vNS and also about to reformat VMWare off one of my servers and replacing it with Debian/KVM - but that's another story and OS - and expecting the disk allocations to radically change.
wilso...@gmail.com
2021-11-15 13:13:20 UTC
Permalink
Post by Shrikant Mitakari
Hi All,
I wanted to check the space occupied by files (and how much is available) in each volume and subovolumes on a node. How can we do that? Which command and utility will be useful.
Thank you !
A quick way I have been checking for large files on a given disk, is to cross over to OSS and run the "du" command. If you have a lot of files on a disk, it will take a little bit of processing to return the results, but it is a quick and easy way I use frequently.

For example, after starting OSH, this command will return the largest 50 files on the disk:
du -a /G/DISK2 | sort -n -r | head -n 50

Loading...