Discussion:
Disk/File fragmentation
(too old to reply)
Igor
2017-06-26 11:30:39 UTC
Permalink
Hello Tandem folks!

Is it possible to generate a list of fragmented files (sorted by # of fragments) on a specific disk? If yes, how? Any available tools for this purpose?

Regards,
Igor
Randall
2017-06-26 19:31:54 UTC
Permalink
Post by Igor
Hello Tandem folks!
Is it possible to generate a list of fragmented files (sorted by # of fragments) on a specific disk? If yes, how? Any available tools for this purpose?
Regards,
Igor
Have a look at DSAP and DCOM in the Guardian Disk and Tape Utilities Reference Manual.
Keith Dick
2017-06-26 22:07:41 UTC
Permalink
Post by Igor
Hello Tandem folks!
Is it possible to generate a list of fragmented files (sorted by # of fragments) on a specific disk? If yes, how? Any available tools for this purpose?
Regards,
Igor
It is not clear to me exactly what you mean when you say "fragmented files". Asking about a fragmented disk makes sense in a NonStop system. That would be a disk that has a lot of small, discontiguous free space areas. The closest to that that occurs for a file is how many blocks of the file currently are allocated to the file, contained data at some point, but currently contain no records.

DSAP can display the free space on a disk. It displays the number of fragments of each size, smallest to largest. You can see a sample of that in the DSAP manual, in the section on "Space Distribution Reports" (page 5-20 in the copy of the manual I am looking at). I believe this is not what you want, so I will not say anything more about it.

DSAP does include a column in its DETAIL output that gives Unused Pages and one that gives Dealloc Pages, but neither of them count unused blocks within a file -- they only report the number of blocks assigned to a file that are beyond the current EOF, not unused blocks before the current EOF of the file. You can use the UNUSED option of DSAP to select files with a given number of unused pages, or more. However that is an absolute number, not relative to the size of the file. If that gives you what you want, you can use it. As far as I know, there is no option to sort the report by the number of unused pages, but perhaps you could post-process the report with a TACL macro or other program you write to put it into that order.

DSAP allows you to capture the raw data it uses about the files into an Enscribe file, and you can use Enform to write a report to display the information in ways that suit you better than the preprogrammed DSAP reports do, so you might make use of that to sort the data the way you want, and perhaps to take the total size of a file into account when selecting which files have "too many" unused pages. The section "Format of the Permanent Work File" gives more information about using the raw data that DSAP uses.

The only utility I know of that displays the unused blocks before the EOF of a file is FUP INFO with the DETAIL or STAT options. Among the results is the number of FREE blocks, but as far as I know, FUP provides no way to select files based on the number of FREE blocks. I guess you could get FUP to produce INFO DETAIL output for some set of files you want to examine for fragmentation and post-process the FUP output with some TACL code (or write a program in another language) to pick out the files that have a large number of free blocks.

There might be someone who already has such a program who would be willing to share it with you, but I do not know of any such program.
Tone
2017-06-27 00:02:56 UTC
Permalink
Post by Igor
Hello Tandem folks!
Is it possible to generate a list of fragmented files (sorted by # of fragments) on a specific disk? If yes, how? Any available tools for this purpose?
Regards,
Igor
As others have mentioned, DSAP is good at analyzing things at a disk level.

If you are talking about files with lots of extents the following TACL
script may be of interest. It currently displays primary, secondary and
number of extents for files with more than 99 extents and accepts a
filename template.

?tacl routine
#frame
#push prev fname pattern done pri sec exts
#if [#argument/value pattern/template]
#set done 0
#set prev
[#loop |do|
#set fname [#filenames/previous [prev],maximum 1/[pattern]]
[#if not [#emptyv fname] |then|
#setmany fname pri sec exts, &
[#fileinfo/fullname,primary,secondary,extentsallocated/[fname]]
[#if exts > 99 |then|
#output /width 26,hold/ [fname]
#output /width 6, hold,justify right/[pri]
#output /width 6, hold,justify right/[sec]
#output /width 6,justify right/[exts]
]
#set prev [fname]
|else|
#set done -1
]
|until| [done]
]
#unframe

It does not do any sorting, but you could capture the output and use
EXCEL to sort.

If you see a file with lots of extents, you can use FUP to list the
extents and see if they are contiguous. For example :

FUP INFO BIGFILE,EXTENTS

If you are talking about badly organised key-sequenced files then check
out the Tandem Reload Analyzer :

http://www.nonstoptra.com/index.htm
Igor
2017-06-27 07:07:26 UTC
Permalink
It looks I was to stingy with my words while asking :)

I really do not wont to change any files, I know that there could be some not used pages (and so could be deallocateable), and that some files can have many allocated used & unused extents (which is the real cause of disk fragmentation). But, are they contiguous on disk physical space?
I am aware of DSAP and DCOM and using it for analyzing disk usage and performing disk defragmentation (we have a NetBatch job which runs DCOM every 2 days). I have put this question because on our test/devl system one of the disks (with a lot of different software) the number of free-space fragments become quite a big number (approx.. 9000, it can be seen with DSAP SHORT report). I am also totally aware that this does not cause any performance problem – but I am curious. Since we recently install & tested two new big application, I would like to know if those files (or any other files) are contiguous on disk – if all allocated extents are following one by another. It could have helped us to better allocate files in case we moved to production.

DCOM is not capable to defragment the disk since there are too many opens files and finishes with errors:
1: Process terminated with warning diagnostics
Termination Info: 2
Subsystem: TANDEM.32.D30
Disk structure diagnostics or I/O errors.

I know that DCOM could do better job if application (that have open files) will be stopped.

I read somewhere that Web/ViewPoint and Guardian Performance Analyzer can produce such a report, but unfortunately, we do not have it.

Does that make any sense to you?
Is there any real need to have file extents in contiguous disk space at all?
Does this simple explanation on Wikipedia apply to NonStop (HDD, not SSD of course) disk at all?
https://en.wikipedia.org/wiki/Defragmentation

Thank you, Keith, for all your hints, but as I said, I really do not want to change any files, only solve disk fragmentation.

Tone, thank you for your macro! Your hint is good and I already know about it, but it could take a lot of time for me to manually check all the files… maybe I really should create another macro :)
Tandem Reload Analyzer: yes, the logic behind is the same, but here for disk fragmentation. Good to know about this tool, I will talk with my colleagues, maybe will find useful.
wbreidbach
2017-06-27 10:20:06 UTC
Permalink
If a file or table does have one extent only there might a s small performance improvement. If you look at a disk where many files are created and deleted again, you usually will see many free extents. DCOM is the only available tool for defragmentation. On a heavily used disk using DCOM will not be really successful.
From my experience the fragmentation within a key-sequenced file, table or index can cause more performance problems than the disk fragmentation.
TRA can analyze the fragmentation, but as far as I know, it does not come for free. FUP is able to analyze the free space within such a file but it dows not give any information about the fragmentation of the data or index level.
We have automatized the reloading of tables and files with a program analyzing the file or table similar to TRA and starting a reload if necessary. So we usually do not need to think about reloads.
Igor
2017-06-27 14:19:10 UTC
Permalink
Interesting:

the FUP INFO <file name>, EXTENTS command lists the extents and starting page as expected, but only on Enscribe files.

It's not working on OSS files. The output is even equal if used with DETAIL and EXTENTS option. Any idea why?

-info $DATA05.ZYQ00001.Z00008N1, extents

$DATA05.ZYQ00001.Z00008N1 27 Jun 2017, 16:06
OSS
PATH: /home/dev0/base24_eps_111_bu11o.tar
OWNER {intentionally hidden by me}
SECURITY: -rw-r--r--
CREATION DATE: 13 May 2015, 12:16
ACCESS TIME: 1 Jun 2017, 22:32
EOF: 866703872


-info $DATA05.ZYQ00001.Z00008N1, detail

$DATA05.ZYQ00001.Z00008N1 27 Jun 2017, 16:06
OSS
PATH: /home/dev0/base24_eps_111_bu11o.tar
OWNER {intentionally hidden by me}
SECURITY: -rw-r--r--
CREATION DATE: 13 May 2015, 12:16
ACCESS TIME: 1 Jun 2017, 22:32
EOF: 866703872
Keith Dick
2017-06-27 17:21:01 UTC
Permalink
Post by Igor
the FUP INFO <file name>, EXTENTS command lists the extents and starting page as expected, but only on Enscribe files.
It's not working on OSS files. The output is even equal if used with DETAIL and EXTENTS option. Any idea why?
-info $DATA05.ZYQ00001.Z00008N1, extents
$DATA05.ZYQ00001.Z00008N1 27 Jun 2017, 16:06
OSS
PATH: /home/dev0/base24_eps_111_bu11o.tar
OWNER {intentionally hidden by me}
SECURITY: -rw-r--r--
CREATION DATE: 13 May 2015, 12:16
ACCESS TIME: 1 Jun 2017, 22:32
EOF: 866703872
-info $DATA05.ZYQ00001.Z00008N1, detail
$DATA05.ZYQ00001.Z00008N1 27 Jun 2017, 16:06
OSS
PATH: /home/dev0/base24_eps_111_bu11o.tar
OWNER {intentionally hidden by me}
SECURITY: -rw-r--r--
CREATION DATE: 13 May 2015, 12:16
ACCESS TIME: 1 Jun 2017, 22:32
EOF: 866703872
I have only speculation about this. I do not know how space allocation for OSS files is done.

As I understand it, disk space allocation on typical Unix systems is done a bit different than the way it is is done for Guardian files. I do not remember the details, but it involves something like progressively larger blocks of space being allocated to a file automatically as it grows. I suppose that when OSS was implemented, something about the way space is allocated to OSS files was done differently enough that the usual way FUP obtained or displayed the extent information did not work, and it was not considered important enough to write the additional code that would have been needed to display the disk space allocation information for OSS files.

Back to your original question. Yes, the way you asked the question originally was not very clear. In theory, there could be a performance advantage in some circumstances if a file's extents were contiguous on disk. That usually is overwhelmed by other factors that affect performance a lot more, so people usually do not make much effort to keep the extents of a file together on disk.

The main issue that led to creation of DCOM is that, as I imagine you understand, when the free space is fragmented, it might be impossible to create a given file because, even though there is plenty of free space available, there are no fragments large enough to contain the primary extent requested for the file. The same thing can prevent a file from growing into a new extent. DCOM moves around the extents of files on the disk, trying to end up with fewer but larger fragments of free space on the disk. It can only move extents of files that are not open (and I have a feeling there is some other restriction that I am not thinking of at the moment), so it cannot always make much progress on consolidating the free space on a disk.

The error you mention you got when running DCOM seems to indicate that DCOM detected a problem with the data on the disk, either a disk I/O error or some invalid data in the disk directory or a file label. I don't know whether you have looked into that further, but it seems to me to be something you should check, since it might mean some of the data on that disk is in danger of being lost or corrupted.

Before DCOM was created, the way to clear up fragmentation on a disk was to do a full backup of the disk to tape, relabel the disk, which makes it completely empty, then restore the full backup that you just made. That ensured that all the free space on the disk was in one large fragment, but is a slow and error-prone procedure. I do not recommend trying that today for most situations. Something that you could do if the disk is in a state that DCOM cannot work on it would be to choose a couple fairly large files, DUP them to another disk (or back them up to tape), purge those files, shut down as many of the programs using the disk as possible, then try to run DCOM. Then you can DUP the files you moved back to the disk, if you want to keep them on that disk. But before you do something like this, check into the reason that DCOM reported that error you mentioned and correct it before trying to reorganize the free space on that disk.
Igor
2017-06-30 05:21:37 UTC
Permalink
Keith, thank you for detailed explanation.

As I mentioned in previous post the problem are open files extents which DCOM is not able to move. The error looks like this:

\PERUN.$DATA05.#Cancel Move from DP2 Swap procedure :12, canceling <---- ERROR
Canceled move: 26130614 16384 2 DEV1ATLF.TL170630

I know that there is nothing I can do about it, except to stop the application and rerun DCOM. Since there are no performance nor file allocation problems, I wouldn't go further with steps you mentioned it (backup/restore, fup dup files, etc.)

I have put a question here because I would just like understand what is going on that disk and to know which files are fragmented.
Randall
2017-07-03 19:00:53 UTC
Permalink
Post by Igor
Keith, thank you for detailed explanation.
\PERUN.$DATA05.#Cancel Move from DP2 Swap procedure :12, canceling <---- ERROR
Canceled move: 26130614 16384 2 DEV1ATLF.TL170630
I know that there is nothing I can do about it, except to stop the application and rerun DCOM. Since there are no performance nor file allocation problems, I wouldn't go further with steps you mentioned it (backup/restore, fup dup files, etc.)
I have put a question here because I would just like understand what is going on that disk and to know which files are fragmented.
You cannot move SWAP files because they are actively mapping directly to through virtual memory. Your assumption that you would have to stop the application is correct - at least the process using it. In some environments, say under Pathway, you can stop unused servers (0 links) to allow some reclamation. Using backup/restore is not a good fix as you will not be able to successfully back-up swap files anyway - you might get part, but backing up an open file with unwritten content, characteristic of virtual memory, is an unsafe operation. There is no real performance benefit to doing this anyway.

For SQL/MX, SQL/MP, and data files, you have much more flexibility. Online reload (FUP), deallocation on unused extents (info in DSAP, done by FUP), can help reclaim disk fragments. Online reload can also speed things by by cleaning up index organization allowing faster primary key access particularly if the application data growth pattern allows you to reduce slack - more PK entries per disk I/O is faster.

Good luck,
Randall

Continue reading on narkive:
Loading...