Collector ems stats based on ssid and event number

Discussion:

(too old to reply)

Pramod Suryawanshi

2020-10-15 18:25:44 UTC

Dear members,
Is there a way to get the ems stats (number of particular event in specific timeframe) based on ssid and/or event number.

Example -
SSID : TANDEM.EMS.01, Event Number : 512 occurred X number of times between timeframe dd-mm hh:min to dd-mm hh:min

JShepherd

2020-10-16 21:55:35 UTC

Permalink

Post by Pramod Suryawanshi
Dear members,
Is there a way to get the ems stats (number of particular event in specific

time

Post by Pramod Suryawanshi
frame) based on ssid and/or event number.
Example -
SSID : TANDEM.EMS.01, Event Number : 512 occurred X number of times between

time

Post by Pramod Suryawanshi
frame dd-mm hh:min to dd-mm hh:min

You have a lot of flexibility in scanning EMS logs if you read them directly.

Reading and deblocking is easy,
all the work is in filtering and displaying the events.

Below are a some snippets from an EMS reader.

A physical record from the EMS logfile can contain multiple logical records
concatenated together, you have to iterate thru the buffer after each read..

zems_val_ssid_def ssid;
zems_ddl_msg_buffer_def ems_buf;
short *pemsbuff; /* for multiple events/record */

erc = FILE_OPEN_(logfilename, logfilename_len, &logfile,
ZSYS_VAL_OPENACC_READONLY,
ZSYS_VAL_OPENEXCL_SHARED);

short process_logfile(void)
{
int32_t done;

for (done = 0; done == 0; )
{

/* from FILE_GETINFOBYNAME_() on the EMS logfile */
read_len = fi.maxreclen;

READX(logfile, (char *)&ems_buf, read_len, &read_len);
FILE_GETINFO_(logfile, &erc);
if (erc != 0)
{
FILE_CLOSE_(logfile);
return(erc);
}

logfile_reads++;
erc = process_events(read_len);
if (erc)
{
break;
}

} /* for */

return(0);
}

short process_events(int32_t bytes_in_buff)
{
pemsbuff = (short *)&ems_buf; /* point to the start of the buffer */

while (bytes_in_buff > 0)
{
msglen = 0;
ems_err = SSGETTKN(pemsbuff, ZSPI_TKN_BUFLEN, (char *)&msglen);
ems_err = SSGETTKN(pemsbuff, ZSPI_TKN_SSID, (char *)&ssid);
ems_err = SSGETTKN(pemsbuff, ZEMS_TKN_EVENTNUMBER,
(char *)&eventnumber);
ems_err = SSGETTKN(pemsbuff, ZEMS_TKN_PROC_DESC, senderid);
if (ems_err)
{
ems_err = SSGETTKN(pemsbuff, ZEMS_TKN_CRTPID, senderid);
}
else
{
flags.process_descr = 1;
}

ems_err = SSGETTKN(pemsbuff, ZEMS_TKN_SYSTEM, (char *)&sysnum);
ems_err = SSGETTKN(pemsbuff, ZEMS_TKN_LOGTIME,
(char *)&ems_logtime);

/* apply filters that don't need the message text */
/* Time, SSID, Eventnumber, process, etc */

/* if you want the displayable text for an event */
ems_err = EMSTEXT((short *)pemsbuff,
(char *)&ems_msg_text,
maxlinelen,
maxlines,
(short*)&actual_len,
/* (short*)&header_key */ ,
0,
0);

/* apply filters that do need the message text */

/* display, count, total by SSID, etc the filtered event */

/* check to see if there are more events in this buffer */
bytes_in_buff = bytes_in_buff - msglen;

/* point to the next logical record */
pemsbuff = &pemsbuff[msglen/2];

} /* while */

Keith Dick

2020-10-17 07:28:30 UTC

Permalink

I wouldn't recommend reading an EMS log file directly. Start a consumer distributor and let it return the individual events to your program.

That way, the only thing you have to do in the program is pull out whatever information you want from the event. You can let the consumer distributor handle opening and closing the log file, selecting which events you want (using an EMS Filter or Filter Table), rolling to the next log file in sequence when you reach the end of the current file, starting at a particular time in the log, etc.

Rob Lesan

2020-10-19 18:14:19 UTC

Permalink

Post by Pramod Suryawanshi
Dear members,
Is there a way to get the ems stats (number of particular event in specific timeframe) based on ssid and/or event number.
Example -
SSID : TANDEM.EMS.01, Event Number : 512 occurred X number of times between timeframe dd-mm hh:min to dd-mm hh:min

We have all tried WAY too hard to get this. Here is a one liner that will do it in OSS:

gtacl -c 'purge emsout';gtacl -c 'emsdist ty p,co $0,ti 00:00,st eof,te emsout';gtacl -c 'fup copy emsout' | grep .*-.*-.*:.*: | awk '{print $4}' | sort | uniq -c | sort -n

The above command purges a Guardian file, starts an EMS distributor on $0 from 00:00 today and stop at the end of the current log writing to a file in my home subvol, the fup copies it out through grep looking for a time and date string. It runs that output via awk and prints the fourth value, then sorts that and gives you a list of unique values sorted by their counts in ascending order.

You can up date the start, stop and collector values to whatever you wish:

ti = start time (set to 00:00)
st = stop time (set to EOF)
co = collector (set to $0)

Keith Dick

2020-10-19 19:47:13 UTC

Permalink

Post by Rob Lesan

gtacl -c 'purge emsout';gtacl -c 'emsdist ty p,co $0,ti 00:00,st eof,te emsout';gtacl -c 'fup copy emsout' | grep .*-.*-.*:.*: | awk '{print $4}' | sort | uniq -c | sort -n
The above command purges a Guardian file, starts an EMS distributor on $0 from 00:00 today and stop at the end of the current log writing to a file in my home subvol, the fup copies it out through grep looking for a time and date string. It runs that output via awk and prints the fourth value, then sorts that and gives you a list of unique values sorted by their counts in ascending order.
ti = start time (set to 00:00)
st = stop time (set to EOF)
co = collector (set to $0)

Thanks. That seems pretty good, though I have not tried it. Have you tested it to be sure it works?

I think it does not do quite what the original request was -- he wanted counts of ssid and/or event number. Since this is only printing one field with the awk command, I think it would not show counts by ssid and event number, but I suppose that would be easy to change by modifying the awk print command to {print $4 $5}.

I think there could be a problem if the text of a long event message contains a timestamp and that timestamp falls into the second line of the event's display. I don't know whether there actually are any event messages that would trigger that problem.

Rob Lesan

2020-10-19 20:34:25 UTC

Permalink

Post by Rob Lesan

Thanks. That seems pretty good, though I have not tried it. Have you tested it to be sure it works?
I think it does not do quite what the original request was -- he wanted counts of ssid and/or event number. Since this is only printing one field with the awk command, I think it would not show counts by ssid and event number, but I suppose that would be easy to change by modifying the awk print command to {print $4 $5}.
I think there could be a problem if the text of a long event message contains a timestamp and that timestamp falls into the second line of the event's display. I don't know whether there actually are any event messages that would trigger that problem.

Thanks for the feedback Keith. I ran this on three different hosts and it worked fine.

You are correct on the ssid vs event number request. Easily modified to generate any combination of values.

I don't see a problem with long lines as the EMSDIST output should always have the time and event number on the same line. The regex I included looks for a timestamp followed by a string with two dots. Again, the user can update that value if necessary to look for anything.

I just tried it again and I ran into a new problem. My outfile (emsout) is too small. The user may wish to hard code the filename and create a larger file.

I will continue to refine this...

Keith Dick

2020-10-19 23:05:13 UTC

Permalink

Post by Rob Lesan

Thanks. That seems pretty good, though I have not tried it. Have you tested it to be sure it works?
I think it does not do quite what the original request was -- he wanted counts of ssid and/or event number. Since this is only printing one field with the awk command, I think it would not show counts by ssid and event number, but I suppose that would be easy to change by modifying the awk print command to {print $4 $5}.
I think there could be a problem if the text of a long event message contains a timestamp and that timestamp falls into the second line of the event's display. I don't know whether there actually are any event messages that would trigger that problem.

Thanks for the feedback Keith. I ran this on three different hosts and it worked fine.
You are correct on the ssid vs event number request. Easily modified to generate any combination of values.
I don't see a problem with long lines as the EMSDIST output should always have the time and event number on the same line. The regex I included looks for a timestamp followed by a string with two dots. Again, the user can update that value if necessary to look for anything.
I just tried it again and I ran into a new problem. My outfile (emsout) is too small. The user may wish to hard code the filename and create a larger file.
I will continue to refine this...

I'm glad you did try it and know that it works.

For the problem of the output file for the formatted events being too small, I am a little puzzled that you said that could be fixed by hard-coding the file name and creating a larger file, since you already have hard-coded a file name for that. I imagine that you only need to add as the second command something like:

gtacl -c 'fup create emsout,type e,rec 80,ext(1000,1000)'

You are correct that the timestamp always is on the first line of the formatted event, and the regular expression will match it and the ssid and event number will also be on that line (unless the site has replaced the default formatting template used to format the header info). The potential issue I had in mind was that if the body of the event, when formatted, included a timestamp that also matches your regular expression and it appears far enough into the formatted text that it appears in the second or following line of the formatted event. Then the grep would match that line, which would not be the beginning of the formatted event, and the awk print would output a line that did not contain the ssid and event number.

Maybe that would be avoided if the regular expression included the symbol to make it match only at the start of the line.

Rob Lesan

2020-10-20 13:41:05 UTC

Permalink

Post by Keith Dick

Post by Rob Lesan

Thanks. That seems pretty good, though I have not tried it. Have you tested it to be sure it works?
I think it does not do quite what the original request was -- he wanted counts of ssid and/or event number. Since this is only printing one field with the awk command, I think it would not show counts by ssid and event number, but I suppose that would be easy to change by modifying the awk print command to {print $4 $5}.
I think there could be a problem if the text of a long event message contains a timestamp and that timestamp falls into the second line of the event's display. I don't know whether there actually are any event messages that would trigger that problem.

Thanks for the feedback Keith. I ran this on three different hosts and it worked fine.
You are correct on the ssid vs event number request. Easily modified to generate any combination of values.
I don't see a problem with long lines as the EMSDIST output should always have the time and event number on the same line. The regex I included looks for a timestamp followed by a string with two dots. Again, the user can update that value if necessary to look for anything.
I just tried it again and I ran into a new problem. My outfile (emsout) is too small. The user may wish to hard code the filename and create a larger file.
I will continue to refine this...

I'm glad you did try it and know that it works.
gtacl -c 'fup create emsout,type e,rec 80,ext(1000,1000)'
You are correct that the timestamp always is on the first line of the formatted event, and the regular expression will match it and the ssid and event number will also be on that line (unless the site has replaced the default formatting template used to format the header info). The potential issue I had in mind was that if the body of the event, when formatted, included a timestamp that also matches your regular expression and it appears far enough into the formatted text that it appears in the second or following line of the formatted event. Then the grep would match that line, which would not be the beginning of the formatted event, and the awk print would output a line that did not contain the ssid and event number.
Maybe that would be avoided if the regular expression included the symbol to make it match only at the start of the line.

Again, thanks for that! It makes much more sense to have the file pre-created and allocated or at least fully qualified in the statement.

FUP CREATE $DSMSCM.TEMP.EMSOUT,TYPE E,EXT (1000,1000),BUFFERED

gtacl -c 'fup purgedata $dsmscm.temp.emsout';gtacl -c 'emsdist ty p,co $0,ti 00:00,st eof,te $dsmscm.temp.emsout';gtacl -c 'fup copy $dsmscm.temp.emsout' | grep ^[0-9] | awk '{print $4,$5}' | sort | uniq -c | sort -n

I also updated the REGEX and added the event number based on your comments. EMSDIST always puts the timestamp in the first column and all other data is indented, so this makes it much simpler.

What else can we add to this?

Randall

2020-10-20 18:45:06 UTC

Thanks. That seems pretty good, though I have not tried it. Have you tested it to be sure it works?
I think it does not do quite what the original request was -- he wanted counts of ssid and/or event number. Since this is only printing one field with the awk command, I think it would not show counts by ssid and event number, but I suppose that would be easy to change by modifying the awk print command to {print $4 $5}.
I think there could be a problem if the text of a long event message contains a timestamp and that timestamp falls into the second line of the event's display. I don't know whether there actually are any event messages that would trigger that problem.

Thanks for the feedback Keith. I ran this on three different hosts and it worked fine.
You are correct on the ssid vs event number request. Easily modified to generate any combination of values.
I don't see a problem with long lines as the EMSDIST output should always have the time and event number on the same line. The regex I included looks for a timestamp followed by a string with two dots. Again, the user can update that value if necessary to look for anything.
I just tried it again and I ran into a new problem. My outfile (emsout) is too small. The user may wish to hard code the filename and create a larger file.
I will continue to refine this...

Again, thanks for that! It makes much more sense to have the file pre-created and allocated or at least fully qualified in the statement.
FUP CREATE $DSMSCM.TEMP.EMSOUT,TYPE E,EXT (1000,1000),BUFFERED
gtacl -c 'fup purgedata $dsmscm.temp.emsout';gtacl -c 'emsdist ty p,co $0,ti 00:00,st eof,te $dsmscm.temp.emsout';gtacl -c 'fup copy $dsmscm.temp.emsout' | grep ^[0-9] | awk '{print $4,$5}' | sort | uniq -c | sort -n
I also updated the REGEX and added the event number based on your comments. EMSDIST always puts the timestamp in the first column and all other data is indented, so this makes it much simpler.
What else can we add to this?

Just a reminder (from my previous post), that the EMSTEXT call used by EMSDIST to format messages is expensive. If you can avoid it to just get the data you want, it's easier on the system.

Cheers,
Randall