Can you give me tips for porting open source projects to NonStop system?

[Keith Dick]

The place I work uses Google Protocol Buffers for formatting messages that get sent between systems of different architectures. In the case of interest to me, that is between a big Java program running on Linux systems and the code I maintain on J-series and L-series NonStop systems.

Many years ago, I ported the then-current version of Google Protocol Buffers to the NonStop system, and it took me a LOT of time to do so. I don't recall exactly how long, but it was at least several weeks, and might have been over a month;. I have used that old version ever since then because of the large effort I expected it would take to port newer versions. But my managers want to move up to the current version of Google Protocol Buffers, so I'll need to dive in and port the current version.

What I'm hoping to get in response to this post is advice about how to go about doing a port of an open source package to the NonStop environment, both J-series and L-series. I recall the general approach of running the configure step, the make step, and the install step that most open source packages use. Google Protocol Buffers did use that approach (I assume the current version still follows that approach). I don't recall the details of the problems I had in the prior port, but I recall having many, many errors in both the configure step and the make step that took a lot of effort to find ways to get around.

What are things that might not be obvious to someone not very experienced in porting stuff to the NonStop system? Don't assume anything is so obvious that it doesn't need to be mentioned. I would much rather have you tell me things I already know than not to learn about something I don't know. Also, what things are there that people sometimes do when trying to port software that might seem like the normal thing to do, but actually should be avoided because they lead to problems?

[Randall]

The place I work uses Google Protocol Buffers for formatting messages that get sent between systems of different architectures. In the case of interest to me, that is between a big Java program running on Linux systems and the code I maintain on J-series and L-series NonStop systems.
Many years ago, I ported the then-current version of Google Protocol Buffers to the NonStop system, and it took me a LOT of time to do so. I don't recall exactly how long, but it was at least several weeks, and might have been over a month;. I have used that old version ever since then because of the large effort I expected it would take to port newer versions. But my managers want to move up to the current version of Google Protocol Buffers, so I'll need to dive in and port the current version.
What I'm hoping to get in response to this post is advice about how to go about doing a port of an open source package to the NonStop environment, both J-series and L-series. I recall the general approach of running the configure step, the make step, and the install step that most open source packages use. Google Protocol Buffers did use that approach (I assume the current version still follows that approach). I don't recall the details of the problems I had in the prior port, but I recall having many, many errors in both the configure step and the make step that took a lot of effort to find ways to get around.
What are things that might not be obvious to someone not very experienced in porting stuff to the NonStop system? Don't assume anything is so obvious that it doesn't need to be mentioned. I would much rather have you tell me things I already know than not to learn about something I don't know. Also, what things are there that people sometimes do when trying to port software that might seem like the normal thing to do, but actually should be avoided because they lead to problems?

What language is the package? There are different issues with different languages. With C, you will need to understand the dependencies. If the project absolutely depends on gcc, which many do, you are not going to be able to port it without significant effort. Understanding what compilers are involved is really important. It also depends on whether you are going a "light" vs. "deep" port. If you are just trying to get it to run, that is one thing, but run efficiently is another.

Good luck,
Randall

[red floyd]

Post by Randall

The place I work uses Google Protocol Buffers for formatting messages that get sent between systems of different architectures. In the case of interest to me, that is between a big Java program running on Linux systems and the code I maintain on J-series and L-series NonStop systems.
Many years ago, I ported the then-current version of Google Protocol Buffers to the NonStop system, and it took me a LOT of time to do so. I don't recall exactly how long, but it was at least several weeks, and might have been over a month;. I have used that old version ever since then because of the large effort I expected it would take to port newer versions. But my managers want to move up to the current version of Google Protocol Buffers, so I'll need to dive in and port the current version.
What I'm hoping to get in response to this post is advice about how to go about doing a port of an open source package to the NonStop environment, both J-series and L-series. I recall the general approach of running the configure step, the make step, and the install step that most open source packages use. Google Protocol Buffers did use that approach (I assume the current version still follows that approach). I don't recall the details of the problems I had in the prior port, but I recall having many, many errors in both the configure step and the make step that took a lot of effort to find ways to get around.
What are things that might not be obvious to someone not very experienced in porting stuff to the NonStop system? Don't assume anything is so obvious that it doesn't need to be mentioned. I would much rather have you tell me things I already know than not to learn about something I don't know. Also, what things are there that people sometimes do when trying to port software that might seem like the normal thing to do, but actually should be avoided because they lead to problems?

What language is the package? There are different issues with different languages. With C, you will need to understand the dependencies. If the project absolutely depends on gcc, which many do, you are not going to be able to port it without significant effort. Understanding what compilers are involved is really important. It also depends on whether you are going a "light" vs. "deep" port. If you are just trying to get it to run, that is one thing, but run efficiently is another.
Good luck,
Randall

In addition to what Randall said, HPE does have an OSS porting guide
available on the documentation site.

[Keith Dick]

Post by red floyd

Post by Randall

The place I work uses Google Protocol Buffers for formatting messages that get sent between systems of different architectures. In the case of interest to me, that is between a big Java program running on Linux systems and the code I maintain on J-series and L-series NonStop systems.
Many years ago, I ported the then-current version of Google Protocol Buffers to the NonStop system, and it took me a LOT of time to do so. I don't recall exactly how long, but it was at least several weeks, and might have been over a month;. I have used that old version ever since then because of the large effort I expected it would take to port newer versions. But my managers want to move up to the current version of Google Protocol Buffers, so I'll need to dive in and port the current version.
What I'm hoping to get in response to this post is advice about how to go about doing a port of an open source package to the NonStop environment, both J-series and L-series. I recall the general approach of running the configure step, the make step, and the install step that most open source packages use. Google Protocol Buffers did use that approach (I assume the current version still follows that approach). I don't recall the details of the problems I had in the prior port, but I recall having many, many errors in both the configure step and the make step that took a lot of effort to find ways to get around.
What are things that might not be obvious to someone not very experienced in porting stuff to the NonStop system? Don't assume anything is so obvious that it doesn't need to be mentioned. I would much rather have you tell me things I already know than not to learn about something I don't know. Also, what things are there that people sometimes do when trying to port software that might seem like the normal thing to do, but actually should be avoided because they lead to problems?

What language is the package? There are different issues with different languages. With C, you will need to understand the dependencies. If the project absolutely depends on gcc, which many do, you are not going to be able to port it without significant effort. Understanding what compilers are involved is really important. It also depends on whether you are going a "light" vs. "deep" port. If you are just trying to get it to run, that is one thing, but run efficiently is another.
Good luck,
Randall

In addition to what Randall said, HPE does have an OSS porting guide
available on the documentation site.

Thanks for the pointer to that manual, Red Floyd.

I took a look at it. It might provide some help, but seems mostly to be advice about how to program the NonStop system, and advice about how to change a program you control so it will port to OSS easier. I don't exactly have control of the open source package I want to port, but I may find those suggestions helpful when making local modifications to the package in order to get it to compile.

[Keith Dick]

Post by Randall

The place I work uses Google Protocol Buffers for formatting messages that get sent between systems of different architectures. In the case of interest to me, that is between a big Java program running on Linux systems and the code I maintain on J-series and L-series NonStop systems.
Many years ago, I ported the then-current version of Google Protocol Buffers to the NonStop system, and it took me a LOT of time to do so. I don't recall exactly how long, but it was at least several weeks, and might have been over a month;. I have used that old version ever since then because of the large effort I expected it would take to port newer versions. But my managers want to move up to the current version of Google Protocol Buffers, so I'll need to dive in and port the current version.
What I'm hoping to get in response to this post is advice about how to go about doing a port of an open source package to the NonStop environment, both J-series and L-series. I recall the general approach of running the configure step, the make step, and the install step that most open source packages use. Google Protocol Buffers did use that approach (I assume the current version still follows that approach). I don't recall the details of the problems I had in the prior port, but I recall having many, many errors in both the configure step and the make step that took a lot of effort to find ways to get around.
What are things that might not be obvious to someone not very experienced in porting stuff to the NonStop system? Don't assume anything is so obvious that it doesn't need to be mentioned. I would much rather have you tell me things I already know than not to learn about something I don't know. Also, what things are there that people sometimes do when trying to port software that might seem like the normal thing to do, but actually should be avoided because they lead to problems?

What language is the package? There are different issues with different languages. With C, you will need to understand the dependencies. If the project absolutely depends on gcc, which many do, you are not going to be able to port it without significant effort. Understanding what compilers are involved is really important. It also depends on whether you are going a "light" vs. "deep" port. If you are just trying to get it to run, that is one thing, but run efficiently is another.
Good luck,
Randall

Thanks for your reply, Randall.

The Google Protocol Buffers package (GPB) is entirely C++. Unless my memory is playing tricks on me, GPB does not depend on any other open source libraries to run, but it does use a number of open source programs in the course of building GPB.

Here is a very brief description of what GPB does, in case it helps understand what is needed to do the port.

GPB has two main parts. One part is a compiler that takes in a description of the messages and the fields that go into them that GPB is going to be responsible for creating and interpreting.

This compiler produces a number of classes which must be compiled and linked to the application program, which uses them to construct or parse messages. Among the methods for a message is one to turn the data in the classes that represents the message into a message buffer that can be sent to another program that will use GPB to parse and use the contents of the message. Also among the methods for a message is one to parse the contents of a message buffer and put its contents into the data of the classes that represents the message.

The other main part of GPB is a bunch of classes that the classes generated by the compiler use in the course of composing, parsing, and otherwise manipulating the message buffers. GPB works entirely in memory -- it does not send or receive the messages it composes or parses.

The main purpose of GPB is to hide the differences in architecture when programs running on different systems exchange messages.

You ask whether GPB depends on gcc. I imagine you want to know whether GPB depends on any extensions to the C++ standard gcc implements, but that are not generally included in other C++ compilers. I believe that was one type of problem I encountered those many years ago when I ported version 2.4.1 of GPB to the NonStop system. I don't remember any problem in particular, but I believe that was one type of problem I had (over and over and over).

Since GPB claims to be intended to support message exchange among systems with a wide range of architectures, I would expect part of their requirements would be not to depend on features unique to gcc, but that seems not to have been the case back then. I'm hoping that Google has removed dependencies on features unique to gcc in the more recent versions, but I do not know whether they have done so. Maybe they mean GPB is to support message exchange among systems with a wide range of architectures that all support gcc. I don't know.

One file on the github project for GPB says these tools are needed to build GPB:

autoconf
automake
libtool
curl
make (for HP Tru64 and AIX gmake is needed instead of make)
g++
unzip

Except for g++, I see all of those are available in the Ituglib open source area, so I can get them,. I did that when I did the port of 2.4.1, and I got it to use c89 for the compiler. My memory of that time is that I had seemingly endless errors from the configure step, where it couldn't figure out how to do things it needed to do, and seemingly endless compiler errors after I beat the configure step. Once I fixed all the compiler errors, I believe it passed all of its tests, and I never had any problem compiling the classes it generated from our message declarations.

Maybe I shouldn't be asking for help before I try building the current version of GPB. I can ask for help if I run into problems I can't fix. I'm just trying to get some specific suggestions that others have found make porting go easier.

[red floyd]

Post by Keith Dick
The Google Protocol Buffers package (GPB) is entirely C++. Unless my memory is playing tricks on me, GPB does not depend on any other open source libraries to run, but it does use a number of open source programs in the course of building GPB.

One note... If it's C++11, you can only build for an L-RVU, not for J.
If it's C++14 or later, you're completely out of luck.

[Randall]

Post by red floyd

The Google Protocol Buffers package (GPB) is entirely C++. Unless my memory is playing tricks on me, GPB does not depend on any other open source libraries to run, but it does use a number of open source programs in the course of building GPB.

One note... If it's C++11, you can only build for an L-RVU, not for J.
If it's C++14 or later, you're completely out of luck.

Big caution that if g++ is required, you are going to have to make a lot of changes. g++ is based on gcc, which cannot be ported to L-series (or J-series).

[Keith Dick]

Post by Randall

Post by red floyd

The Google Protocol Buffers package (GPB) is entirely C++. Unless my memory is playing tricks on me, GPB does not depend on any other open source libraries to run, but it does use a number of open source programs in the course of building GPB.

One note... If it's C++11, you can only build for an L-RVU, not for J.
If it's C++14 or later, you're completely out of luck.

Big caution that if g++ is required, you are going to have to make a lot of changes. g++ is based on gcc, which cannot be ported to L-series (or J-series).

Well, those last two posts are a bit discouraging. Thanks for not holding back on bad news!

Is there some fairly simple way to tell whether the C++ source code is C++11, C++14, or some other version? The instructions for building GPB I have found so far do not say anything about what version of C++ it needs. Maybe it will be stated somewhere I have not yet seen.

What is it about those versions of C++ that prevents them from being compiled on J-series or L-series? Is that only true if certain features of the language are used in the source code, or are those versions of C++ so fundamentally different from what c89 or c99 can compile that there truly is zero hope?

I was under the impression that g++ was just the name given to the gcc executable when it was to expect C++ syntax in the source files it was given to compile. Is that impression wrong in some way?

I have to say that these warnings do seem to confirm my experience from almost 10 years ago when porting GPB version 2.4.1. I had to make a LOT of changes to the C++ source files, though I don't remember any details of what those changes were. I do have the original source files and the modified source files that I finally was able to get to compile. I could use diff to see what all the changes were, but I figured that would not be very helpful, given that I believe 3.x was a rewrite.

I see a very large number of open source packages in the ITUGLIB open source library. Do none of those packages use gcc or g++? I thought that gcc was the most common compiler in the open source world. If a large number of the ITUGLIB contents did use gcc or g++ in their original environment, how were you able to get them compiled for the NonStop system?

How about the other packages listed as being required to build GPB? Especially autoconf, automake, and libtool. Are there known difficulties in using them on NonStop systems when building other open source packages, or do they work well on NonStop systems? Any tips or tricks needed to help get them to work on NonStop systems? I recall whatever tools were used in the configure step of building GPB 2.4.1 gave me a huge number of problems 10 years ago.

[Randall]

Post by Randall

Post by red floyd

The Google Protocol Buffers package (GPB) is entirely C++. Unless my memory is playing tricks on me, GPB does not depend on any other open source libraries to run, but it does use a number of open source programs in the course of building GPB.

One note... If it's C++11, you can only build for an L-RVU, not for J.
If it's C++14 or later, you're completely out of luck.

Big caution that if g++ is required, you are going to have to make a lot of changes. g++ is based on gcc, which cannot be ported to L-series (or J-series).

Well, those last two posts are a bit discouraging. Thanks for not holding back on bad news!
Is there some fairly simple way to tell whether the C++ source code is C++11, C++14, or some other version? The instructions for building GPB I have found so far do not say anything about what version of C++ it needs. Maybe it will be stated somewhere I have not yet seen.
What is it about those versions of C++ that prevents them from being compiled on J-series or L-series? Is that only true if certain features of the language are used in the source code, or are those versions of C++ so fundamentally different from what c89 or c99 can compile that there truly is zero hope?
I was under the impression that g++ was just the name given to the gcc executable when it was to expect C++ syntax in the source files it was given to compile. Is that impression wrong in some way?
I have to say that these warnings do seem to confirm my experience from almost 10 years ago when porting GPB version 2.4.1. I had to make a LOT of changes to the C++ source files, though I don't remember any details of what those changes were. I do have the original source files and the modified source files that I finally was able to get to compile. I could use diff to see what all the changes were, but I figured that would not be very helpful, given that I believe 3.x was a rewrite.
I see a very large number of open source packages in the ITUGLIB open source library. Do none of those packages use gcc or g++? I thought that gcc was the most common compiler in the open source world. If a large number of the ITUGLIB contents did use gcc or g++ in their original environment, how were you able to get them compiled for the NonStop system?
How about the other packages listed as being required to build GPB? Especially autoconf, automake, and libtool. Are there known difficulties in using them on NonStop systems when building other open source packages, or do they work well on NonStop systems? Any tips or tricks needed to help get them to work on NonStop systems? I recall whatever tools were used in the configure step of building GPB 2.4.1 gave me a huge number of problems 10 years ago.

There is no easy way to tell, other than looking in Makefile.am or Makefile.in and find the compiler references. If you can read configure.ac, that would provide some info. Running after running autoconf you should be able to run conf_script_floss_cc and look at the output. That should report what is missing and whether gcc is required.

The C++ differences are numerous. The syntax has evolved over time. C++11 is only available on L-series. Much relates to initialization, which does create big problems for older compilers.

Any of the ITUGLIB packages that had hard dependencies would have been rewritten - we have not done a gcc port in at least 8 years, maybe 15. I can only speak as far back as about 2010. The autoconf versions in coreutils are one version back and that can cause problems for newer products, so you might get stuck there.

You might want to use git as a comparison tool on GPB to see what changed over time on a file by file basis. If you have a port, starting a new branch with that port off the base version might be helpful, and then merging in the newer changes on top of your port might make things easier.

If you get this to work, you might want to consider submitting it to ITUGLIB for long-term posterity. We do try to keep packages up to date, particularly if the core team is willing to take on NonStop port adaptations.

Regards,
Randall