Rewinding this repo and encouraging community

[gordonwoodhull]

I work with @lkoutsofios at AT&T Labs Research and volunteered to see if I could help restore the repo to a state the community is comfortable with. This was last discussed in #1464 (comment) and onward.

I am not an expert in ksh or the technical issues, and I don't know the entire history, but my understanding is that

1. ksh93u+ is the final official release from AT&T. This can be found in the branch 2012-08-01-master
2. A later version, ksh93v-, was contributed by the original authors after they left AT&T, but this version is not considered stable. This can be found in the tag ksh93v
3. @krader1961 has been maintaining the master branch for some years. He has tried to streamline the codebase (and only support ksh, not the rest of ast), but this has caused compatibility issues.

The original authors of ksh have all left AT&T. We still have plenty of users inside the company, but no one who can maintain and manage the project.

My proposal is:

• Move the current master to a branch "ksh2020" (or whatever is appropriate).
• Clearly mark README.md and CHANGELOG.md on that branch as unmaintained, point to segfault with extended globs #1464 and this issue for further info
• Reset the master branch to 2012-08-01-master
• Remove external collaborators from the repo. Since no one in AT&T is maintaining this, it doesn't make sense to have this fork be the active one. Git and Github allow anyone to take the code in whatever direction they want, in their own space.
• Clean up README.md to reflect the current state: "This repo contains the AT&T ksh93u+ and experimental ksh93v- versions of AST. There may be later forks of AST and ksh out there."
• Change CONTRIBUTING.md and issue template to be clear we are not actively maintaining this fork and do not promise to review anything.
• Shorten the repo description to "AST - AT&T Software Technology" (Streamline this repository's description #1441)

After this is complete, we at AT&T may contribute patches to ksh93u+ in the master branch. We probably won't review or accept contributions unless they are trivial (build issues, etc.) Eventually, we will archive this repo.

Currently I don't think anyone in the company plans to create a repo just for ksh. However, I think this is fairly straightforward and can be taken up by the community.

If anyone wants to create a community fork, I'd suggest creating a new GitHub Organization. This is free for open source projects. The bigger commitment is time, making contribution and maintenance guidelines clear, possibly creating a code of conduct, etc. Of course if anyone wants to fork this repo to an existing organization, they are welcome to do that too.

[jghub]

@gorodnwoodhull: my most recent comments are here: #1464 (comment)

sorry for overlooking this (better) place. will take note from now on...

[gordonwoodhull]

I went ahead with removing the external collaborators and fixing the description. I'll wait to see if there is further discussion about the other points before continuing with those.

[lkoutsofios]

thanks @gordonwoodhull for helping out.

my original motivation for going along with @krader1961's proposal was to keep at least KSH alive since no one else was volunteering to work on the AST project as a whole. but I think we were all clear that backwards compatibility was mandatory. KSH's strongest advantage over other SH/KSH clones was its programming interface. if the new version breaks old scripts, there's no point in using it. it seems from the recent issues on this repo that this isn't working out. so @gordonwoodhull's changes seem like the way to go. we'll see what happens with KSH. I do have some fixes for the u+ version that should get it to build and work on all recent distros of redhat, ubuntu, and opensuse, that I can push out. unfortunately, I don't have the time to be more involved.

[DavidMorano]

Just some friendly observational comments:

I have tried to follow both the recent discussion about rewinding the repository, et cetera, and the progress on this project since @siteshwar and then @krader1961 started making major contributions back in 2017. For the record, my own experience w/ KSH dates from its very (albeit public) inception, back in 1983 (I was at Bell Labs at the time and following). I have used KSH almost entirely exclusively and continuously since that time to the present (including very recently on ksh2020). Some items:

• this code is very complicated and somewhat messy by the best modern coding practices (that may be something of a gross understatement); it is NOT suited for the casual programmer -- or team of programmers -- to dive into; despite people possibly feeling otherwise, I think that @krader1961 is-was fairly competent as a programmer and developer; where @krader1961 seemed to possibly lack something was his understanding of the various historical customer uses of KSH over the last 37 years -- being much more intensive and far-reaching than he seemed to realize; I will not dwell on this now, but KSH was Perl before Perl was, and it was Python before Python was; it had substantial enterprise applications written in it before the end of the 1980s (and the first book only came out in 1988)

• the code creates a program that is substantially defined by three standards: 1) POSIX, 2) the KSH book (by David Korn), and 3) the existing behavior of the ksh93u+ branch; this is not something to be trifled with lightly; extreme care needs to be exercised to not break any existing documented or expected behavior (this may be an area were @krader1961 might have appeared to not be as sensitive as people would have expected)

• despite some possible (arguably relatively minor) personality issues, @krader1961 has put an exceptionally enormous amount of work into fixing problems that even existed in ksh93u+ that were not always readily observed or complained about; to replicate @krader1961's work starting from scratch at the ksh93u+ branch will be extremely non-trivial (please, @siteshwar give your feeling on this if you would)

• I am aware that even ksh93u+ had severe bugs and problems that needed to be eventually fixed; many of these did not manifest readily for most users, but they needed to be addressed eventually none-the-less; these are things such as strays writes, buffer overflows, and the like; there were a variety of minor (and not so minor) problems also that had been in there for about 20+ years (also not readily complained about by most regular uses); @krader1961 had made a very substantial start at getting some of the more serious of these (the memory related stuff) fixed (with the help of, among other things, ASAN and Valgrind); starting again from ksh93u+ means a huge amount of dog-work in fixing up again what @krader1961 has already done

• not all distros are equal! some distros are more important and have more say in the world than others! sadly for some, the Red-Hat distro is more important now-a-days than pretty much all other distros combined (yes my opinion!); and I am not one who even likes Linux derived distros in the first place! ; Linux itself is very much inferior in many ways to older non-Linux distros (yes that is also my opinion); so the short of it is, that I think that @siteshwar should have more say in how KSH proceeds into the future than all the rest of us (distro maintainers or not) combined! I am sorry, but that is what I feel should happen; @siteshwar is not a dumb guy; he knows 'the score"; he knows full well that maintaining KSH is like managing fire in your hands; he works for Red Hat!!! Red Hat understands managing real (hard core, porn) enterprise (and other) customers and their requirements and demands; I would ask that more deference be given to @siteshwar's thoughts on this whole matter going forward; no joke

• my fear is that if the ksh2020 branch is essentially abandoned in favor of starting from ksh93u+ again, a huge amount of time and work will have been forfeited; I want to hear from @siteshwar more of this point; this kind of decision should not be taken lightly

• even separating out KSH from the rest of the AST code was not a totally trivial endeavor; @siteshwar and @krader1961 did this w/ ksh2020 at no little (no little) effort in itself; although I had some trepidation at the time, I feel that this was a useful and important thing to do to modernize the KSH code base in particular; this is just one example of something that would have to be done again if starting from scratch

• my own preference going forward would be something along the lines of the following: 1) bring the ksh93u+ branch to a new state (calling it something else, ksh93uu+ whatever) so that it can be easily compiled by the community (easily, maybe using make, autotools, CMake, whatever) so that distros have a fairly reliable and readily accessible base to port from (in the near term), but then 2) continue w/ the ksh2020 branch and make it more conforming to historical and expected behavior; ksh2020 would be the base for future -- possibly feature enhanced -- versions going forward

• and finally, for those who think that just using ksh93u+ is some sort of magical cookie to the golden land, it is very likely that making any fixes to ksh93+ (which will likely be required) will break some conforming behavior in unexpected ways; this is likely part of what happened with ksh2020 -- the act of fixing some previous bad stuff, exposed some bad stuff that was still in there already from the past, but not previously readily observable by most customers; this happens all of the time with old code, but to get through it one has to often just confront bug after bug and plow ahead through the hard times; for myself, I feel that ksh2020 has started down that path (as hard as it may be); starting fresh from ksh93u+ may never get past the hump of fixing some of its worst bugs lurking within it

• KSH is a complicated and still -- in many quarters -- a mission critical program; in my opinion, it needs the attention and contributions from the entire world to focus on a single main code base branch in order for it to flourish into the future, rather than just staying "useable" at the ksh93u+ level; ksh93u+ had (has) many bugs that need to be eventually fixed if there is any hope for real future feature enhancements to likely work; the code base needs to be modernized also (not a trifle in itself)

• just my two cents on this whole thing!

[jelmd]

@gordonwoodhull, I think you got it right and your proposal is good.

However, some nits:

* [ ]  Move the current master to a branch "ksh2020" (or whatever is appropriate).

Yes. But to avoid confusion [1] [2], that people use it as an official or stable version (actually what happened with version v-), README.md as well as CHANGELOG.md in this branch should make it very clear, that this is an experimental branch (e.g. by adding an appropriate level 1 header at the top) and the mentioned articles refer to an experimental release (or that there is no ksh2020.*, etc.).

* [ ]  Rewind the master branch to 2012-08-01-master

Adding @lkoutsofios patches is a good idea - thanks @lkoutsofios for spending your time! Direct merge, separate *.patch files or PR, whatever you like is probably ok.

* [ ]  Clean up README.md to reflect the current state, and point to any active forks from there. If desired, transfer relevant issues to child forks.

If no-one at AT&T has time to maintain it, or prefers one fork over another, it should be kept neutral and just mention, that there are forks which are not AT&T driven. I think, if there is a valuable, active fork, people interested in ksh93 will find it by themselves or by getting hints from others...

[gordonwoodhull]

@DavidMorano thanks for your perspective. I don't know the history but I agree in principle with what you're saying.

I will second that I hope ksh2020 starts an organization, perhaps led by @siteshwar and @krader1961, starting from what is currently on master.

[gordonwoodhull]

@jelmd, sure, sounds good - I've edited the tasks above.

[jghub]

@DavidMorano, I appreciate your perspective, especially concerning the "KSH was Perl before Perl etc" aspect and its implications.

regarding what has been achieved (or not...) in ksh2020 I respect your assessment, although it seems too "understanding" in my view. I would readily agree that I cannot really judge the technical difficulties involved when rewriting/rephrasing the C-code . I am in the position of a "typical" KSH user who uses the KornShell language for scripting w/o knowing the C innards of the ksh program .

let's say I cannot build a car but I can drive it and if I drive ksh2020 around the block I notice strange noises from the motor department and assorted malfunctions. it's slower, too... before the overhaul the car was just fine, only had some peculiarities the driver knew (mostly) to avoid. so while the overhaul obviously involved technical skills (at the C-level) beyond my own I can judge that the job has not produced something better but something worse.

I maintain that is the current state of ksh2020. choose any metric you like (except better formatted C-code and "tidiness").

as you explained to me elsewhere (I forgot so far to again thank you for this!) #1449 (comment), you especially value the malloc vs. vmalloc changes for reasons of your special interests although you also agree that vmalloc might be uncritical for the 99.9% crowd not writing their own (multi-threaded) builtins. of course, if it turns out, that indeed replacing vmalloc is clearly better (but for whom and according to what measures would have to be decided) one might consider to do that to 93u+ at some point down the road. I am not sure that there is clear evidence that this is worth it -- but not for me to decide....

and it sure would have been the responsibility of @krader1961 and @siteshwar to listen to people like you and many others (I was a latecomer in the queue of people raising concerns...) from the start and adjust their actions how to deal with the code, the features etc. I feel they (at least krader) had the wrong agenda. and the decision to start from the 93v- beta is not comprehensible to me, given the circumstances (known buggy beta state, no advice from dgk etc.).

I do not know of any "problem/bug driven" TODO list with which ksh2020 was started. there was the "we must first overhaul the code to adhere to modern standards" and the fuzzy "keep ksh relevant" approach and that without any adequate understanding of either the code or KornShell. the issue history is there for everybody to go through (it is not that extremely long): a clear story evolves of misconceptions, false claims and -- and that was the central problem throughout -- an incomprehensible rigidity and refusal to adjust course and actions in the small or in the big issues on the side of krader.

so I believe your rather benevolent description of krader's "possible (arguably relative minor) personality issues" is objectively not accurate (and this is not irrelevant since it crucially has determined what has happened in ksh2020 -- it could have been a different project -- and is also crucial for deciding how to proceed from here).

for me this implies that ksh2020 should (no: must!), indeed pursue its goals independently of ksh93. I also insist that terminology-wise there is need for a strict and hopefully agreed-upon "branding". it was proposed to call ksh2020 "krsh" for obvious reasons, but I really believe the "ksh2020" name is more sensible and "catchy" and accurate enough, so why not call it that for the next 25 years just like ksh93 still is called that...

altogether I am still not quite sure, what exactly the goals of ksh2020 are: so far it has eliminated most or all of the new features of 93v- since they could not debug those (being, so far, not yet or not fully functional in 93v-) while incompatibilities and regressions and new bugs crept in and performance went southward. maybe, that all can be fixed and at some point ksh2020 is better than a (patched) 93u+ but I really do not see that right now.

to reiterate: in my view, ksh93 and ksh2020 development should not proceed in a common repository. this can not work out. what could work out in my view is a friendly coexistence (in the long run, at least, when potential hard feelings have waned) and, hopefully, bug fix exchange (the latter for the time being could very well be unidirectional from ksh2020 to ksh93u+ for the rather few cases (AFAICS) where ksh2020 has fixed things that apply to 93u+ and not only to 93v-).

I am looking forward to implementation of the strategy outlined by @gordonwoodhull: even if ksh2020 could proof at some point to be the "future", I believe making ksh93u+ easily available again for everybody ASAP is much more important right now. this requires separate projects as the history of the ksh2020 project proofs beyond reasonable doubt in my view.

[gordonwoodhull]

I clarified that we won't immediately archive the repo, and we'll contribute what patches we have, but we probably won't accept contributions unless they are trivial.

I'll proceed with the plan in the next week.

[DavidMorano]

@jelmd
I think that we agree (and most everyone agrees) with the growing consensus around the following:

• that a new workable version of KSH be made out of the ksh93u+ branch as its base, possibly replacing NMAKE w/ something else; this would serve the needs of distros that need a very reliable (as much as can be expected given everything) and behaviorally compliant KSH

• further, I have no objection to VMALLOC staying in the ksh93u+ follow-on product; if I ever want to use this new KSH product for myself, I will just replace VMALLOC w/ a regular system one as I did in the past

• neither do I take any issue w/ trying to bring the new (ksh93u+ derived) product up-to-date w/ any patches or fixes that can readily be put into it (the issue here is the time to do this at the level that ksh2020 has already achieved)

• I do not want to waste ksh2020, and neither do @krader1961 and @siteshwar as far as I would guess; but it seems that the consensus is that they open a new unique repo for its development, separate from this ATT/AST repo

• my guess is that @krader1961 and @siteshwar will likely continue to develop and orient ksh2020 as a possible next generally-released KSH also, and I encourage that work; but my preference would be that it try to adhere more to the standardized and expected behavior of prior KSHs -- circumscribed as I stated previously around 1) POSIX, 2) the KSH book (Korn), and 3) ksh93u+ behavior

For myself, I admit that I am optimistic that the benefits that @krader1961 has put into ksh2020 (much of it quite subtle, like cleaning up the various NV preprocessor flags mess, to name just one) more than offsets the regressions that people have complained about. But, yes, this is opinion on my part. But based on following many or almost all of @krader1961 commits. I have seen regressions in ksh2020 myself (like in the form of unexpected crashes). But in fairness, there were some unexpected crashes in many of the production KSHs over the last couple decades also. My hope is that, accidentally or not, @krader1961 might have actually fixed some of these (again, perhaps even by accident as it were).

Having a follow-on version of ksh93u+ available for distros (whatever its name is, I do not care about any names really) will take the pressure off of the ksh2020 development, so that they can concentrate on cleaning it up so that it appears to perform at least as reliably and conformally as the old ksh93u+ did (or its follow-on replacement).

I also have to admit that I hope that ksh2020 succeeds totally eventually. I have seen, and sometimes tried to understand, the code in ksh93u+ and it is often quite difficult (and I look at tons of bad code all of the time for my living). I tend to get spooked by and biased against code that has memory buffer overflows and the like, and I very much welcomed @krader1961 and other's initial attempts to clean some or much of it up -- as much as is exposed by bad behavior or by development tools (like ASAN or Valgrind, et cetra). I do not feel that the code base of ksh93u+ is really fit for the future. I would like to see the code come to a place where both fixes and feature enhancements become at some point much easier. But I think that the right path for this (following the project as I have) is with ksh2020, simply due to the amount of cleanup put into it (being perceived as useful or not to the end user). I could be wrong, but we will now see. Neither do I discourage future major fixes or enhancements to the new ksh93u+ branch. Let the race begin, as it were. The end code-builders or end-users can decide for themselves which one is the real future (or not).

[jelmd]

@DavidMorano, I totally understand your concerns. But what we need right now is to get back the trust from vendors and users and therefore the outlined plan is IMHO the right way to go.

FWIW, I'm basically repeating, what has been already said:

At the moment all distros ship a ksh93 package, have certain bugs already fixed with their own patches and the process to build the package, or apply a small patch from time to time is almost a no-brainer. Vendors/developers/users may even have a knowledge base, FAQ, ... wrt. workarounds for existing bugs/corner cases so that even patching is not really needed, as @jghub already pointed out. So on the vendor side there is no need to invest a lot of/any time and they can continue to ship it until a new star appears on the sky, which makes it redundant.

Redundant means for me full compatibility to ksh93u+. E.g. the getopts self documenting as well as the time conversion and formatter feature are essential and unique (skim over acme-ksh scripts to get some more details if you like). Last but not least I do not like to waste a lot of time to find out, which of several dozen man pages actually contains a description of the feature I wanna use (like zsh), and than skim over several pages again to finally find it. ${builtin} --man is such a useful thing, that I do not really care about the size of the ksh93 binary.

Anyway, new stuff like native json support, possibly ksh thread safety, ... would be nice to have, yes. But I think, even changing the build system only would requires a lot of "review" time at least on distro maintainers' side to make sure, that they still ship a stable/usable version. Debian has shown, that this is probably too challenging, decided to just trust the most active members of the repo and bundled something that at least I do not want to see on any of the 100+ zones/containers I need to manage (or others, which use scripts I wrote in the last 20+ years). Therefore, encouraging people to fork and give it a distinguishable name seems to be the best option. If it is good, distros will pick it up pretty fast and do not need to care about compatibility wrt. ksh93* ...

[kdudka]

But what we need right now is to get back the trust from vendors and users and therefore the outlined plan is IMHO the right way to go.

The necessary precondition to get back the trust of Red Hat is that upstream is able to competently and reasonably fast react on reports of important security issues. For example, how quickly would your team be able to fix the following security issue in ksh93u+?

https://access.redhat.com/security/cve/CVE-2019-14868

[siteshwar]

I would ask that more deference be given to @siteshwar's thoughts on this whole matter going forward; no joke

I give up and move on with my life. I made all my efforts with the goodwill to help the community and it turned out that I achieved quite the opposite. I hope that someone else would pick up this project and would make progress in the direction that's more acceptable to the community.

[DavidMorano]

@siteshwar

I give up and move on with my life. ...

I am very sorry to hear your decision on this matter.
I think that you were the best thing to happen to KSH in years!
Having RedHat (as represented by yourself) in the middle of decisions regarding KSH was likely the best thing to happen to KSH since the old haughty days of AT&T (pretty much long gone now).
I feel badly that the general community did not seem to show you more respect (deference).
Thanks very much for the service that you did give to this project over these past years.

All the best with whatever you do going forward!