[TriLUG] Piping, redirection and shellscipts: 3/5/2025 7pm Eastern Standard time

[Steve Litt via TriLUG]

Hi all,

Where: GoLUG: https://meet.jit.si/golug
When: Wednesday, 3/5/2025 7pm sharp Eastern Standard time
      Arrive 15 minutes early for Microphone check & discussion
Who: Every attendee. This is a free-for-all discussion

I've taken piping, redirection and shellscripts for granted the last
two decades. I'm probably not the only one. And it's probable that
each of us uses these things a little differently and could benefit
from everybody else's use of piping, redirection and shellscripts.

Piping and redirection, possibly aided by a good text editor, bestow on
us a spectacular ad-hoc query system. Yesterday I was tasked with
getting every domain name I owned into my list at
444domains.com/domains , which is created by a shellscript and Python
program that read a Yaml file and convert it to the web page. So I had
to do the following:

* Screen scrape all my domain names from ionos, put in file

* Screen scrape all my domain names from Spaceship.Com, put in another
  file

* Use Vim to remove everything but the domains themselves

* Sort both files

* Create a Python program to sort my Yaml file

* Run diff between each file and the sorted domains of the Yaml file
    - The output with the left arrows need to be added to Yaml file

* Re-sort the Yaml file

* Run the shellscript to create the web page.

This sounds straightforward, and if I were infallible it would be. But
I made lots of mistakes, including forgetting to sort one file,
producing lots of duplicates in the diff. For this reason, having the
system be ad-hoc, complete with sed, diff, grep, etc was wonderful.

And with the knowledge learned, if I expect this to be an ongoing
thing, which I don't, I could put the whole process into a shellscript
to be run at will.

The world has been trying for decades, with lackluster success, to
create "reusable code". To this day, piping, redirection, shellscripts,
and a good curation of fast, tested and known good processes like grep,
sed, cut, sort, etc yield by far the quickest and easiest way to reuse
code. Nowadays you can even go GUI by using the zenity command.

Mine is just one example. I bet every one of you has a unique example
of using piping, redirection and shellscripts to do something that
would have been a massive program in Perl, Python, etc. The GoLUG
meeting Wednesday night is where you can talk about your unique
examples, and learn from others.

Also up for discussion are our favorite shells (bash, /bin/sh, dash,
ksh, csh, etc), and the role of shellscript checkers like shellcheck.
And the age old question: When is something so big that it should be
done in C, Python, Rust, Lazarus, etc.

This link (thanks Kyle) https://www.youtube.com/watch?v=tc4ROCJYbm0 
has given me the inspiration to try harder to do it with piping,
redirection, shellscripts and utilities before busting out my Python
interpreter, and if I do go Python, do it as a "do one thing" process
inputting from stdin and outputting to stdout.

Hope to see you there.

SteveT

Steve Litt 
GoLUG Publicity Coordinator