[TriLUG] Piping, redirection and shellscipts: 3/5/2025 7pm Eastern Standard time

[William Sutton via TriLUG]

Oh yeah.  This is a really important topic, and I can't count the number 
of weird shell command pipes that I've had to come up with over the years. 
Possibly the weirdest was the NetBackup commands that require user input 
to verify that you want to do something, the solution for which was

echo "y" |$some_netbackup_command

William Sutton

On Sun, 2 Mar 2025, Steve Litt via TriLUG wrote:

> Hi all,
>
> Where: GoLUG: https://meet.jit.si/golug
> When: Wednesday, 3/5/2025 7pm sharp Eastern Standard time
>      Arrive 15 minutes early for Microphone check & discussion
> Who: Every attendee. This is a free-for-all discussion
>
> I've taken piping, redirection and shellscripts for granted the last
> two decades. I'm probably not the only one. And it's probable that
> each of us uses these things a little differently and could benefit
> from everybody else's use of piping, redirection and shellscripts.
>
> Piping and redirection, possibly aided by a good text editor, bestow on
> us a spectacular ad-hoc query system. Yesterday I was tasked with
> getting every domain name I owned into my list at
> 444domains.com/domains , which is created by a shellscript and Python
> program that read a Yaml file and convert it to the web page. So I had
> to do the following:
>
> * Screen scrape all my domain names from ionos, put in file
>
> * Screen scrape all my domain names from Spaceship.Com, put in another
>  file
>
> * Use Vim to remove everything but the domains themselves
>
> * Sort both files
>
> * Create a Python program to sort my Yaml file
>
> * Run diff between each file and the sorted domains of the Yaml file
>    - The output with the left arrows need to be added to Yaml file
>
> * Re-sort the Yaml file
>
> * Run the shellscript to create the web page.
>
> This sounds straightforward, and if I were infallible it would be. But
> I made lots of mistakes, including forgetting to sort one file,
> producing lots of duplicates in the diff. For this reason, having the
> system be ad-hoc, complete with sed, diff, grep, etc was wonderful.
>
> And with the knowledge learned, if I expect this to be an ongoing
> thing, which I don't, I could put the whole process into a shellscript
> to be run at will.
>
> The world has been trying for decades, with lackluster success, to
> create "reusable code". To this day, piping, redirection, shellscripts,
> and a good curation of fast, tested and known good processes like grep,
> sed, cut, sort, etc yield by far the quickest and easiest way to reuse
> code. Nowadays you can even go GUI by using the zenity command.
>
> Mine is just one example. I bet every one of you has a unique example
> of using piping, redirection and shellscripts to do something that
> would have been a massive program in Perl, Python, etc. The GoLUG
> meeting Wednesday night is where you can talk about your unique
> examples, and learn from others.
>
> Also up for discussion are our favorite shells (bash, /bin/sh, dash,
> ksh, csh, etc), and the role of shellscript checkers like shellcheck.
> And the age old question: When is something so big that it should be
> done in C, Python, Rust, Lazarus, etc.
>
> This link (thanks Kyle) https://www.youtube.com/watch?v=tc4ROCJYbm0 
> has given me the inspiration to try harder to do it with piping,
> redirection, shellscripts and utilities before busting out my Python
> interpreter, and if I do go Python, do it as a "do one thing" process
> inputting from stdin and outputting to stdout.
>
> Hope to see you there.
>
> SteveT
>
> Steve Litt 
> GoLUG Publicity Coordinator
> -- 
> This message was sent to: William <william at trilug.org>
> To unsubscribe, send a blank message to trilug-leave at trilug.org from that address.
> TriLUG mailing list : https://www.trilug.org/mailman/listinfo/trilug
> Unsubscribe or edit options on the web	: https://www.trilug.org/mailman/options/trilug/william%40trilug.org
> Welcome to TriLUG: https://trilug.org/welcome