Over the decades I’ve seen the praise and critique of the “Unix way”. I don’t remember reading about the aspect I’ll describe below. This might be a banality, and I haven’t invested much thought into it either. I’m just curious to hear your opinions.
At the core of the Unix way is that tools should do just one thing. Users very often need results that can only be achieved after doing several things. With only single-purpose tools at hand, users have to integrate them. In other words, the tool authors delegate integration to the users.
Here’s the idea I want to discuss. In programming languages, integrating various modules to achieve a goal is already a complex task because we often have to handle numerous failure modes and edge cases. In shell scripts this is much worse, since shells are more limited than programming languages. Writing a tool that does only one thing looks like avoiding the most complicated task. Which leads to a conclusion that the Unix way only works well in a rather small fraction of use cases; the majority of cases are better served by different approaches.
What do you think?
I like tools to be separated enough that I can do scripted/CLI calls with them through standard interfaces (as opposed to being a GUI/TUI monolith). All sorts of stuff can be done when tools output structured data like JSON that’s easy to parse and manipulate, or being able to interact with DBus APIs or services in other ways.
What I don’t like is when core tools try to be “smart” and reimplement things that other tools already do and end up making them more difficult to work with. Like package management, we already have Nix, Mise, and a billion other package managers, we don’t need another one. And turning the compiler into a package manager (cough Typst, .NET) is just stupid. Also programs editing their own configuration files is annoying as well, I can edit them myself and I don’t want them touching my edits, and if other people don’t want to do that then maybe a custom interface could be put in a UI/add-on of some sort that’s not part of the core tooling.
Also web browsers and web applications are close to bottom of the barrel in terms of scriptability. The stuff the Nix ecosystem had to do to make Firefox configurable is nightmarish and still janky, and Electron is an overall mess. Although with everyone wanting to make the programs work with AI agents now, maybe this will improve.
turning the compiler into a package manager (cough Typst, .NET) is just stupid
Do you mean Typst’s
import "@preview"function?Yes. Also I’m trying to figure out if there’s a way to keep the versions in sync between files because it doesn’t seem to have a lockfile or central version management either
I think you’re missing the difference between professional software development and a QnD script thrown together to accomplish a task.
The UNIX philosophy is also related to The Agile Manifesto and “worse is better”.
People are supposed to have the power to solve their problems. Unix provides primitives and tools to people, and they are expected to learn enough to use the system. It’s from a very different time. Well before the “don’t make me think” crowd showed up.
In programming languages, integrating various modules to achieve a goal is already a complex task because we often have to handle numerous failure modes and edge cases.
Now ignore all of the failure modes and edge cases except those that a script is going to directly run into, and that should simplify the requirements. This is “worse is better”.
Perl, Python, and Ruby are supposed to be approachable and easy to get started with. They were never supposed to be industrial programming languages like C, C++, Ada, Pascal, Java, etc., and what you’re noticing is the software industry not understanding there needs to be levels which address the different needs of people. Sometimes people need a simple tool, and not everyone needs to master the professional tool which takes two hours to set up and adjust.
The software industry destroying easy, simple to use software is another discussion.
So, it is me “missing the difference between professional software development and a QnD script” or the whole software industry? ;)
ignore all of the failure modes and edge cases except those that a script is going to directly run into
I’ve seen lots of cases of catastrophically wrong expectations of what the script will run into. Both for casual personal scripts, and professionally written scripts. I dare to claim that the majority of script authors cannot properly estimate this.
What does QnD mean here? Sorry if it’s a stupid question, but I tried searching google and Wikipedia and got nothing relevant
QnD is Quick ‘n Dirty.
Quickly thrown together with dirty code.
Ah yes! I know the expression and the meaning, but I’ve never seen that abbreviation before! You learn something everyday! Thank you very much! (And I agree with your original comment)
Sometimes I’m looking for a throwaway script that only does one thing and it doesn’t necessarily need to be pretty. I’m not looking to make it publicly used or even to be repeatable. Like the famous Todd Howard quote, “it just works.” QnD is the best option.
On the other hand a programmer rarely knows exactly what I’m trying to accomplish. Do I want to count the characters in a textfile? Or the words? Do I want to sort the files by size? age? filename? And which of those am I trying to move? What if I want to delete all files older than 30 days which don’t have the word “elephant” in them? I mean sure, you can implement all of that in a very complex program with a very convoluted user interface… Or you do it the Unix way and give me seperate (simple) tools to count, a tool to sort, search text… and a way to connect it to the delete tool.
Continuing the train of thought from the post: an approach that might serve you better is not a universal program that does everything. It is rather a custom program that does what you need. It very likely will apply reusable code, but in the form of modules instead of programs. It will be much easier to write well than to write a bash script well.
I mean that’s basically the same thing, just a different user interface. I think from a technical perspective it doesn’t matter a lot whether we write down the commands as text/words and connect them with “|” symbols or whether we pick them from a long list of symbols and arrange them as puzzle pieces or connect them with pipes or arrows drawn with the mouse.
In my experience, programming languages offer more control than just connecting inputs and outputs, and it’s easier to get right
And generally should be more efficient too. Bash and this unix-style tool calling was not designed for efficiency. Which is fine for most use cases, but should be avoided for tools you regular use.
To be more detailed, every time you call a program using bash, your bash program has to fork into two identical processes. Then all that work of forking is wasting as one of the forks then execs to replace itself with the program you called. And that happens with each tool called.
Using other programs, like Python, can more efficiently exec other programs without the same overhead of forking. And for longer running jobs, Python is much more efficient as it converts itself to bytecode, whereas bash is purely interpreted.
The main downside of Python is its longer time to start. So for simpler jobs, bash can be better.
Most computer stuff is tradeoffs anyway. I mean we’re looking at different manifestations of “efficiency” here. On the one hand we’re wasting fractions of a milliseconds here and there. On the other hand this might allow us to execute code in parallel, offer modularization. I can combine wildly different programs from different manufacturers, written in different programming languages. I can install new ones and execute them straight away without reloading my shell…
But I’d argue this doesn’t even matter all that much. We’re talking tiny fractions of a second in most use-cases. The real efficiency is me sitting in front of my computer and saving 10 minutes of time because it’s easy and straightforward to make the computer do the task I want it to do.
I’m not a big fan of BASH. I do use it. But mostly when it makes sense and it’s the right tool. And I keep forgetting how to do if-else-statements, when to use single or double brackets… It occasionally breaks in weird ways and it generally needs a lot of background knowledge to write good bash scripts. Python is designed with a bit of a different use-case in mind. But it can also be used to write quick scripts to sort files, do some admin stuff and automate things. Oftentimes it’s a good and comfortable tool to use.
By making smaller tools you’re allowing for a shell scripting distinction to exist where without it you would have a lot more specialized tools that do “more” but don’t integrate well. Kind of like the webbrowser.
If I understood your idea correctly, my proposal is the opposite of what you mentioned. I argue for more of better integrated software that does less.
I think the Unix way is a great ideal if you develop composable cli tools, functions or libraries. In each its easier to reason and recombine a thing, that does one specific thing. More complex user problems are then solved by combining multiple simpler components. Lots of modern graphical user interfaces neither think about what the user might want to do, just offering everything, nor prescribe a specific workflow, which just leaves users that have a only a vague idea of what they want to do or a porgram could do, confused.
Being more unixy would be great for mobile apps as well. iOS for example has these app shortcuts which expose functionality of an app to be combined with other apps featuers. I think once siri AI takes hold this could be done. I think, however, the monetization realities of most software will always pressure them to do lots of things and lock the user in.
More complex user problems are then solved by combining multiple simpler components.
What I am trying to say is that this solution is very likely to be poor, not taking into account various failures and edge cases.
If those edge cases and failure modes don’t apply to the situation, then it’s a good thing. I’ve watched many engineers over the years invest hours in making clean and robust solutions for one-off tasks that could have been scripted in a few minutes and in a few lines of python/perl/ruby/bash. 🤷♂️
I mean one often thinks they don’t apply and then is surprised by breakage. Or worse: it goes unnoticed. How many problems have happened because someone hasn’t considered whitespace in filenames?
That old “rm -rf ./$datadir”, indeed. I used “shellcheck” for years to help catch the simple stuff… Back to the topic at hand: Ultimately we have personal goals, and we cannot rely on the author’s of other software to know and implement everything we want. Especially when it comes to UIs, it’s often a take it or leave it affair.
In this respect, AI software development has been a boon. I’m increasingly creating software for myself in highly performant type-safe languages, using native APIs… If it’s more than a simple script (but also not too complex), modern LLMs are going to do a decent job, faster than I could do it (whereas before I often wouldn’t have started, due to limited time). I can mod an existing app that’s almost good enough, in order to scratch a personal itch. Or reimplement it from scratch in my preferred form. Small composable apps, tools, and libraries are perfect for this. And most importantly, I don’t have to slop up the Internet with my overly opinionated but perfect for me projects. 😉
The UNIX philosophy does look like “do one thing and do it well”, but it’s really a means of working in such a way that one doesn’t introduce scope creep to a point of Drupal.
Initially I had some words in the post about the philosophy opposing some other approaches, and I can guess that there were some Drupal-like examples at the time. Is it possible that the approach was taken to the extreme and the baby was spilled with the water?
Well, these things come in cycles. In the 90s everything had it’s own tool, then tools like Drupal and WordPress, office and image editing suites came along… And we went back to using simpler tools with app stores. And then we got orchestration with K8s. And so on.
We’re going through another a big change in software, and we’ll see more. The big, complex swiss army knife software suites again.
I think a big portion of the unix philosophy is to be easy for development. Not for consumers or customers.
The idea is that it will be easier for developers more complex software and flexible software if the pieces they are working with are well defined.
Well, my post describes the experience of a developer using these tools. “User” here is supposed to mean the person who combines tools to build a more complex piece of software.
Yeah, and Unix is built around the paradigm of letting people build the tools they need. People are supposed to be skilled in using the computer as a tool to accomplish tasks.
Originally, IT wasn’t a defined business function. Scientists, accountants, and others, were given access to compute resources, and they were expected to learn the tools to master their jobs. Part of that, was learning how to program the computers.
People are supposed to be skilled
Software is supposed to make simple things simple, while complex things should be possible. I’d expect iterating over a list of files to be simple, but everyone has to remember to include special handling of whitespaces in filenames. That’s just one example.
Qualification: I am a certified moron, I used nano as my IDE for 8 years because I did not know about VS code. I still structure my python like the qbasic I learned in school. I have never mastered object oriented thinking.
Write your scrips one chunk at a time, make them modular, make intermediate scripts if you have to, and have each part output a data structure that you nail down early in the project. Having a ass-load of little parts chained together means you can tweak or entirely replace tools at a later date without fucking the whole process.
If your project is a monolithic do-all be-all monster it means even the tiniest change involves reevaluating your entire codebase. You’re giving yourself all the problems of cuckcoding with none of the time savings.
Many commenters assume I propose do-all be-all monsters. Looks like I failed to explain my idea of custom-made programs for specific use-cases.
Writing a tool that does only one thing looks like avoiding the most complicated task. Which leads to a conclusion that the Unix way only works well in a rather small fraction of use cases; the majority of cases are better served by different approaches.
The alternative to do-one-thing is do-lots-of-things.
Users very often need results that can only be achieved after doing several things. With only single-purpose tools at hand, users have to integrate them. In other words, the tool authors delegate integration to the users.
You describe a need for all-singing, all-dancing solutions over single use tools.
You know there are colors beyond black and white? There’s a spectrum of alternatives.
I’m arguing for a little more singing and a little more dancing well-integrated solutions. Like the
-deleteparameter to thefindtool is the step in the right direction. Or do you think that this parameter is superfluous and good oldrmis perfectly fine?You’re gonna think I’m a asshole but, chaining find and rm seems like the best way because you get to keep all the options of rm instead of having only the less flexible option of -deleting every match.
Hey get a load of this guy says he can’t “code” but then lays out (actual) practical advice for coding.
My github account is a warcrime, everything I know I have learned from running Debian, reading wiki’s, and stealing code from stack overflow.
Yeah, that’s how you learn tho. I think all of us have just done that and you can get pretty good with just that. And that pipeline of functions thinking
In one way it is a simpler task to fix a single tool, than update a bunch of complex tools that suffer a problem.
BUT this was before the GIT method of pulling changes into a main branch, so maybe nowadays it doesn’t matter as much as their is much less effort updating a bunch of tools for a problem, than manually editing and recompiling.
Srill when we talk to users about our software use we alsways recommend using as much out of the box functions to get done what you need, rather than coding new tools for the Open API. Because a software update (should) fix all your workflow and automation that you have leveraged, rather than recoding dozens of apo addons.
It would be interesting to compare the number of problems caused by buggy tools versus poor integration!
The Unix way isn’t for people who can’t integrate those tools.
Ever been in the drivers seat of an 18 wheeler? Even just getting into one requires more physical exertion than getting into your car. When you’re up there you have a million gauges and controls meant to provide the operator the tools they need for any situation they might encounter and to interface with a galaxy of stuff that can be hooked up to the back. You need a special license which encapsulates this different paradigm of use, operating a gigantic vehicle under much different and more variable conditions than the everyday driver (in addition to approaching normal road conditions and functions like turning left with a different outlook and awareness).
It’s why I often tell people not to switch to Linux. It’s not for them.
I’ve been professionally developing software for two decades. I can integrate those tools. Still, I find it unnecessarily more difficult than integrating programming language modules. And adjusting my solution to use ready-made, well-integrated complex software is significantly easier, all while being more stable.
Okay. Build a weird farm use only modified v10 excursion with half the body removed and a gooseneck receiver welded on.
If you find it too limiting to tape together the standard utilities then do what your intro to programming class did when you learned c and write your own utilities and libraries, zip them up and store them on a cheap vps you can remember the ip or fqdn of and yank them down onto whatever system you happen to be on, pardon the turn of phrase, like a normal person.
The reason a particular style of pressure fitting is mandated by law to be used for hydraulic connections between cabs and trailers isn’t because it’s the most versatile, repairable or even most secure type but because its manual of operation is simple, straightforward and very importantly: excludes off label or dangerous uses.
An alternative to the standard utilities or Unix way is something you want because you think it will be easier and faster and more versatile. It might be that way for you. Combination wrenches tend to have the sizes and shapes they do to accommodate working around the spaces their size of fitting tends to be in and because the wrench snapping in half when you try to operate it with a breaker bar is the tools way of telling you to begin destructive removal of the fastener instead of continuing to pour pb blaster into it.
Unix is dead. Everyone ships RESTful services + web UI. Underlying platform doesn’t make a difference. You could ship it on Windows server and no one would notice.
The programmers and ops would probably notice because it now takes ten times the effort to automate stuff and customize things for their clients. I mean there is a reason why Linux is the market leader for any kind of serious behind-the-scenes computer stuff. But yeah, there’s enough abstraction layers in-between. The end-users would probably get pretty much the same service. Maybe at a slightly different price point. And it sometimes has an effect on IT infrastructure at work. You’d either get a more traditional workspace or Office365 and cloud only.
Even if so, discussing dead things could be interesting as well 🦖️
