ksh vs bash: setting variable in piped loops are lost

[Georges]

I'm a long time unix (not linux) programmer, mainly shell scripts.
Unix does not know about bash, but rather ksh or sh.

So I often build stuff like this:

n=0
du | sort -n | while read size dir
do
  if [ "$size" -gt 100000 ]
  then
    n=$((n+1))
  fi
done
echo "Found $n too big files"

the question is not how to reformulate this differently using awk or perl.
The question is:

in ksh this script returns the correct value
in bash it always returns 0

This is because in ksh the last command in the pipe runs in the current process, whereas in bash the first command runs in the current proces. Result: the modified variables are lost.

I'm still puzzled that his is the case and that nobody really cares about.
Maybe I'm missing the point and there is some simple environment variable or other setting to change to enable ksh compatible behaviour.

Georges

[mssever]

Originally Posted by Georges I'm a long time unix (not linux) programmer, mainly shell scripts. Unix does not know about bash, but rather ksh or sh. So I often build stuff like this: n=0 du | sort -n | while read size dir do if [ "$size" -gt 100000 ] then n=$((n+1)) fi done echo "Found $n too big files" the question is not how to reformulate this differently using awk or perl. The question is: in ksh this script returns the correct value in bash it always returns 0 This is because in ksh the last command in the pipe runs in the current process, whereas in bash the first command runs in the current proces. Result: the modified variables are lost. I'm still puzzled that his is the case and that nobody really cares about. Maybe I'm missing the point and there is some simple environment variable or other setting to change to enable ksh compatible behaviour. Georges

I guess that's something I've never really tried before--maybe because I've always used bash. I don't think that the bash devs care a whole lot about ksh, but I understand that zsh is nearly 100% compatible with ksh--plus a bunch of added features. You might be interested in it. It's available from the main repo. I haven't used it much, but then, bash does everything I need...

I suppose a bash kludge could be to use tmp files rather than a pipeline, but that isn't very elegant.

[Georges]

yes, "apt-get install ksh" is a solution too.

So you never use longer pipe constructs with loops etc?
That's the real power of shell programming, else one could as well use perl.

zsh also behaves correctly.

Only bash is different. The developers probably think they can reinvent the wheel in square form. Do you think there is any chance that a bug report will help?

[Arndt]

Originally Posted by Georges yes, "apt-get install ksh" is a solution too. So you never use longer pipe constructs with loops etc? That's the real power of shell programming, else one could as well use perl. zsh also behaves correctly. Only bash is different. The developers probably think they can reinvent the wheel in square form. Do you think there is any chance that a bug report will help?

Googling for "bash pipe subprocesses order" shows that the question has been asked before, but I couldn't find what the bash developers' official stand is.

[malagant]

I know this is a dead thread but I wanted to add something in case someone comes looking at this for a solution. My thanks as always to the mksh team. The solution involves (m)ksh co-processes. Sounds like you may be able to do something similar with bash using named pipe pairs (one for read, one for write) if you do a little research in that area.

This works in mksh 35b:

n=0
du | sort -n |& 
while read -p size dir
 
do
  if [ "$size" -gt 1000 ]
  then
    n=$((n+1))
  fi
done
echo "Found $n too big files"

Basically this is straight from the mksh man page. mksh is the successor to pdksh and several times smaller and faster than bash or zsh. Think of it as a step up from dash at almost 100% POSIX compliance. If you come from a Solaris, HP-UX or AIX shop and are used to ksh but miss the nice touches of the newer shells then you should give mksh a try. As a bonus it is UTF aware (zsh isn't nor is bash without add-ons, I believe) and is also actively being developed.

[Georges]

that's like a kludge with temporary files we mentioned earlier.
And there are no co-processes in bash either.
How can this be the default shell in linux. It lacks several great inventions which make unix stand out.

Thanks for the hint on mksh, I will give it a try (setting it as default shell).

Georges

[mirabilos]

Originally Posted by malagant My thanks as always to the mksh team.

That would be me thanks.

Originally Posted by malagant du | sort -n |& Basically this is straight from the mksh man page.

Actually, in your case, this would be better:

(du | sort -n) |&

I seem to recall, though, that these parenthesēs are implicit in recent versions of mksh. (These also build on a fair larger number of platforms – ULTRIX, OSF/1, QNX, … – and with much more compilers than just gcc.)

[eightmillion]

I just spent 15 minutes trying to do this in pure bash before realizing that this thread is nearly 2 years old. Anyway, here's what I came up with. It's not pretty, but it works.

#!/bin/bash
n=0;a=0;x=0
for line in `du | sort -n`;do
if [ $line -eq $line 2> /dev/null ];then
 size[$n]=$line
 n=$(($n+1))
fi
done
while [ $x -lt ${#size[*]} ];do
  if [ ${size[$x]} -gt 100000 ];then
    a=$(($a+1))
  fi
  x=$(($x+1))
done
echo "Found $a too big files"
exit

[Cracauer]

Unfortunately bash is plain and simply correct when it comes to bourne shell script according to POSIX. You got the pipe, you are in a child process and nothing you do influences the parent process.

This is really annoying since it pretty much disables all sane ways to deal with filenames that contain any number of wired characters, sequences of multiple spaces one after another etc.

You never can use
find . ... | while read foo ; echo $foo ; do

Nor can you use
find . -print0 | xargs -0 ...
invoking anything under the script's control that would allow you to count.

The only way to make use of this in the loop solution is to return everything you want for status by echoing them and putting the whole loop into $(...). If you need stdout for other purposes inside the loop you gotta start placing a bunch of bloddy `1>&3) | ... 3>&1 | grep`.

[geirha]

I guess the prettiest way to handle it in bash would be something like this:

#!/bin/bash
exec 3< <(du | sort -n)  

n=0
while read size dir; do
  [ $size -gt 1000 ] && ((n++))
done <&3
exec 3<&-

echo "Found $n too big files"