Bash - pair each line of file2019 Community Moderator ElectionCommand line tool to “cat” pairwise expansion of all rows in a fileCost-efficiently pair each line of a file with all othersHow to rewrite multiline path into one-line relative pathMerge two files line by line with the delimiter triple pipe symbol “|||”How do I remove all but the file name (with no extension) from a full file path?Insert string or line after last instance of a specific search variable, in a loopcopy every line from a text file that contains a number greater than 5000How to insert a line from file A above the FIRST LINE in file Bextract lines from a file based on sequential pair of patterns, and output to separate filesscript to parse file for two consecutive lines of unequal lengthHow to add a path before the first character of each line in a file.txt and save the same file with those edits?How can I add 10 lines from a file (file2) to another one after 2 lines (file1)?

Air travel with refrigerated insulin

"Oh no!" in Latin

When is the exact date for EOL of Ubuntu 14.04 LTS?

Remove all of the duplicate numbers in an array of numbers - Javascript

How much do grades matter for a future academia position?

How to get directions in deep space?

What is the tangent at a sharp point on a curve?

Did I make a mistake by ccing email to boss to others?

Why is the sun approximated as a black body at ~ 5800 K?

win_unzip does not extract file

Make a Bowl of Alphabet Soup

Slur or Tie when they are mixed?

Distinction between 地平線 【ちへいせん】 and 水平線 【すいへいせん】

Why do Radio Buttons not fill the entire outer circle?

What is the meaning of "You've never met a graph you didn't like?"

Should I warn a new PhD Student?

Why does the frost depth increase when the surface temperature warms up?

My co-worker is secretly taking pictures of me

How to preserve electronics (computers, iPads and phones) for hundreds of years

Extracting patterns from a text

Giving feedback to someone without sounding prejudiced

Showing mass murder in a kid's book

How to understand "he realized a split second too late was also a mistake"

Adjusting bounding box of PlotLegends in TimelinePlot



Bash - pair each line of file



2019 Community Moderator ElectionCommand line tool to “cat” pairwise expansion of all rows in a fileCost-efficiently pair each line of a file with all othersHow to rewrite multiline path into one-line relative pathMerge two files line by line with the delimiter triple pipe symbol “|||”How do I remove all but the file name (with no extension) from a full file path?Insert string or line after last instance of a specific search variable, in a loopcopy every line from a text file that contains a number greater than 5000How to insert a line from file A above the FIRST LINE in file Bextract lines from a file based on sequential pair of patterns, and output to separate filesscript to parse file for two consecutive lines of unequal lengthHow to add a path before the first character of each line in a file.txt and save the same file with those edits?How can I add 10 lines from a file (file2) to another one after 2 lines (file1)?










9















This question is strongly related to this and this question. I have a file that contains several lines where each line is a path to a file. Now I want to pair each line with each different line (not itself). Also a pair A B is equal to a B A pair for my purposes, so only one of these combinations should be produced.



Example



files.dat reads like this in a shorthand notation, each letter is a file path (absolute or relative)



a
b
c
d
e


Then my result should look something like this:



a b
a c
a d
a e
b c
b d
b e
c d
c e
d e


Preferrably I would like to solve this in bash. Unlike the other questions, my file list is rather small (about 200 lines), so using loops and RAM capacity
pose no problems.










share|improve this question









New contributor




Enno is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.




















  • Does it have to be in bash proper, or just something available via the bash commandline? Other utilities are better positioned to process text.

    – Jeff Schaller
    Mar 17 at 14:15











  • @JeffSchaller Something accessible via the bash commandline. I was a bit unclear, sorry

    – Enno
    Mar 17 at 14:17












  • This is almost becoming a Code Golf :P

    – Richard de Wit
    2 days ago






  • 1





    As a general rule, as long as you need to do something non-trivial, use your favourite scripting language over BASH. It will be less fragile (for example, against special characters or spaces), and much easier to expand whenever you need it (if you need three, or filter some of them away). Python or Perl should be installed in almost any Linux box, so they are good choices (unless you are working on embedded systems, like Busybox).

    – Davidmh
    2 days ago















9















This question is strongly related to this and this question. I have a file that contains several lines where each line is a path to a file. Now I want to pair each line with each different line (not itself). Also a pair A B is equal to a B A pair for my purposes, so only one of these combinations should be produced.



Example



files.dat reads like this in a shorthand notation, each letter is a file path (absolute or relative)



a
b
c
d
e


Then my result should look something like this:



a b
a c
a d
a e
b c
b d
b e
c d
c e
d e


Preferrably I would like to solve this in bash. Unlike the other questions, my file list is rather small (about 200 lines), so using loops and RAM capacity
pose no problems.










share|improve this question









New contributor




Enno is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.




















  • Does it have to be in bash proper, or just something available via the bash commandline? Other utilities are better positioned to process text.

    – Jeff Schaller
    Mar 17 at 14:15











  • @JeffSchaller Something accessible via the bash commandline. I was a bit unclear, sorry

    – Enno
    Mar 17 at 14:17












  • This is almost becoming a Code Golf :P

    – Richard de Wit
    2 days ago






  • 1





    As a general rule, as long as you need to do something non-trivial, use your favourite scripting language over BASH. It will be less fragile (for example, against special characters or spaces), and much easier to expand whenever you need it (if you need three, or filter some of them away). Python or Perl should be installed in almost any Linux box, so they are good choices (unless you are working on embedded systems, like Busybox).

    – Davidmh
    2 days ago













9












9








9


0






This question is strongly related to this and this question. I have a file that contains several lines where each line is a path to a file. Now I want to pair each line with each different line (not itself). Also a pair A B is equal to a B A pair for my purposes, so only one of these combinations should be produced.



Example



files.dat reads like this in a shorthand notation, each letter is a file path (absolute or relative)



a
b
c
d
e


Then my result should look something like this:



a b
a c
a d
a e
b c
b d
b e
c d
c e
d e


Preferrably I would like to solve this in bash. Unlike the other questions, my file list is rather small (about 200 lines), so using loops and RAM capacity
pose no problems.










share|improve this question









New contributor




Enno is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.












This question is strongly related to this and this question. I have a file that contains several lines where each line is a path to a file. Now I want to pair each line with each different line (not itself). Also a pair A B is equal to a B A pair for my purposes, so only one of these combinations should be produced.



Example



files.dat reads like this in a shorthand notation, each letter is a file path (absolute or relative)



a
b
c
d
e


Then my result should look something like this:



a b
a c
a d
a e
b c
b d
b e
c d
c e
d e


Preferrably I would like to solve this in bash. Unlike the other questions, my file list is rather small (about 200 lines), so using loops and RAM capacity
pose no problems.







shell-script text-processing






share|improve this question









New contributor




Enno is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











share|improve this question









New contributor




Enno is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









share|improve this question




share|improve this question








edited Mar 17 at 14:18









Jeff Schaller

43.7k1161141




43.7k1161141






New contributor




Enno is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









asked Mar 17 at 14:14









EnnoEnno

1483




1483




New contributor




Enno is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.





New contributor





Enno is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.






Enno is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.












  • Does it have to be in bash proper, or just something available via the bash commandline? Other utilities are better positioned to process text.

    – Jeff Schaller
    Mar 17 at 14:15











  • @JeffSchaller Something accessible via the bash commandline. I was a bit unclear, sorry

    – Enno
    Mar 17 at 14:17












  • This is almost becoming a Code Golf :P

    – Richard de Wit
    2 days ago






  • 1





    As a general rule, as long as you need to do something non-trivial, use your favourite scripting language over BASH. It will be less fragile (for example, against special characters or spaces), and much easier to expand whenever you need it (if you need three, or filter some of them away). Python or Perl should be installed in almost any Linux box, so they are good choices (unless you are working on embedded systems, like Busybox).

    – Davidmh
    2 days ago

















  • Does it have to be in bash proper, or just something available via the bash commandline? Other utilities are better positioned to process text.

    – Jeff Schaller
    Mar 17 at 14:15











  • @JeffSchaller Something accessible via the bash commandline. I was a bit unclear, sorry

    – Enno
    Mar 17 at 14:17












  • This is almost becoming a Code Golf :P

    – Richard de Wit
    2 days ago






  • 1





    As a general rule, as long as you need to do something non-trivial, use your favourite scripting language over BASH. It will be less fragile (for example, against special characters or spaces), and much easier to expand whenever you need it (if you need three, or filter some of them away). Python or Perl should be installed in almost any Linux box, so they are good choices (unless you are working on embedded systems, like Busybox).

    – Davidmh
    2 days ago
















Does it have to be in bash proper, or just something available via the bash commandline? Other utilities are better positioned to process text.

– Jeff Schaller
Mar 17 at 14:15





Does it have to be in bash proper, or just something available via the bash commandline? Other utilities are better positioned to process text.

– Jeff Schaller
Mar 17 at 14:15













@JeffSchaller Something accessible via the bash commandline. I was a bit unclear, sorry

– Enno
Mar 17 at 14:17






@JeffSchaller Something accessible via the bash commandline. I was a bit unclear, sorry

– Enno
Mar 17 at 14:17














This is almost becoming a Code Golf :P

– Richard de Wit
2 days ago





This is almost becoming a Code Golf :P

– Richard de Wit
2 days ago




1




1





As a general rule, as long as you need to do something non-trivial, use your favourite scripting language over BASH. It will be less fragile (for example, against special characters or spaces), and much easier to expand whenever you need it (if you need three, or filter some of them away). Python or Perl should be installed in almost any Linux box, so they are good choices (unless you are working on embedded systems, like Busybox).

– Davidmh
2 days ago





As a general rule, as long as you need to do something non-trivial, use your favourite scripting language over BASH. It will be less fragile (for example, against special characters or spaces), and much easier to expand whenever you need it (if you need three, or filter some of them away). Python or Perl should be installed in almost any Linux box, so they are good choices (unless you are working on embedded systems, like Busybox).

– Davidmh
2 days ago










6 Answers
6






active

oldest

votes


















6














Use this command:



awk ' name[$1]++ 
END PROCINFO["sorted_in"] = "@ind_str_asc"
for (v1 in name) for (v2 in name) if (v1 < v2) print v1, v2
' files.dat


PROCINFO may be a gawk extension. 
If your awk doesn’t support it,
just leave out the PROCINFO["sorted_in"] = "@ind_str_asc" line
and pipe the output into sort (if you want the output sorted).



(This does not require the input to be sorted.)






share|improve this answer






























    7














    $ join -j 2 -o 1.1,2.1 file file | awk '!seen[$1,$2]++ && !seen[$2,$1]++'
    a b
    a c
    a d
    a e
    b c
    b d
    b e
    c d
    c e
    d e


    This assumes that no line in the input file contains any whitespace. It also assumes that the file is sorted.



    The join command creates the full cross product of the lines in the file. It does this by joining the file with itself on a non-existing field. The non-standard -j 2 may be replaced by -1 2 -2 2 (but not by -j2 unless you use GNU join).



    The awk command reads the result of this and only outputs results that are pairs that has not yet been seen.






    share|improve this answer

























    • What do you mean by "the file is sorted"? Sorted by which criteria?

      – Enno
      Mar 17 at 14:36











    • @Enno Sorted the way sort -b would sort it. join require sorted input files.

      – Kusalananda
      Mar 17 at 14:44



















    7














    A python solution.
    The input file is fed to itertools.combinations from the standard library, which generates 2-length tuples that are formatted and printed to standard output.



    python3 -c 'from itertools import combinations
    with open("file") as f:
    lines = (line.rstrip() for line in f)
    lines = (" ".format(x, y) for x, y in combinations(lines, 2))
    print(*lines, sep="n")
    '





    share|improve this answer






























      6














      If you have ruby installed:



      $ ruby -0777 -F'n' -lane '$F.combination(2) c' ip.txt
      a b
      a c
      a d
      a e
      b c
      b d
      b e
      c d
      c e
      d e



      • -0777 slurp entire file (should be okay as it is mentioned in OP that file size is small)


      • -F'n' split based on newline, so each line will be an element in $F array


      • $F.combination(2) generate combinations 2 elements at a time


      • c print as required

      • if input file can contain duplicates, use $F.uniq.combination(2)



      for 3 elements at a time:



      $ ruby -0777 -F'n' -lane '$F.combination(3) c' ip.txt
      a b c
      a b d
      a b e
      a c d
      a c e
      a d e
      b c d
      b c e
      b d e
      c d e





      With perl (not generic)



      $ perl -0777 -F'n' -lane 'for $i (0..$#F) 
      for $j ($i+1..$#F)
      print "$F[$i] $F[$j]n" ' ip.txt
      a b
      a c
      a d
      a e
      b c
      b d
      b e
      c d
      c e
      d e




      With awk



      $ awk ' a[NR]=$0 
      END for(i=1;i<=NR;i++)
      for(j=i+1;j<=NR;j++)
      print a[i], a[j] ' ip.txt
      a b
      a c
      a d
      a e
      b c
      b d
      b e
      c d
      c e
      d e





      share|improve this answer
































        3














        Here's one in pure shell.



        test $# -gt 1 || exit
        a=$1
        shift
        for f in "$@"
        do
        echo $a $f
        done
        exec /bin/sh $0 "$@"


        Example:



        ~ (137) $ sh test.sh $(cat file.dat)
        a b
        a c
        a d
        a e
        b c
        b d
        b e
        c d
        c e
        d e
        ~ (138) $





        share|improve this answer










        New contributor




        EdC is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
        Check out our Code of Conduct.















        • 1





          Command substitution strips trailing newlines, so you're better off with something like <file.dat xargs test.sh than test.sh $(cat file.dat)

          – iruvar
          2 days ago


















        0














        Using Perl we can do it as shown:



        $ perl -lne '
        push @A, $_} exit
        a=$1
        shift
        for f in "$@"
        do
        echo $a $f
        done
        exec /bin/sh $0 "$@"


        Example:



        ~ (137) $ sh test.sh $(cat file.dat)
        a b
        a c
        a d
        a e
        b c
        b d
        b e
        c d
        c e
        d e
        ~ (138) $





        shareimprove this answer

























          0












          0








          0







          Using Perl we can do it as shown:



          $ perl -lne '
          push @A, $_
          while ( @A )
          my $e = shift @A;
          print "$e $_" for @A;

          ' input.txt





          share{
          while ( @A )
          my $e = shift @A;
          print "$e $_" for @A;

          ' input.txt






          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered 2 days ago









          Rakesh SharmaRakesh Sharma

          342115




          342115




















              Enno is a new contributor. Be nice, and check out our Code of Conduct.









              draft saved

              draft discarded


















              Enno is a new contributor. Be nice, and check out our Code of Conduct.












              Enno is a new contributor. Be nice, and check out our Code of Conduct.











              Enno is a new contributor. Be nice, and check out our Code of Conduct.














              Thanks for contributing an answer to Unix & Linux Stack Exchange!


              • Please be sure to answer the question. Provide details and share your research!

              But avoid


              • Asking for help, clarification, or responding to other answers.

              • Making statements based on opinion; back them up with references or personal experience.

              To learn more, see our tips on writing great answers.




              draft saved


              draft discarded














              StackExchange.ready(
              function ()
              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f506815%2fbash-pair-each-line-of-file%23new-answer', 'question_page');

              );

              Post as a guest















              Required, but never shown





















































              Required, but never shown














              Required, but never shown












              Required, but never shown







              Required, but never shown

































              Required, but never shown














              Required, but never shown












              Required, but never shown







              Required, but never shown







              Popular posts from this blog

              Quoting Keynes in a lectureIs differentiated instruction permitted by universities?How to make students learn prerequisitesUnsatisfactory Instructor Evaluations: balancing of expectations of engineering studentsWhat is the difference between a “statistician”, “applied statistician”, and an academic applying advanced stats within their field?Listing in reference section, but not quotingHow to efficiently use time while preparing for a class?Graduate Admissions: Teaching Emphasisstrategies for sharing teaching information with universities I don't personally have contacts withIs there an efficient way to give a large class of students feedback about their assignments?Is it unreasonable to expect students to read the lecture notes before attending the first class?

              Rank groups within a grouped sequence of TRUE/FALSE and NAGrouping functions (tapply, by, aggregate) and the *apply familyCharacters counting and subletting specific patternsWhat is the purpose of setting a key in data.table?data.table vs dplyr: can one do something well the other can't or does poorly?how to make a bar plot for a list of dataframes?How to group by unique values in a list in RPandas - Alternative to rank() function that gives unique ordinal ranks for a columnRank within group in for loop in RData transformation: from dyadic to observational data in RGetting map from purrr to work with paste0

              Are all passive ability checks floors for active ability checks?Does passive perception supersede active perception?Which skills can be used passively?Active Opposition with Free-Form Professions in Fate5E Trap/Ambush/Stealth Mechanics VS Passive Perception ConfusionInteraction between perception and stealth in obscured conditionsHow does Keen Sight affect Passive Perception?Are all d20 rolls either attacks, saves or ability checks?Can players declare that they are making a specific ability check?Can I see a Hidden creature that is not obscured at all?Can a Stealth check ever be made passively?Is this alternate version of the Observant feat balanced?What is the minimum amount of skill points per HD?