lcpan tips 015: Munging lcpan text output with ‘td’

About this series: a collection of short, daily blog posts about lcpan tips/recipes. Some posts will also end up in the upcoming App::lcpan::Manual::Cookbook POD to be included in the App-lcpan distribution.

About lcpan: an application to download and index a mini CPAN mirror on your local filesystem, so in effect you will have something like your own CPAN with a command-line tool (or perl API) to query and extract information from your mirror. I find it perfect for my own personal use when working offline.

Often the output of lcpan is too wide to view on your terminal, causing wrapped text that’s hard to read, e.g.:

% lcpan related-mods Text::Roman
+------------------------------+------------------------------------------------------------------+--------------+-----------------------+-----------------------+-----
--+---------------------------+----------+
| module                       | abstract                                                         | num_mentions | num_mentions_together | pct_mentions_together | scor
e | dist                      | author   |
+------------------------------+------------------------------------------------------------------+--------------+-----------------------+-----------------------+-----
--+---------------------------+----------+
| Math::Roman                  | Arbitrary sized Roman numbers and conversion from and to Arabic. | 5            | 3                     | 60                    | 540 
  | Math-Roman                | TELS     |
| Convert::Number::Roman       | Convert Between Western and Roman Numeral Systems                | 2            | 2                     | 100                   | 400 
  | Convert-Number-Roman      | DYACOB   |
| Roman                        | functions for converting between Roman and Arabic numerals       | 4            | 2                     | 50                    | 200 
  | Roman                     | CHORNY   |
| Roman::Unicode               | Make roman numerals, using the Unicode characters for them       | 4            | 2                     | 50                    | 200 
  | Roman-Unicode             | BDFOY    |
| Acme::MetaSyntactic::roman   | The roman theme                                                  | 1            | 1                     | 100                   | 100 
  | Acme-MetaSyntactic-Themes | BOOK     |
| Acme::Roman                  | Do maths like Romans did                                         | 1            | 1                     | 100                   | 100 
  | Acme-Roman                | FERREIRA |
| Convert::Number::Digits      | Convert Digits Between the Scripts of Unicode.                   | 1            | 1                     | 100                   | 100 
  | Convert-Number-Digits     | DYACOB   |
| Language::Befunge::lib::ROMA | Roman numerals extension                                         | 1            | 1                     | 100                   | 100 
  | Language-Befunge          | JQUELIN  |
| Convert::Number::Coptic      | Convert Between Western and Coptic Numeral Systems               | 4            | 1                     | 25                    | 25  
  | Convert-Number-Coptic     | DYACOB   |
+------------------------------+------------------------------------------------------------------+--------------+-----------------------+-----------------------+-----
--+---------------------------+----------+

In a GUI terminal emulator, you usually can shrink the font size so more characters can fit in a single row. For example, in Konsole you can press Ctrl-[-] to do this. However, as the font becomes smaller it’s harder to read. Sometimes you just want to read some columns and ignore the others.

Since lcpan is using Perinci::CmdLine framework, its text output is actually a data structure, as you can see if you ask it to output in JSON format instead:

% lcpan related-mods Text::Roman --json
[
   {
      "abstract" : "Arbitrary sized Roman numbers and conversion from and to Arabic.",
      "author" : "TELS",
      "dist" : "Math-Roman",
      "module" : "Math::Roman",
      "num_mentions" : 5,
      "num_mentions_together" : 3,
      "pct_mentions_together" : 60,                                                                                                                                    
      "score" : 540                                                                                                                                                    
   },                                                                                                                                                                  
   {                                                                                                                                                                   
      "abstract" : "Convert Between Western and Roman Numeral Systems",                                                                                                
      "author" : "DYACOB",
      "dist" : "Convert-Number-Roman",
      "module" : "Convert::Number::Roman",
      "num_mentions" : 2,
      "num_mentions_together" : 2,
      "pct_mentions_together" : 100,
      "score" : 400
   },
   {
      "abstract" : "functions for converting between Roman and Arabic numerals",
      "author" : "CHORNY",
      "dist" : "Roman",
      "module" : "Roman",
      "num_mentions" : 4,
      "num_mentions_together" : 2,
      "pct_mentions_together" : 50,
      "score" : 200
   },
   {
      "abstract" : "Make roman numerals, using the Unicode characters for them",
      "author" : "BDFOY",
      "dist" : "Roman-Unicode",
      "module" : "Roman::Unicode",
      "num_mentions" : 4,
      "num_mentions_together" : 2,
      "pct_mentions_together" : 50,
      "score" : 200
   },
   {
      "abstract" : "The roman theme",
      "author" : "BOOK",
      "dist" : "Acme-MetaSyntactic-Themes",
      "module" : "Acme::MetaSyntactic::roman",
      "num_mentions" : 1,
      "num_mentions_together" : 1,
      "pct_mentions_together" : 100,
      "score" : 100
   },
   {
      "abstract" : "Do maths like Romans did",
      "author" : "FERREIRA",
      "dist" : "Acme-Roman",
      "module" : "Acme::Roman",
      "num_mentions" : 1,
      "num_mentions_together" : 1,
      "pct_mentions_together" : 100,
      "score" : 100
   },
   {
      "abstract" : "Convert Digits Between the Scripts of Unicode.",
      "author" : "DYACOB",
      "dist" : "Convert-Number-Digits",
      "module" : "Convert::Number::Digits",
      "num_mentions" : 1,
      "num_mentions_together" : 1,
      "pct_mentions_together" : 100,
      "score" : 100
   },
   {
      "abstract" : "Roman numerals extension",
      "author" : "JQUELIN",
      "dist" : "Language-Befunge",
      "module" : "Language::Befunge::lib::ROMA",
      "num_mentions" : 1,
      "num_mentions_together" : 1,
      "pct_mentions_together" : 100,
      "score" : 100
   },
   {
      "abstract" : "Convert Between Western and Coptic Numeral Systems",
      "author" : "DYACOB",
      "dist" : "Convert-Number-Coptic",
      "module" : "Convert::Number::Coptic",
      "num_mentions" : 4,
      "num_mentions_together" : 1,
      "pct_mentions_together" : 25,
      "score" : 25
   }
]

Now you can use a JSON munging tool like jq to filter the fields of each record, and then later render the JSON back to text table using tool like pretty, e.g.:

% lcpan related-mods Text::Roman --json | jq '[ .[] | {module, abstract} ]'
[
  {
    "module": "Math::Roman",
    "abstract": "Arbitrary sized Roman numbers and conversion from and to Arabic."
  },
  {
    "module": "Convert::Number::Roman",
    "abstract": "Convert Between Western and Roman Numeral Systems"
  },
  {
    "module": "Roman",
    "abstract": "functions for converting between Roman and Arabic numerals"
  },
  {
    "module": "Roman::Unicode",
    "abstract": "Make roman numerals, using the Unicode characters for them"
  },
  {
    "module": "Acme::MetaSyntactic::roman",
    "abstract": "The roman theme"
  },
  {
    "module": "Acme::Roman",
    "abstract": "Do maths like Romans did"
  },
  {
    "module": "Convert::Number::Digits",
    "abstract": "Convert Digits Between the Scripts of Unicode."
  },
  {
    "module": "Language::Befunge::lib::ROMA",
    "abstract": "Roman numerals extension"
  },
  {
    "module": "Convert::Number::Coptic",
    "abstract": "Convert Between Western and Coptic Numeral Systems"
  }
]
% lcpan related-mods Text::Roman --json | jq '[ .[] | {module, abstract} ]' | pretty
┌──────────────────────────────────────────────────────────────────────────────┐
│ abstract                                        module                       │
│                                                                              │
│ Arbitrary sized Roman numbers and conversion    Math::Roman                  │
│ from and to Arabic.                                                          │
│ Convert Between Western and Roman Numeral       Convert::Number::Roman       │
│ Systems                                                                      │
│ functions for converting between Roman and      Roman                        │
│ Arabic numerals                                                              │
│ Make roman numerals, using the Unicode          Roman::Unicode               │
│ characters for them                                                          │
│ The roman theme                                 Acme::MetaSyntactic::roman   │
│ Do maths like Romans did                        Acme::Roman                  │
│ Convert Digits Between the Scripts of           Convert::Number::Digits      │
│ Unicode.                                                                     │
│ Roman numerals extension                        Language::Befunge::lib::ROMA │
│ Convert Between Western and Coptic Numeral      Convert::Number::Coptic      │
│ Systems                                                                      │
└──────────────────────────────────────────────────────────────────────────────┘

Or, if you want to make sure that the order of the columns is maintained:

% lcpan related-mods Text::Roman --json | jq '[ .[] | [.module, .abstract] ]' | pretty
┌──────────────────────────────────────────────────────────────────────────────┐
│ column0                        column1                                       │
│                                                                              │
│ Math::Roman                    Arbitrary sized Roman numbers and conversion  │
│                                from and to Arabic.                           │
│ Convert::Number::Roman         Convert Between Western and Roman Numeral     │
│                                Systems                                       │
│ Roman                          functions for converting between Roman and    │
│                                Arabic numerals                               │
│ Roman::Unicode                 Make roman numerals, using the Unicode        │
│                                characters for them                           │
│ Acme::MetaSyntactic::roman     The roman theme                               │
│ Acme::Roman                    Do maths like Romans did                      │
│ Convert::Number::Digits        Convert Digits Between the Scripts of         │
│                                Unicode.                                      │
│ Language::Befunge::lib::ROMA   Roman numerals extension                      │
│ Convert::Number::Coptic        Convert Between Western and Coptic Numeral    │
│                                Systems                                       │
└──────────────────────────────────────────────────────────────────────────────┘

There’s an easier way though, using td. It takes a JSON input, processes it according to the given command, and then renders it back to text table (among other formats). For example:

% lcpan related-mods Text::Roman --json | td select module abstract
+------------------------------+------------------------------------------------------------------+
| module                       | abstract                                                         |
+------------------------------+------------------------------------------------------------------+
| Math::Roman                  | Arbitrary sized Roman numbers and conversion from and to Arabic. |
| Convert::Number::Roman       | Convert Between Western and Roman Numeral Systems                |
| Roman                        | functions for converting between Roman and Arabic numerals       |
| Roman::Unicode               | Make roman numerals, using the Unicode characters for them       |
| Acme::MetaSyntactic::roman   | The roman theme                                                  |
| Acme::Roman                  | Do maths like Romans did                                         |
| Convert::Number::Digits      | Convert Digits Between the Scripts of Unicode.                   |
| Language::Befunge::lib::ROMA | Roman numerals extension                                         |
| Convert::Number::Coptic      | Convert Between Western and Coptic Numeral Systems               |
+------------------------------+------------------------------------------------------------------+

In the above command, we tell td to just select a couple of columns from the table.

There are a few other td commands available, e.g. to sort rows based on one or more columns, to perform sum/count/and so on. So basically td is the equivalent of Unix toolbox command cut, head, tail, sum, and so on, except that it operates on a table data structure instead of text stream.

There’s an extra convenience put in if you use td. If Pipe::Find module is available, Perinci::CmdLine can use it to check that the program called “td” is at the right side of the pipeline and automatically defaults to JSON output, so you don’t need to specify --json:

% lcpan related-mods Text::Roman | td select module abstract

Note that this tip is not only for lcpan, but for all Perinci::CmdLine-based CLI applications.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s