lcpan tips 022: Testing all dependents

About this series: A collection of short blog posts about lcpan tips/recipes. Some posts will also end up in the upcoming App::lcpan::Manual::Cookbook POD to be included in the App::lcpan distribution. First article is here. See the whole series.

About lcpan: an application to download and index a mini CPAN mirror on your local filesystem, so in effect you will have something like your own CPAN with a command-line tool (or perl API) to query and extract information from your mirror.

When you release a new version of a module, CPAN Testers will test your module on a variety of platforms. Sometimes, when they happen to test other modules that depend on your module, they will also uncover problems that is caused by your new version breaking your dependents. However, this is not done systematically or completely. If you want to make sure your changes do not break your dependents, you can test for yourself.

Suppose I’m making some changes to Regexp::Pattern. Let’s see who depends on this module:

% lcpan rdeps Regexp::Pattern -R --phase runtime --rel requires
+----------------------------------+-----------+--------------+-------------+
| dist                             | author    | dist_version | req_version |
+----------------------------------+-----------+--------------+-------------+
| App-Licensecheck                 | JONASS    | v3.0.36      | 0           |
| App-RegexpPatternUtils           | PERLANCAR | 0.003        | 0           |
| Bencher-Scenarios-RegexpPattern  | PERLANCAR | 0.003        | 0           |
| Regexp-Common-RegexpPattern      | PERLANCAR | 0.001        | 0           |
|   Bencher-Scenarios-RegexpCommon | PERLANCAR | 0.02         | 0           |
| Test-Regexp-Pattern              | PERLANCAR | 0.004        | v0.2.7      |
+----------------------------------+-----------+--------------+-------------+

Force reinstalling all dependents

One way to run tests for all dependents is by force-(re)installing all those distributions. One way to do that:

% for mod in `lcpan rdeps Regexp::Pattern -R | td select dist | perl -pe's/^\s+//' | dist2mod`; do
    echo "Installing $mod ..."
    lcpanm --force $mod
  done

dist2mod is provided by App::DistUtils.

Extracting tarballs

Another way to test all dependents is to extract them to a temporary directory then run prove on each. lcpan provides the handy subcommand extract-dist to do this. But you might want to install the dependents all first to pull their dependencies.

% mkdir test-deps
% cd test-deps
% for dist in `lcpan rdeps Regexp::Pattern -R | td select dist`; do
    mkdir $dist && cd $dist && lcpan extract-dist $dist && cd ..
done

We can now test each dependent distro one by one, or all in one go:

% for dist in *; do
    echo "Testing $dist ..."
    cd $dist/*; prove -lr; cd ../..
done
Advertisements

lcpan tips 018: How did I use lcpan in 2018?

About this series: A collection of short blog posts about lcpan tips/recipes. Some posts will also end up in the upcoming App::lcpan::Manual::Cookbook POD to be included in the App::lcpan distribution. First article is here. See the whole series.

About lcpan: an application to download and index a mini CPAN mirror on your local filesystem, so in effect you will have something like your own CPAN with a command-line tool (or perl API) to query and extract information from your mirror. I find it perfect for my own personal use when working offline.

lcpan remains as one of my most-used tools when doing Perl/CPAN development. I use it to search for modules to do some task (usually when I am offline or too lazy to open a browser tab to MetaCPAN, but also to do some grep-ing against the search results). I also use it sometimes to find modules related to a specific module, as keyword searches or seeing the SEE ALSO section of PODs are sometimes insufficient.

The following is a simplistic shell history analysis on how I use lcpan (on my main laptop, at least). The history I have on my laptop is from Nov 2017 to Jan 2019, so that more or less reflects how I used lcpan during 2018.

lcpan and lcpanm

% history | perl -lne's/.+?\]//; s/^pg //; next unless /^lcpanm?\s/; s/(^\S+).*/$1/; print' | freqtable
459	  lcpan
332     lcpanm

I install modules from local mirror using lcpanm quite often.

PAGE_RESULT=1

"pg " is a shell alias which I define as:

alias pg='PAGE_RESULT=1'

because that lets me navigate the output of lcpan with a PAGER (which I have defined as 'less -FRSX').

Most often used subcommands

% history | perl -lne's/.+?\]//; s/^pg //; next unless /^lcpan\s/; s/(^\S+ \S+).*/$1/; print' | freqtable
175	lcpan mods
107	lcpan doc
 35	lcpan scripts
 32	lcpan deps
 17	lcpan rdeps
 16	lcpan related-mods
 14	lcpan src
 11	lcpan stats-last-index-time
 10	lcpan upd
...

I use lcpan most often to:

  • search for modules (lcpan mods);
  • read documentation of modules which I don't have installed yet (lcpan doc);
  • search for scripts (lcpan scripts);
  • see dependencies (lcpan deps);
  • see reverse dependencies (lcpan rdeps);
  • search for related modules (lcpan related-mods);
  • see source code of modules I don't have installed yet (lcpan src);
  • checking the last update date of the index (lcpan stats-last-index-time);
  • updating the mirror/index (lcpan upd which is shortcut for lcpan update).

Finding modules

% history | perl -lne's/.+?\]//; s/^pg //; next unless /^lcpan mods\s/; print' | sort -u
lcpan mods
lcpan mods average
lcpan mods average -l
lcpan mods average|wc -l
lcpan mods binary search -l
lcpan mods bitfin
lcpan mods bitflip
lcpan mods bitgrail
lcpan mods bloomberg
lcpan mods bluetooth
lcpan mods bluetooth -l
lcpan mods b perlstring
lcpan mods ccxt
lcpan mods cpan release
lcpan mods cpan release -l
lcpan mods DateTime::Format
lcpan mods DateTime::Format duration
lcpan mods DateTime::Format::Japanese
lcpan mods datetime iso8601
lcpan mods datetime iso8601 -l
lcpan mods dbi string
lcpan mods dbi string -l
lcpan mods dbi table
lcpan mods dbi table -l
lcpan mods Dist::Zilla::Plugin add -l
lcpan mods Dist::Zilla::Plugin:: --author PERLANCAR -l
lcpan mods Dist::Zilla::Plugin Module
lcpan mods Dist::Zilla::Plugin Module -l
lcpan mods eval
lcpan mods eval -l
lcpan mods fifo -l
lcpan mods File::Slurper
lcpan mods File::Slurper -l
lcpan mods finance crypto
lcpan mods float util
lcpan mods histo
lcpan mods histog -l
lcpan mods histogr -l
lcpan mods histo -l
lcpan mods inside eval -l
lcpan mods interpo
lcpan mods inventory
lcpan mods inventory -l
lcpan mods inventory|wc -l
lcpan mods -l array rank
lcpan mods -l bin groups
lcpan mods -l c encode
lcpan mods -l cli hub
lcpan mods -l compare
lcpan mods -l data cmp
lcpan mods -l dbi csv
lcpan mods -l dbix conn
lcpan mods -l dbix shortcut
lcpan mods -l freq table
lcpan mods -l groups
lcpan mods -l http tiny
lcpan mods -l list rank
lcpan mods -l module abstract
lcpan mods -l module info
lcpan mods -l module pod
lcpan mods -ln role
lcpan mods -l ord
lcpan mods -l ordina
lcpan mods -l ordinaq
lcpan mods -l permute
lcpan mods -l pod abstract
lcpan mods -l Regexp::Common::
lcpan mods -l Regexp::Pattern
lcpan mods -l return level
lcpan mods -l stock exchange
lcpan mods -l test2 tool
lcpan mods -l Test::Approximate
lcpan mods -l test compare
lcpan mods -l Test::Deep::
lcpan mods -l throttle
lcpan mods -l Tickit
lcpan mods -l Tickit Grid
lcpan mods -l tie array
lcpan mods -l Versioning dot
lcpan mods -l who
lcpan mods --namespace Acme::CPANLists
lcpan mods --namespace Acme::CPANLists -l
lcpan mods --namespace Acme::CPANModules -l
lcpan mods --namespace Acme::CPANModuless -l
lcpan mods --namespace Bencher -l
lcpan mods --namespace Bencher::Scenario
lcpan mods --namespace Bencher::Scenario -l
lcpan mods --namespace Data::Sah::Coerce::perl
lcpan mods --namespace Data::Sah::Coerce::perl::str
lcpan mods --namespace DateTime::Format
lcpan mods --namespace DateTime::Format -l
lcpan mods --namespace Graphics::ColorNames
lcpan mods --namespace Graphics::ColorNames|xargs lcpanm -n
lcpan mods --namespace Log::ger
lcpan mods --namespace Log::Ger
lcpan mods --namespace Log::ger|grep -i dump
lcpan mods --namespace String -l
lcpan mods --namespace WordList::Char
lcpan mods -n Archive::Tar
lcpan mods -n digit
lcpan mods -n digit -l
lcpan mods -n digit|wc -l
lcpan mods nearest -l
lcpan mods near -l
lcpan mods -n fifo
lcpan mods -n generic
lcpan mods -n generic -l
lcpan mods -nl Archive::Tar
lcpan mods -nl gen pw
lcpan mods -nl genpw
lcpan mods -nl pass gen
lcpan mods -nl pwd gen
lcpan mods -n pass gen
lcpan mods -n permute digit
lcpan mods Number::Format
lcpan mods Number Format -l
lcpan mods Number::Format -l
lcpan mods pass templat
lcpan mods pass templat -l
lcpan mods percent
lcpan mods percent Sah
lcpan mods percent|wc -l
lcpan mods perinci usage
lcpan mods perl release -l
lcpan mods permute lis
lcpan mods purchase
lcpan mods purchase -l
lcpan mods purchase price
lcpan mods python -l
lcpan mods qr decode
lcpan mods qr decoded
lcpan mods random norm -l
lcpan mods redact
lcpan mods regexp common
lcpan mods regexp common cc
lcpan mods regexp common credit
lcpan mods regexp common -l
lcpan mods ssh client -l
lcpan mods stack trace -l
lcpan mods stock
lcpan mods stock -l
lcpan mods stock|wc -l
lcpan mods Test::Deep:: -l
lcpan mods test path
lcpan mods Text::Histogram
lcpan mods time -l
lcpan mods time of day -l
lcpan mods timeofday -l
lcpan mods version dot
lcpan mods version scheme
lcpan mods version scheme|grep -v Google
lcpan mods who -l
lcpan mods WordList::CryptoCurrency::Catalog::Name

I do keyword searches a lot, and when the keyword is not specific enough I usually add "pg" and "-l" to let me navigate and search further with less. Sometimes I also do namespace searching (--namespace). I search for my own modules a lot, usually because I forget the exact name.

Finding related modules

Modules I tried to find related modules of:

% history | perl -lne's/.+?\]//; next unless /^lcpan related-mods\s/; print' | sort -u

; lcpan related-mods alias::module

lcpan related-mods Data::Diff
lcpan related-mods Data::Throttler
lcpan related-mods Data::Valve
lcpan related-mods IO::Tee
lcpan related-mods Number::Tolerant
lcpan related-mods Package::Alias
lcpan related-mods String::JS
lcpan related-mods Test::Deep
lcpan related-mods utf8

Finding scripts

% history | perl -lne's/.+?\]//; s/^pg //; next unless /^lcpan scripts\s/; print' | sort -u
lcpan scripts bin -l
lcpan scripts count
lcpan scripts count -l
lcpan scripts dateconv
lcpan scripts dateconv -l
lcpan scripts envres
lcpan scripts group -l
lcpan scripts histogram -l
lcpan scripts http-tiny
lcpan scripts interval
lcpan scripts interval -l
lcpan scripts lineno
lcpan scripts linenum
lcpan scripts line number
lcpan scripts lino
lcpan scripts linum
lcpan scripts -l org2html
lcpan scripts -l org-to-html
lcpan scripts parse-nik
lcpan scripts parse-nik -l
lcpan scripts parse num
lcpan scripts _pause
lcpan scripts perl
lcpan scripts perllint
lcpan scripts pick
lcpan scripts pick -l
lcpan scripts pick-l
lcpan scripts rand
lcpan scripts rand -l
lcpan scripts resolution
lcpan scripts resolution -l
lcpan scripts throttle
lcpan scripts zodiac -l

I search for my own scripts a lot too, since I have almost a thousand (~880) of them on CPAN. It's a bit challenging trying to keep the naming organized. When tab completion doesn't help, lcpan comes to the rescue.

The simplistic simulation mode in rename

About the rename script

rename is a Perl script that lets you use Perl expression or code to rename files. It's convenient for mass-renaming. For example, to convert all filenames in the current directory to lowercase:

% rename -e '$_=lc' *

rename was written by Larry himself. It was released in 1989 as part of perl 3.0 as an example script put under eg/rename. Later, together with a bunch of other old Perl scripts that were not compiling cleanly under strict/-w, it was removed from the perl source tree prior to the release of Perl 5.8 in 2000. It was then republished on CPAN under File::Rename since 2005 by Robin Baker (RBAKER).

About the simulation (dry-run) mode

One of the useful things that rename provides is simulation (-n) mode. In this mode, the files are not actually renamed: rename only prints how it would rename them. For example:

% mkdir test-area
% cd test-area
% touch FOO Bar baz
% rename -n -e '$_ = lc' *
rename(Bar, bar)
rename(FOO, foo)

rename was (and is) a very simple script, measuring at only about 130 lines of code. This simplicity shows in the simplistic simulation mode. As it shows the files as they are being renamed, rename does not keep track of the new filenames that would spring into existence (or old filenames that would disappear). For example:

% rm *
% touch 1 2 3
% rename -n -e '$_ = 3' *
1 not renamed: 3 already exists
2 not renamed: 3 already exists

rename correctly refuse overwriting 3 because it already exists (unless when instructed via -f). However:

% rename -n -e '$_ = 4' *
rename(1, 4)
rename(2, 4)
rename(3, 4)

If you remove the -n option, rename would still behave correctly:

% rename -e '$_ = 4' *
2 not renamed: 4 already exists
3 not renamed: 4 already exists

But the simulation mode does not show you that.

An alternative simulation mode

Another script on CPAN, perlmv, is an alternative that offers a better simulation mode:

% perlmv -d -e'$_ = 4' *
DRYRUN: move `1` -> `4`
DRYRUN: move `2` -> `4.1`
DRYRUN: move `3` -> `4.2`

Note that perlmv does automatic suffixing to avoid overwriting existing files.

Remembering terminal background color for each remote host we SSH to

Entering (the right) command to a wrong SSH session is no fun. To help me be aware that I am typing commands to a (the correct) remote host, I usually set the background color of my Konsole tab (manually, by right-clicking and then switching the profile). Inspired by this post, I decided to make my own SSH wrapper in Perl: sshwrap-hostcolor.

The wrapper

The sshwrap-hostcolor script is a wrapper that will execute ssh with the arguments you passed to it, but do some stuff before and after that. To try out this wrapper, first install it from CPAN:

% cpanm -n App::sshwrap::hostcolor

then alias it to ssh:

% alias ssh=sshwrap-hostcolor

(You might want to put the above line to your shell's startup file, if you decide to use this script permanently after all.)

Now ssh somewhere:

% ssh user@one.example.com

then change the terminal background color. After you exit, the wrapper will restore the original background color. But if you ssh to the same user+host again later, the background color that you set (and saved by the script to ~/.sshwrap-hostcolor.history) will be restored. See video demonstration.

If you want the wrapper to automatically assign a random color to a new user+host, you can set the environment variable SSHWRAP_HOSTCOLOR_AUTO to random-dark (or random-light, if you use a light background color), e.g.:

% export SSHWRAP_HOSTCOLOR_AUTO=random-dark

See video demonstration.

The completion script

The wrapper comes with its own tab completion script, which you can install using:

% complete -C _sshwrap-hostcolor ssh

This completion script can complete user+host argument for you, and when there's a single completion with known background color, the completion script will immediately change the terminal background color during completion. See video demonstration. This might or might not be to your liking, but you have the option to install the completion if you want this behavior.

Additional stuff

Note that this wrapper will only work with an XTerm-compatible terminal emulation software. See the list here.

While writing the wrapper script, I also wrote a few other CLI utilities.

get-term-bgcolor to programmatically get the terminal's current background color.

set-term-bgcolor to programmatically set the terminal's current background color. This script recognizes RGB color code (e.g. 00002b) or color names (e.g. black, darkblue).

show-color-swatch to show the list of color names and their codes. Use it as such:

% show-color-swatch X | less -R

Oh, and the set-term-bgcolor script also features tab completion (activated via complete -C set-term-bgcolor set-term-bgcolor) so you can complete color names, e.g.:

% set-term-bgcolor dark<Tab><Tab>

Removing password from PDF files

Electronic billing statement is great. Paperless, easier to archive, and you don't need to care if you leave the house for a month or move addresses. But what's the most irritating thing about credit card e-statements? That the banks decide to encrypt the PDF with some easy-to-guess password like YYYYMMDD of your birth date, or FirstNameDDMMYY, or some other variation. But each bank picks a different password, naturally.

So the first thing I do after downloading an e-statement is to run them through my remove-pdf-password script. It's basically a wrapper for "qpdf" and will try a series of passwords, then decrypt the PDF in-place. You put the passwords in ~/.config/remove-pdf-password.conf, e.g.:

passwords = PASS1  ; anz
passwords = PASS2  ; hsbc
passwords = PASS2  ; uob
...

The script is distributed with the App::PDFUtils distribution. There's the counterpart script in the distribution, add-pdf-password, in case you want to return the favor. The next time some bank staff requests your documents, you can send them in an email encrypted. In the email you write, "the password is the name of the bank followed by the phone number of the bank, but replace every f with f*ckyou …"

Quickly generate CSV or TSV from SQL query

Sometimes to produce CSV or TSV from information in a MySQL database, all you have to do is:

echo 'some SELECT query ...' | mysql mydb >data.tsv

but for more complex things, you'll often need to use scripting like with Perl to do some munging first. When you're ready to output the data with:

$sth->fetchrow_arrayref
$sth->fetchrow_hashref

the data is in a Perl data structure. How to quickly output CSV? I've looked at DBI::Format and DBIx::CSVDumper and they are cumbersome especially if you want something brief for one-liners: you need to instantiate a separate object with some parameters, then call an additional one or more methods. You might as well just do this with the standard Text::CSV (or Text::CSV_XS):

use Text::CSV;
my $csv = Text::CSV->new;

...
$csv->print(\*STDOUT, $sth->{NAME});
while (my $row = $sth->fetchrow_arrayref) {
     $csv->print(\*STDOUT, $row);
}

For TSV it's even shorter (note: fail spectacularly when data contains Tab character, but that is relatively rare):

say join("\t", @{$sth->{NAME}});
while (my $row = $sth->fetchrow_arrayref) {
     say join("\t", @$row);
}

My modules DBIx::CSV and DBIx::TSV attempt to offer something even briefer (for CSV, at least). Instead of fetchrow_arrayref or fetchall_arrayref (or selectrow_arrayref, selectall_arrayref), you use fetchrow_csv and friends:

print $sth->fetchall_csv;

This automatically prints the header too, and if you don't want header:

print $sth->fetchall_csv_noheader;

For even more text table formats, like ASCII table, Org table, Markdown table, and so on, I've also written DBIx::TextTableAny.

I've also just written DBIx::Conn::MySQL after being tired from writing the gazillionth one-liner to connect to MySQL database. Now, instead of:

% perl -MDBI -E'my $dbh = DBI->connect("dbi:mysql:database=mydb", "username", "password"); $sth = $dbh->prepare(...)'

you can now write this:

% perl -MDBI::Conn::MySQL=mydb -E'$sth = $dbh->prepare(...)'

Generating random passwords according to patterns with genpw

There are several modules on CPAN for generating random passwords, but last time I checked, none of them are flexible enough. Different websites or applications have different requirements (sometimes ridiculous ones) for passwords. True, most modules allow setting minimum/maximum length, and some of them allow customizing the number of special characters, or number of digits, but what I want is some sort of template/pattern that the generator would follow. So I created genpw.

Basic functionality

genpw is your usual random password generator CLI. When run without any arguments, it returns a single random password (8-20 characters long, comprising letters/digits):

% genpw
J9K3ZjBVR

To return several passwords:

% genpw 5
wAYftKsS
knaY7MOBbcvFFS3L1wyW
oQGz62aF
sG1A9reVOe
Zo8GoFEq

To set exact length or minimum/maximum of length:

% genpw -l 4
1CN4
% genpw --min-len 12 --max-len 14
nqBKX5lyyx7B

Patterns

In addition to the above basic customization, genpw allows you to specify a pattern (or several patterns to pick randomly, for that matter). A pattern is a string that is similar to a printf pattern where conversion sequences like %d will be replaced with actual random characters. Here are the available conversions:

%l   Random Latin letter (A-Z, a-z)
%d   Random digit (0-9)
%h   Random hexdigit (0-9a-f)
%a   Random letter/digit (Alphanum) (A-Z, a-z, 0-9; combination of %l and %d)
%s   Random ASCII symbol, e.g. "-" (dash), "_" (underscore), etc.
%x   Random letter/digit/ASCII symbol (combination of %a and %s)
%m   Base64 character (A-Z, a-z, 0-9, +, /)
%b   Base58 character (A-Z, a-z, 0-9 minus IOl0)
%B   Base56 character (A-Z, a-z, 0-9 minus IOol01)
%%   A literal percent sign
%w   Random word

You can specify %NC (where N is a positive integer, and C is the conversion) to mean a sequence of random characters (or a random word) with the exact length of N. Or %N$MC (where N and M are positive integers) to mean random characters (or word) with length between N and M.

Unsupported conversion will be unchanged, like in printf.

Some examples:

Generate random digits between 10 and 12 characters long:

% genpw -p '%10$12d'
55597085674

Generate a random UUID:

% genpw -p '%8h-%4h-%4h-%4h-%12h'
ff26d142-37a8-ecdf-c7f6-8b6ae7b27695

Like the above, but in uppercase:

% genpw -p '%8h-%4h-%4h-%4h-%12h' -U
22E13D9E-1187-CD95-1D05-2B92A09E740D

Words

The %w conversion in pattern mean to replace with a random word. Words are fetched from STDIN (and will be shuffled before use). For example, the command below will generate password in the form of a random word + 4 random digits:

% genpw -p '%w%4d' < /usr/share/dict/words
shafted0412

Instead of from STDIN, you can also fetch words from the various WordList::* modules available on CPAN (and installed locally). A separate CLI provides this functionality: genpw-wordlist, which is basically just a wrapper to extract the words from the wordlist module(s) then feed it to App::genpw. By default, if you don't specify -w option(s) to select wordlist modules, the CLI will use WordList::EN::Enable:

% genpw-wordlist -p '%w%4d'
sedimentologists8542

Generate 5 passwords comprising 8-character word followed by between four to six random digits:

% genpw-wordlist 5 -p '%8w-%4$6d'
alcazars-218050
pathogen-4818
hogmanes-212244
pollened-3206
latinity-21053

Configuration file

To avoid you from having to type patterns again and again, you can use a configuration file. For example, put this in $HOME/genpw.conf:

[profile=uuid]
patterns = %8h-%4h-%4h-%4h-%12h
case = upper

then you can just say:

% genpw -P uuid
7B8FB8C6-0D40-47A3-78B7-848A114E5C6D

Speed

The speed is not great, around 2000 passwords/sec on my laptop, though I believe this should not matter for most use-cases.

Closing

genpw is a flexible random password generator where you can specify patterns or templates for the password. I'm still working on a few things, like how to enable secure random source in the most painless way. There are also various CLI variants to generate specific kinds of passwords as straightforward as possible: genpw-base56, genpw-base64, genpw-id, and the aforementioned genpw-wordlist.