Removing password from PDF files

Electronic billing statement is great. Paperless, easier to archive, and you don't need to care if you leave the house for a month or move addresses. But what's the most irritating thing about credit card e-statements? That the banks decide to encrypt the PDF with some easy-to-guess password like YYYYMMDD of your birth date, or FirstNameDDMMYY, or some other variation. But each bank picks a different password, naturally.

So the first thing I do after downloading an e-statement is to run them through my remove-pdf-password script. It's basically a wrapper for "qpdf" and will try a series of passwords, then decrypt the PDF in-place. You put the passwords in ~/.config/remove-pdf-password.conf, e.g.:

passwords = PASS1  ; anz
passwords = PASS2  ; hsbc
passwords = PASS2  ; uob

The script is distributed with the App::PDFUtils distribution. There's the counterpart script in the distribution, add-pdf-password, in case you want to return the favor. The next time some bank staff requests your documents, you can send them in an email encrypted. In the email you write, "the password is the name of the bank followed by the phone number of the bank, but replace every f with f*ckyou …"


Quickly generate CSV or TSV from SQL query

Sometimes to produce CSV or TSV from information in a MySQL database, all you have to do is:

echo 'some SELECT query ...' | mysql mydb >data.tsv

but for more complex things, you'll often need to use scripting like with Perl to do some munging first. When you're ready to output the data with:


the data is in a Perl data structure. How to quickly output CSV? I've looked at DBI::Format and DBIx::CSVDumper and they are cumbersome especially if you want something brief for one-liners: you need to instantiate a separate object with some parameters, then call an additional one or more methods. You might as well just do this with the standard Text::CSV (or Text::CSV_XS):

use Text::CSV;
my $csv = Text::CSV->new;

$csv->print(\*STDOUT, $sth->{NAME});
while (my $row = $sth->fetchrow_arrayref) {
     $csv->print(\*STDOUT, $row);

For TSV it's even shorter (note: fail spectacularly when data contains Tab character, but that is relatively rare):

say join("\t", @{$sth->{NAME}});
while (my $row = $sth->fetchrow_arrayref) {
     say join("\t", @$row);

My modules DBIx::CSV and DBIx::TSV attempt to offer something even briefer (for CSV, at least). Instead of fetchrow_arrayref or fetchall_arrayref (or selectrow_arrayref, selectall_arrayref), you use fetchrow_csv and friends:

print $sth->fetchall_csv;

This automatically prints the header too, and if you don't want header:

print $sth->fetchall_csv_noheader;

For even more text table formats, like ASCII table, Org table, Markdown table, and so on, I've also written DBIx::TextTableAny.

I've also just written DBIx::Conn::MySQL after being tired from writing the gazillionth one-liner to connect to MySQL database. Now, instead of:

% perl -MDBI -E'my $dbh = DBI->connect("dbi:mysql:database=mydb", "username", "password"); $sth = $dbh->prepare(...)'

you can now write this:

% perl -MDBI::Conn::MySQL=mydb -E'$sth = $dbh->prepare(...)'

List of new CPAN distributions – Jun 2018

dist author version abstract
AWS-XRay FUJIWARA 0.01 AWS X-Ray tracing library
Acme-CPANModules-Import-CPANRatings-User-perlancar PERLANCAR 0.001 Modules mentioned by CPANRatings user perlancar
Acme-CPANModules-Import-CPANRatings-User-stevenharyanto PERLANCAR 0.001 Modules mentioned by CPANRatings user stevenharyanto
Alt-Devel-CallParser-ButWorking TOBYINK 0.002 Devel::CallParser patched to fix RT#110623
Apache2_4-AuthCookieMultiDBI DETAILS 0.01 An AuthCookie module backed by a DBI database for apache 2.4.
App-ConMenu MMUELLER 1.00 Very simple Menu For Console commands Platform Agnostic
App-Cpanx JACOBG 0.01 A CPAN downloader script
App-CreateAcmeCPANModulesImportCPANRatingsModules PERLANCAR 0.001 Create Acme::CPANModules::Import::CPANRatings::User::* modules
App-ExifUtils PERLANCAR 0.001 Utilities related to EXIF
App-Greple-msdoc UTASHIRO 0.01 Greple module for access MS office documents
App-JC-Client BYTERAZOR 0.001 command line client for JIRA
App-ListPerlReleases PERLANCAR 0.001 List Perl releases
App-MarkFiles BRENNEN v0.0.1 some utility functions for marking and operating on files
App-Prove-Plugin-Elasticsearch TEODESIAN 0.001 Prove Plugin to upload test results to elastic search as they are executed
App-ReportPrereqs SKIRMESS 0.001 Report prerequisite versions
App-Unding BORISD 0.001 dark magic encrypted wallet
App-boxmuller TULAMILI 0.23 Provides the command which produces Gaussian distributed random numbers, as well as log-normal distributed numbers.
App-buffer TULAMILI 0.01 Reads the whole input and then writes into STDOUT. Useful to copy/paste operation.
App-cryp-arbit PERLANCAR 0.001 Cryptocurrency arbitrage utility
App-linenum PERLANCAR 0.001 Add line number to lines
App-np05bctl TTKCIAR 1.03 Command line utility for controlling NP-05B networked power switch
App-optex-msdoc UTASHIRO 0.01 module to replace MS document by its text contents
App-readbuffer TULAMILI 0.02 Reads the whole input and then writes into STDOUT. Useful to copy/paste operation.
App-saikoro TULAMILI 0.22 A random number (matrix) generator of uniform distributions. Saikoro is a Japanese dice.
App-showreverse KLPTWO 0.0.2 given an ip block in cidr notation, show all reverse IP lookups
App_JC BYTERAZOR 0.001 command line client for JIRA
App_JC_Client BYTERAZOR 0.001 command line client for JIRA
App_JIRA_Client BYTERAZOR 0.001 command line client for JIRA
Bencher-Scenarios-MemoryCacheModules PERLANCAR 0.001 Scenarios to benchmark memory cache modules
Bot-BasicBot-Pluggable-Module-DateTimeCalc GENE 0.01 Calculate date-time operations
CLI-Coin-Toss TULAMILI 0.32 Several random number generators by CLI (Command Line Interface) are provided.
CLI-Contingency-Table TULAMILI 0.51 Command line programs for making a contingency tables in 1-way, 2-way.
CLI-KeyValue-Hack TULAMILI 0.07 Provides CLI commands for key-value text data files in TSV.
CLI-LaTeX-Table TULAMILI 0.51 The great new CLI::LaTeX::Table!
CLI-Numbers-Hack TULAMILI 0.21 commands for handling a bunch of numbers for `finding denominator', `N-th min/max', `cumulative sum' and so on.
CLI-Table-Key-Finder TULAMILI 0.31 These CLIs help you to find key column(s) of a table, as fast as possible, hopefully.
CLI-Table-Util TULAMILI 0.05 If you are given table text file, what would you do? This provides CLI commands "colsummary", "colsplit", "colchop" and so on.
CLI-TextLines-Hack TULAMILI 0.06 The great new CLI::TextLines::Hack!
CLI-TextLines-Utils TULAMILI 0.08 The great new CLI::TextLines::Utils!
CPAN-Cpanorg-Auxiliary JKEENAN 0.01 Methods used in infrastructure
CPP-Boost-Mini MRAQ v1.67.0 C++ Boost library (no perl interface).
Catmandu-FileStore HOCHSTEN 1.10 Namespace for packages that can make files persistent
Class-XSConstructor TOBYINK 0.001 a super-fast (but limited) constructor in XS
Code-TidyAll-Plugin-TSLint SHLOMIF 0.001 Use tslint with tidyall
Commandable PEVANS 0.01 utilities for commandline-based programs
Const-Dual BAMBR 0.01 numeric constants that know their names
DBD-MariaDB PALI 0.90_01 MariaDB and MySQL driver for the Perl5 Database Interface (DBI)
DBIx-TextTableAny PERLANCAR 0.001 Generate text table from SQL query result using Text::Table::Any
Dancer2-Plugin-FontSubset YANICK 0.0.1 Generate font subsets on-the-fly
Dancer2-Plugin-Showterm YANICK 0.0.1 Replay terminal typescript captures
Dancer2-Session-DatabasePlugin AKALINUX 1.0001 Dancer2 Session implementation for databases
Dancer2-Template-Mustache YANICK 0.0.1 Wrapper for the Mustache template system
Data-Edit-Conversion PRBRENAN 20180610 Perform a restartable series of steps in parallel
Data-Format ROZCOVO 0.2 Perl module to format data
Data-Format-Validate ROZCOVO 0.1 Perl module to validate data
Data-MuForm-Model-DBIC GSHANK 0.01 MuForm class with DBIC model already applied
Data-Random-NL WATERKIP 1.0 Tools for generating random Dutch numbers
Data-Sah-Coerce-perl-str-str_to_cryptoexchange_currency_pair PERLANCAR 0.001 Coerce string into cryptoexchange currency pair, e.g. LTC/USD
Data-Sah-Coerce-perl-str-str_to_currency_pair PERLANCAR 0.001 Coerce string into currency pair, e.g. USD/IDR
Data-Sah-Coerce-perl-str-str_to_fiat_or_cryptocurrency_code PERLANCAR 0.001 Coerce string containing fiat/cryptocurrency code/name/safename to uppercase code
Data-Sah-CoerceBundle-Num-str_num_en PERLANCAR 0.001 Use Parse::Number::EN to parse number
Data-Sah-CoerceBundle-Num-str_num_id PERLANCAR 0.001 Use Parse::Number::ID to parse number
Data-Validate-DNS-SSHFP MSCHOUT 0.01 Validate DNS SSH Fingerprint (SSHFP) Record Values
Data-Validate-DNS-TLSA MSCHOUT 0.01 Validate DNS Transport Layer Security Association (TLSA) Record Values
DateTime-Format-ISO8601-Format PERLANCAR 0.001 Format DateTime as ISO8601 date/time string
Devel-KYTProf-Logger-XRay FUJIWARA 0.01 Logger for AWS::XRay
Device-GBA ATHREEF 0.001 Perl Interface to the Gameboy Advance
Device-Power-Synaccess-NP05B TTKCIAR 1.02 Manage and monitor the Synaccess NP-05B networked power strip
Dist-Zilla-Plugin-ContributorCovenant KYZN 0.01 Add Contributor Covenant as Code of Conduct
Dist-Zilla-Plugin-XSVersion JEFFOBER 0.01 a thing
Dist-Zilla-PluginBundle-Author-JMASLAK JMASLAK 0.001 JMASLAK's Plugin Bundle
Dist-Zilla-PluginBundle-Author-MINTLAB WATERKIP 0.01 An plugin bundle for all distributions by MINTLAB
Dist-Zilla-Role-CheckPackageDeclared PERLANCAR 0.001 Role to check if a package is provided by your distribution
Dita-Project PRBRENAN 20180530 Dita conversion utilities.
Elasticsearch-Model AMIRI 0.0.1 Does one thing only: helps to deploy a Moose model and accompanying document classes to Elasticsearch.
Electronics-SigGen-FY3200 PEVANS 0 control a FeelTech FY32xx signal generator
Email-SendGrid-V3 GSG 0.90 Class for building a message to be sent through the SendGrid v3 Web API
EventStore-Tiny MEMOWE 0.2 A minimal event sourcing framework.
FFI-TinyCC-Inline PLICEASE 0.27_01 Embed Tiny C code in your Perl program
File-Update SHLOMIF 0.0.1 update/modify/mutate a file only on change in contents.
Finance-AMEX-Transaction THINC 0.001 Parse AMEX transaction files: EPRAW, EPPRC, EPTRN, CBNOT, GRRCN
Finance-Currency-Convert-BCA PERLANCAR 0.152 Convert currency using BCA (Bank Central Asia)
Format ROZCOVO 0.2
HiD-BibtexPage BYTERAZOR 0.001 HiD Bibtex publication list page generator
HiD-Generator-GitRepositories BYTERAZOR 0.1 HiD GitRepository listing generator
Log-ger-UseDataDumper PERLANCAR 0.001 Use Data::Dumper (with nicer defaults) to dump data structures
Log-ger-UseJSON PERLANCAR 0.001 Use JSON::MaybeXS to dump data structures (as JSON)
Module-Extract-DeclaredVersion BDFOY 1.021 Extract the version of Perl a module declares
MojoX-ConfigAppStart GRYPHON 1.01 Start a Mojolicious application with Config::App
Mojolicious-Command-generate-DBIxCustomModel WFSO v1.0.1 generate DBIx::Custom model directory structure
Mojolicious-Plugin-Prometheus-Shared-FastMmap TYLDUM 0.1_01 Mojolicious Plugin
MooX-XSConstructor TOBYINK 0.001 glue between Moo and Class::XSConstructor
Net-Mattermost-Bot MIKEJONES 0.01 A base class for Mattermost bots.
Net-RDAP GBROWN 0.1 an interface to the Registration Data Access Protocol (RDAP).
Net-TL1UDP PCARTER 1.02 Transaction Language 1 (TL-1) UDP Interface
OpenGL-Sandbox NERDVANA 0.01_1 Easy access to a variety of OpenGL prototyping tools
OpenGL-Sandbox-V1 NERDVANA 0.01_1 Portions of OpenGL::Sandbox which depend on OpenGL 1.x API
OpenGL-Sandbox-V1-FTGLFont NERDVANA 0.01_1 Wrapper around FTGL Font library which renders TrueType in OpenGL 1.x
OpenOffice-OODoc-HeadingStyles VANHOESEL 0.01 utilities for manipulating OpenOffice::OODoc objects
OpenOffice-OODoc-InsertDocument VANHOESEL 0.01 insert, merge or append OpenOffice::OODoc objects
PAUSE-Permissions-MetaCPAN SKAJI 0.001 get module permissions from MetaCPAN API
PMVersions-Util PERLANCAR 0.001 Utilities related to pmversions.ini
Parse-Date-Month-ID PERLANCAR 0.001 Parse month name from Indonesian text
Password-Policy-Rule-Pwned HOUSTON 0.00 Compare text to known pwned passwords list
Perinci-Sub-XCompletionBundle-App-cryp PERLANCAR 0.001 Completion routines for the 'cryp' app family
Perl-Critic-Policy-Variables-RequireHungarianNotation MZIESCHA v0.0.1 prohibit hungarian notation
Perl-Critic-TooMuchCode GUGOD 0.01 perlcritic add-ons that generally check for un-needed stuffs.
Plack-Middleware-LogStderr-StderrToLogger AMALEK 0.001 Everything printed to STDERR sent to psgix.logger or other logger
Plack-Middleware-XRay FUJIWARA 0.01 Plack middleware for AWS X-Ray tracing
Python-Version SLOYD 0.0000_01 Python PEP440 compatible version string parser in Perl
Python-Versions SLOYD 0.0000_01 Python PEP440 compatible version string parser in Perl
RPi-GPIOExpander-MCP23017 STEVEB 0.01 Interface to the MCP23017 GPIO Expander Integrated Circuit over I2C
SMS-Send-CZ-Bulkgate RADIUSCZ 1.000 SMS::Send driver for Bulkgate – Czech Republic
Search-Natural ROZCOVO 0.1
Serge-Sync-Plugin-TranslationService-phraseapp DRAGOSV 0.900.0 Serge PhraseApp translation server ( synchronization plugin
Task-FreecellSolver-Testing-MultiConfig SHLOMIF 0.0.1 install the CPAN dependencies of the Freecell Solver multi-config test suite.
Template-Plugin-Filter-IDN MSCHOUT 0.01 Template Toolkit plugin for encoding and decoding International Domain Names.
Test-RequiredMinimumDependencyVersion SKIRMESS 0.001 Require a minimum version for your dependencies
Test-Spelling-Comment SKIRMESS 0.001 TODO
Test2-Tools-HTTP PLICEASE 0.07 Test HTTP / PSGI
Test2-Tools-HTTP-UA-Mojo PLICEASE 0.02 Mojo user agent wrapper for Test2::Tools::HTTP
TestML1 INGY 0.55 A Generic Software Testing Meta Language
Text-Editor-Perl PRBRENAN 20180616 Perl source code head-less editor written in Perl.
Text-NGrammer NIDS 0.01 Pure Perl extraction of n-grams and skip-grams
Text-Shingle NIDS 0.01 Pure Perl implementation of shingles for pieces of text
Text-Table-LTSV PERLANCAR 0.001 Generate LTSV
Text-Table-TSV PERLANCAR 0.001 Generate TSV
Tie-Reduce TOBYINK 0.001 a scalar that reduces its old and new values to a single value
WWW-YNAB DOY 0.01 Wrapper for the YNAB API
WebService-HIBP DDICK 0.01 An interface to the Have I Been Pwned webservice at
device-power-synaccess-np05b TTKCIAR 1.00 Manage and monitor the Synaccess NP-05B networked power strip
voiceIt-voiceIt2 HASSANIS 0

List of new CPAN distributions – May 2018

dist author version abstract
Acme-CPANModules-BloomFilters PERLANCAR 0.001 Bloom filter modules on CPAN
Acme-CPANRatings PERLANCAR 0.001 A dummy module just so we can rate CPAN Ratings (the site itself)
Acme-PERLANCAR-Test-Versioning PERLANCAR 0.001 Test versioning
Acme-UNIVERSAL-can-t TOBYINK 0.001 the opposite of UNIVERSAL::can
Action-CircuitBreaker HANGY 0.1 Module to try to perform an action, with an option to suspend execution after a number of failures.
Alien-TidyHTML5 RRWO v0.1.0 Download and install HTML Tidy
Alien-libdeflate KIWIROY 0.01 Fetch/build/stash the libdeflate headers and libs for libdeflate
Apache2-Camelcadedb MBARBON 0.01 mod_perl2 integration for Devel::Camelcadedb
App-Environ-Que KAKTUS 0.1 Perl library to enqueue tasks in Ruby Que
App-GnuplotUtils PERLANCAR 0.001 Utilities related to plotting data using gnuplot
App-Prove-Plugin-TermTable PLICEASE 0.02 Set the size of the console for Term::Table
App-Sandy TMILLER 0.17 A straightforward and complete next-generation sequencing read simulator
App-fiatx PERLANCAR 0.001 Convert two currencies using current rate
App-ypath BDFOY 0.012 Extract information from YAML
Authen-Passphrase-Argon2 LNATION 0.01 Store and check password using Argon2
Backup-Hanoi BORISD 0.001 select backup according to algo
Bencher-Scenarios-ProgressAny PERLANCAR 0.001 Scenarios to benchmark Progress::Any
Binance-API TASKULA 1.00 Perl implementation for Binance API
Bing-ContentAPI BGERRARD 1.00 Perl interface to the Bing Ads Content API
Business-OnlinePayment-Mock OAXLIN 0.001 A backend for mocking fake results in the Business::OnlinePayment environment
CLI-Files-Utils TULAMILI 0.103 File bunch utilies 'dirhier' and 'madeafter' to supplement the functionality of the "ls" and "find" commands.
CLI-LatexTable TULAMILI 0.116 Copy-paste almost any kind of table such as Excel
CLI-latextable TULAMILI 0.114 The great new CLI::latextable!
CXC-Optics-Prescription-LVS DJERIUS 0.01 a really awesome library
Cache-Method-Cache-FastMmap RRWO v0.1.0 Cache method results using Cache::FastMmap
Catmandu-HTML HOCHSTEN 0.01 Modules for handling HTML data within the Catmandu framework
Class-Method-Cache-FastMmap RRWO v0.1.1 Cache method results using Cache::FastMmap
Code-TidyAll-Plugin-Flake8 SHLOMIF 0.0.1 run flake8 using Code::TidyAll
DBIx-Class-Helper-ResultSet-WindowFunctions RRWO v0.1.0 Add support for window functions to DBIx::Class
DBIx-Class-Relationship-Abbreviate ALBATROSS 0.001 allows you to abbreviate result class names in your relationships
Data-MuFormX-Registry JJNAPIORK 0.001 Manage a hierachry of Data::MuForm classes
Data-Sah-Coerce-perl-str-str_to_cryptocurrency_code PERLANCAR 0.001 Coerce string containing cryptocurrency code/name/safename to code
Data-Sah-Coerce-perl-str-str_to_cryptocurrency_safename PERLANCAR 0.001 Coerce string containing cryptocurrency code/name/safename to safename
Data-Sah-Coerce-perl-str-str_to_cryptoexchange_safename PERLANCAR 0.001 Coerce string containing cryptoexchange code/name/safename to safename
Data-Sah-CoerceBundle-App-cryp PERLANCAR 0.001 Coercion rules for the 'cryp' app family
Date-Indiction RPAVLOV 0.01 Compute a year's indiction as used in old Russian chronicles
Date-Vruceleto RPAVLOV 0.01 Compute year's vruceleto and solar cycle as used in old Russian calendar
Digest-HighwayHash MGV 0.000_001 fast strong hash function
Docker-Registry JLMARTIN 0.02 A client for talking to Docker Registries
Dot RSLOVERS v1.0.0 The beginning of a Perl universe
Email-Stuffer-TestLinks BINARY 0.01 validates links in HTML emails sent by Email::Stuffer>send_or_die()
Eval-Reversible GSG 0.90 Evals with undo stacks
File-ByLine JMASLAK 1.000 Line-by-line file access loops
Finance-Currency-FiatX PERLANCAR 0.001 Convert fiat currency using current rate
Float-Truncate MCCHEUNG 0.01 Ttruncate Float decimal length by special length
Games-Axmud ASLEWIS v1.1.0 Axmud, a modern Multi-User Dungeon (MUD) client written in Perl5 / GTK2
HTML5-DOM ZHUMARIN 1.00 Super fast html5 DOM library with css selectors (based on Modest/MyHTML)
IO-K8s JLMARTIN 0.01 Objects representing things found in the Kubernetes API
ISO-639_1 LDIDRY 0.01 ISO 639-1 Language informations
Image-Find-Loops PRBRENAN 20180502 Find loops in an image.
JTM-Boilerplate JMASLAK 1.020 Default Boilerplate for Joelle Maslak's Code
JavaScript-V8-CommonJS CAFEGRATZ 0.01 Modules/1.0 for JavaScript::V8
Koha-Contrib-ARK FREDERICD 1.0.0 ARK Management
Kubectl-CLIWrapper JLMARTIN 0.01 Module to use the Kubernetes API via the kubectl CLI
Lingua-RU-Declension MALLEN 0.001 Decline Russian pronouns, adjectives and nouns
Log-Log4perl-Layout-ColoredPatternLayout TOMGRACEY 0.01 multicolor log messages
Log-Log4perl-Shortcuts STEVIED 0.006 shortcut functions to make log4perl even easier
Log-ger-Level-Like-Syslog PERLANCAR 0.001 Define logging levels like those described in RFC 3164 (syslog protocol)
Math-Business-BlackscholesMerton BINARY 1.23 Algorithm of Math::Business::BlackScholesMerton for binary and non-binary options
Model-Envoy HOWARS 0.1 A Moose Role that can be used to build a model layer that keeps business logic separate from your storage layer.
Mojo-SAML JBERGER 0.01 A SAML2 toolkit using the Mojo toolkit
Mojolicious-Plugin-Cron DMANTO 0.014 a Cron-like helper for Mojolicious and Mojolicious::Lite projects
Mojolicious-Plugin-GistGithubProxy RENEEB 0.01 Mojolicious::Plugin::GistGithubProxy – a small proxy that can be useful when you embed gists in your website
Mojolicious-Plugin-GoogleFontProxy RENEEB 0.01 a small proxy that can be useful when you embed gists in your website
Mojolicious-Plugin-NoIndex RENEEB 0.01 add meta tag to HTML output to define a policy for robots
Mojolicious-Plugin-NoReferrer RENEEB 0.01 add meta tag to HTML output to define a referrer policy
Mojolicious-Plugin-PromiseActions MRAMBERG 0.01 Automatic async and error handling for Promises
Mojolicious-Plugin-SecurityHeader RENEEB 0.01 Mojolicious Plugin
MooX-AttributeFilter VRURG 0.001000 Implements 'filter' option for Moo-class attributes
MooseX-AttributeFilter KITTEN 0.01 MooX::AttributeFilter with cute antlers OwO
MooseX-Storage-MaybeDeferred UFOBAT v0.0.1 A role for the less indecesive programmers
MouseX-Types-Enum IKENOX 1.00 Object-oriented, Java-like enum type declaration based on Mouse
PMLTQ-Command-printtrees MATY 0.0.1 generate svg trees for given treebank
Parse-SSA DISI 1.0 SSA/ASS Parser
Path-Class-Tiny BAREFOOT 0.01 a Path::Tiny wrapper for Path::Class compatibility
Perinci-CmdLine-sero PERLANCAR 0.001 Perinci::CmdLine::Lite subclass for App::sero
Plack-Middleware-Auth-FCGI GUIMARD 0.01 authentication middleware that query remote FastCGI server
Plack-Middleware-Camelcadedb MBARBON 0.02 interactive debugging for Plack applications
Qgoda GUIDO v0.9.2 The Qgoda Static Site Generator
RPi-RTC-DS3231 STEVEB 0.01 Interface to the DS3231 Real-Time Clock IC over I2C
Sah-Schemas-App-cryp PERLANCAR 0.001 Various Sah schemas for the 'cryp' app family
Serge-Sync-Plugin-TranslationService-mojito DRAGOSV 0.9 Serge Mojito translation server ( synchronization plugin
Serge-Sync-Plugin-TranslationService-zanata DRAGOSV 0.9 Serge Zanata translation server ( synchronization plugin
Software-LicenseMoreUtils DDUMONT 0.001 More utilities and a summary for Software::License
Stor JASEI 0.10.2 Save/retrieve a file to/from primary storage
TaskPipe TOMGRACEY 0.01 A task management framework for building web scrapers and crawlers
Template-Plugin-Path-Tiny RRWO v0.1.0 use Path::Tiny objects from within templates
Test-Conditions MMCCLENN 0.8 test multiple conditions across a large data structure or list in a simple and compact way
Test-Spec-Acceptance MEIS 0.01 Aliases for acceptance-like testing using Test::Spec
Test2-Tools-JSON-Pointer PLICEASE 0.01 Compare parts of JSON string to data structure using JSON pointers VERSION
Test2-Tools-SkipUntil DFARRELL 0.01 skip tests until a date is reached
Text-PerlPP CXW v0.3.1 Perl preprocessor: process Perl code within any text file
Types-Bool FERREIRA 0.1.0 Booleans as objects for Perl
Unix-OpenBSD-Random JMATES 0.01 interface to arc4random(3) on OpenBSD
WebService-AcousticBrainz GENE 0.01 Access to the AcousticBrainz API
WebService-Pokemon KIANMENG 0.03 A module to access the Pokémon data through RESTful API from
WebService-RESTCountries KIANMENG 0.01 A Perl module to interface with the REST Countries ( webservice.
WebService-Swapi KIANMENG 0.1.6 A Perl module to interface with the Star Wars API ( webservice.
Webservice-Pokemon KIANMENG 0.001 A module to access the Pokémon data through RESTful API from
binance-perl-api TASKULA 1.00 Perl implementation for Binance API
cppAdaptive MFJONKER 0.01 cppAdaptive2 XS
cppAdaptive2 MFJONKER 0.01 cppAdaptive2 XS
ssa DISI 0


This post is a "thinking out loud" post, about an unimplemented feature I'm planning.

The problem

The most often complaint I get when I release a Perinci::CmdLine -based Perl application is the huge dependencies of Perinci::CmdLine (currently Perinci::CmdLine::Lite only has 24 direct non-core dependencies, but recursively it has 98 non-core dependencies in 92 unique distributions). By the way, you can produce these numbers "very easily" using lcpan and td:

% lcpan mods -lx Perinci::CmdLine::Lite
| module                 | version | abstract                                              | dist                 | author    | rel_mtime            | is_core |
| Perinci::CmdLine::Lite | 1.812   | A Rinci/Riap-based command-line application framework | Perinci-CmdLine-Lite | PERLANCAR | 2018-05-01T08:42:19Z | 0       |
% lcpan deps Perinci::CmdLine::Lite --noinclude-core | wc -l
% lcpan deps Perinci::CmdLine::Lite --noinclude-core -R | wc -l
% lcpan deps Perinci::CmdLine::Lite --noinclude-core -R --flatten | \
    td select module | lcpan mod2dist | td select value | sort | uniq | wc -l

Sort of ironic because years ago I used to mock Moose 's high number of dependencies and avoid it like the plague, and then ending up creating the same situation with my own distribution. Correction: a much worse situation than the current Moose:

% lcpan mods -lx Moose
| module | version | abstract                              | dist  | author | rel_mtime            | is_core |
| Moose  | 2.2010  | A postmodern object system for Perl 5 | Moose | ETHER  | 2018-02-16T22:01:37Z | 0       |
% lcpan deps Moose --noinclude-core | wc -l
% lcpan deps Moose --noinclude-core -R | wc -l
% lcpan deps Moose --noinclude-core -R --flatten | \
    td select module | lcpan mod2dist | td select value | sort | uniq | wc -l

I arrange the modules into many distributions because I am trying to keep things modular. So when I only need a specific subset of functionality, I don't have to pull the whole thing (and along with it its large list of dependencies). Maybe I went overboard? Maybe. Nevertheless.

The high number of dependencies presents an inconvenience and annoyance when users want to install my application, especially since CPAN clients like cpan and cpanm default to testing distributions before installing them.

One solution: Perinci::CmdLine::Inline

One solution I created for this problem is Perinci::CmdLine::Inline (PC:Inline) which basically "pre-assembles" the application with sort of a "mini", "embedded" Perinci::CmdLine during distribution build time, so that when a user installs the application she doesn't need to get Perinci::CmdLine anymore. This also has another benefit of faster application startup time due to the "pre-assembling" thingy. But this pre-assembling has some downsides too. Whenever I create a new version of PC:Inline, I will have to rebuild all the applications again. PC:Inline also does not (read: will not, because of the lack of Riap layer) have all the features of a proper Perinci::CmdLine. I use PC:Inline only for simpler applications that need to be very light (has no non-core dependencies, or starts fast) like hr, wordlist, zodiac-of.

Another solution: fatpacking, datapacking

Another solution is fatpacking or datapacking. I've tried this in the past with the pause script, and it works rather well. Except that when it comes to packaging the CPAN distribution as a Debian package, the Debian policy forbids "convenience copies" of code.

Past solution: lumping

Yet another solution which I've tried in the past is what I call "lumping": in a lump distribution I include modules from other distributions (the dependencies) but leave them unindexed. For example, I created a distribution called Perinci-CmdLine-Any-Lumped that contains modules from Perinci-CmdLine-Any (Perinci::CmdLine::Any) as well as all its recursive pure-perl non-core dependencies, like Data::Sah, Data::Sah::Coerce, and so on. After installing just one lump distribution, a user will have perhaps 100+ extra (but "hidden") modules from various distributions on her system. The perl interpreter will see those extra modules fine. The modules are "hidden" only in the sense that the original distributions of those modules are not listed as installed (because they aren't).

I abandoned this solution because it feels really dirty. The extra modules lumped into the lump distribution are actually "orphan" modules because the lump distribution does not publicly confess that it includes the modules. And I'm pretty sure lump distributions are not good candidates for Debian perl packages, but it's okay.

Another proposed solution: clumping

This post will describe another solution which I'm thinking of (with an equally stupid name so if the idea ended up being really stupid, the name would have already fitted): clumping.

A clump distribution is named "Clump-SOMETHING" and will contain modules from several other distributions (called "source distributions"). The purpose of a clump distribution is to package several other distributions as a single distribution for the purpose of reducing the number of dependencies for the end-user.

For example, a clump distribution called "Clump-Data-Sah" will contain modules from Data-Sah as well as Data-Sah-Coerce, Data-Sah-Format, and so on.


To help build this distribution, a Dist::Zilla plugin will be created and used: DZP:Clump. To use this plugin, we list the source distributions that we want to include:


The plugin will gather module files from the source distributions as well as merge the dependencies from the source distributions into the dependencies for the clump distribution.

Dependencies that end up being in the clump distribution can be "netted out". For example, Data-Sah depends on Data::Sah::Coerce which is in the Data-Sah-Coerce distribution which is also another source distribution in the clump, so this dependency now does not need to be specified in Clump-Data-Sah.

Clumping and module versions

The version number of a module inside the clump distribution will be the original version joined by ".0". So for example, if the original version number of module M1 is "0.001" then M1's version number in the clump distribution is "0.001.0". If M2's original version is "0.1.2" then M2's version in the clump is "" and so on. As far as perl concerns the two version numbers in each case are the same:

version->parse("0.001") == version->parse("0.001.0")
version->parse("0.1.2") == version->parse("")

The module version will have to be checked to satisfy the above relationship and if it does not, a new release will need to be made in the source distribution first to remedy this. For example, M3's original version is 0.02. If we append ".0" to it to become "0.02.0" then the new version will be less than the original version:

version->parse("0.02") > version->parse("0.02.0")
# because 0.02   is 0.020
# and     0.02.0 is 0.002.0

To include M3 in the clump, we will need to make a new release of M3 in the original source distribution first, say of version 0.030. Then M3 version 0.030 can now be included in the clump, as version 0.030.0.

Yes, this means the clump distribution will contain different version numbers. Which is usually not recommended for a "normal" Perl distribution but is appropriate here.

Clumping and the PAUSE indexer

When the clump distribution is released, PAUSE will index the clump distribution and now modules included in the clump will be indexed as belonging to the clump distribution instead of their original distribution.

When user installs one of these modules, she will be getting it (and automatically a lot of other modules too) from the clump distribution, thus reducing the number of distributions she needs to install to satisfy all the dependencies of an application.

Developing modules that are included in clumps

Ideally, modules are still developed in its original source distribution, e.g. Data::Sah::Coerce in Data-Sah-Coerce. When I want to release a new version of Data::Sah::Coerce, I can just release a new version of Data-Sah-Coerce. Now the module will be indexed by PAUSE as belonging to the new Data-Sah-Coerce.

As more and more modules are being "unclumped" as new releases of the source distributions come along, the level of inconvenience to end-users will once again increase. To remedy this, from time to time I can release a new clump distribution again that contains newer "snapshots" of modules.

This is the reason why version of modules in the clump is being kept the same (albeit with extra ".0"): so version bump in the original source distribution will be able to "eclipse" the clumped version on CPAN later. Even when the original distribution bumps using an extra subversion, e.g. 0.001 to 0.001.1 it will still eclipse the clumped version. The next clumped version will be

Clumping vs lumping

This clump solution is cleaner than the lumping solution because in the former case, no modules are "hidden". Basically clumping is lumping, but the included modules are "acknowledged" and properly indexed. The $orig_version . ".0" thing is really the only novel element here. Because the modules are now not hidden, their dependencies must now also be handled.

As to Debian packaging, the source distributions and not the clump distribution are the ones that will be packaged so there should not be an issue with "convenience copies" or "bundling".

Generating random passwords according to patterns with genpw

There are several modules on CPAN for generating random passwords, but last time I checked, none of them are flexible enough. Different websites or applications have different requirements (sometimes ridiculous ones) for passwords. True, most modules allow setting minimum/maximum length, and some of them allow customizing the number of special characters, or number of digits, but what I want is some sort of template/pattern that the generator would follow. So I created genpw.

Basic functionality

genpw is your usual random password generator CLI. When run without any arguments, it returns a single random password (8-20 characters long, comprising letters/digits):

% genpw

To return several passwords:

% genpw 5

To set exact length or minimum/maximum of length:

% genpw -l 4
% genpw --min-len 12 --max-len 14


In addition to the above basic customization, genpw allows you to specify a pattern (or several patterns to pick randomly, for that matter). A pattern is a string that is similar to a printf pattern where conversion sequences like %d will be replaced with actual random characters. Here are the available conversions:

%l   Random Latin letter (A-Z, a-z)
%d   Random digit (0-9)
%h   Random hexdigit (0-9a-f)
%a   Random letter/digit (Alphanum) (A-Z, a-z, 0-9; combination of %l and %d)
%s   Random ASCII symbol, e.g. "-" (dash), "_" (underscore), etc.
%x   Random letter/digit/ASCII symbol (combination of %a and %s)
%m   Base64 character (A-Z, a-z, 0-9, +, /)
%b   Base58 character (A-Z, a-z, 0-9 minus IOl0)
%B   Base56 character (A-Z, a-z, 0-9 minus IOol01)
%%   A literal percent sign
%w   Random word

You can specify %NC (where N is a positive integer, and C is the conversion) to mean a sequence of random characters (or a random word) with the exact length of N. Or %N$MC (where N and M are positive integers) to mean random characters (or word) with length between N and M.

Unsupported conversion will be unchanged, like in printf.

Some examples:

Generate random digits between 10 and 12 characters long:

% genpw -p '%10$12d'

Generate a random UUID:

% genpw -p '%8h-%4h-%4h-%4h-%12h'

Like the above, but in uppercase:

% genpw -p '%8h-%4h-%4h-%4h-%12h' -U


The %w conversion in pattern mean to replace with a random word. Words are fetched from STDIN (and will be shuffled before use). For example, the command below will generate password in the form of a random word + 4 random digits:

% genpw -p '%w%4d' < /usr/share/dict/words

Instead of from STDIN, you can also fetch words from the various WordList::* modules available on CPAN (and installed locally). A separate CLI provides this functionality: genpw-wordlist, which is basically just a wrapper to extract the words from the wordlist module(s) then feed it to App::genpw. By default, if you don't specify -w option(s) to select wordlist modules, the CLI will use WordList::EN::Enable:

% genpw-wordlist -p '%w%4d'

Generate 5 passwords comprising 8-character word followed by between four to six random digits:

% genpw-wordlist 5 -p '%8w-%4$6d'

Configuration file

To avoid you from having to type patterns again and again, you can use a configuration file. For example, put this in $HOME/genpw.conf:

patterns = %8h-%4h-%4h-%4h-%12h
case = upper

then you can just say:

% genpw -P uuid


The speed is not great, around 2000 passwords/sec on my laptop, though I believe this should not matter for most use-cases.


genpw is a flexible random password generator where you can specify patterns or templates for the password. I'm still working on a few things, like how to enable secure random source in the most painless way. There are also various CLI variants to generate specific kinds of passwords as straightforward as possible: genpw-base56, genpw-base64, genpw-id, and the aforementioned genpw-wordlist.

List of new CPAN distributions – Apr 2018

dist author version abstract
Acme-CPANModules-PortedFrom-NPM PERLANCAR 0.001 Modules/applications that are ported (or inspired from) NPM libraries
Acme-Pointer CODEHEX 0.01 We can access to data using address as the string
Acme-ZeroWidth VTI 0.01 Zero-width fingerprinting
Aliyun NSNAKE 0.9 阿里云SDK
Apache2-AuthCASpbh PHENSON 0.10 CAS SSO integration for Apache/mod_perl
App-DHCPClientUtils HQJATU 0.01 ISC dhclient lease information
App-ModuleVersionDiffUtils PERLANCAR 0.001 Utilities to diff stuffs from two different versions of a module
App-QRCodeUtils PERLANCAR 0.001 Utilities related to QR Code
App-cryp-exchange PERLANCAR 0.001 Interact with cryptoexchanges
App-cryp-mn PERLANCAR 0.001 Manage your masternodes
App-indodax PERLANCAR 0.023 CLI for
App-rank PERLANCAR 0.001 Rank lines of text files
App-rs RSLOVERS 1.01 package manager of RSLinux
Astro-Units SPAJAI 1.10 Astronomical unit conversion with high precision and large number support
Astro-units SPAJAI 1.00
Business-BR-CNJ-Classes DLIMA 0 Convert brazilian CNJ (Conselho Nacional de Justiça) class numbers and abreviations to full text names/description.
Business-BR-CNJ-Format DLIMA 0 Format brazilian CNJ (Conselho Nacional de Justiça) numbers, from all numbers to correctly ponctuated strings.
Business-ID-NKK PERLANCAR 0.001 Parse Indonesian family card number (nomor kartu keluarga, NKK)
CPAN-Test-Dummy-Perl5-UseUnsafeINC-Fail MIYAGAWA 0.01 Fails when PERL_USE_UNSAFE_INC is set
CPAN-Test-Dummy-Perl5-UseUnsafeINC-One MIYAGAWA 0.01 x_use_unsafe_inc 1
Catalyst-Plugin-ServeFile JJNAPIORK 0.001 A less opinionated, minimal featured way to serve static files.
Class-Slot JEFFOBER 0.01 Simple, efficient, comple-time class declaration
Command-Runner SKAJI 0.001 run external commands and Perl code refs
Crypt-OpenSSL-Guess AKIYM 0.01 Guess OpenSSL include path
DBIx-Result-Convert-JSONSchema MALISHEW 0.01 Convert DBIx result schema to JSON schema
Dancer2-Plugin-Map-Tube MANWAR 0.01 Dancer2 add-on for Map::Tube.
Data-Cuid ZAKAME 0.01 collision-resistant IDs
Data-SExpression-Util MGV 0.000_001 routines for processing linked lists
Data-Sah-CoerceRule-array-str_int_range PERLANCAR 0.001 Coerce array of ints from string in the form of "INT1-INT2"
Data-Sah-CoerceRule-array-str_int_range_and_comma_sep PERLANCAR 0.001 Coerce array of ints from comma-separated ints/int ranges
DateTime-Format-Genealogy NHORNE 0.01 Create a DateTime object from a Genealogy Date
Device-Gyro-IXBLUE TROXEL 1.0 Perl module to interface with a IXBLUE Gyro
Dist-Zilla-MintingProfile-FB11 ALTREUS 0.016 Mints a new FB11 site
Dist-Zilla-Plugin-ReadmeAddDevInfo RENEEB 0.01 Dist::Zilla::Plugin::ReadmeAddDevInfo – add info how to contribute to the project
Dita-Validate PRBRENAN 20180421 In situ validation of XML parse trees representing OASIS DITA documents.
EPFL-Service-Open WILLBELL 1.00 Open the EPFL website (service) associated with the
File-ContentStore BOOK 1.000 A store for file content built with hard links
File-Copy-Recursive-Reduced JKEENAN 0.001 Recursive copying of files and directories within Perl 5 toolchain
Finance-Indodax PERLANCAR 0.008 Trade with using Perl
Game-TextPatterns JMATES 0.01 generate patterns of text
Geo-GDAL-FFI AJOLMA 0.01 A foreign function interface to GDAL
Geo-Privacy DWRIGHT 0.01 Information about privacy/GDPR regulations by state
Graphics-Grid SLOYD 0.0000_01 An incomplete port of the R "grid" library to Perl
Image-Find-Paths PRBRENAN 20180429 Locate the longest line in an image.
JIP-DataPath VLADIMIRZ 0.01 provides a way to access data elements in a deep, complex and nested data structure.
Linux-Perl FELIPE 0.01 Linux system calls with pure Perl
Locale-Country-EU MATTSIZLE 0.001 Module to check if a country is in the European Union ( EU ) using various ISO data formats
Log-Log4perl-Appender-LogGer PERLANCAR 0.001 Log to Log::ger
Map-Tube-Frankfurt RENEEB 0.01 Map::Tube::Frankfurt – interface to the Frankfurt S- and U-Bahn map
Map-Tube-Madrid MANWAR 0.01 Interface to the Madrid Metro Map.
Map-Tube-Nuremberg STELIM 0.01 Map::Tube::Nuremberg – interface to the Nuremberg U-Bahn map
Map-Tube-Server MANWAR 0.01 Dancer2 based server for Map::Tube.
MarkMoreDown KIAVASH 0.000030 markdown parser
Markmoredown KIAVASH 0.000031 markdown parser
Math-Utils-XS GONZUS 0.000002 Some math utility functions implemented in XS
Menlo-Legacy MIYAGAWA 1.9019 Legacy internal and client support for Menlo
Mojo-Feed DOTAN 0.13 Mojo::DOM-based parsing of RSS & Atom feeds
Mojolicious-Command-generate-localplugin RENEEB 0.01 Plugin generator command
Mojolicious-Plugin-CSPHeader LDIDRY 0.01 Mojolicious Plugin to add Content-Security-Policy header to every HTTP response.
Mojolicious-Plugin-ExportExcel WFSO 0.01 The great new Mojolicious::Plugin::ExportExcel!
Mojolicious-Plugin-GzipStatic LDIDRY 0.01 Mojolicious Plugin to to compress the static files before serving them.
Mojolicious-Plugin-NamedHelpers TYLDUM 0.01 Mojolicious Plugin
MooX-Async-Console CHOHAG 0.103 Attach a console to an async module
Music-ScaleNote GENE 0.01 Position of notes in a scale
Net-ISC-DHCPClient JATU 0.01 ISC dhclient lease reader
Oracle-ZFSSA-Client WHINDS 0.01 Oracle ZFS Storage Appliance RESTful API Connector
Outlook-Mail-Count LINUS 0.01 The great new Outlook::Mail::Count!
PDL-Cluster MOOCOW v1.54.001 Perl interface to the C Clustering Library with PDL support
PERLCORE-Predicates-IsEven ECARROLL 0.01 An implementation of JS is-even
PGObject-Util-AsyncPool EINHVERFR 0.01 An Async Connection Pooler for PGObject
PMLTQ-Suggest MATY 0.0.1 Tool for generating simple PMLTQ query based on given nodes
Panda-next SYBER v0.1.0 Replacement for next::method, performing very fast, and super::method (next-by-mro), with Perl and C++ interface.
RT-Extension-InlineAudioVideoAttachments GIBUS 0.01 RT-Extension-InlineAudioVideoAttachments Extension
Report_HTML_DB HIMEWEN 0.01 DBI Model Class
Role-NotSoTiny FERREIRA 0.1.0 Experiment with Role::Tiny / Role::NotSoTiny->make_role()
SMS-Send-Iletimerkezi DUMLU 0.01 SMS::Send driver for
Serge-Smartcat PDORONIN 0.01 Maintain tag
Serge AFAN v1.3.1 Solution for continuous localization
Shorthand-Entropy-UseLocal PERLANCAR 0.001 Use local entropy source
Table-PreAnalysis TULAMILI 0.001 The great new Table::PreAnalysis!
Table-PreCheck TULAMILI 0.001 The great new Table::PreCheck!
Task-Bootstrap-Perl-CPAN SCHWIGON 0.003 Dependencies for App::Bootstrap::Perl – CPAN bundle
Task-Kensho-All DBOOK 0.39 Install all of Task::Kensho
Taskwarrior-Kusarigama-Plugin-Command-Shelf SSORICHE 0.001 Move tasks to and from the shelf.
Test-Pod-Links SKIRMESS 0.001 Test Pod for invalid HTTP/S links
Text-Gitignore VTI 0.01 Match .gitignore patterns
Text-Markmoredown KIAVASH 0.000032 MarkMoreDown parser
Tree-Interval-Fast AVULLO v0.0.1 Perl extension for efficient creation and manipulation of interval trees.
UUID-URandom DAGOLDEN 0.001 UUIDs based on /dev/urandom or the Windows Crypto API
Web-Mention JMAC 0.1 Implementation of the IndieWeb Webmention protocol
Web-Microformats2 JMAC 0.02 Read Microformats2 metadata from HTML or JSON
WebService-GetSongBPM GENE 0.01 Access to the API
WebService-S3-Tiny CVLIBRARY 0.001 Perl module for using S3 or compatible APIs
Webservice-Swapi KIANMENG 0.1.0 A Perl module to interface with the Star Wars API webservice.
WordList-Mod PERLANCAR 0.001 Modified wordlist
WordList-Mod-BinarySearch PERLANCAR 0.001 Provide word_exists() that uses binary search
WordList-Mod-Bloom PERLANCAR 0.001 Provide word_exists() that uses bloom filter
WordList-Number-Prime-First10000 PERLANCAR 0.001 First 10,000 prime numbers
WordList-Test-Dynamic-Number-10000 PERLANCAR 0.001 10 million numbers from "00001" to "10000"
WordList-Test-Dynamic-Number-10Million PERLANCAR 0.001 10 million numbers from "00000001" to "10000000"
WordList-Test-Number-1Million PERLANCAR 0.001 1 million numbers from "0000001" to "1000000"
Yahoo-Weather-Api SPAJAI 1.00 The great new 2018 Yahoo::Weather::Api
astro-unit SPAJAI 1.00 Astronomical unit conversion with high precision and large number support
common-header ZDM v0.1.0 re-exporting the set of standard perl pragmas
continuous-delivery-template JOENIO 0.01 continuous delivery workflow dockerhub, gitlab, and cpan
locale-country-eu MATTSIZLE 0.001 Perl extension for determining if a country is within the EU using various ISO values
slot JEFFOBER 0.01 Simple, efficient, comple-time class declaration