Org stuffs: 001: Moving/removing done todo items

14 October 202014 October 2020 / perlancar / Leave a comment

About this series: A collection of short blog posts related to manipulating Org documents using Perl. This is the first article. See the whole series.

A bit of history. I started using org-mode in 2011. Emacs I had been using much earlier, but as also happened with many others, org-mode really clinched the deal. Slowly but sure, I migrated my notes, and later on, my todo list, into this format. Also in that year I started the Org::Parser project. It’s not perfect, and at one point educated_foo said it was pointless because just like only perl can parse Perl, only emacs’ org-mode can truly parse Org. However, I manage to do some useful tricks with the module. (BTW, I missed the guy.)

Being a heavily procrastinating and utterly lazy person, with lots of ideas and abandoned projects, my todo file grew to thousands of items (last count, 4600+ undone todo items and 1900+ done dones).

I currently organize my todo.org like the following: there are hierarchical categories which I write strictly as level-1 headings, with subcategories written inline in the heading using “cat > subcat > subsubcat” notation. I keep these level-1 headings sorted. Example:

* 00unorganized [0/0]
* finance [0/0]
* health [0/0]
* leisure > vacation [0/0]
* pet [0/0]
* person > kid 1 > education [0/0]
* person > kid 1 > health [0/0]
* person > kid 2 > health [0/0]
* person > kid 2 > health [0/0]
* person > kids > education [0/0]
* person > kids > health [0/0]
* person > mom > health [0/0]
* proj > blog > blog title 1 [0/0]
* proj > blog > blog title 2 [0/0]
* proj > book > working title 1 [0/0]
* proj > book > working title 2 [0/0]
* proj > perl [0/0]
* proj > perl > ansitable [0/0]
* proj > perl > debian [0/0]
* proj > perl > lcpan [0/0]
* proj > perl > perinci [0/0]
* proj > perl > wordlist [0/0]
* reading > books to read [0/0]
* watching > movies to watch [0/0]
* watching > tv series to watch [0/0]

Actual todo items are always written as level-2 headings under one of these level-1 categories. This way, I always know that todo items are at level-2, which makes some things easier to find and manipulate. For example:

* proj > perl > lcpan [0/0]
** IDEA [2020-10-04 Sun] lcpan: subcommand to star/favorite modules/dists/authors
** IDEA [2020-10-04 Sun] lcpan: subcommand to show whether our favorite modules/dists/authors have a release in the past week/month/etc
** WISHLIST [2020-10-04 Sun] lcpan: subcommand to retrieve number of ++ from metacpan
** IDEA [2020-02-24 Mon] lcpan: subcommand to check circular dependency
** TODO [2020-05-10 Sun] lcpan: replace Archive::Tar with Archive::Tar::Wrapper

If you’re wondering about the timestamps, my habit is to put a creation date at the beginning of each headline’s title. I myself feel it’s rather unclean and have tried to put “logs” inside the entries, but it’s nice being able to quickly see how old a todo item is, so I’ve kept the habit alive.

The actual “proj > perl” category itself is quite large (currently 3000+), as I put todo items for various smaller projects there and only create a subcategory for a few notable projects.

Over time, I collect hundreds of done todo items in various categories, including in this large “proj > perl” category. I want to remove the done items, but archive it in other file (done.org), under the same category headlines.

Emacs offers some archiving functions for this, and I’m sure with some elisp code it can be instructed to do the above, but with my very limited eslip-fu I shuddered at how much time I have to spend to produce such code. Thus, Org::Parser to the rescue. The result is move-done-todos. It’s not as concise as I like it to be, because the Org::Parser’s API still stinks and lacking when it comes to building node objects or doing some basic manipulation. But at least it works.

To use it:

% move-done-todos todo.org done.org

By default, the script operates in dry-run (simulation) mode, for safety. If you are sure with the results, you add the --no-dry-run option:

% move-done-todos todo.org done.org --no-dry-run

The nice thing about this script is that it can auto-create the categories in done.org, so moving is only a single command-line invocation process. I do strive to make everything in the world a single command-line invocation process.

If you want to remove the done todo items from your todo file, which you might prefer because the done todo items are already recorded in the revision history of your repository anyway, you can omit the done filename:

% move-done-todos todo.org --no-dry-run

There are things that I want the script to be able to do, but not yet, like updating the counter cookies. That, for now, is currently still noted as an undone item in my todo.org.

CLI programs and scripts I use daily 1: net{on,off}, bt{on,off}, {,un}pause-browsers

11 September 202011 September 2020 / perlancar / Leave a comment

Hey, another mini-article series! Since I use the CLI a lot and also publish a bunch of CLI scripts on CPAN, this series will serve mainly as a list of mentions for my scripts, as well as for other CLI programs and scripts I use daily.

Let's start with browsing and power-saving.

Browsing is one of everybody's main activities nowadays. Unless you browse using text-based browsers like lynx and links, you'll invariably run into the web obesity crisis (a.k.a. web bloat problem) affecting your battery life. Even if you only open a news article, your laptop may still issue thousands of requests to hundreds of domains to the various analytic tracking beacons, repeatedly every second or so for as long as the tab exists. Multiply that with 5, 10, or 50 tabs. (I seem to have a problem, er, work best, when I keep lots of tabs for various purposes open. You might also correctly guess that my desk is messy.) The result, your browser processes are eating between 20% to 50% or more CPU even when you are not focused on any of browser windows. I often find myself just typing on a terminal emulator and seeing that my system load is 0.75 or even greater thatn 1.00, only because I have Firefox open. Linux desktop often not being the most power-efficient environment, this certainly does not help.

I try to actively conserve power whenever I am not close to an available power outlet. So whenever I am coding or writing and not needing the Internet, I issue this:

% netoff; pause-browsers

netoff (and its counterpart neton) will use NetworkMonitor to turn off (or on) networking. pause-browsers will kill -STOP browser processes, which I find, though relatively blunt, is convenient to save power. There doesn't seem an addon or menu setting to let you freeze one or more tabs or globally disable and re-enable JavaScript instantly. The various ways to disable Javascript (including javascript.enabled Firefox config, or uMatrix or uBlock addon) usually require you to refresh the page, and/or only work for a single site or tab. Other ways to freeze tabs will simply clear the tab content and require you to reload, which for me defeat the purpose of having many tabs open.

When you want to browse again, you can issue:

% neton; btoff; unpause-browsers

btoff is for turning Bluetooth back off, because NetworkMonitor always wants to turn my bluetooth on when I turn my networking on, regardless of the previous bluetotoh state. Aside from {pause,unpause}-browsers, I also made scripts for specific browsers: pause-firefox, pause-chrome, pause-vivaldi, pause-opera. You can also only pause/unpause (as well as kill) processes of certain Unix users, if you happen to run several in a single system (which I often do, before using Multi-Account Containers addon).

Multi-Account Containers Firefox addon

I made a separate blog post for this. Basically I use this addon and have lots of containers created. I write scripts to sort, list, or modify/delete them from the CLI.

Suspending

As any Linux laptop user experience, automatic suspending by closing the lid is randomly screwy, some brands more so than the others. I find this invocation more reliable (oh it still randomly fails too from time to time, but much less often):

# sleep 3; pm-suspend

The sleep part is to give me time to lock the screen (e.g. press Ctrl-Alt-L), because otherwise your laptop will wake up unlocked.

Checking the battery

I find:

% acpi

often faster than moving my graphical pointer to the battery widget on the desktop tray. As a bonus, there's acpi -V which will give you lots more information.

The joy of piping tables on the command line

11 June 2020 / perlancar / Leave a comment

Staying home a lot lately has given me the chance, among other things, to organize my personal media files (photos, videos, audios) and update my remote backups on Google Photos and other cloud services.

To work with these files, I've updated some of my CLI scripts like media-info and td, as well as created a few others like delete-all-empty-dirs, show-duplicate-files, and reencode-videos.

I just want to show again (I've blogged about this once before) that it's absolutely wonderful piping tables on the command-line. While piping bytes on Unix remains as one of the simplest yet most powerful and flexible interprocess communications, to increase the convenience, one can put a layer of encoding/decoding on top of this to pass around higher-level entities. PowerShell, for example, goes full-blown objects. Which can be very convenient sometimes. I choose to output and pipe plain data structures in most of my CLI scripts, particularly table data. With this, I can do things like:

Show a table of data for my videos:

% media-info *
+---------------+--------------+------------+---------+----------+-------------------------+--------+----------------+---------------+-----------+--------------+-----------+--------------+--------------------+-------------------+-----------+---------------------+-------------+
| audio_bitrate | audio_format | audio_rate | backend | duration | media                   | rotate | type_from_name | video_bitrate | video_dar | video_format | video_fps | video_height | video_longest_side | video_orientation | video_sar | video_shortest_side | video_width |
+---------------+--------------+------------+---------+----------+-------------------------+--------+----------------+---------------+-----------+--------------+-----------+--------------+--------------------+-------------------+-----------+---------------------+-------------+
| 59392         | AAC          | 44100      | Ffmpeg  | 59.81    | 2019-08-27 14.40.38.mp4 |        | video          | 1245184       |           | H264         | 59.67     | 848          | 848                | portrait          |           | 400                 | 400         |
| 262144        | AAC          | 48000      | Ffmpeg  | 97.56    | 2019-08-27 19.06.06.mp4 | 180    | video          | 964608        |           | H264         | 30.03     | 352          | 640                | landscape         |           | 352                 | 640         |
| 262144        | AAC          | 48000      | Ffmpeg  | 91.16    | 2019-08-27 19.08.59.mp4 | 180    | video          | 1012736       |           | H264         | 30.04     | 352          | 640                | landscape         |           | 352                 | 640         |
| 262144        | AAC          | 48000      | Ffmpeg  | 91.93    | 2019-08-27 19.10.07.mp4 | 180    | video          | 1006592       |           | H264         | 30.04     | 352          | 640                | landscape         |           | 352                 | 640         |
| 262144        | AAC          | 48000      | Ffmpeg  | 89.58    | 2019-08-27 19.11.24.mp4 | 90     | video          | 1025024       |           | H264         | 30.04     | 352          | 640                | landscape         |           | 352                 | 640         |
| 262144        | AAC          | 48000      | Ffmpeg  | 318.36   | 2019-08-27 19.22.55.mp4 | 180    | video          | 705536        |           | H264         | 29.96     | 352          | 640                | landscape         |           | 352                 | 640         |
| 261120        | AAC          | 48000      | Ffmpeg  | 46.88    | 2019-08-27 19.23.28.mp4 | 180    | video          | 1451008       |           | H264         | 30.06     | 352          | 640                | landscape         |           | 352                 | 640         |
|               |              |            | Ffmpeg  | 5.67     | 2019-08-29 11.01.11.mp4 |        | video          | 3174400       | 9:16      | H264         | 19.94     | 1280         | 1280               | portrait          | 1:1       | 720                 | 720         |
| 131072        | AAC          | 48000      | Ffmpeg  | 18.07    | 2019-08-29 13.56.10.mp4 | 90     | video          | 1851392       |           | H264         | 29.95     | 352          | 640                | landscape         |           | 352                 | 640         |
| 63488         | AAC          | 44100      | Ffmpeg  | 28.89    | 2019-08-29 17.04.49.mp4 | 90     | video          | 1520640       |           | H264         | 29.97     | 480          | 848                | landscape         |           | 480                 | 848         |
| 63488         | AAC          | 44100      | Ffmpeg  | 14.97    | 2019-08-29 17.04.56.mp4 |        | video          | 3277824       | 9:16      | H264         | 29.93     | 1280         | 1280               | portrait          | 1:1       | 720                 | 720         |
| 63488         | AAC          | 44100      | Ffmpeg  | 73.87    | 2019-08-29 17.05.10.mp4 | 90     | video          | 1303552       |           | H264         | 29.97     | 480          | 848                | landscape         |           | 480                 | 848         |
| 64512         | AAC          | 44100      | Ffmpeg  | 51.24    | 2019-08-29 17.05.30.mp4 | 90     | video          | 1374208       |           | H264         | 29.98     | 480          | 848                | landscape         |           | 480                 | 848         |
| 262144        | AAC          | 48000      | Ffmpeg  | 9.11     | 2019-08-30 07.16.24.mp4 |        | video          | 1959936       |           | H264         | 30.22     | 352          | 640                | landscape         |           | 352                 | 640         |
+---------------+--------------+------------+---------+----------+-------------------------+--------+----------------+---------------+-----------+--------------+-----------+--------------+--------------------+-------------------+-----------+---------------------+-------------+

Show the total duration:

% media-info * | td sum
+---------------------+----------+
| key                 | value    |
+---------------------+----------+
| audio_bitrate       | 2279424  |
| audio_format        | 0        |
| audio_rate          | 604500   |
| backend             | 0        |
| duration            | 997.1    |
| media               | 0        |
| rotate              | 1350     |
| type_from_name      | 0        |
| video_bitrate       | 21872640 |
| video_dar           | 0        |
| video_format        | 0        |
| video_fps           | 439.8    |
| video_height        | 7664     |
| video_longest_side  | 11072    |
| video_orientation   | 0        |
| video_sar           | 0        |
| video_shortest_side | 6096     |
| video_width         | 9504     |
+---------------------+----------+

Only select videos that have bitrates higher than 3MiB/s:

% media-info * | td grep '$_->{video_bitrate} > 3_000_000'
+---------------+--------------+------------+---------+----------+-------------------------+----------------+---------------+-----------+--------------+-----------+--------------+--------------------+-------------------+-----------+---------------------+-------------+
| audio_bitrate | audio_format | audio_rate | backend | duration | media                   | type_from_name | video_bitrate | video_dar | video_format | video_fps | video_height | video_longest_side | video_orientation | video_sar | video_shortest_side | video_width |
+---------------+--------------+------------+---------+----------+-------------------------+----------------+---------------+-----------+--------------+-----------+--------------+--------------------+-------------------+-----------+---------------------+-------------+
|               |              |            | Ffmpeg  | 5.67     | 2019-08-29 11.01.11.mp4 | video          | 3174400       | 9:16      | H264         | 19.94     | 1280         | 1280               | portrait          | 1:1       | 720                 | 720         |
| 63488         | AAC          | 44100      | Ffmpeg  | 14.97    | 2019-08-29 17.04.56.mp4 | video          | 3277824       | 9:16      | H264         | 29.93     | 1280         | 1280               | portrait          | 1:1       | 720                 | 720         |
+---------------+--------------+------------+---------+----------+-------------------------+----------------+---------------+-----------+--------------+-----------+--------------+--------------------+-------------------+-----------+---------------------+-------------+

Re-encode those high-bitrate videos:

% media-info * | td grep '$_->{video_bitrate} > 3_000_000' | while read f; do reencode-videos "$f"; done

Show portrait videos:

% media-info * | td grep '$_->{video_height} > $_->{video_width}'

~~—————~~————–+————+———+———-+————————-+—————-+—————+———–+————–+———–+————–+——————–+——————-+———–+———————+————-+

audio_bitrate

audio_format

audio_rate

backend

duration

media

type_from_name

video_bitrate

video_dar

video_format

video_fps

video_height

video_longest_side

video_orientation

video_sar

video_shortest_side

video_width

59392	AAC	44100	Ffmpeg	59.81	2019-08-27 14.40.38.mp4	video	1245184		H264	59.67	848	848	portrait		400	400
			Ffmpeg	5.67	2019-08-29 11.01.11.mp4	video	3174400	9:16	H264	19.94	1280	1280	portrait	1:1	720	720
63488	AAC	44100	Ffmpeg	14.97	2019-08-29 17.04.56.mp4	video	3277824	9:16	H264	29.93	1280	1280	portrait	1:1	720	720

If I want to meticulously edit the filenames, I can convert the table as CSV:

% media-info * --format csv > media.csv

or:

% media-info * | td2csv > media.csv

Edit the CSV in a spreadsheet, add the column name new_filename, and rename the files again on the CLI:

% csv2td media.csv | td as-aohos | td map 'qq(mv -i "$_->{media}" "$_->{new_filename}")'

mv -i "2019-08-27 14.40.38.mp4" "2019-08-27 14.40.38 – blah blah.mp4" mv -i "2019-08-27 19.06.06.mp4" "2019-08-27 19.06.06 – some description.mp4" mv -i "2019-08-27 19.08.59.mp4" "2019-08-27 19.08.59 – some other description.mp4" …

And pipe it to bash if you have verified the output:

% csv2td media.csv | td as-aohos | td map 'qq(mv -i "$_->{media}" "$_->{new_filename}")' | bash

lcpan tips 023: What’s new (including whatsnew)

11 June 202011 June 2020 / perlancar / Leave a comment

About this series: A collection of short blog posts about lcpan tips/recipes. Some posts will also end up in the upcoming App::lcpan::Manual::Cookbook POD to be included in the App::lcpan distribution. First article is here. See the whole series.

About lcpan: an application to download and index a mini CPAN mirror on your local filesystem, so in effect you will have something like your own CPAN with a command-line tool (or perl API) to query and extract information from your mirror.

The previous tip was posted in January 2019. I thought I'd post a status update of what has been added to lcpan since then.

New subcommands

As usual, a bunch of subcommands have been added, most of them for convenience, can be composed from existing subcommands, and not that groundbreaking. For example, the extract-dist subcommands extracts a distro tarball. lcpan already has extract-mod, but sometimes we want to specify a distro name instead of a module. This could almost as easily be expressed using lcpan extract-mod and lcpan dist-mods:

% lcpan extract-dist libwww-perl

can also be written as:

% lcpan extract-mod `lcpan dist-mods libwww-perl | head -n1`

Or, the author, dist, module, release, script subcommands accompany the existing plural versions authors et al to retrieve only a single entity with an exact name search.

The following subcommands have also added: rdeps-scripts, dist-rdeps, heaviest-dists, most-depended-authors, most-mentioned-mods, most-mentioned-scripts, and whatsnew (more on this one below).

Recording of creation and last modification times

One of the most important new features is that lcpan now stores creation time and last modification time of most entities, like authors, modules, dependencies, as well as mentions. Everytime you update the index using lcpan update, when a database record is added or updated, the rec_ctime and rec_mtime columns are set respectively. This means you can query whether a distribution, or author, or dependency is new or recently added. Along with this, many subcommands now sport the filtering options like –added-since, –updated-since, –added-or-updated-since, or the more convenient options like –added-since-last-index-update or –update-since-last-n-index-updates.

For example, to see what modules are added in the last index update:

% lcpan mods -l --added-since-last-index-update

Or, to see what authors are added or updated after Jun 1, 2020 (assuming this year is 2020):

% lcpan authors -l --added-or-updated-since 'jun 1'

Of course, the creation and update times are based on the time you perform lcpan update, since PAUSE itself does not record creation/modification times on the CPAN index files. So if your local CPAN index is only created from scratch today, everything will be new.

Whatsnew

One new subcommand is particularly convenient: whatsnew. It will display modules, distributions, and authors that are recently added/updated, by default since the last index update. And if you specify the –my-author option, or put it in your lcpan.conf:

[subcommand=whatsnew]
my_author=PERLANCAR

then whatsnew will also display new reverse dependencies and new mentions to one of your modules. The output is in the form of Org document, which you can view as-is using a pager, or view using Emacs, directly from the command-line:

% lcpan whatsnew --pager
% lcpan whatsnew --view

There's a demo video for this.

lcpan tips 022: Testing all dependents

9 January 20199 January 2019 / perlancar / 1 Comment

When you release a new version of a module, CPAN Testers will test your module on a variety of platforms. Sometimes, when they happen to test other modules that depend on your module, they will also uncover problems that is caused by your new version breaking your dependents. However, this is not done systematically or completely. If you want to make sure your changes do not break your dependents, you can test for yourself.

Suppose I’m making some changes to Regexp::Pattern. Let’s see who depends on this module:

% lcpan rdeps Regexp::Pattern -R --phase runtime --rel requires
+----------------------------------+-----------+--------------+-------------+
| dist                             | author    | dist_version | req_version |
+----------------------------------+-----------+--------------+-------------+
| App-Licensecheck                 | JONASS    | v3.0.36      | 0           |
| App-RegexpPatternUtils           | PERLANCAR | 0.003        | 0           |
| Bencher-Scenarios-RegexpPattern  | PERLANCAR | 0.003        | 0           |
| Regexp-Common-RegexpPattern      | PERLANCAR | 0.001        | 0           |
|   Bencher-Scenarios-RegexpCommon | PERLANCAR | 0.02         | 0           |
| Test-Regexp-Pattern              | PERLANCAR | 0.004        | v0.2.7      |
+----------------------------------+-----------+--------------+-------------+

Force reinstalling all dependents

One way to run tests for all dependents is by force-(re)installing all those distributions. One way to do that:

% for mod in `lcpan rdeps Regexp::Pattern -R | td select dist | perl -pe's/^\s+//' | dist2mod`; do
    echo "Installing $mod ..."
    lcpanm --force $mod
  done

dist2mod is provided by App::DistUtils.

Extracting tarballs

Another way to test all dependents is to extract them to a temporary directory then run prove on each. lcpan provides the handy subcommand extract-dist to do this. But you might want to install the dependents all first to pull their dependencies.

% mkdir test-deps
% cd test-deps
% for dist in `lcpan rdeps Regexp::Pattern -R | td select dist`; do
    mkdir $dist && cd $dist && lcpan extract-dist $dist && cd ..
done

We can now test each dependent distro one by one, or all in one go:

% for dist in *; do
    echo "Testing $dist ..."
    cd $dist/*; prove -lr; cd ../..
done

lcpan tips 018: How did I use lcpan in 2018?

8 January 2019 / perlancar / Leave a comment

lcpan remains as one of my most-used tools when doing Perl/CPAN development. I use it to search for modules to do some task (usually when I am offline or too lazy to open a browser tab to MetaCPAN, but also to do some grep-ing against the search results). I also use it sometimes to find modules related to a specific module, as keyword searches or seeing the SEE ALSO section of PODs are sometimes insufficient.

The following is a simplistic shell history analysis on how I use lcpan (on my main laptop, at least). The history I have on my laptop is from Nov 2017 to Jan 2019, so that more or less reflects how I used lcpan during 2018.

lcpan and lcpanm

% history | perl -lne's/.+?\]//; s/^pg //; next unless /^lcpanm?\s/; s/(^\S+).*/$1/; print' | freqtable
459	  lcpan
332     lcpanm

I install modules from local mirror using lcpanm quite often.

PAGE_RESULT=1

"pg " is a shell alias which I define as:

alias pg='PAGE_RESULT=1'

because that lets me navigate the output of lcpan with a PAGER (which I have defined as 'less -FRSX').

Most often used subcommands

% history | perl -lne's/.+?\]//; s/^pg //; next unless /^lcpan\s/; s/(^\S+ \S+).*/$1/; print' | freqtable
175	lcpan mods
107	lcpan doc
 35	lcpan scripts
 32	lcpan deps
 17	lcpan rdeps
 16	lcpan related-mods
 14	lcpan src
 11	lcpan stats-last-index-time
 10	lcpan upd
...

I use lcpan most often to:

search for modules (lcpan mods);
read documentation of modules which I don't have installed yet (lcpan doc);
search for scripts (lcpan scripts);
see dependencies (lcpan deps);
see reverse dependencies (lcpan rdeps);
search for related modules (lcpan related-mods);
see source code of modules I don't have installed yet (lcpan src);
checking the last update date of the index (lcpan stats-last-index-time);
updating the mirror/index (lcpan upd which is shortcut for lcpan update).

Finding modules

% history | perl -lne's/.+?\]//; s/^pg //; next unless /^lcpan mods\s/; print' | sort -u
lcpan mods
lcpan mods average
lcpan mods average -l
lcpan mods average|wc -l
lcpan mods binary search -l
lcpan mods bitfin
lcpan mods bitflip
lcpan mods bitgrail
lcpan mods bloomberg
lcpan mods bluetooth
lcpan mods bluetooth -l
lcpan mods b perlstring
lcpan mods ccxt
lcpan mods cpan release
lcpan mods cpan release -l
lcpan mods DateTime::Format
lcpan mods DateTime::Format duration
lcpan mods DateTime::Format::Japanese
lcpan mods datetime iso8601
lcpan mods datetime iso8601 -l
lcpan mods dbi string
lcpan mods dbi string -l
lcpan mods dbi table
lcpan mods dbi table -l
lcpan mods Dist::Zilla::Plugin add -l
lcpan mods Dist::Zilla::Plugin:: --author PERLANCAR -l
lcpan mods Dist::Zilla::Plugin Module
lcpan mods Dist::Zilla::Plugin Module -l
lcpan mods eval
lcpan mods eval -l
lcpan mods fifo -l
lcpan mods File::Slurper
lcpan mods File::Slurper -l
lcpan mods finance crypto
lcpan mods float util
lcpan mods histo
lcpan mods histog -l
lcpan mods histogr -l
lcpan mods histo -l
lcpan mods inside eval -l
lcpan mods interpo
lcpan mods inventory
lcpan mods inventory -l
lcpan mods inventory|wc -l
lcpan mods -l array rank
lcpan mods -l bin groups
lcpan mods -l c encode
lcpan mods -l cli hub
lcpan mods -l compare
lcpan mods -l data cmp
lcpan mods -l dbi csv
lcpan mods -l dbix conn
lcpan mods -l dbix shortcut
lcpan mods -l freq table
lcpan mods -l groups
lcpan mods -l http tiny
lcpan mods -l list rank
lcpan mods -l module abstract
lcpan mods -l module info
lcpan mods -l module pod
lcpan mods -ln role
lcpan mods -l ord
lcpan mods -l ordina
lcpan mods -l ordinaq
lcpan mods -l permute
lcpan mods -l pod abstract
lcpan mods -l Regexp::Common::
lcpan mods -l Regexp::Pattern
lcpan mods -l return level
lcpan mods -l stock exchange
lcpan mods -l test2 tool
lcpan mods -l Test::Approximate
lcpan mods -l test compare
lcpan mods -l Test::Deep::
lcpan mods -l throttle
lcpan mods -l Tickit
lcpan mods -l Tickit Grid
lcpan mods -l tie array
lcpan mods -l Versioning dot
lcpan mods -l who
lcpan mods --namespace Acme::CPANLists
lcpan mods --namespace Acme::CPANLists -l
lcpan mods --namespace Acme::CPANModules -l
lcpan mods --namespace Acme::CPANModuless -l
lcpan mods --namespace Bencher -l
lcpan mods --namespace Bencher::Scenario
lcpan mods --namespace Bencher::Scenario -l
lcpan mods --namespace Data::Sah::Coerce::perl
lcpan mods --namespace Data::Sah::Coerce::perl::str
lcpan mods --namespace DateTime::Format
lcpan mods --namespace DateTime::Format -l
lcpan mods --namespace Graphics::ColorNames
lcpan mods --namespace Graphics::ColorNames|xargs lcpanm -n
lcpan mods --namespace Log::ger
lcpan mods --namespace Log::Ger
lcpan mods --namespace Log::ger|grep -i dump
lcpan mods --namespace String -l
lcpan mods --namespace WordList::Char
lcpan mods -n Archive::Tar
lcpan mods -n digit
lcpan mods -n digit -l
lcpan mods -n digit|wc -l
lcpan mods nearest -l
lcpan mods near -l
lcpan mods -n fifo
lcpan mods -n generic
lcpan mods -n generic -l
lcpan mods -nl Archive::Tar
lcpan mods -nl gen pw
lcpan mods -nl genpw
lcpan mods -nl pass gen
lcpan mods -nl pwd gen
lcpan mods -n pass gen
lcpan mods -n permute digit
lcpan mods Number::Format
lcpan mods Number Format -l
lcpan mods Number::Format -l
lcpan mods pass templat
lcpan mods pass templat -l
lcpan mods percent
lcpan mods percent Sah
lcpan mods percent|wc -l
lcpan mods perinci usage
lcpan mods perl release -l
lcpan mods permute lis
lcpan mods purchase
lcpan mods purchase -l
lcpan mods purchase price
lcpan mods python -l
lcpan mods qr decode
lcpan mods qr decoded
lcpan mods random norm -l
lcpan mods redact
lcpan mods regexp common
lcpan mods regexp common cc
lcpan mods regexp common credit
lcpan mods regexp common -l
lcpan mods ssh client -l
lcpan mods stack trace -l
lcpan mods stock
lcpan mods stock -l
lcpan mods stock|wc -l
lcpan mods Test::Deep:: -l
lcpan mods test path
lcpan mods Text::Histogram
lcpan mods time -l
lcpan mods time of day -l
lcpan mods timeofday -l
lcpan mods version dot
lcpan mods version scheme
lcpan mods version scheme|grep -v Google
lcpan mods who -l
lcpan mods WordList::CryptoCurrency::Catalog::Name

I do keyword searches a lot, and when the keyword is not specific enough I usually add "pg" and "-l" to let me navigate and search further with less. Sometimes I also do namespace searching (--namespace). I search for my own modules a lot, usually because I forget the exact name.

Finding related modules

Modules I tried to find related modules of:

% history | perl -lne's/.+?\]//; next unless /^lcpan related-mods\s/; print' | sort -u

; lcpan related-mods alias::module

lcpan related-mods Data::Diff
lcpan related-mods Data::Throttler
lcpan related-mods Data::Valve
lcpan related-mods IO::Tee
lcpan related-mods Number::Tolerant
lcpan related-mods Package::Alias
lcpan related-mods String::JS
lcpan related-mods Test::Deep
lcpan related-mods utf8

Finding scripts

% history | perl -lne's/.+?\]//; s/^pg //; next unless /^lcpan scripts\s/; print' | sort -u
lcpan scripts bin -l
lcpan scripts count
lcpan scripts count -l
lcpan scripts dateconv
lcpan scripts dateconv -l
lcpan scripts envres
lcpan scripts group -l
lcpan scripts histogram -l
lcpan scripts http-tiny
lcpan scripts interval
lcpan scripts interval -l
lcpan scripts lineno
lcpan scripts linenum
lcpan scripts line number
lcpan scripts lino
lcpan scripts linum
lcpan scripts -l org2html
lcpan scripts -l org-to-html
lcpan scripts parse-nik
lcpan scripts parse-nik -l
lcpan scripts parse num
lcpan scripts _pause
lcpan scripts perl
lcpan scripts perllint
lcpan scripts pick
lcpan scripts pick -l
lcpan scripts pick-l
lcpan scripts rand
lcpan scripts rand -l
lcpan scripts resolution
lcpan scripts resolution -l
lcpan scripts throttle
lcpan scripts zodiac -l

I search for my own scripts a lot too, since I have almost a thousand (~880) of them on CPAN. It's a bit challenging trying to keep the naming organized. When tab completion doesn't help, lcpan comes to the rescue.

The simplistic simulation mode in rename

21 December 201821 December 2018 / perlancar / Leave a comment

About the rename script

rename is a Perl script that lets you use Perl expression or code to rename files. It's convenient for mass-renaming. For example, to convert all filenames in the current directory to lowercase:

% rename -e '$_=lc' *

rename was written by Larry himself. It was released in 1989 as part of perl 3.0 as an example script put under eg/rename. Later, together with a bunch of other old Perl scripts that were not compiling cleanly under strict/-w, it was removed from the perl source tree prior to the release of Perl 5.8 in 2000. It was then republished on CPAN under File::Rename since 2005 by Robin Baker (RBAKER).

About the simulation (dry-run) mode

One of the useful things that rename provides is simulation (-n) mode. In this mode, the files are not actually renamed: rename only prints how it would rename them. For example:

% mkdir test-area
% cd test-area
% touch FOO Bar baz
% rename -n -e '$_ = lc' *
rename(Bar, bar)
rename(FOO, foo)

rename was (and is) a very simple script, measuring at only about 130 lines of code. This simplicity shows in the simplistic simulation mode. As it shows the files as they are being renamed, rename does not keep track of the new filenames that would spring into existence (or old filenames that would disappear). For example:

% rm *
% touch 1 2 3
% rename -n -e '$_ = 3' *
1 not renamed: 3 already exists
2 not renamed: 3 already exists

rename correctly refuse overwriting 3 because it already exists (unless when instructed via -f). However:

% rename -n -e '$_ = 4' *
rename(1, 4)
rename(2, 4)
rename(3, 4)

If you remove the -n option, rename would still behave correctly:

% rename -e '$_ = 4' *
2 not renamed: 4 already exists
3 not renamed: 4 already exists

But the simulation mode does not show you that.

An alternative simulation mode

Another script on CPAN, perlmv, is an alternative that offers a better simulation mode:

% perlmv -d -e'$_ = 4' *
DRYRUN: move `1` -> `4`
DRYRUN: move `2` -> `4.1`
DRYRUN: move `3` -> `4.2`

Note that perlmv does automatic suffixing to avoid overwriting existing files.

Remembering terminal background color for each remote host we SSH to

26 September 20185 October 2018 / perlancar / Leave a comment

Entering (the right) command to a wrong SSH session is no fun. To help me be aware that I am typing commands to a (the correct) remote host, I usually set the background color of my Konsole tab (manually, by right-clicking and then switching the profile). Inspired by this post, I decided to make my own SSH wrapper in Perl: sshwrap-hostcolor.

The wrapper

The sshwrap-hostcolor script is a wrapper that will execute ssh with the arguments you passed to it, but do some stuff before and after that. To try out this wrapper, first install it from CPAN:

% cpanm -n App::sshwrap::hostcolor

then alias it to ssh:

% alias ssh=sshwrap-hostcolor

(You might want to put the above line to your shell's startup file, if you decide to use this script permanently after all.)

Now ssh somewhere:

% ssh user@one.example.com

then change the terminal background color. After you exit, the wrapper will restore the original background color. But if you ssh to the same user+host again later, the background color that you set (and saved by the script to ~/.sshwrap-hostcolor.history) will be restored. See video demonstration.

If you want the wrapper to automatically assign a random color to a new user+host, you can set the environment variable SSHWRAP_HOSTCOLOR_AUTO to random-dark (or random-light, if you use a light background color), e.g.:

% export SSHWRAP_HOSTCOLOR_AUTO=random-dark

See video demonstration.

The completion script

The wrapper comes with its own tab completion script, which you can install using:

% complete -C _sshwrap-hostcolor ssh

This completion script can complete user+host argument for you, and when there's a single completion with known background color, the completion script will immediately change the terminal background color during completion. See video demonstration. This might or might not be to your liking, but you have the option to install the completion if you want this behavior.

Additional stuff

Note that this wrapper will only work with an XTerm-compatible terminal emulation software. See the list here.

While writing the wrapper script, I also wrote a few other CLI utilities.

get-term-bgcolor to programmatically get the terminal's current background color.

set-term-bgcolor to programmatically set the terminal's current background color. This script recognizes RGB color code (e.g. 00002b) or color names (e.g. black, darkblue).

show-color-swatch to show the list of color names and their codes. Use it as such:

% show-color-swatch X | less -R

Oh, and the set-term-bgcolor script also features tab completion (activated via complete -C set-term-bgcolor set-term-bgcolor) so you can complete color names, e.g.:

% set-term-bgcolor dark<Tab><Tab>

Removing password from PDF files

2 July 20182 July 2018 / perlancar / Leave a comment

Electronic billing statement is great. Paperless, easier to archive, and you don't need to care if you leave the house for a month or move addresses. But what's the most irritating thing about credit card e-statements? That the banks decide to encrypt the PDF with some easy-to-guess password like YYYYMMDD of your birth date, or FirstNameDDMMYY, or some other variation. But each bank picks a different password, naturally.

So the first thing I do after downloading an e-statement is to run them through my remove-pdf-password script. It's basically a wrapper for "qpdf" and will try a series of passwords, then decrypt the PDF in-place. You put the passwords in ~/.config/remove-pdf-password.conf, e.g.:

passwords = PASS1  ; anz
passwords = PASS2  ; hsbc
passwords = PASS2  ; uob
...

The script is distributed with the App::PDFUtils distribution. There's the counterpart script in the distribution, add-pdf-password, in case you want to return the favor. The next time some bank staff requests your documents, you can send them in an email encrypted. In the email you write, "the password is the name of the bank followed by the phone number of the bank, but replace every f with f*ckyou …"

Quickly generate CSV or TSV from SQL query

2 July 20182 July 2018 / perlancar / Leave a comment

Sometimes to produce CSV or TSV from information in a MySQL database, all you have to do is:

echo 'some SELECT query ...' | mysql mydb >data.tsv

but for more complex things, you'll often need to use scripting like with Perl to do some munging first. When you're ready to output the data with:

$sth->fetchrow_arrayref
$sth->fetchrow_hashref

the data is in a Perl data structure. How to quickly output CSV? I've looked at DBI::Format and DBIx::CSVDumper and they are cumbersome especially if you want something brief for one-liners: you need to instantiate a separate object with some parameters, then call an additional one or more methods. You might as well just do this with the standard Text::CSV (or Text::CSV_XS):

use Text::CSV;
my $csv = Text::CSV->new;

...
$csv->print(\*STDOUT, $sth->{NAME});
while (my $row = $sth->fetchrow_arrayref) {
     $csv->print(\*STDOUT, $row);
}

For TSV it's even shorter (note: fail spectacularly when data contains Tab character, but that is relatively rare):

say join("\t", @{$sth->{NAME}});
while (my $row = $sth->fetchrow_arrayref) {
     say join("\t", @$row);
}

My modules DBIx::CSV and DBIx::TSV attempt to offer something even briefer (for CSV, at least). Instead of fetchrow_arrayref or fetchall_arrayref (or selectrow_arrayref, selectall_arrayref), you use fetchrow_csv and friends:

print $sth->fetchall_csv;

This automatically prints the header too, and if you don't want header:

print $sth->fetchall_csv_noheader;

For even more text table formats, like ASCII table, Org table, Markdown table, and so on, I've also written DBIx::TextTableAny.

I've also just written DBIx::Conn::MySQL after being tired from writing the gazillionth one-liner to connect to MySQL database. Now, instead of:

% perl -MDBI -E'my $dbh = DBI->connect("dbi:mysql:database=mydb", "username", "password"); $sth = $dbh->prepare(...)'

you can now write this:

% perl -MDBI::Conn::MySQL=mydb -E'$sth = $dbh->prepare(...)'

perlancar's blog

#perl #programming

cli

Org stuffs: 001: Moving/removing done todo items

CLI programs and scripts I use daily 1: net{on,off}, bt{on,off}, {,un}pause-browsers

Multi-Account Containers Firefox addon

Suspending

Checking the battery

The joy of piping tables on the command line

lcpan tips 023: What’s new (including whatsnew)

New subcommands

Recording of creation and last modification times

Whatsnew

lcpan tips 022: Testing all dependents

Force reinstalling all dependents

Extracting tarballs

lcpan tips 018: How did I use lcpan in 2018?

lcpan and lcpanm

PAGE_RESULT=1

Most often used subcommands

Finding modules

Finding related modules

Finding scripts

The simplistic simulation mode in rename

About the rename script

About the simulation (dry-run) mode

An alternative simulation mode

Remembering terminal background color for each remote host we SSH to

The wrapper

The completion script

Additional stuff

Removing password from PDF files

Quickly generate CSV or TSV from SQL query