lcpan tips 013: Finding modules

About this series: a collection of short, daily blog posts about lcpan tips/recipes. Some posts will also end up in the upcoming App::lcpan::Manual::Cookbook POD to be included in the App-lcpan distribution.

About lcpan: an application to download and index a mini CPAN mirror on your local filesystem, so in effect you will have something like your own CPAN with a command-line tool (or perl API) to query and extract information from your mirror. I find it perfect for my own personal use when working offline.

This post is a recap of the various ways you can find CPAN modules using lcpan.

lcpan mods KEYWORD …. This can search module names and abstracts (and perhaps POD/source full-text search in the future?) For example, lcpan mods color rgb will search for modules that have “color” or “rgb” in their names/abstracts. If you only want to search in module name, use --query-type name. If you only want to search in abstract, use --query-type abstract. To search for exact module name, use --query-type exact-name. To list modules that match “color” OR “rgb” (instead of the default AND logic), add --or.

lcpan mods –author CPANID. You might also want to list modules that are published by a certain author.

lcpan mods –namespace NAMESPACE. Suitable for when you are looking for modules that are in a known namespace, for example Dist::Zilla plugins (in Dist::Zilla::Plugin namespace) or PSGI servers (in Plack::Server namespace).

lcpan related-mods. Another way of searching modules are by using the “related modules” feature. This is discussed in a previous post. For example, you are searching for HTTP clients. You can start by looking at: lcpan related-mods LWP. The downside of this method is: you won’t find obscure modules that are not mentioned by other modules/POD. Since this feature is using the mentions information to find related modules.

Advertisement

lcpan tips 012: Querying the SQLite database directly

About this series: a collection of short, daily blog posts about lcpan tips/recipes. Some posts will also end up in the upcoming App::lcpan::Manual::Cookbook POD to be included in the App-lcpan distribution.

About lcpan: an application to download and index a mini CPAN mirror on your local filesystem, so in effect you will have something like your own CPAN with a command-line tool (or perl API) to query and extract information from your mirror. I find it perfect for my own personal use when working offline.

As you might have notices, lcpan indexes the CPAN mirror into a SQLite database. By default the database is the index.db at the top-level directory of the mirror (so, by default it’s ~/cpan/index.db.

You can see the schema for the database in the App::lcpan source code. Copy-pasted here:

    install => [
        'CREATE TABLE author (
             cpanid VARCHAR(20) NOT NULL PRIMARY KEY,
             fullname VARCHAR(255) NOT NULL,
             email TEXT
         )',
 
        'CREATE TABLE file (
             id INTEGER NOT NULL PRIMARY KEY,
             name TEXT NOT NULL,
             cpanid VARCHAR(20) NOT NULL REFERENCES author(cpanid),
 
             mtime INT,
             size INT,
 
             -- file status: ok (archive type is known, content can be listed,
             -- and at least some files can be extracted), nofile (file does not
             -- exist in mirror), unsupported (archive type is not supported,
             -- e.g. rar, pm.gz), err (cannot be opened/extracted for some
             -- reason)
 
             file_status TEXT,
             file_error TEXT,
 
             -- META.* processing status: ok (meta has been extracted and
             -- parsed), err (META.json/META.yml has some error), nometa (no
             -- META.json/META.yml found).
 
             meta_status TEXT,
             meta_error TEXT,
 
             -- POD processing status: ok (POD has been extracted and
             -- parsed/indexed).
 
             pod_status TEXT,
 
             -- sub processing status: ok (sub names have been parsed/indexed)
 
             sub_status TEXT,
 
             has_metajson INTEGER,
             has_metayml INTEGER,
             has_makefilepl INTEGER,
             has_buildpl INTEGER
        )',
        'CREATE UNIQUE INDEX ix_file__cpanid__name ON file(cpanid,name)',
 
        # files inside the release archive file
        'CREATE TABLE content (
             id INTEGER NOT NULL PRIMARY KEY,
             file_id INTEGER NOT NULL REFERENCES file(id),
             path TEXT NOT NULL,
             package TEXT, -- only the first package declaration will be recorded
             mtime INT,
             size INT -- uncompressed size
        )',
        'CREATE UNIQUE INDEX ix_content__file_id__path ON content(file_id, path)',
        'CREATE INDEX ix_content__package ON content(package)',
 
        'CREATE TABLE module (
             id INTEGER NOT NULL PRIMARY KEY,
             name VARCHAR(255) NOT NULL,
             cpanid VARCHAR(20) NOT NULL REFERENCES author(cpanid), -- [cache]
             file_id INTEGER NOT NULL,
             version VARCHAR(20),
             version_numified DECIMAL,
             content_id INTEGER REFERENCES content(id),
             abstract TEXT
         )',
        'CREATE UNIQUE INDEX ix_module__name ON module(name)',
        'CREATE INDEX ix_module__file_id ON module(file_id)',
        'CREATE INDEX ix_module__cpanid ON module(cpanid)',
 
        'CREATE TABLE script (
             id INTEGER NOT NULL PRIMARY KEY,
             file_id INTEGER NOT NULL REFERENCES file(id), -- [cache]
             cpanid VARCHAR(20) NOT NULL REFERENCES author(cpanid), -- [cache]
             name TEXT NOT NULL,
             content_id INT REFERENCES content(id),
             abstract TEXT
        )',
        'CREATE UNIQUE INDEX ix_script__file_id__name ON script(file_id, name)',
        'CREATE INDEX ix_script__name ON script(name)',
 
        'CREATE TABLE mention (
             id INTEGER NOT NULL PRIMARY KEY,
             source_file_id INT NOT NULL REFERENCES file(id), -- [cache]
             source_content_id INT NOT NULL REFERENCES content(id),
             module_id INTEGER, -- if mention module and module is known (listed in module table), only its id will be recorded here
             module_name TEXT,  -- if mention module and module is unknown (unlisted in module table), only the name will be recorded here
             script_name TEXT   -- if mention script
        )',
        'CREATE UNIQUE INDEX ix_mention__module_id__source_content_id   ON mention(module_id, source_content_id)',
        'CREATE UNIQUE INDEX ix_mention__module_name__source_content_id ON mention(module_name, source_content_id)',
        'CREATE UNIQUE INDEX ix_mention__script_name__source_content_id ON mention(script_name, source_content_id)',
 
        'CREATE TABLE namespace (
            name VARCHAR(255) NOT NULL,
            num_sep INT NOT NULL,
            has_child BOOL NOT NULL,
            num_modules INT NOT NULL
        )',
        'CREATE UNIQUE INDEX ix_namespace__name ON namespace(name)',
 
        'CREATE TABLE dist (
             id INTEGER NOT NULL PRIMARY KEY,
             name VARCHAR(90) NOT NULL,
             cpanid VARCHAR(20) NOT NULL REFERENCES author(cpanid), -- [cache]
             abstract TEXT,
             file_id INTEGER NOT NULL,
             version VARCHAR(20),
             version_numified DECIMAL,
             is_latest BOOLEAN -- [cache]
         )',
        'CREATE INDEX ix_dist__name ON dist(name)',
        'CREATE UNIQUE INDEX ix_dist__file_id ON dist(file_id)',
        'CREATE INDEX ix_dist__cpanid ON dist(cpanid)',
 
        'CREATE TABLE dep (
             file_id INTEGER,
             dist_id INTEGER, -- [cache]
             module_id INTEGER, -- if module is known (listed in module table), only its id will be recorded here
             module_name TEXT,  -- if module is unknown (unlisted in module table), only the name will be recorded here
             rel TEXT, -- relationship: requires, ...
             phase TEXT, -- runtime, ...
             version VARCHAR(20),
             version_numified DECIMAL,
             FOREIGN KEY (file_id) REFERENCES file(id),
             FOREIGN KEY (dist_id) REFERENCES dist(id),
             FOREIGN KEY (module_id) REFERENCES module(id)
         )',
        'CREATE INDEX ix_dep__module_name ON dep(module_name)',
        # 'CREATE UNIQUE INDEX ix_dep__file_id__module_id ON dep(file_id,module_id)', # not all module have module_id anyway, and ones with module_id should already be correct because dep is a hash with module name as key
 
        'CREATE TABLE sub (
             id INTEGER NOT NULL PRIMARY KEY,
             file_id INTEGER NOT NULL REFERENCES file(id), --[cache]
             content_id INTEGER NOT NULL REFERENCES content(id),
             name TEXT NOT NULL,
             linum INTEGER NOT NULL
         )',
        'CREATE UNIQUE INDEX ix_sub__name__content_id ON sub(name, content_id)',
 
    ], # install
...

You can also peek at what SQL queries each subcommand is doing by installing Log::Any::DBI::Query and then, for example:

% PERL5OPT=-MLog::Any::DBI::Query TRACE=1 lcpan stats
lcpan: [pericmd] Running hook_before_action ...
lcpan: [pericmd] Running action_call() ...
lcpan: [pericmd] Riap request: action=call, url=/App/lcpan/Cmd/stats/handle_cmd
lcpan: Connecting to SQLite database at /home/s1/cpan/index.db ...
lcpan: SQL query (prepare): {{PRAGMA database_list}}
lcpan: SQL query (execute): {{[]}}
lcpan: SQL query (prepare): {{SELECT *
FROM
(
SELECT NULL         TABLE_CAT
     ,              TABLE_SCHEM
     , tbl_name     TABLE_NAME
     ,              TABLE_TYPE
     , NULL         REMARKS
     , sql          sqlite_sql
FROM (
    SELECT 'main' TABLE_SCHEM, tbl_name, upper(type) TABLE_TYPE, sql
    FROM sqlite_master
UNION ALL
    SELECT 'temp' TABLE_SCHEM, tbl_name, 'LOCAL TEMPORARY' TABLE_TYPE, sql
    FROM sqlite_temp_master
UNION ALL
    SELECT 'main' TABLE_SCHEM, 'sqlite_master'      tbl_name, 'SYSTEM TABLE' TABLE_TYPE, NULL sql
UNION ALL
    SELECT 'temp' TABLE_SCHEM, 'sqlite_temp_master' tbl_name, 'SYSTEM TABLE' TABLE_TYPE, NULL sql
)
)
 WHERE TABLE_NAME LIKE 'meta'
 ORDER BY TABLE_TYPE, TABLE_SCHEM, TABLE_NAME
}}
lcpan: SQL query (execute): {{[]}}
...

In the future I might make query-peeking easier, e.g. adding a --db-trace option or something.

lcpan tips 011: Listing modules you depend on

About this series: a collection of short, daily blog posts about lcpan tips/recipes. Some posts will also end up in the upcoming App::lcpan::Manual::Cookbook POD to be included in the App-lcpan distribution.

About lcpan: an application to download and index a mini CPAN mirror on your local filesystem, so in effect you will have something like your own CPAN with a command-line tool (or perl API) to query and extract information from your mirror. I find it perfect for my own personal use when working offline.

If you happen to be a CPAN author (and if you don’t, why not start now?) you can list all the modules that your distributions depend on using the subcommand author-deps. This subcommand is not that different from deps but instead of looking at dependencies of a single distribution, author-deps collects all your distributions and retrieves the dependencies information from all those distributions. Example:

% lcpan author-deps PERLANCAR ;# list all the modules that I depend on

There are a few options like filtering only core or non-core modules, filtering only dependencies for certain phase and relationship, and filtering only modules from certain authors (or not from certain authors).

There’s another related subcommand you might find interested if you happen to have more than one distribution on CPAN (and if you only have one, why not start uploading more?): author-deps-by-dependent-count. This is exactly like author-deps but for each module will count the number of your distributions that depend on it. So, at the top of the result list you’ll see modules that you “most depend on” (use over and over again across distributions).

Here’s an example of result for my distributions:

% lcpan author-deps-by-dependent-count perlancar
+--------------------------------------------------------+-----------+------------+---------+-----------------+
| module                                                 | author    | version    | is_core | dependent_count |
+--------------------------------------------------------+-----------+------------+---------+-----------------+
| Log::Any::IfLOG                                        | PERLANCAR | 0.07       | 0       | 123             |
| Perinci::CmdLine::Any                                  | PERLANCAR | 0.12       | 0       | 101             |
| strict                                                 | SHAY      | 1.09       | 1       | 77              |
| warnings                                               | SHAY      | 1.34       | 0       | 77              |
| experimental                                           | LEONT     | 0.016      | 0       | 69              |
| Dist::Zilla                                            | RJBS      | 5.043      | 0       | 46              |
| Complete::Util                                         | PERLANCAR | 0.45       | 0       | 36              |
| JSON                                                   | MAKAMAKA  | 2.90       | 0       | 36              |
| Exporter                                               | TODDR     | 5.72       | 1       | 29              |
| Data::Dmp                                              | PERLANCAR | 0.14       | 0       | 25              |
| Perinci::Sub::Util                                     | PERLANCAR | 0.45       | 0       | 25              |
| File::Which                                            | PLICEASE  | 1.19       | 0       | 24              |
| WordList                                               | PERLANCAR | 0.001001   | 0       | 24              |
| Module::Path::More                                     | PERLANCAR | 0.30       | 0       | 22              |
| Perinci::Sub::Gen::AccessTable                         | PERLANCAR | 0.55       | 0       | 22              |
| File::Slurper                                          | LEONT     | 0.008      | 0       | 21              |
| Module::List                                           | ZEFRAM    | 0.003      | 0       | 21              |
| Perinci::Sub::Normalize                                | PERLANCAR | 0.15       | 0       | 21              |
| Module::Patch                                          | PERLANCAR | 0.22       | 0       | 20              |
| Complete::Common                                       | PERLANCAR | 0.22       | 0       | 19              |
| Moo                                                    | HAARG     | 2.000002   | 0       | 17              |
...

I’m in the middle of a process of listing core modules for my distributions (previously I didn’t), so currently more than half of my distributions do not yet list core prereqs. Over time, core modules like strict or warnings should be at the top since I use them practically always (except when I use Moo or Moose, in which case strict/warnings are already enabled by those modules.)

Filtering out my own modules, we get:

% lcpan author-deps-by-dependent-count perlancar --module-author-isnt perlancar
+-------------------------------------------+-----------+------------+---------+-----------------+
| module                                    | author    | version    | is_core | dependent_count |
+-------------------------------------------+-----------+------------+---------+-----------------+
| strict                                    | SHAY      | 1.09       | 1       | 77              |
| warnings                                  | SHAY      | 1.34       | 0       | 77              |
| experimental                              | LEONT     | 0.016      | 0       | 69              |
| Dist::Zilla                               | RJBS      | 5.043      | 0       | 46              |
| JSON                                      | MAKAMAKA  | 2.90       | 0       | 36              |
| Exporter                                  | TODDR     | 5.72       | 1       | 29              |
| File::Which                               | PLICEASE  | 1.19       | 0       | 24              |
| File::Slurper                             | LEONT     | 0.008      | 0       | 21              |
| Module::List                              | ZEFRAM    | 0.003      | 0       | 21              |
| Moo                                       | HAARG     | 2.000002   | 0       | 17              |
| Capture::Tiny                             | DAGOLDEN  | 0.34       | 0       | 16              |
| utf8                                      | SHAY      | 1.17       | 1       | 16              |
| DBI                                       | TIMB      | 1.634      | 0       | 15              |
| Data::Dump                                | GAAS      | 1.23       | 0       | 15              |
| List::MoreUtils                           | REHSACK   | 0.413      | 0       | 15              |
| Moose                                     | ETHER     | 2.1605     | 0       | 15              |
| DateTime                                  | DROLSKY   | 1.21       | 0       | 14              |
| Log::Any                                  | DAGOLDEN  | 1.038      | 0       | 14              |
...

lcpan tips 010: Updating your installed Perl modules

About this series: a collection of short, daily blog posts about lcpan tips/recipes. Some posts will also end up in the upcoming App::lcpan::Manual::Cookbook POD to be included in the App-lcpan distribution.

About lcpan: an application to download and index a mini CPAN mirror on your local filesystem, so in effect you will have something like your own CPAN with a command-line tool (or perl API) to query and extract information from your mirror. I find it perfect for my own personal use when working offline.

The cpan-outdated script (or its faster variant cpan-outdated-coro) can compare the version of installed CPAN modules against the version that is indexed in modules/02packages.details.txt.gz file in a CPAN mirror. If your local CPAN mirror is sufficiently up to date (i.e. you have just or quite recently updated it using lcpan update), you can use your local CPAN mirror for this purpose. This can be a bit faster than using a CPAN mirror on the Internet:

% time cpan-outdated-coro
M/ML/MLEHMANN/AnyEvent-7.12.tar.gz
K/KA/KAZEBURO/Apache-LogFormat-Compiler-0.33.tar.gz
D/DA/DAGOLDEN/Capture-Tiny-0.34.tar.gz
L/LE/LEEJO/CGI-4.26.tar.gz
E/ET/ETHER/Moose-2.1605.tar.gz
D/DR/DROLSKY/DateTime-Format-Strptime-1.64.tar.gz
D/DR/DROLSKY/DateTime-TimeZone-1.95.tar.gz
I/IS/ISHIGAKI/DBD-SQLite-1.50.tar.gz
R/RI/RIBASUSHI/DBIx-Class-0.082821.tar.gz
...

real    0m14.681s
user    0m7.664s
sys     0m0.284s

% time cpan-outdated-coro --mirror file:/home/perlancar/cpan
...

real    0m8.030s
user    0m7.464s
sys     0m0.196s

The saving is basically just from not having to download the (currently ~1.7MB) modules/02packages.details.txt.gz file.

What can give you more saving is the installation part, because each of the release file is also already available on your local filesystem.

% cpan-outdated-coro --mirror file:/home/perlancar/cpan | cpanm -n --mirror file:/home/perlancar/cpan --mirror-only

Using cpm can be even faster:

% cpan-outdated-coro -p --mirror file:/home/perlancar/cpan | xargs cpm install --mirror file:/home/perlancar/cpan

lcpan tips 009: Finding subroutines

About this series: a collection of short, daily blog posts about lcpan tips/recipes. Some posts will also end up in the upcoming App::lcpan::Manual::Cookbook POD to be included in the App-lcpan distribution.

About lcpan: an application to download and index a mini CPAN mirror on your local filesystem, so in effect you will have something like your own CPAN with a command-line tool (or perl API) to query and extract information from your mirror. I find it perfect for my own personal use when working offline.

About the feature

As of version 0.78, lcpan supports indexing of subroutines. This is still an experimental version, and not enabled by default. To enable it, you’ll have to update your index using:

% lcpan update --no-skip-sub-indexing

Or, you can also put this in your ~/lcpan.conf:

[subcommand=update]
skip_sub_indexing=0

What’s implemented so far is also very basic. lcpan parses each indexed module’s .pm file using Compiler::Lexer (I skip PPI because I’d reckon it would be rather annoyingly slow. There is also Perl::Lexer but the module’s POD gives a hard warning against using it and I don’t know if it can execute Perl code during lexing, which would be unsafe.) Subroutine declaration is searched and the subroutine name collected, along with the declaration’s line number in the source file.

I haven’t skipped any special subroutine names (e.g. new, DESTROY, AUTOLOAD) except the ones prefixed with underscore. I haven’t parsed any subroutine signature (either from the perl 5.20+ new signature feature, or from the POD). I also haven’t tried to guess whether a subroutine is method/static method/regular function/role/all of them. And lastly, there is no support yet for things like Moose or other fancy modules like Function::Parameters (which lets you define subroutine using the fun keyword).

Searching for functions

I also don’t know yet how useful this feature is going to be. I know that in one or two occasions I’m looking to use a function from a CPAN module, but I don’t remember the module’s name. I remember the function’s name but not exactly. In that case, I can try searching the function:

% lcpan subs rgb -l --query-type name ;# search any function that contain the string 'rgb' in its name
% lcpan subs 'convert%color%' -l --query-type name ;# search using SQL LIKE expression
% lcpan subs 'set_position' -l --query-type exact-name ;# search for exact name

And basically that’s about it.

I plan (and am also looking for ideas on how) to make this more useful. For example, indexing the calling/using relationship between subroutines, so you can know which subroutines are most popular/used. Or, creating statistics of the average number of subroutines per module, the average subroutine name’s length, the average subroutine’s length (in number of lines), and so on. Ideas welcome.

lcpan tips 008: Finding related modules

About this series: a collection of short, daily blog posts about lcpan tips/recipes. Some posts will also end up in the upcoming App::lcpan::Manual::Cookbook POD to be included in the App-lcpan distribution.

About lcpan: an application to download and index a mini CPAN mirror on your local filesystem, so in effect you will have something like your own CPAN with a command-line tool (or perl API) to query and extract information from your mirror. I find it perfect for my own personal use when working offline.

Suppose you are using or evaluating a CPAN module. You want to find an alternative for that module, or just want to know what other modules are related to that module. lcpan (version 0.79 or later) can help you with this using the subcommand related-mods.

This subcommand utilizes the “mentions” information built by lcpan from parsing the PODs of CPAN modules/scripts. It finds related modules by listing what modules that tend to be mentioned together with the target module.

First example, let’s list modules related to Carp::Always:

% lcpan related-mods Carp::Always --format text-pretty | less -S
+---------------------------------+-------------------------------------------------------------------+--------------+-----------------------+-----------------------+--------+-----------+
| module                          | abstract                                                          | num_mentions | num_mentions_together | pct_mentions_together | score  | author    |
+---------------------------------+-------------------------------------------------------------------+--------------+-----------------------+-----------------------+--------+-----------+
| Carp::Always::Color             | Carp::Always, but with color                                      | 9            | 4                     | 44.44                 | 711.04 | DOY       |
| Regexp::Debugger                | Visually debug regexes in-place                                   | 7            | 3                     | 42.86                 | 385.74 | DCONWAY   |
| V                               | Print version of the specified module(s).                         | 9            | 3                     | 33.33                 | 299.97 | ABELTJE   |
| App::cpanoutdated               | detect outdated CPAN modules in your environment.                 | 24           | 4                     | 16.67                 | 266.72 | TOKUHIROM |
| Carp::Source::Always            | Warns and dies with stack backtraces and source code context      | 3            | 2                     | 66.67                 | 266.68 | MARCEL    |
| Devel::bt                       | Automatic gdb backtraces on errors                                | 3            | 2                     | 66.67                 | 266.68 | FLORA     |
| Module::Install::AuthorRequires | declare author-only dependencies                                  | 3            | 2                     | 66.67                 | 266.68 | FLORA     |
| MooseX::Types::LoadableClass    | ClassName type constraint with coercion to load the class.        | 12           | 3                     | 25                    | 225    | ETHER     |
| List::AllUtils                  | Combines List::Util and List::MoreUtils in one bite-sized package | 30           | 4                     | 13.33                 | 213.28 | DROLSKY   |
| App::Software::License          | Command-line interface to Software::License                       | 4            | 2                     | 50                    | 200    | ETHER     |
+---------------------------------+-------------------------------------------------------------------+--------------+-----------------------+-----------------------+--------+-----------+

num_mentions is the total number of POD documents that mention a related module. So, the in above example, Carp::Always::Color is mentioned in 9 POD documents. (BTW, you can see each individual mention using the lcpan mentions --mentioned-module Carp::Always::Color command.) num_mentions_together is the number of POD documents that mention both Carp::Always and the related module (in this case, Carp::Always::Color, and there are 4 POD documents that mention both C:A and C:A:C). pct_mentions_together is the ratio of num_mentions_together divided by num_mentions, expressed in percentage.

score is used to sort the list and is currently calculated as the square of num_mentions_together multiplied by pct_num_mentions_together. BTW, the scoring still needs to be tuned at this point of writing. So, a --sort option is also provided if you want to sort using, e.g. num_mentions_together or pct_mentions_together.

Also, only related modules having a minimum score of 200 is listed. This number is also currently arbitrarily chosen and still needs to be tuned. You might see that some related modules, e.g. Devel::Confess doesn’t make the cut. To see more results, use the --min-score option, e.g.:

% lcpan related-mods Carp::Always --min-score 0 --format text-pretty | less -S

Note that only modules that are mentioned in POD documents can be used. If you have an obscure module that nobody ever mentions, you can’t find other modules related to it, e.g.:

% lcpan related-mods Module::CoreList::More
lcpan: ERROR 400: No mentions for module(s)

The quality of the result depends on how all CPAN authors references other modules in their CPAN module documentation (e.g., in See Also section).

And of course, there are other ways to find related CPAN modules. lcpan related-mods only utilizes data from PODs of CPAN modules/scripts. The other simplest thing you can do to find related modules is listing modules in the same namespace, if the module happens to be put in a clear and specific namespace. Or, simply by searching for some names and keywords. For example, to find modules related to Dist::Zilla::Plugin::TidyAll, one can simply list all other modules in the Dist::Zilla::Plugin namespace(lcpan mods --namespace Dist::Zilla::Plugin -l). There aren’t too many modules in that namespace there yet. Or, to search for other file slurping modules, you can simply do an lcpan mods slurp -l or lcpan mods file slurp -l, which will search from the module names as well as abstract.

There are also other sources of information outside of it, like CPANRatings or other Perl websites, Google search, and the community itself (e.g. asking on perl IRC channels or PerlMonks), and so on.

Some other examples so you can see the results:

% lcpan related-mods Sort::ByExample
+-------------------------------+-----------------------------------------------------------------+--------------+-----------------------+-----------------------+-------+-----------+
| module                        | abstract                                                        | num_mentions | num_mentions_together | pct_mentions_together | score | author    |
+-------------------------------+-----------------------------------------------------------------+--------------+-----------------------+-----------------------+-------+-----------+
| Bencher::Scenario::SortBySpec |                                                                 | 1            | 1                     | 100                   | 100   | PERLANCAR |
| Data::Dump::Filtered          | Pretty printing with filtering                                  | 2            | 1                     | 50                    | 50    | GAAS      |
| Sort::BySpec                  | Sort array (or create a list sorter) according to specification | 2            | 1                     | 50                    | 50    | PERLANCAR |
+-------------------------------+-----------------------------------------------------------------+--------------+-----------------------+-----------------------+-------+-----------+

% lcpan related-mods Module::Path
+--------------------+-------------------------------------------------------+--------------+-----------------------+-----------------------+---------+-----------+
| module             | abstract                                              | num_mentions | num_mentions_together | pct_mentions_together | score   | author    |
+--------------------+-------------------------------------------------------+--------------+-----------------------+-----------------------+---------+-----------+
| Module::Version    | Get module versions                                   | 10           | 9                     | 90                    | 7290    | XSAWYERX  |
| App::IODUtils      | IOD utilities                                         | 9            | 8                     | 88.89                 | 5688.96 | PERLANCAR |
| App::DistUtils     | Collection of utilities related to Perl distributions | 8            | 7                     | 87.5                  | 4287.5  | PERLANCAR |
| App::DzilUtils     | Collection of CLI utilities for Dist::Zilla           | 8            | 7                     | 87.5                  | 4287.5  | PERLANCAR |
| App::LedgerUtils   | Command-line utilities related Ledger files           | 8            | 7                     | 87.5                  | 4287.5  | PERLANCAR |
| App::ProgUtils     | Command line to manipulate programs in PATH           | 8            | 7                     | 87.5                  | 4287.5  | PERLANCAR |
| App::WeaverUtils   | Collection of CLI utilities for Pod::Weaver           | 8            | 7                     | 87.5                  | 4287.5  | PERLANCAR |
| App::GitUtils      | Day-to-day command-line utilities for git             | 9            | 7                     | 77.78                 | 3811.22 | PERLANCAR |
| App::PMUtils       | Command-line utilities related to Perl modules        | 10           | 7                     | 70                    | 3430    | PERLANCAR |
| App::PlUtils       | Command-line utilities related to Perl scripts        | 10           | 7                     | 70                    | 3430    | PERLANCAR |
| Pod::Weaver        | weave together a Pod document from an outline         | 151          | 8                     | 5.3                   | 339.2   | RJBS      |
| Module::Path::More | Get path to locally installed Perl module             | 4            | 2                     | 50                    | 200     | PERLANCAR |
+--------------------+-------------------------------------------------------+--------------+-----------------------+-----------------------+---------+-----------+

% lcpan related-mods WWW::Mechanize --format text-pretty --min-score 3000
+------------------------------------------+-------------------------------------------------------+--------------+-----------------------+-----------------------+----------+----------+
| module                                   | abstract                                              | num_mentions | num_mentions_together | pct_mentions_together | score    | author   |
+------------------------------------------+-------------------------------------------------------+--------------+-----------------------+-----------------------+----------+----------+
| WWW::Scraper::ISBN::Record               | Book Record class for L module.   | 39           | 33                    | 84.62                 | 92151.18 | BARBIE   |
| WWW::Scraper::ISBN                       | Retrieve information about books from online sources. | 43           | 34                    | 79.07                 | 91404.92 | BARBIE   |
| WWW::Scraper::ISBN::Driver               | Driver class for WWW::Scraper::ISBN module.           | 40           | 33                    | 82.5                  | 89842.5  | BARBIE   |
| Test::WWW::Mechanize                     | Testing-specific WWW::Mechanize subclass              | 53           | 34                    | 64.15                 | 74157.4  | PETDANCE |
| HTML::Form                               | Class that represents an HTML form element            | 58           | 33                    | 56.9                  | 61964.1  | GAAS     |
| LWP::UserAgent                           | Web user agent class                                  | 1431         | 61                    | 4.26                  | 15851.46 | ETHER    |
| Finance::Bank::LloydsTSB                 | Check your bank accounts from Perl                    | 39           | 18                    | 46.15                 | 14952.6  | ASPIERS  |
| WWW::Mechanize::Shell                    | An interactive shell for WWW::Mechanize               | 15           | 12                    | 80                    | 11520    | CORION   |
| Template::Extract                        | Use TT2 syntax to extract data from documents         | 25           | 14                    | 56                    | 10976    | AUDREYT  |
| WWW::Mechanize::Link                     | Link object for WWW::Mechanize                        | 11           | 10                    | 90.91                 | 9091     | ETHER    |
| Jifty::Test::WWW::Mechanize              | Subclass of L with              | 14           | 10                    | 71.43                 | 7143     | ALEXMV   |
| WWW::Mechanize::Cached                   | Cache response to be polite                           | 16           | 10                    | 62.5                  | 6250     | OALDERS  |
| HTML::TokeParser                         | Alternative HTML::Parser interface                    | 66           | 16                    | 24.24                 | 6205.44  | GAAS     |
| IO::Socket::SSL                          | a SSL socket interface class                          | 256          | 24                    | 9.38                  | 5402.88  | SULLR    |
| HTTP::Response                           | HTTP style response message                           | 692          | 33                    | 4.77                  | 5194.53  | ETHER    |
| Plagger::Plugin::Subscription::HatenaRSS | HatenaRSS Subscription via OPML                       | 7            | 7                     | 100                   | 4900     | MIYAGAWA |
| Crypt::SSLeay                            | OpenSSL support for LWP                               | 145          | 19                    | 13.1                  | 4729.1   | NANIS    |
| HTML::Form::Input                        |                                                       | 8            | 7                     | 87.5                  | 4287.5   | GAAS     |
| WWW::Mechanize::Pluggable                | A WWW::Mechanize that's custmomizable via plugins     | 13           | 8                     | 61.54                 | 3938.56  | MCMAHON  |
| HTTP::Cookies                            | HTTP cookie jars                                      | 108          | 16                    | 14.81                 | 3791.36  | GAAS     |
| Plagger::Plugin::Notify::Campfire        | Notification bot for Campfire                         | 6            | 6                     | 100                   | 3600     | MIYAGAWA |
| WWW::Sucksub::Divxstation                | automated access to divxstation.com                   | 6            | 6                     | 100                   | 3600     | TFOUCART |
| Test::Pod                                | check for POD errors in files                         | 101          | 15                    | 14.85                 | 3341.25  | ETHER    |
| Devel::NYTProf                           | Powerful fast feature-rich Perl source code profiler  | 55           | 12                    | 21.82                 | 3142.08  | TIMB     |
| Test::Pod::Coverage                      | Check for pod coverage in your distribution.          | 89           | 14                    | 15.73                 | 3083.08  | NEILB    |
+------------------------------------------+-------------------------------------------------------+--------------+-----------------------+-----------------------+----------+----------+

lcpan tips 007: Showing new dists

About this series: a collection of short, daily blog posts about lcpan tips/recipes. Some posts will also end up in the upcoming App::lcpan::Manual::Cookbook POD to be included in the App-lcpan distribution.

About lcpan: an application to download and index a mini CPAN mirror on your local filesystem, so in effect you will have something like your own CPAN with a command-line tool (or perl API) to query and extract information from your mirror. I find it perfect for my own personal use when working offline.

Here‘s a simple wrapper script for lcpan update (hopefully it will be integrated into some kind of hook/plugin later for lcpan) that shows new dists after updating the mirror. It simply list distributions newer than the last index time of the mirror.

Sample output:

% lcpan-update-and-show-new-dists
+------------------------------------------+----------+----------+------------------------------------------------------+----------+------------+-----------------------------------------------------------------------+
| dist                                     | author   | version  | release                                              | rel_size | rel_mtime  | abstract                                                              |
+------------------------------------------+----------+----------+------------------------------------------------------+----------+------------+-----------------------------------------------------------------------+
| Map-Tube-Barcelona                       | MANWAR   | 0.28     | Map-Tube-Barcelona-0.28.tar.gz                       | 17971    | 1455621083 | Interface to the Barcelona Metro Map.                                 |
| Data-Peek                                | HMBRAND  | 0.45     | Data-Peek-0.45.tgz                                   | 60698    | 1455619251 | Modified and extended debugging facilities                            |
| PONAPI-Server                            | MICKEY   | 0.002015 | PONAPI-Server-0.002015.tar.gz                        | 89130    | 1455617402 | PONAPI - Perl implementation of {JSON:API} (http://jsonapi.org/) v1.0 |
| Math-MPFR                                | SISYPHUS | 3.32     | Math-MPFR-3.32.tar.gz                                | 109598   | 1455617315 | Perl interface to the MPFR (floating point) library                   |
| Inferno-RegMgr                           | POWERMAN | v1.0.0   | Inferno-RegMgr-v1.0.0.tar.gz                         | 16742    | 1455617143 | Keep connection to OS Inferno's registry(4) and it tasks              |
| Tapper-Config                            | TAPPER   | 5.0.1    | Tapper-Config-5.0.1.tar.gz                           | 13075    | 1455616295 | Tapper - Context sensitive configuration hub for all Tapper libs      |
| Tapper-MCP                               | TAPPER   | 5.0.1    | Tapper-MCP-5.0.1.tar.gz                              | 86189    | 1455615949 | Tapper - Central master control program of Tapper automation          |
| Tapper-Metadata                          | TAPPER   | 5.0.1    | Tapper-Metadata-5.0.1.tar.gz                         | 19863    | 1455613737 | Save and search Metadata points by database                           |
| Scientist                                | LANCEW   | 0.003    | Scientist-0.003.tar.gz                               | 2693     | 1455612965 | Perl module inspired by https://github.com/github/scientist           |
| Plack-Middleware-StackTrace-LinkedSource | BAYASHI  | 0.04     | Plack-Middleware-StackTrace-LinkedSource-0.04.tar.gz | 13330    | 1455612268 | Adding links to library source codes in stacktrace                    |
| Narada                                   | POWERMAN | v2.3.4   | Narada-v2.3.4.tar.gz                                 | 103789   | 1455611796 | framework for ease deploy and support microservice projects           |
| Date-Pregnancy                           | JONASBN  | 0.06     | Date-Pregnancy-0.06.tar.gz                           | 12680    | 1455610099 | calculate birthdate and week numbers for a pregnancy                  |
| Net-OpenNebula                           | STDWEIRD | 0.305.0  | Net-OpenNebula-0.305.0.tar.gz                        | 31673    | 1455610012 | Access OpenNebula RPC via Perl.                                       |
| File-Rsync-Mirror-Recent                 | ANDK     | v0.4.2   | File-Rsync-Mirror-Recent-0.4.2.tar.bz2               | 77280    | 1455606667 | unknown                                                               |
| Algorithm-Networksort-Chooser            | FRACTAL  | 0.110    | Algorithm-Networksort-Chooser-0.110.tar.gz           | 5695     | 1455604080 | unknown                                                               |
| Mojo-Weixin                              | SJDY     | v1.0.3   | Mojo-Weixin-1.0.3.tar.gz                             | 765865   | 1455603274 | A Weixin Client Framework base on Mojolicious                         |
| MooseX-AttributeHelpers                  | ETHER    | 0.25     | MooseX-AttributeHelpers-0.25.tar.gz                  | 54105    | 1455600583 | (DEPRECATED) Extend your attribute interfaces                         |
| MooseX-Daemonize                         | ETHER    | 0.21     | MooseX-Daemonize-0.21.tar.gz                         | 44322    | 1455599805 | Role for daemonizing your Moose based application                     |
+------------------------------------------+----------+----------+------------------------------------------------------+----------+------------+-----------------------------------------------------------------------+

This can be an alternative to seeing recent dists on MetaCPAN. The script’s version shows you the dist’s abstract, author, and file size.

lcpan tips 006: lcpanm

About this series: a collection of short, daily blog posts about lcpan tips/recipes. Some posts will also end up in the upcoming App::lcpan::Manual::Cookbook POD to be included in the App-lcpan distribution.

About lcpan: an application to download and index a mini CPAN mirror on your local filesystem, so in effect you will have something like your own CPAN with a command-line tool (or perl API) to query and extract information from your mirror. I find it perfect for my own personal use when working offline.

lcpanm is a thin wrapper for cpanm and comes with the App-lcpan distribution (which means that when you install lcpan you automatically get lcpanm). It sets --mirror to your local CPAN mirror. Also, its tab completion uses the lcpan database to be able to complete CPAN module names.

Some examples:

% lcpanm -n File::Lock[tab]
Lock    Lock::    lockf    Lockfile

lcpan tips 005: Monitoring new/changed dependants and mentions

About this series: a collection of short, daily blog posts about lcpan tips/recipes. Some posts will also end up in the upcoming App::lcpan::Manual::Cookbook POD to be included in the App-lcpan distribution.

About lcpan: an application to download and index a mini CPAN mirror on your local filesystem, so in effect you will have something like your own CPAN with a command-line tool (or perl API) to query and extract information from your mirror. I find it perfect for my own personal use when working offline.

One thing I’d like to keep an eye on whenever I update my CPAN mirror is whether there are new distributions that declare dependencies on one of my modules (new dependants), or whether a distribution no longer depends on my modules. Or, whether someone now mentions one of my modules/scripts (new mentions), or whether someone no longer mentions them.

I plan to add some sort of hook mechanism to lcpan, so that after lcpan update is run, the above things can be performed. But, until that feature is implemented, I’m using a wrapper script like this.

When the script is run:

% lcpan-update-and-diff-rdeps-and-mentions PERLANCAR

then your CPAN mirror is updated with lcpan update, and after that a diff output (or nothing, if nothing changes) will be shown detailing new/changed dependants and mentions for modules/scripts by author PERLANCAR.

lcpan tips 004: Who mentions my modules/scripts?

About this series: a collection of short, daily blog posts about lcpan tips/recipes. Some posts will also end up in the upcoming App::lcpan::Manual::Cookbook POD to be included in the App-lcpan distribution.

About lcpan: an application to download and index a mini CPAN mirror on your local filesystem, so in effect you will have something like your own CPAN with a command-line tool (or perl API) to query and extract information from your mirror. I find it perfect for my own personal use when working offline.

Since 0.67, lcpan indexes scripts in addition to modules. To do this, lcpan uses a heuristic: all files inside release tarballs that have paths matching ^([^/]+/)?(bin/scripts?)/([^/]+)$. It might miss a few and also get a few false-positives. But the alternative way of parsing Makefile.PL (e.g. from WriteMakefile argument EXE_FILES) is considered unsafe.

Also since 0.67, lcpan parses POD of .pm/.pod files and scripts (extracted from release tarballs) and indexes POD links as well as module names that are mentioned in the PODs.

To view this information from the command-line, lcpan provides the mentions subcommand. For example, to list who mentions my module Text::ANSITable:

% lcpan mentions --mentioned-module Text::ANSITable
+-----------------+---------------+-----------------------------------------------+------------------+---------------------------------------------------------------------------------+
| module          | module_author | release                                       | mentioner_author | content_path                                                                    |
+-----------------+---------------+-----------------------------------------------+------------------+---------------------------------------------------------------------------------+
| Text::ANSITable | PERLANCAR     | App-DistUtils-0.08.tar.gz                     | PERLANCAR        | App-DistUtils-0.08/bin/packlist-for                                             |
| Text::ANSITable | PERLANCAR     | Bencher-Scenario-TextTableModules-0.04.tar.gz | PERLANCAR        | Bencher-Scenario-TextTableModules-0.04/lib/Bencher/Scenario/TextTableModules.pm |
| Text::ANSITable | PERLANCAR     | Color-Theme-0.01.tar.gz                       | PERLANCAR        | Color-Theme-0.01/lib/Color/Theme.pm                                             |
| Text::ANSITable | PERLANCAR     | Color-Theme-0.01.tar.gz                       | PERLANCAR        | Color-Theme-0.01/lib/Color/Theme/Role.pm                                        |
| Text::ANSITable | PERLANCAR     | Color-Theme-0.01.tar.gz                       | PERLANCAR        | Color-Theme-0.01/lib/Color/Theme/Util.pm                                        |
| Text::ANSITable | PERLANCAR     | Complete-Module-0.24.tar.gz                   | PERLANCAR        | Complete-Module-0.24/lib/Complete/Module.pm                                     |
| Text::ANSITable | PERLANCAR     | Data-Format-Pretty-Console-0.35.tar.gz        | PERLANCAR        | Data-Format-Pretty-Console-0.35/lib/Data/Format/Pretty/Console.pm               |
| Text::ANSITable | PERLANCAR     | Dist-Util-0.06.tar.gz                         | PERLANCAR        | Dist-Util-0.06/lib/Dist/Util.pm                                                 |
| Text::ANSITable | PERLANCAR     | Dist-Zilla-Plugin-Preload-0.01.tar.gz         | PERLANCAR        | Dist-Zilla-Plugin-Preload-0.01/lib/Dist/Zilla/Plugin/Preload.pm                 |
| Text::ANSITable | PERLANCAR     | Perinci-CmdLine-1.48.tar.gz                   | PERLANCAR        | Perinci-CmdLine-1.48/lib/Perinci/CmdLine/Manual/FAQ.pod                         |
| Text::ANSITable | PERLANCAR     | Perinci-CmdLine-Any-Lumped-0.09.tar.gz        | PERLANCAR        | Perinci-CmdLine-Any-Lumped-0.09/lib/Text/ANSI/Util.pm                           |
| Text::ANSITable | PERLANCAR     | Term-Detect-Software-0.21.tar.gz              | PERLANCAR        | Term-Detect-Software-0.21/lib/Term/Detect/Software.pm                           |
| Text::ANSITable | PERLANCAR     | Text-ANSI-Util-0.21.tar.gz                    | PERLANCAR        | Text-ANSI-Util-0.21/lib/Text/ANSI/Util.pm                                       |
| Text::ANSITable | PERLANCAR     | Text-Table-Any-0.01.tar.gz                    | PERLANCAR        | Text-Table-Any-0.01/lib/Text/Table/Any.pm                                       |
| Text::ANSITable | PERLANCAR     | Text-Table-Manifold-1.00.tgz                  | RSAVAGE          | Text-Table-Manifold-1.00/lib/Text/Table/Manifold.pm                             |
| Text::ANSITable | PERLANCAR     | Tickit-Widget-Table-0.214.tar.gz              | TEAM             | Tickit-Widget-Table-0.214/lib/Tickit/Widget/Table.pm                            |
| Text::ANSITable | PERLANCAR     | Tickit-Widget-Table-0.214.tar.gz              | TEAM             | Tickit-Widget-Table-0.214/lib/Tickit/Widget/Table.pod                           |
| Text::ANSITable | PERLANCAR     | Tickit-Widget-Table-Paged-0.004.tar.gz        | TEAM             | Tickit-Widget-Table-Paged-0.004/lib/Tickit/Widget/Table/Paged.pm                |
| Text::ANSITable | PERLANCAR     | Tickit-Widget-Table-Paged-0.004.tar.gz        | TEAM             | Tickit-Widget-Table-Paged-0.004/lib/Tickit/Widget/Table/Paged.pod               |
| Text::ANSITable | PERLANCAR     | App-lcpan-0.73.tar.gz                         | PERLANCAR        | App-lcpan-0.73/bin/lcpan                                                        |
+-----------------+---------------+-----------------------------------------------+------------------+---------------------------------------------------------------------------------+

As you can see, there’s a lot of self mentions there (one of my modules/scripts mentioning my own module). To filter those out, add the option --mentioner-author-isnt. You can add more than one instance of this option.

% lcpan mentions --mentioned-module Text::ANSITable --mentioner-author-isnt PERLANCAR
+-----------------+---------------+----------------------------------------+------------------+-------------------------------------------------------------------+
| module          | module_author | release                                | mentioner_author | content_path                                                      |
+-----------------+---------------+----------------------------------------+------------------+-------------------------------------------------------------------+
| Text::ANSITable | PERLANCAR     | Text-Table-Manifold-1.00.tgz           | RSAVAGE          | Text-Table-Manifold-1.00/lib/Text/Table/Manifold.pm               |
| Text::ANSITable | PERLANCAR     | Tickit-Widget-Table-0.214.tar.gz       | TEAM             | Tickit-Widget-Table-0.214/lib/Tickit/Widget/Table.pm              |
| Text::ANSITable | PERLANCAR     | Tickit-Widget-Table-0.214.tar.gz       | TEAM             | Tickit-Widget-Table-0.214/lib/Tickit/Widget/Table.pod             |
| Text::ANSITable | PERLANCAR     | Tickit-Widget-Table-Paged-0.004.tar.gz | TEAM             | Tickit-Widget-Table-Paged-0.004/lib/Tickit/Widget/Table/Paged.pm  |
| Text::ANSITable | PERLANCAR     | Tickit-Widget-Table-Paged-0.004.tar.gz | TEAM             | Tickit-Widget-Table-Paged-0.004/lib/Tickit/Widget/Table/Paged.pod |
+-----------------+---------------+----------------------------------------+------------------+-------------------------------------------------------------------+

Top modules/scripts/authors ranked by mention count

Aside from the mentions subcommand to list individual mentions, there are also various *-by-mention-count subcommands to rank modules/scripts/authors by mention count. Some examples:

% lcpan authors-by-mod-mention-count | head -n 10
SWALTERS        49901
ETHER   9000
SUNDQUIST       4767
RJBS    4609
JRADCLIFF       3672
RIBASUSHI       3616
SHAY    3226
DROLSKY 3042
PMATTHEWS       2442
ALEXMV  2032

The number after each author is number of mentions by other authors. Whenever a module by the author gets mentioned in another module/script’s POD, it gets a count (this means, we are counting per content/mentioning POD). There are other count types that can be used for the ranking: per-dist (where each distribution that mentions a module by author is counted as one count, so an author will get a maximum of one count per mentioning dist), per-author (where an author will get a maximum of one count per mentioning author).

% lcpan authors-by-mod-mention-count --count-per dist | head -n 10
lcpan authors-by-mod-mention-count --count-per dist | head -n 10
ETHER   3535
RJBS    2296
SHAY    1643
SMUELLER        1294
DROLSKY 1111
GAAS    1076
GBARR   961
EXODIST 931
MIYAGAWA        838
DAGOLDEN        726
% lcpan authors-by-mod-mention-count --count-per author | head -n 10
ETHER   1255
RJBS    866
SMUELLER        730
SHAY    714
GAAS    601
GBARR   592
DROLSKY 543
EXODIST 470
TODDR   389
SHLOMIF 387