lcpan tips 008: Finding related modules

About this series: a collection of short, daily blog posts about lcpan tips/recipes. Some posts will also end up in the upcoming App::lcpan::Manual::Cookbook POD to be included in the App-lcpan distribution.

About lcpan: an application to download and index a mini CPAN mirror on your local filesystem, so in effect you will have something like your own CPAN with a command-line tool (or perl API) to query and extract information from your mirror. I find it perfect for my own personal use when working offline.

Suppose you are using or evaluating a CPAN module. You want to find an alternative for that module, or just want to know what other modules are related to that module. lcpan (version 0.79 or later) can help you with this using the subcommand related-mods.

This subcommand utilizes the “mentions” information built by lcpan from parsing the PODs of CPAN modules/scripts. It finds related modules by listing what modules that tend to be mentioned together with the target module.

First example, let’s list modules related to Carp::Always:

% lcpan related-mods Carp::Always --format text-pretty | less -S
+---------------------------------+-------------------------------------------------------------------+--------------+-----------------------+-----------------------+--------+-----------+
| module                          | abstract                                                          | num_mentions | num_mentions_together | pct_mentions_together | score  | author    |
+---------------------------------+-------------------------------------------------------------------+--------------+-----------------------+-----------------------+--------+-----------+
| Carp::Always::Color             | Carp::Always, but with color                                      | 9            | 4                     | 44.44                 | 711.04 | DOY       |
| Regexp::Debugger                | Visually debug regexes in-place                                   | 7            | 3                     | 42.86                 | 385.74 | DCONWAY   |
| V                               | Print version of the specified module(s).                         | 9            | 3                     | 33.33                 | 299.97 | ABELTJE   |
| App::cpanoutdated               | detect outdated CPAN modules in your environment.                 | 24           | 4                     | 16.67                 | 266.72 | TOKUHIROM |
| Carp::Source::Always            | Warns and dies with stack backtraces and source code context      | 3            | 2                     | 66.67                 | 266.68 | MARCEL    |
| Devel::bt                       | Automatic gdb backtraces on errors                                | 3            | 2                     | 66.67                 | 266.68 | FLORA     |
| Module::Install::AuthorRequires | declare author-only dependencies                                  | 3            | 2                     | 66.67                 | 266.68 | FLORA     |
| MooseX::Types::LoadableClass    | ClassName type constraint with coercion to load the class.        | 12           | 3                     | 25                    | 225    | ETHER     |
| List::AllUtils                  | Combines List::Util and List::MoreUtils in one bite-sized package | 30           | 4                     | 13.33                 | 213.28 | DROLSKY   |
| App::Software::License          | Command-line interface to Software::License                       | 4            | 2                     | 50                    | 200    | ETHER     |
+---------------------------------+-------------------------------------------------------------------+--------------+-----------------------+-----------------------+--------+-----------+

num_mentions is the total number of POD documents that mention a related module. So, the in above example, Carp::Always::Color is mentioned in 9 POD documents. (BTW, you can see each individual mention using the lcpan mentions --mentioned-module Carp::Always::Color command.) num_mentions_together is the number of POD documents that mention both Carp::Always and the related module (in this case, Carp::Always::Color, and there are 4 POD documents that mention both C:A and C:A:C). pct_mentions_together is the ratio of num_mentions_together divided by num_mentions, expressed in percentage.

score is used to sort the list and is currently calculated as the square of num_mentions_together multiplied by pct_num_mentions_together. BTW, the scoring still needs to be tuned at this point of writing. So, a --sort option is also provided if you want to sort using, e.g. num_mentions_together or pct_mentions_together.

Also, only related modules having a minimum score of 200 is listed. This number is also currently arbitrarily chosen and still needs to be tuned. You might see that some related modules, e.g. Devel::Confess doesn’t make the cut. To see more results, use the --min-score option, e.g.:

% lcpan related-mods Carp::Always --min-score 0 --format text-pretty | less -S

Note that only modules that are mentioned in POD documents can be used. If you have an obscure module that nobody ever mentions, you can’t find other modules related to it, e.g.:

% lcpan related-mods Module::CoreList::More
lcpan: ERROR 400: No mentions for module(s)

The quality of the result depends on how all CPAN authors references other modules in their CPAN module documentation (e.g., in See Also section).

And of course, there are other ways to find related CPAN modules. lcpan related-mods only utilizes data from PODs of CPAN modules/scripts. The other simplest thing you can do to find related modules is listing modules in the same namespace, if the module happens to be put in a clear and specific namespace. Or, simply by searching for some names and keywords. For example, to find modules related to Dist::Zilla::Plugin::TidyAll, one can simply list all other modules in the Dist::Zilla::Plugin namespace(lcpan mods --namespace Dist::Zilla::Plugin -l). There aren’t too many modules in that namespace there yet. Or, to search for other file slurping modules, you can simply do an lcpan mods slurp -l or lcpan mods file slurp -l, which will search from the module names as well as abstract.

There are also other sources of information outside of it, like CPANRatings or other Perl websites, Google search, and the community itself (e.g. asking on perl IRC channels or PerlMonks), and so on.

Some other examples so you can see the results:

% lcpan related-mods Sort::ByExample
+-------------------------------+-----------------------------------------------------------------+--------------+-----------------------+-----------------------+-------+-----------+
| module                        | abstract                                                        | num_mentions | num_mentions_together | pct_mentions_together | score | author    |
+-------------------------------+-----------------------------------------------------------------+--------------+-----------------------+-----------------------+-------+-----------+
| Bencher::Scenario::SortBySpec |                                                                 | 1            | 1                     | 100                   | 100   | PERLANCAR |
| Data::Dump::Filtered          | Pretty printing with filtering                                  | 2            | 1                     | 50                    | 50    | GAAS      |
| Sort::BySpec                  | Sort array (or create a list sorter) according to specification | 2            | 1                     | 50                    | 50    | PERLANCAR |
+-------------------------------+-----------------------------------------------------------------+--------------+-----------------------+-----------------------+-------+-----------+

% lcpan related-mods Module::Path
+--------------------+-------------------------------------------------------+--------------+-----------------------+-----------------------+---------+-----------+
| module             | abstract                                              | num_mentions | num_mentions_together | pct_mentions_together | score   | author    |
+--------------------+-------------------------------------------------------+--------------+-----------------------+-----------------------+---------+-----------+
| Module::Version    | Get module versions                                   | 10           | 9                     | 90                    | 7290    | XSAWYERX  |
| App::IODUtils      | IOD utilities                                         | 9            | 8                     | 88.89                 | 5688.96 | PERLANCAR |
| App::DistUtils     | Collection of utilities related to Perl distributions | 8            | 7                     | 87.5                  | 4287.5  | PERLANCAR |
| App::DzilUtils     | Collection of CLI utilities for Dist::Zilla           | 8            | 7                     | 87.5                  | 4287.5  | PERLANCAR |
| App::LedgerUtils   | Command-line utilities related Ledger files           | 8            | 7                     | 87.5                  | 4287.5  | PERLANCAR |
| App::ProgUtils     | Command line to manipulate programs in PATH           | 8            | 7                     | 87.5                  | 4287.5  | PERLANCAR |
| App::WeaverUtils   | Collection of CLI utilities for Pod::Weaver           | 8            | 7                     | 87.5                  | 4287.5  | PERLANCAR |
| App::GitUtils      | Day-to-day command-line utilities for git             | 9            | 7                     | 77.78                 | 3811.22 | PERLANCAR |
| App::PMUtils       | Command-line utilities related to Perl modules        | 10           | 7                     | 70                    | 3430    | PERLANCAR |
| App::PlUtils       | Command-line utilities related to Perl scripts        | 10           | 7                     | 70                    | 3430    | PERLANCAR |
| Pod::Weaver        | weave together a Pod document from an outline         | 151          | 8                     | 5.3                   | 339.2   | RJBS      |
| Module::Path::More | Get path to locally installed Perl module             | 4            | 2                     | 50                    | 200     | PERLANCAR |
+--------------------+-------------------------------------------------------+--------------+-----------------------+-----------------------+---------+-----------+

% lcpan related-mods WWW::Mechanize --format text-pretty --min-score 3000
+------------------------------------------+-------------------------------------------------------+--------------+-----------------------+-----------------------+----------+----------+
| module                                   | abstract                                              | num_mentions | num_mentions_together | pct_mentions_together | score    | author   |
+------------------------------------------+-------------------------------------------------------+--------------+-----------------------+-----------------------+----------+----------+
| WWW::Scraper::ISBN::Record               | Book Record class for L module.   | 39           | 33                    | 84.62                 | 92151.18 | BARBIE   |
| WWW::Scraper::ISBN                       | Retrieve information about books from online sources. | 43           | 34                    | 79.07                 | 91404.92 | BARBIE   |
| WWW::Scraper::ISBN::Driver               | Driver class for WWW::Scraper::ISBN module.           | 40           | 33                    | 82.5                  | 89842.5  | BARBIE   |
| Test::WWW::Mechanize                     | Testing-specific WWW::Mechanize subclass              | 53           | 34                    | 64.15                 | 74157.4  | PETDANCE |
| HTML::Form                               | Class that represents an HTML form element            | 58           | 33                    | 56.9                  | 61964.1  | GAAS     |
| LWP::UserAgent                           | Web user agent class                                  | 1431         | 61                    | 4.26                  | 15851.46 | ETHER    |
| Finance::Bank::LloydsTSB                 | Check your bank accounts from Perl                    | 39           | 18                    | 46.15                 | 14952.6  | ASPIERS  |
| WWW::Mechanize::Shell                    | An interactive shell for WWW::Mechanize               | 15           | 12                    | 80                    | 11520    | CORION   |
| Template::Extract                        | Use TT2 syntax to extract data from documents         | 25           | 14                    | 56                    | 10976    | AUDREYT  |
| WWW::Mechanize::Link                     | Link object for WWW::Mechanize                        | 11           | 10                    | 90.91                 | 9091     | ETHER    |
| Jifty::Test::WWW::Mechanize              | Subclass of L with              | 14           | 10                    | 71.43                 | 7143     | ALEXMV   |
| WWW::Mechanize::Cached                   | Cache response to be polite                           | 16           | 10                    | 62.5                  | 6250     | OALDERS  |
| HTML::TokeParser                         | Alternative HTML::Parser interface                    | 66           | 16                    | 24.24                 | 6205.44  | GAAS     |
| IO::Socket::SSL                          | a SSL socket interface class                          | 256          | 24                    | 9.38                  | 5402.88  | SULLR    |
| HTTP::Response                           | HTTP style response message                           | 692          | 33                    | 4.77                  | 5194.53  | ETHER    |
| Plagger::Plugin::Subscription::HatenaRSS | HatenaRSS Subscription via OPML                       | 7            | 7                     | 100                   | 4900     | MIYAGAWA |
| Crypt::SSLeay                            | OpenSSL support for LWP                               | 145          | 19                    | 13.1                  | 4729.1   | NANIS    |
| HTML::Form::Input                        |                                                       | 8            | 7                     | 87.5                  | 4287.5   | GAAS     |
| WWW::Mechanize::Pluggable                | A WWW::Mechanize that's custmomizable via plugins     | 13           | 8                     | 61.54                 | 3938.56  | MCMAHON  |
| HTTP::Cookies                            | HTTP cookie jars                                      | 108          | 16                    | 14.81                 | 3791.36  | GAAS     |
| Plagger::Plugin::Notify::Campfire        | Notification bot for Campfire                         | 6            | 6                     | 100                   | 3600     | MIYAGAWA |
| WWW::Sucksub::Divxstation                | automated access to divxstation.com                   | 6            | 6                     | 100                   | 3600     | TFOUCART |
| Test::Pod                                | check for POD errors in files                         | 101          | 15                    | 14.85                 | 3341.25  | ETHER    |
| Devel::NYTProf                           | Powerful fast feature-rich Perl source code profiler  | 55           | 12                    | 21.82                 | 3142.08  | TIMB     |
| Test::Pod::Coverage                      | Check for pod coverage in your distribution.          | 89           | 14                    | 15.73                 | 3083.08  | NEILB    |
+------------------------------------------+-------------------------------------------------------+--------------+-----------------------+-----------------------+----------+----------+

Advertisement