lcpan tips 014: Why use lcpan?

About this series: a collection of short, daily blog posts about lcpan tips/recipes. Some posts will also end up in the upcoming App::lcpan::Manual::Cookbook POD to be included in the App-lcpan distribution.

About lcpan: an application to download and index a mini CPAN mirror on your local filesystem, so in effect you will have something like your own CPAN with a command-line tool (or perl API) to query and extract information from your mirror. I find it perfect for my own personal use when working offline.

Okay, this is not so much a tip but a background story. And this post should be the first in the series. Anyway.

So, why would you want to use lcpan?

The first and foremost reason for me is internet connectivity and bandwidth. Back in around 2001 I think, to save bandwidth for the company, our server kept mirrors of Linux distributions (RedHat at the time, and later Debian) as well as CPAN. I started keeping these mirrors too on my harddisk, copied from the company’s mirrors, as I worked between two cities and in my hometown Internet availability sucked. At home I only had 56k land-line modems, which due to the bad/noisy phone lines in my neighborhood, almost always connected at lower speeds like 28.8k or even 14.4k. Updating OS packages online would be glacially slow.

A few years ago, I switched to using CPAN::Mini and updated my local mirror directly instead of using our company’s CPAN mirror. I also started keeping CPAN mini mirror on my laptop so CPAN is always with me whenever I go. That helped a lot when hacking on the plane or in cars.

Around last year, I started hacking on WWW::PAUSE::Simple, along with its CLI pause. Someone posted a comment on my blog post about this module, saying that the dependencies are out of hand. True, in the name of modularity, the framework (later trimmed down version) which I use for the CLI has a total dependency of about 150+ modules.

So I was looking for a way to reduce this dependency, and looks like fatpacking the script would work.

However, producing a fatpacked script that would include all the necessary dependencies proved to be somewhat tricky. The default tracer that App::FatPacker provides which traps require() during BEGIN phase didn’t cut it, as a lot of the module loading happens at the runtime phase. Also, for some modules like Data::Sah or quite a few others, we need to include all the modules in its distribution. This is because depending on the program’s execution branch, different data types would need to be validated and different Data::Sah type handler modules would be loaded to produce the validators. And lastly, some dependencies are optional/untraceable using runtime tracing, so we need to look instead at the distribution’s dependency information (in META.yml or META.json) to be able to trace them.

Before lcpan was written, this information was not available offline, so that means during fatpacking the script (which I have to do multiple times during development) I have to launch lots of API requests to MetaCPAN, which is annoyingly slow.

There is CPAN::SQLite which also creates SQLite database index from a local CPAN mirror, but it only indexes dists/modules/files from modules/02packages.details.txt.gz. No META.{yml,json} information is extracted and indexed.

Thus, lcpan was born to index dependency information so I could build a fatpacked pause. The rest is history (and feature creep :p).

In some next post I will share all the other ways I’ve used lcpan for.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s