pericmd 037: More on tab completion (8): x.schema.entity

If you have a lot of functions each with several arguments, you can end up with adding a lot of completion routines for them. And chances are, a lot of them have the same completion routines, for example many of your “user” arguments probably are to be completed with a list of Unix usernames, “file” arguments are filenames, and so on.

Instead of explicitly adding completion routines, you can now also just add an x.schema.entity attribute to your function argument’s specification to mark what kind of entity this argument is.

Perinci::CmdLine will then look for corresponding Perinci::Sub::ArgEntity::* module and inside the module will search for complete_arg_val function. This function will then be used for the completion routine.

There is also the corresponding x.schema.element_entity to be used for the element_completion property.

Currently there is only a handful (or less than a handful) of Perinci::Sub::ArgEntity::* modules on CPAN. I plan to add some more. I also plan to create a Dist::Zilla plugin so that CLI applications having x.schema.{entity,element_entity} attributes in their Rinci metadata will have the corresponding Perinci::Sub::ArgEntity:* modules searched on CPAN and added automatically as prerequisites.

Also of note is that the x.schema.entity attribute is useful for other purposes too, but this is the topic for another blog post.

Afterthought

x.schema.entity is a temporary workaround. Since this is just an issue of types, eventually this information should be put in the schema property, where instead of something like ["str*", min_len=>1, max_len=>255] you specify a schema like ["filename*"] and then the tab completion subsystem can get hints from the type. Sah is capable of this subtyping, but the implementation modules haven’t caught up yet, so I’m keeping the schemas to basic/builtin types and add more refined information in the x.schema.entity attributes for now.

 

pericmd 036: More on tab completion (7): –config-profile

Back to tab completion on today’s post. In this post I want to show you how the completion of --config-profile works. I think this is a nice feature of Perinci::CmdLine and studying the code highlights the mechanism of the tab completion routine.

What is –config-profile for?

As already discussed in a previous blog post, a configuration file can consist of several profiles (or perhaps “scenarios”) which can be selected by the user via command-line option. For example:

# in ~/.config/myapp.conf
foo=1

[profile=p20]
foo=21
bar=22

# in /etc/myapp.conf
bar=2
baz=3

[profile=p10]
foo=11
bar=12

So if you run myapp --config-profile=p10 you’ll get (foo=11, bar=12, baz=3). If you run myapp --config-profile=p20 you’ll get (foo=21, bar=22, baz=3). And if you run without using any profile: myapp you’ll get (foo=1, bar=2, baz=3).

There is completion available for --config-profile:

% myapp --config-profile <tab><tab>
p10     p20

How the completion routine works

In Perinci/CmdLine/Base.pm, in the definition of common options (%copts), we have the following code:

    config_profile => {
        getopt  => 'config-profile=s',
        summary => 'Set configuration profile to use',
        handler => sub {
            my ($go, $val, $r) = @_;
            $r->{config_profile} = $val;
        },
        completion => sub {
            # return list of profiles in read config file

            my %args = @_;
            my $word    = $args{word} // '';
            my $cmdline = $args{cmdline};
            my $r       = $args{r};

            # we are not called from cmdline, bail (actually we might want to
            # return list of programs anyway, but we want to read the value of
            # bash_global_dir et al)
            return undef unless $cmdline;

            # since this is common option, at this point we haven't parsed
            # argument or even read config file. so we need to do that first.
            {
                # this is not activated yet
                $r->{read_config} = 1;

                my $res = $cmdline->parse_argv($r);
                #return undef unless $res->[0] == 200;
            }

            # we are not reading any config file, return empty list
            return [] unless $r->{config};

            my @profiles;
            for (keys %{$r->{config}}) {
                if (length $r->{subcommand_name}) {
                    push @profiles, $1
                        if /\A\Q$r->{subcommand_name}\E \s+ profile=(.+)/x;
                } else {
                    push @profiles, $1 if /\Aprofile=(.+)/;
                }
            }

            require Complete::Util;
            Complete::Util::complete_array_elem(
                array=>[sort @profiles], word=>$word, ci=>1);
        },
    },

Below is the description of how the completion routine works. Lines 210-212 takes in the arguments. Aside from the familiar word, we also have cmdline and r. cmdline contains the Perinci::CmdLine object and is passed to the completion routine so the routine can access the attributes of the command-line application, as well as know that it is run in the context of CLI (completion routine can also be run under different context, like GUI or HTTP over AJAX for autocompletion of form fields). r is a hash structure that contains per-request data, and is explained in the POD documentation of Perinci::CmdLine::Base, for example: format (the output format chosen by user), action (the action chosen, like “help” when program should display help message and exit, “version” to display version and exit, or “call” to execute Riap function, which is the main action), and so on.

r also contains information about the configuration paths and content. Lines 221-227 force the reading of config files via calling the parse_argv() method. After this, if we are reading any config files, the configuration data will be in $r->{config} and we just need to grab all the available profiles (lines 232-240).

Finally we call the usual complete_array_elem to filter the list of profile names with word and return the final answer (line 242-244).

pericmd 035: Common options

We’ll take a short break from discussing tab completion. Let’s turn to one thing that we’ve taken for granted so far: common options. These are things like --help, --version, --format and so on. Where are they defined and how can we customize them?

When we instantiate our CLI application object with Perinci::CmdLine::Any->new, it will return either a Perinci::CmdLine::Lite (the default) or Perinci::CmdLine::Classic object, depending on some conditions/parameters. These classes are all subclasses of Perinci::CmdLine::Base.

Perinci::CmdLine::Base defines the following package variable:

our %copts = (

    version => {
        getopt  => "version|v",
        usage   => "--version (or -v)",
        handler => sub {
            my ($go, $val, $r) = @_;
            $r->{action} = 'version';
            $r->{skip_parse_subcommand_argv} = 1;
        },
    },

    help => {
        getopt  => 'help|h|?',
        summary => 'Display this help message',
        usage   => "--help (or -h, -?)",
        handler => sub {
            my ($go, $val, $r) = @_;
            $r->{action} = 'help';
            $r->{skip_parse_subcommand_argv} = 1;
        },
        order => 0, # high
    },

    format => {
        getopt  => 'format=s',
        summary => 'Choose output format, e.g. json, text',
        handler => sub {
            my ($go, $val, $r) = @_;
            $r->{format} = $val;
        },
    },
 
    ...

Later, each of Perinci::CmdLine::Lite and Perinci::CmdLine::Classic will copy from this hash into new object’s common_opts attribute. Some options will not be included, for example all the config-related common options (--config-profile, --config-path, --no-config), will be skipped if we don’t want config file support (read_config attribute is set to false). You get the final set of options in the common_opts attribute.

Customizing common options

Suppose you don’t want any common option (this makes the CLI app quite incovenient to use, but for the sake of the example…):

# myapp
use Perinci::CmdLine::Any;
Perinci::CmdLine::Any->new(common_opts=>{}, url=>'...');

When this program is run:

% myapp --help
Unknown option: help
ERROR 500: GetOptions failed

% myapp -v
Unknown option: v
ERROR 500: GetOptions failed

What you would like to do is sometimes just remove one or two common options or rename them, usually because they happen to conflict with your function argument. For example, suppose your function has an argument called format and you want the common option --format to be renamed to --output-format. You can already see the structure of the format common option from the previous listing, so:

# myapp
use Perinci::CmdLine::Any;
my $app = Perinci::CmdLine::Any->new(url=>'...');
$app->common_opts->{format}{getopt} = 'output-format=s';
$app->run;

pericmd 034: More on tab completion (6): Completion modules

In this post we’ll take a tour on what completion modules have been written and how you can write one yourself.

Completing Perl modules and distributions

Complete::Module, previously mentioned, can complete Perl module names. There are some other derivative modules which I wrote which are basically shortcuts for convenience like Complete::Dist::Zilla (for listing modules in the namespace of /^Dist::Zilla::(Plugin,PluginBundle,Role)::/) or Complete::Pod::Weaver (for modules in /^Pod::Weaver::(Role,Plugin,PluginBundle,Section)::/). There is also Complete::Riap which currently only supports local URL (like “/App/fatten/fatten” or “pl:/WWW/PAUSE/Simple/list_files”) and also complete from Perl modules and functions.

Complete::Dist can complete from list of locally installed Perl distributions. I’m using it App::DistUtils (“uninstall-dist <tab>”).

Unix stuffs

Complete::Unix provides several functions for completing Unix-related things like user names, UIDs, group names, GIDs, PIDs, as well as process names. They can be useful for writing Unix or Unix-related utilities. For example, I’m using it for App::ShellCompleter::emacs (“emacs –user <tab>”).

Complete::Man can complete from list of available manpages. So far I haven’t had the need to use it yet though.

Others

complete_prog() from Complete::Util can complete from list of programs available on PATH.

Creating your own completion routine/module

Let’s create an example of completing from the list of time zone names. In a Unix machine, there is usually a directory structure in /usr/share/zoneinfo/ which we can readily use (we just need to filter out some files). Since this is just filesystem path, we can utilize Complete::Util‘s complete_file() for this.

First create a module called Complete::TZ by creating file named “lib/Complete/TZ.pm” with the following content:

package Complete::TZ;

use strict;
use warnings;
use Exporter qw(import);
our @EXPORT_OK = qw(complete_tz);

sub complete_tz {
    require Complete::Util;
    my %args = @_;
    Complete::Util::complete_file(
        starting_path => '/usr/share/zoneinfo',
        handle_tilde => 0,
        allow_dot => 0,
        filter => sub {
            return 0 if $_[0] =~ /\.tab$/;
            1;
        },

        word => $word,
    );
}
1;

Let’s test this module:

% perl -Ilib -MComplete::TZ=complete_tz -MDD -E'dd complete_tz(word=>"asia/j")'
["Asia/Jakarta", "Asia/Jayapura", "Asia/Jerusalem"]

If you later package this module as a proper CPAN module and install it, you’ll be able to use it in your CLI app as with other completion routines, e.g.:

use Complete::TZ qw(complete_tz);

$SPEC{yourapp} = {
    v => 1.1,
    args => {
        tz => {
            summary => 'Select a timezone name',
            schema => 'str*',
            completion => \&complete_tz,
        },
        ...
    },
};
...

When you use your CLI app, you’ll then get a convenience of tab completion for timezone names, e.g.:

% yourapp --tz asia/j<tab><tab>
Asia/Jakarta   Asia/Jayapura   Asia/Jerusalem

pericmd 033: More on tab completion (5): Hash answer

Simpler complete_*() functions will just return the list of words as an array, e.g.:

complete_array_elem(word=>"a", array=>[qw/an apple a day keeps the doctor away/]);
# -> ["a", "an", "apple", "away"]

But actually completion routine can also return a hash structure. This is described in Complete. For example:

{words=>["a", "an", "apple", "away"]}

Aside from words which is the meat of the answer, a hash can contain some other keys for metadata/hints for formatting. I’m just going to mention 2 of such keys. The others you can read for yourself in the Complete or Complete::Bash’s POD (or will perhaps be discussed in later posts.)

path_sep

As you probably noticed from the previous blog post (pericmd 032), complete_module() function from Complete::Module routines return a hash with path_sep key set to the path separator:

complete_module(word=>"Complete::P");
# -> {words=>["Complete::Path", "Complete::Pod::Weaver"], path_sep=>"::"}

complete_module(word=>"Complete/Po", dig_leaf=>0);
# -> {words=>["Complete/Pod/"], path_sep=>"/"}

The path_sep key is useful to give hints for, e.g. Complete::Bash (the module that formats the completion answer to stdout for bash). In bash, when a completion routine returns a single entry, bash will add a space to let user move to the next word since the completion is already unambiguous. For example, you already type “Cha” before pressing Tab, and there is only a single file that begins with “Cha” (“Changes”). bash will then change the current word “Cha” to “Changes” and add a space. This is fine when we are completing file/non-folder entity, but suppose the single completion you have for “li” is the directory “lib/”, and bash automatically adds a space after “lib/”. Normally you’d want to have “lib/” right before cursor and without any space, so you can press Tab again to drill down inside that directory. Bash does not provide a way for programs to hint this behavior, so the Complete::Bash employs a trick of returning ["lib/", "lib/ "] instead of just ["lib/"]. This makes the completion ambiguous (there are two candidates) and bash replaces current word “li” with the common substring of the answers “lib/” without any space.

So path_sep will make Complete::Bash format the output of complete_module() in the previous example:

complete_module(word=>"Complete/Po", dig_leaf=>0);
# -> {words=>["Complete/Pod/"], path_sep=>"/"}

into this final output for bash (the second entry has an extra space at the end):

Complete/Pod/
Complete/Pod/ 

But this answer will not get an additional entry since it’s already ambiguous:

complete_module(word=>"Complete::P");
# -> {words=>["Complete::Path", "Complete::Pod::Weaver"], path_sep=>"::"}
Complete::Path
Complete::Pod::Weaver

In short, you can a path_sep key if you are returning a path-like thing and you want to let user drill down. If you do have a custom path-like completion, though, consider using Complete::Path which provides some other nice features like previously discussed (pericmd 032).

esc_mode

This key gives hints to Complete::Bash on how to escape the output, as there are some possible variant depending on what the user expects. A value of shellvar tells Complete::Bash not to escape “$” (dollar sign) to “\$”. Normally if we are completing filenames, for example, we’d want dollar signs to be escaped to avoid shell variable interpretation. But if we are completing shell variables, we do want the “$” to stay as “$”.

pericmd 032: More on tab completion (4): Completing paths

There are several kinds of tree-like entities that can be addressed using a path. Filesystem is one, another is a hierarchy of Perl module/package (yet another is Riap URL, which we haven’t really covered in depth, but suffice to say that local Riap URL also map to Perl packages in Perl-based application). A module called Complete::Path is used as a backend to complete all these kinds of path. Each specific type of path is then completed using a higher-level function which uses Complete::Path, but they support the same settings that Complete::Path supports/respects.

Completing filesystem path

Function complete_file in Complete::Util can be used to complete filesystem path (that is, files and directories). There is a filter option which can be a simple string like "d" to only use directories and not files or "x" to only include files (and directories) that have their executable bit set, or as complex as you want since it can also be a coderef.

Let’s try using the function directly. Suppose we have a directory containing these files:

% mkdir tmp
% cd tmp
% mkdir dir1 dir2 Dir3 dir2/dir4
% touch file1 file2-a file2_b File3 dir2/file4 dir2/dir4/file5

Then this code:

% perl -MComplete::Util=complete_file -MData::Dump -E'dd complete_file(word=>"d")'
["Dir3/", "dir1/", "dir2/"]

Note how directories are automatically appended with path separator character (in this case, /). This is for convenience to let you press Tab again directly to dig a filesystem deeper into subdirectories without typing the path separator character manually.

The map_case option. complete_file() also accepts map_case option (will be passed to Complete::Path) which, if turned on (by default it is), will regard underscore (_) and dash (-) as the same character. This is for convenience to let you use dash (which does not require pressing the Shift key on US keyboards) for completing words that might use underscores as separators. Example:

% perl -MComplete::Util=complete_file -MData::Dump -E'dd complete_file(word=>"file2-")'
["file2-a", "file2_b"]

The exp_im_path option. exp_im_path is short for “expand intermediate paths” and is another convenience option which by default is turned on (can be turned off globally by setting environment COMPLETE_OPT_EXP_IM_PATH to 0). This option lets you type only one or a few characters of intermediate paths. For example:

% perl -MComplete::Util=complete_file -MData::Dump -E'dd complete_file(word=>"d/d/f")'
["dir2/dir4/file5"]

This is akin to a shell wildcard like d*/d*/f*.

Note that by default, expansion is limited only when each intermediate path is only 1 or 2 characters long. As to why this is done, the documentation for Complete module contains the gory details.

The dig_leaf option. This is another convenience option (again, by default is turned on and can be turned off using COMPLETE_OPT_DIG_LEAF=0), which lets Complete::Path dig immediately several levels down if it finds only a single directory in the intermediate paths. For example:

% perl -MComplete::Util=complete_file -MData::Dump -E'dd complete_file(word=>"dir2/")'
["dir2/dir4/file5", "dir2/file4"]

Inside dir2 there is only a single file (file4) and a single subdirectory (dir4). Instead of settling with those, since there is only a single directory, Complete::Path will dig inside dir4 and add the files inside it to the completion answer. If dir4 in turn only contains a single subdirectory, the process is repeated. The effect is, if you have a deep directory structure, e.g. lib/TAP/Parser/Iterator/Stream.pm and you happen to have only a single file like that and no other intermediate paths, you just have to type “lib” (or even “l/”, due to exp_im_path setting) and voila, the whole path is completed using a single Tab press instead of you having to Tab-Tab-Tab your way into the deep directory.

Completing Perl module names

Perl module names can be completed using the complete_module function in Complete::Module module. Since Perl modules also form a hierarchical namespace, the function also calls Complete::Path::complete_path as its backend and shares the same support for options like exp_im_path and dig_leaf. Let’s see some examples:

% perl -MComplete::Module=complete_module -MData::Dump -E'dd complete_module(word=>"TA")'
{
  path_sep => "/",
  words => ["TAP/", "TableDef", "Taint/", "Task/Weaken", "tainting"],
}
% perl -MComplete::Module=complete_module -MData::Dump -E'dd complete_module(word=>"TAP::")'
{
  path_sep => "::",
  words => [
    "TAP::Base",
    "TAP::Formatter::",
    "TAP::Harness",
    "TAP::Harness::",
    "TAP::Object",
    "TAP::Parser",
    "TAP::Parser::",
  ],
}

Wait, why is the path separator still “/”, shouldn’t it be “::” (double colon)? Yes, this is for convenience when doing bash completion. Path separator will only become “::” if the word already contains “::”. Otherwise . See the documentation of Complete::Module (or some of my old blog posts) for more details.

You can force using “::” by specifying path_sep argument:

% perl -MComplete::Module=complete_module -MData::Dump -E'dd complete_module(word=>"TA", path_se=>"::")'
{
  path_sep => "::",
  words => ["TAP::", "TableDef", "Taint::", "Task::Weaken", "tainting"],
}

Also, why does instead of array of words, the function returns a hash structure instead? This allows for setting metadata (like the path_sep key above) useful for hints when formatting the completion. The hash completion answer structure will be discussed in the next blog post.

Another convenience that the function provides is some common shortcuts like “dzp” automatically being expanded to “Dist/Zilla/Plugin/”, “pws” to “Pod/Weaver/Section/” and so on. This list of shortcuts can be customized, even from the environment variable.

Let’s see the complete_module() function in action in an actual CLI program. Install App::PMUtils from CPAN. It contains several CLI apps like pmversion or pmpath:

% pmversion t/ansit<tab>
% pmversion Text/ANSITable _
0.39

% pmpath dat<tab><tab>
Data/            DateTime         DateTimePP       
Date/            DateTime/        DateTimePPExtra  
% pmpath date/<tab>
% pmpath Date/<tab><tab>
Date/Format     Date/Language   Date/Language/  Date/Parse      
% pmpath Date/f<tab>
% pmpath Date/Format _
/home/s1/perl5/perlbrew/perls/perl-5.18.4/lib/site_perl/5.18.4/Date/Format.pm

% pmversion dzb<tab>
% pmversion Dist/Zilla/PluginBundle/<tab>
% pmversion Dist/Zilla/PluginBundle/a/perla<tab>
% pmversion Dist/Zilla/PluginBundle/Author/PERLANCAR _
0.33

pericmd 031: More on tab completion (3): case sensitivity, Complete

Continuing from the previous blog’s example, you’ll notice that by default tab completion is case-insensitive:

% mycomp2 --baz h<tab>
% mycomp2 --baz H<tab><tab>
HISTCONTROL     HISTIGNORE      HISTTIMEFORMAT  
HISTFILESIZE    HISTSIZE        HOME            

This is because most completion routines, including complete_env() used above, or complete_array_elem() used often in custom completion routines, offer ci (case-insensitive) option which defaults to $Complete::OPT_CI which in turn default to environment variable COMPLETE_OPT_CI or 1.

If you turn off case-insensitivity, e.g. by:

% export COMPLETE_OPT_CI=0

then the above completion will no longer work:

% mycomp2 --baz h<tab><tab>

Alternatively if you need to set case (in-)sensitivity specifically in a routine, you can turn it on or off explicitly. For example:

# explicitly turn on case-insensitivity, regardless of COMPLETE_OPT_CI or $Complete::OPT_CI setting
complete_array_elem(array=>\@array, word=>$args{word}, ci=>1);

There are several other settings in Complete that are observed by the other Complete::* modules. Most of these settings’ default are optimized for convenience. This will be covered in the next blog post.

pericmd 030: More on tab completion (2): Completing arguments, element_completion

Like the previous post, this blog post still focuses on tab completion, particularly on completing arguments.

Aside from option names and option values, Perinci::CmdLine can also complete arguments. In Perinci::CmdLine, command-line arguments will also be fed to function as function arguments. The function argument to which the command-line argument(s) will be fed to must be specified with the pos (for positional) property, and optionally with the greedy property. Example:

use Perinci::CmdLine::Any;

our %SPEC;
$SPEC{mycomp2} = {
    v => 1.1,
    args => {
        foo => {
            schema => 'str*',
            pos => 0,
            req => 1,
        },
        bar => {
            schema => 'str*',
            pos => 1,
        },
        baz => {
            schema => 'str*',
        },
    },
};
sub mycomp2 {
    my %args = @_;
    [200, "OK", join(
        "",
        "foo=", $args{foo}//'', " ",
        "bar=", $args{bar}//'', " ",
        "baz=", $args{baz}//'',
    )];
}

Perinci::CmdLine::Any->new(
    url => '/main/mycomp2',
)->run;

In the above program, the argument foo will map to the first
command-line argument (pos=0), bar to the second command-line argument
(pos=1), while baz does not map to any command-line argument (must be specified as command-line option, e.g. --baz val). Of course, the positional arguments can also be specified as command-line options too, although they cannot be both command-line options and arguments at the same time.

% mycomp2
ERROR 400: Missing required argument(s): foo

% mycomp2 1
foo=1 bar= baz=

% mycomp2 --foo 1
foo=1 bar= baz=

% mycomp2 1
ERROR 400: You specified option --foo but also argument #0

% mycomp2 1 2
foo=1 bar=2 baz=

% mycomp2 1 --bar 2
foo=1 bar=2 baz=

% mycomp2 1 --bar 2 2
ERROR 400: You specified option --bar but also argument #1

% mycomp2 1 2 --baz 3
foo=1 bar=2 baz=3

% mycomp2 1 2 3
ERROR 400: There are extra, unassigned elements in array: [3]

As you can see from the last example, Perinci::CmdLine will complain if there are extra arguments that are not unassigned to any function argument. What if we want a function argument to slurp all the remaining command-line arguments? We can declare a function argument as an array and set the pos property as well as set greedy property to true to express that the argument is slurpy.

use Perinci::CmdLine::Any;

our %SPEC;
$SPEC{mycomp2} = {
    v => 1.1,
    args => {
        foo => {
            schema => 'str*',
            pos => 0,
            req => 1,
        },
        bar => {
            schema => ['array*', of=>'str*'],
            pos => 1,
            greedy => 1,
        },
        baz => {
            schema => 'str*',
        },
    },
};
sub mycomp2 {
    my %args = @_;
    [200, "OK", join(
        "",
        "foo=", $args{foo}//'', " ",
        "bar=", ($args{bar} ? "[".join(",",@{$args{bar}})."]" : ''), " ",
        "baz=", $args{baz}//'',
    )];
}

Perinci::CmdLine::Any->new(
    url => '/main/mycomp2',
)->run;

When run:

% mycomp2 1
foo=1 bar= baz=

% mycomp2 1 2
foo=1 bar=[2] baz=

% mycomp2 1 2 3 4
foo=1 bar=[2,3,4] baz=

Now, since command-line arguments map to function arguments, to specify completion for it we just need to put a completion property to the metadata, just like any other argument.

#!/usr/bin/env perl

use Complete::Util qw(complete_array_elem complete_env);
use Perinci::CmdLine::Any;

our %SPEC;
$SPEC{mycomp2} = {
    v => 1.1,
    args => {
        foo => {
            schema => 'str*',
            pos => 0,
            req => 1,
            cmdline_aliases => {f=>{}},
            completion => sub {
                my %args = @_;
                complete_array_elem(
                    word  => $args{word},
                    array => [qw/apple banana blackberry blueberry/],
                ),
            },
        },
        bar => {
            schema => ['array*', of=>'str*'],
            pos => 1,
            greedy => 1,
            element_completion => sub {
                my %args = @_;
                complete_array_elem(
                    word    => $args{word},
                    array   => [qw/snakefruit durian jackfruit/],
                    exclude => $args{args}{bar},
                );
            },
        },
        baz => {
            schema => 'str*',
            completion => \&complete_env,
        },
    },
};
sub mycomp2 {
    my %args = @_;
    [200, "OK", join(
        "",
        "foo=", $args{foo}//'', " ",
        "bar=", ($args{bar} ? "[".join(",",@{$args{bar}})."]" : ''), " ",
        "baz=", $args{baz}//'',
    )];
}

Perinci::CmdLine::Any->new(
    url => '/main/mycomp2',
)->run;

Completion works when function argument is fed as command-line option (including aliases) or command-line argument. Let’s test the completion for foo (note: from this point onwards, I assume you have activated bash completion for the script, as described in the previous post pericmd 029):

% mycomp2 <tab><tab>
-\?               blackberry        --format          --no-config
apple             blueberry         -h                -v
banana            --config-path     --help            --version
--bar             --config-profile  --json            
--baz             --foo             --naked-res      

% mycomp2 b<tab>
banana      blackberry  blueberry   

% mycomp2 --foo <tab><tab>
apple       banana      blackberry  blueberry   

% mycomp2 -f <tab><tab>
apple       banana      blackberry  blueberry   

From the last program listing, you’ll see several new things. First is that the bar argument uses the element_completion property (line 27) instead of completion. This is because bar itself is an argument with type of array (of string), and we are completing the element, not the array itself:

% mycomp2 --bar <tab><tab>
durian      jackfruit   snakefruit  

% mycomp2 --bar durian --bar <tab><tab>
jackfruit   snakefruit  

% mycomp2 --bar durian --bar jackfruit --bar <tab>
% mycomp2 --bar durian --bar jackfruit --bar snakefruit _

You’ll also notice that if a bar value has been specified, the choice will be removed from the offering in completion for the subsequent --bar option value. This is because we are using the exclude option in complete_array_elem() (line 32). The $args{args} contains the function arguments that have been formed at that point.

And lastly, in line 38, you see a new function complete_env which can complete from environment variable names. Since both complete_env() and completion routine expect hash argument as well, and the only required argument is also word, we can pass the subroutine reference directly. Let’s see it in action:

% mycomp2 --baz H
HISTCONTROL     HISTIGNORE      HISTTIMEFORMAT  
HISTFILESIZE    HISTSIZE        HOME            

pericmd 029: More on tab completion (1)

The next several blog posts will focus on tab completion.

Let’s get right to it with a simple example. Put the code below to mycomp, chmod +x the file, and put it somewhere in your PATH (e.g. /usr/local/bin or $HOME/bin if your PATH happens to have $HOME/bin as an entry):

#!/usr/bin/env perl

use 5.010;
use strict;
use warnings;

use Perinci::CmdLine::Any;

our %SPEC;
$SPEC{mycomp} = {
    v => 1.1,
    args => {
        int1 => {
            schema => [int => min=>1, max=>30],
        },
        str1 => {
            schema => [str => in=>[qw/foo bar baz qux quux/]],
        },
        str2 => {
            schema => ['str'],
        },
    },
};
sub mycomp {
    [200];
}

Perinci::CmdLine::Any->new(
    url => '/main/mycomp',
)->run;

Activate bash completion by executing this command in your shell:

% complete -C mycomp mycomp

If your script happens to live outside PATH, e.g. in /path/to/mycomp, you can instead use:

% complete -C /path/to/mycomp mycomp

but normally your CLI programs will reside in PATH, so the above command is for testing only.

Now to test completion:

% mycomp <tab><tab>
-\?               .gitignore        --json            perl-App-hello/
--config-path     -h                mycomp            --str1
--config-profile  hello             --naked-res       -v
--format          --help            --no-config       --version
.git/             --int1            pause/      

As you can see, by default Perinci::CmdLine gives you a list of known options as well as files and directives in the current directory.

% mycomp -<tab><tab>
-\?               -h                --naked-res       --version
--config-path     --help            --no-config       
--config-profile  --int1            --str1            
--format          --json            -v  

If the current word (the word being completed at the cursor) is “-“, Perinci::CmdLine assumes that you want to complete an option name so it doesn’t give a list of files/dirs. (What if, in the rare case, there is a file beginning with a dash and you want to complete it? You can use ./-.)

If the option name can be completed unambiguously:

% mycomp --i<tab><tab>

then it will be completed directly without showing list of completion candidates (underscore _ shows the location of cursor):

% mycomp --int1 _

Perinci::CmdLine can also complete option values. Now let’s press tab again to complete:

% mycomp --int1 <tab><tab>
1   11  13  15  17  19  20  22  24  26  28  3   4   6   8   
10  12  14  16  18  2   21  23  25  27  29  30  5   7   9   

From the argument schema ([int => min=>1, max=>30]), Perinci::CmdLine can provide a list of numbers from 1 to 30 as completion candidates. Now let’s try another argument:

% mycomp --str1=<tab><tab>
bar   baz   foo   quux  qux   

The schema ([str => in=>[qw/foo bar baz qux quux/]]) also helps Perinci::CmdLine provide a completion list. Now another argument:

% mycomp --str2 <tab><tab>
.git/            hello            mycomp~          perl-App-hello/  
.gitignore       mycomp           pause/           

What happened? Since the schema (['str']) doesn’t provide any hints about possible values, Perinci::CmdLine falls back to completing using files/dirs in the current directory. Of course, you can also do something like:

% mycomp --str2 ../../foo<tab><tab>

to list other directories.

This is all nice and good, but the power of tab completion comes with custom completion: when we are able to provide our own completion to option values (and arguments). Let’s try that by adding a completion routine in our Rinci metadata:

use Complete::Util qw(complete_array_elem);

$SPEC{mycomp} = {
    v => 1.1,
    args => {
        int1 => {
            schema => [int => min=>1, max=>30],
            completion => sub {
                my %args = @_;
                my $word = $args{word};

                # let's provide a list of numbers from 1 to current day of month
                my $mday = (localtime)[3];
                complete_array_elem(word=>$word, array=>[1..$mday]);
            },
        },
        str1 => {
            schema => [str => in=>[qw/foo bar baz qux quux/]],
        },
        str2 => {
            schema => ['str'],
        },
    },
};

You see a couple of things new here. First is the completion routine which is supplied in the completion property of the argument specification. A completion routine will receive a hash of arguments (the most important argument is word, there are other arguments and we will get to it later). A completion routine is expected to return an array of words or a hash (see Complete for the specification of the “completion answer”). Second is the use of the module Complete::Util and a function from the module called complete_array_elem which will return an array filtered by $word as prefix. The module contains some more utility functions which we will discuss later.

Now let’s test it (assuming today is Feb 27th, 2015):

% mycomp --int1 <tab><tab>
1   11  13  15  17  19  20  22  24  26  3   5   7   9   
10  12  14  16  18  2   21  23  25  27  4   6   8   

Debugging completion

When we write completion code, we might make mistakes. For example, suppose we forget to use Complete::Util qw(complete_array_elem); then when we test it, we might get unexpected result:

% mycomp --int1 <tab><tab>
.git/            hello            mycomp~          perl-App-hello/  
.gitignore       mycomp           pause/   

Why is Perinci::CmdLine showing files/dirs from current directory instead?

To help debug problems when doing custom completion, you can use the testcomp utility (install it via cpanm App::CompleteUtils). To use testcomp, specify the command and arguments and put ^ (caret) to signify where the cursor is supposed to be. So type:

% testcomp mycomp --int1 ^
[testcomp] COMP_LINE=<mycomp --int1 >, COMP_POINT=14
[testcomp] exec(): ["/mnt/home/s1/perl5/perlbrew/perls/perl-5.18.4/bin/perl","-MLog::Any::Adapter=ScreenColoredLevel","mycomp"]
[pericmd] -> run(), @ARGV=[]
[pericmd] Checking env MYCOMP_OPT: <undef>
[pericmd] Running hook_after_get_meta ...
[comp][periscomp] entering Perinci::Sub::Complete::complete_cli_arg(), words=["--int1",""], cword=1, word=<>
[comp][compgl] entering Complete::Getopt::Long::complete_cli_arg(), words=["--int1",""], cword=1, word=<>
[comp][compgl] invoking routine supplied from 'completion' argument to complete option value, option=<--int1>
[comp][periscomp] entering completion routine (that we supply to Complete::Getopt::Long)
[comp][periscomp] completing option value for a known function argument, arg=<int1>, ospec=<int1=i>
[comp][periscomp] invoking routine supplied from 'completion' argument
[comp][periscomp] result from 'completion' routine: <undef>
[comp][periscomp] entering complete_arg_val, arg=<int1>
[comp][periscomp] invoking routine specified in arg spec's 'completion' property
[comp][periscomp] completion died: Undefined subroutine &main::complete_array_elem called at mycomp line 22.
[comp][periscomp] no completion from metadata possible, declining
[comp][periscomp] leaving complete_arg_val, result=<undef>
[comp][periscomp] leaving completion routine (that we supply to Complete::Getopt::Long)
[comp][compgl] adding result from routine: <undef>
[comp][compgl] entering default completion routine
[comp][compgl] completing with file, file=<>
[comp][compgl] leaving default completion routine, result={path_sep => "/",words => [".git/",".gitignore","hello","mycomp","mycomp~","pause/","perl-App-hello/"]}
[comp][compgl] adding result from default completion routine
[comp][compgl] leaving Complete::Getopt::Long::complete_cli_arg(), result={path_sep => "/",words => [".git/",".gitignore","hello","mycomp","mycomp~","pause/","perl-App-hello/"]}
[comp][periscomp] leaving Perinci::Sub::Complete::complete_cli_arg(), result={path_sep => "/",words => [".git/",".gitignore","hello","mycomp","mycomp~","pause/","perl-App-hello/"]}
[pericmd] Running hook_display_result ...
.git/
.gitignore
hello
mycomp
mycomp~
pause/
perl-App-hello/
[pericmd] Running hook_after_run ...
[pericmd] exit(0)

From the debug output, you can see the error message and realize that the completion routine dies. You’ll also know that Perinci::CmdLine then falls back to using using files/dirs.

pericmd 028: Environment support

Aside from config file, environment can also be a convenient way to input things. It’s more “secure” than command-line options because a casual “ps ax” command won’t show the values of environment variables unlike the command-line. By default, only the user running the program and the superuser can see the environment values of a process (exposed via /proc/PID/environ which has 0700 mode, while /proc/PID/cmdline has 0444 mode).

Perinci::CmdLine gives you environment support. By default, it reads PROGNAME_OPT for you (the value will be prepended to the command-line option). It has a higher precedence than config files (i.e. it can override values from config files) but can be overriden by command-line options.

Let’s see an example with pause. If you run without any configuration file:

% pause ls
ERROR 400: Missing required argument(s): password, username

If we set PAUSE_OPT environment to:

% export PAUSE_OPT="--username PERLANCAR --password xxx"

then run the program again:

% pause ls
...

then username and password arguments have been set from the environment.

Turning off environment (and configuration)

If you don’t want your CLI program to read from environment or configuration, you can turn these features off via the read_env and read_config attributes, respectively:

Perinci::CmdLine::Any->new(
    ...
    read_env => 0,
    read_config => 0,
)->run;