Experimenter CLI ================ Overview -------- The jFed experimenter CLI is used to create slivers easily from the command line. The tool takes a command as first argument, and several options depending on the command. The supported commands are: ``create`` Create one or more slivers. And optionally the slice used. Ansible playbooks can also be executed once everything is ready. ``delete`` Delete the specified slivers. ``status`` Get the status of the specified slice. ``manifest`` Locally store the manifest RSpec of the specified slice (use ``--manifest`` to override the default location). This can also be used to generate `ansible `_ config files, using the ``--ansible-dir`` option, and to execute ansible playbooks. ``renew`` Renew the expiration time of the slivers ``userinfo`` Fetch user info: urn, slices and projects This command requires minimum version ``develop 172`` of the jFed CLI ``poa`` PerformOperationalAction. On emulab based testbeds this can be used to reload or restart nodes, and to get a console URL. ``createslice`` Create a slice, but allocate no resources (= create no slivers) ``slice-info`` Retrieve slice info. This also tries to retrieve a list of all slivers. It does not query the sliver status. (use ``status`` for that). The first 4 commands are based around the user's request RSpec. So, ``delete``, ``status`` and ``renew`` all need the request rspec as parameter. These commands also require slice information (to create or use a slice). The ``userinfo`` command does not require as much information, as it only fetches user information. The ``poa`` command requires a target sliver URN and an action to execute. The ``createslice`` and ``slice-info` command only require user and slice information. Download and Run ---------------- Go to `the jFed download page `_ and download the jFed CLI package at the bottom of the page. You can also `download a development version `_. Note that the top build on that page is the latest version. Click on the build of your choice, and on the next page, download the "jFed CLI (archive)". In both cases, you will download a file called ``jfed_cli.tar.gz``. Extract this file to a directory and go to that directory. Here you will see ``experimenter-cli.jar`` and a ``lib`` dir. These are both required to run the Experimenter CLI. Linux CLI example:: user@laptop ~/Downloads $ tar xfz jfed_cli.tar.gz user@laptop ~/Downloads $ cd jfed_cli user@laptop ~/Downloads/jfed_cli $ ls automated-testing-5.2.0-SNAPSHOT.jar experimenter-cli-5.2.0-SNAPSHOT.jar lib probe-cli-5.2.0-SNAPSHOT.jar user@laptop ~/Downloads/jfed_cli $ java -jar experimenter-cli.jar Syntax: jfed-experimenter-cli [command_options ... ] Available Commands: create,delete,status,renew Usage ----- The tool is a java jar, so you need to use java to run it. The create command takes a number of mandatory arguments: ``-p `` The PEM login file that identifies you as an SFA user. This file contains one or more certificates, and a matching private key. Example content:: -----BEGIN RSA PRIVATE KEY----- Proc-Type: 4,ENCRYPTED DEK-Info: DES-EDE3-CBC,B9AFE3F639D6D355 oDj5CQZ68EqY... (multiple lines) ... -----END RSA PRIVATE KEY----- -----BEGIN CERTIFICATE----- MIIEBzF... (multiple lines) ... -----END CERTIFICATE----- ``-P `` The plaintext password that locks the private key. If this option is not provided, and a key is requested, the password will be requested on ``stdin`` by the CLI. Note that not all private keys are password protected and that you can remove the password from a private key using the openssl tool:: openssl rsa < password_protected_file.pem > unprotected_file.pem (important: the ``unprotected_file.pem`` file will only contain the unprotected RSA key. Any certificate in ``password_protected_file.pem`` is not copied to ``unprotected_file.pem``. This can be done manually) The following options are mandatory for all but the ``userinfo`` command: ``--rspec `` The request RSpec that specifies the sliver(s) to be created. ``--slice `` The name of the slice in which the slivers(s) need to be created. This can be an existing slice, or a slice that needs to be created. In the later case, you need to also privide the ``--create-slice`` option in order to confirm that the experimenter CLI must create the slice for you. You may choose to either provide just the slice name, or the full slice URN. Both will work. ``--project-name `` or ``-S `` In case you used the ``--slice`` option to specify the slice name instead of the URN, you might want to specifiy the project name (sometimes refered to as `sub authority`). If you do not specify a project name, no project will be provided. Some authorities do not allow this. You can also specify the special value ``CHOOSE_AUTOMATICALLY`` as project name. In that case, the CLI will contact the SA and request a list of projects you are a member of, and automatically select the last project. There are also a number of optional arguments. These are for settings for which the experimenter CLI has reasonable defaults. The arguments include: ``--expiration-hours `` This option is used to specify when the slivers should expire. If a slice is created, this is the slice expiration time as well. This argument is specified as the number of hours from the current date. The default for this option is 2 hours. Note that for both slice and sliver expiration time, the server might not honour your request, and return a longer or shorter time than what you requested. ``--ssh-keys`` This option can be used to specify which ssh public keys should be added to the nodes when creating slivers. This option takes a comma seperated list of public key source identifiers. The default value is: ``usercert,rspec,shareduserallkeys`` The 4 supported identifiers are: ``usercert`` The certificate used to authenticate contains a public key, which will be used as an ssh public key and added to created slivers. This way, you can login to the nodes using the matching private key. ``userkeys`` The server contains information on user public SSH keys. If this identifier is added, this information is requested, and the public ssh keys of the user are added when creating slivers. ``shareduserallkeys`` Add all ssh keys that are stored on the server for all users the slice is shared with. This will also request these users login certificate SSH key from the server. ``rspec`` The RSpec file can contain a list of public SSH keys. If this identifier is added, these are added when creating slivers. To add public keys to an RSpec, add this as last child element of the ```` element:: ssh-rsa AAAAB.... foo@laptop You may also need to add the correct namespace to the ```` element:: xmlns:jfed-ssh-keys="http://jfed.iminds.be/rspec/ext/jfed-ssh-keys/1" Full example:: ssh-rsa AAAAB.... foo@laptop ``--share-slice`` This options allows sharing the created or existing slice with other users. It expects a list of users, seperated by commas. Users can be specified by either the user URN, or by the short username. You can also specify the special value "PROJECTUSERS", which will automatically share the slice with all users in the project. This is nice to use together with the ``shareduserallkeys`` option of ``--ssh-keys`` (which is set by default). ``--manifest `` The file in which to store the manifest RSpec received when creating the sliver(s). This manifest typically contains details such as hostnames and SSH login details. By default, the manifest is saved in ``manifest-.rspec`` ``--nowait`` By default, the experimenter CLI will wait until the slivers are ready before stopping. If this option is specified, it will not wait, bu exit once the slivers have started initialising. (The manifest is saved in both cases). ``--call-log `` This will store detailed information about all calls to a specified file. It is an XML based format. ``--slice-recover-info `` A file into which slice recover info will be stored. This can be used by the jFed GUI (if copied to the correct directory). (This functionality is not needed anymore if combining the latest CLI and GUI versions) ``--stitching`` If the RSpec requires stitching, allow it. By default, RSpecs that require stitching cause an abort ``--rewrite-rspec`` Parse the provided RSpec, and reconstruct it again, before sending it to the server. This can help prevent some server sides bug caused by valid but atypical RSpecs. ``--bind-rspec `` Set the component_manager_urn for all "unbound" nodes in the provided request RSpec to the specified component manager. This option is allows using RSpecs that are not bound to a specific AM. ``--output-format`` The output format for the user info. Options are ``text`` and ``json``. The default is ``json``. ``--action`` and ``--target-sliver`` The options for the ``poa`` command. ``--delete-on-create-failure`` If there is a failure in a call made to create the sliver(s), delete all resources everywhere before exiting. ``--delete-on-become-ready-failure`` If there is a failure in a call while waiting for the sliver(s) to become ready, delete all resources everywhere before exiting. An example:: java -jar experimenter-cli.jar create -s slice1 -S myproject --create-slice -p mypem.pem -P mypass --rspec 1node.rspec --expiration-hours 1 Here, the user identified by `mypem.pem` create the slice `slice1` in project `myproject`, with a slice expiration of 1 hour (the server might use a longer slice expiration than requested). Then, a sliver is made, containing the node specified in `1node.rspec`. The CLI will wait until this sliver is reay, and will save the manifest RSpec in the file `manifest-slice1.rspec`. To delete the created sliver you would use the command:: java -jar experimenter-cli.jar create -s slice1 -S myproject -p mypem.pem -P mypass --rspec 1node.rspec To fetch user info (from the user in mypem.pem), you would use this command:: java -jar experimenter-cli.jar userinfo -p mypem.pem -P mypass To reload a node, you would use this command:: java -jar experimenter-cli.jar poa --action restart --target-sliver urn:publicid:IDN+wall2.ilabt.iminds.be+sliver+73775 -p mypem.pem -P mypass -s slice1 -S myproject To create a slice (and get text info instead of JSON info), you would use this command:: java -jar experimenter-cli.jar createslice --output-format text -p mypem.pem -P mypass -s slice1 -S myproject To request slice info (in JSON format), you would use this command:: java -jar experimenter-cli.jar slice-info -p mypem.pem -P mypass -s slice1 -S myproject Ansible support --------------- There are a number of options which interact with `ansible `_. The main option is used to write the ansible config files for the experiment: ``--ansible-dir `` This option is available for the ``create`` and ``manifest`` command. It writes ansible config files to the specified directory. The dir will be created if it doesn't exist. If it does exist, any existing files will be overwritten. The ansible files can be used with ansible commands, or with ansible playbooks. These can be used to create scripts that execute commands on different nodes. **Important**: In most cases (but not all), your private key will be copied to the created ansible dir. You should take the security considerations of this into account. Next to ansible config files, `Fabric `_ files and ssh config files are also created. The ssh config file enables easy access to the nodes, even when an SSH proxy needs to be used. SCP access is also a lot easier using this file. An example:: ~ $ cd ansible-files/ ~/ansible-files/$ ansible all -m ping ~/ansible-files/$ ssh -F ssh-config node0 ~/ansible-files/$ scp -F ssh-config /work/file.tgz node0:/tmp/ A number of optinos is available to automatically execute ansible playbooks with the CLI:: --ansible-add-playbook Add an ansible playbook to execute. The argument specifies the source, and optionally (after a space) the target (this option can be specified multiple times) --ansible-allow-any-playbook-outputfile Allow any local file as execute_ansible_playbook output_file. (default: only use the basename of the specified output_file, and use the --ansible-dir as directory) --ansible-allow-playbook-inputfile Allow local files in execute_ansible_playbook source. (default: only allow URLs as source) --ansible-debug Set debug flags when calling ansible playbook. --ansible-execute-rspec-playbooks Execute all ansible playbooks found within request Rspec execute_ansible_playbook elements. (default: ignore them) --ansible-playbook-exe Executable for the ansible playbook. Default is "ansible-playbook" and is usually fine. On some systems might need to be "/usr/bin/python2 /usr/bin/ansible-playbook" As can be seen, ansible playbooks can either be specified on the command line, using the `--ansible-add-playbook` option, or they can be specified inside the request RSpec file. In the last case, the playbook{s} in the RSpec are only executed if the `--ansible-execute-rspec-playbooks` options is specified. To specify playbooks in the RSpec, add the following directly under the rspec element (so not in a node or link):: The output file will be stored in the dir specified by `--ansible-dir`, unless the output file is a full path and the `--ansible-allow-any-playbook-outputfile` options is specified. The source must be an URL, unless `--ansible-allow-playbook-inputfile` is specified, in which case it is also allowed to be the path to a local file. `--ansible-playbook-exe` can be used to change the ansible command. Note that it allows spaces, so it can also be used to pass additional arguments to the ansible executable. Speaksfor credential support ---------------------------- Speaksfor credentials enable tools to execute calls on behalf of users, without having access to the user's private key or password. Speaksfor credentials are typically generated by a webpage on the user's authority, and passed to the tool. (The jFed probe can also generate speaksfor credentials, but that is not a typical use case.) The tool then includes these credentials in all calls to the AMs. If the provided speaksfor credential is valid, the AMs will execute the requested call on behalf of the user, instead of on behalf of the tool (and thus use the access rights of the user, not those of the tool). In the AM logs, it is visible everywhere that the tool speaks for the user. Note that speaksfor credentials have a limited validity, so tools cannot speak on hehalf of the user forever. Some servers support "chainded" speaksfor. In this scenario, multiple speaksfor credentials are provided. For example, if valid speaksfor credentials are privided to allow tool A to speaks for tool B, and tool B to speaks for user C. Then, the AM will allow tool A to speak for user C. The following command line option is used for speaksfor support: ``--speaks-for `` Include a speaksFor certificate in all calls, enabling you to make calls on behalf of another user. This option can be specified multiple times, and/or the provided file may contain multiple speaksfor credentials. In these cases, multiple speaksfor credential are sent to the AM. The experimenter CLI will check scenarios with multiple speaksfor credentials, if the list of credentials forms no valid chain, it will fail before making any AM calls. Command Line Help ----------------- If you do not enter any argument, the general syntax and the list of available commands will be returned. If you add a command as argument, but nothing else, specific help for the command will be returned. Help for the ``create`` command:: user@laptop ~/Downloads/jfed_cli $ java -jar experimenter-cli.jar create Command line argument Syntax error: Missing required options: s, r usage: jfed-experimenter-cli [command_options ... ] Available commands: create,delete,status,renew,userinfo,sliceinfo,createslice create options: --abort-if-slivers-exist Do not create any slivers if any sliver already exists on any of the authorities. -am,--am-api The AM Api version to use ("2" and "3" supported, default: choose automatically) --ansible-dir The dir in which to save ansible config files. Will be created if it doesn't exist. Will overwrite any existing files. --authorities-file The xml file containing the list of known authorities. Default: choose automatically --bind-rspec Bind any unbound nodes in the RSpec to a specified authority. This leaves bound nodes as they are. (this imples --rewrite-rspec) -c,--context-file The properties file containing context details (only the login info in the context file is used by this tool) --call-log A file into which all call details will be stored --clearinghouse Fetch certificates etc from geni clearinghouse first --create-slice Create the slice if it doesn't exist. (if it exists, this option is ignored) -d,--debug extra debugging output --delete-on-become-ready-failure If there is a failure in a call while waiting for the sliver(s) to become ready, delete all resources everywhere before exiting. --delete-on-create-failure If there is a failure in a call made to create the sliver(s), delete all resources everywhere before exiting. -e,--expiration-hours The number of hours after which this slice expires. This is an optional argument. default: 2 hours in the future --expiration-date The date and time at which this slice expires. In RFC3339 format. This is an optional argument. default: 2 hours in the future --fake Do everything, expect actually making the slice and sliver calls. Useful for debugging syntax. (will make calls relating to retrieving user data) --fake-sliver Do everything, expect actually making the sliver calls. (will make calls relating to retrieving user data and creating or retrieving slice) -k,--ssh-keys