Connect Lightning nodes on regtest with CLN

We are in the terminal, in the lightning repository, we source the script contrib/startup_regtest.sh that we have already used in the previous episodes. This script provides commands and aliases to startup local nodes with bitcoind all running on regtest. This is useful to test things out, by hand:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ source contrib/startup_regtest.sh
lightning-cli is /usr/bin/lightning-cli
lightningd is /usr/bin/lightningd

First we start two nodes running on regtest by running the following command:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ start_ln 2
Bitcoin Core starting
awaiting bitcoind...
Making "default" bitcoind wallet.
[1] 25354
[2] 25388
Commands:
        l1-cli, l1-log,
        l2-cli, l2-log,
        bt-cli, stop_ln, fund_nodes

We see that Bitcoin Core has been started with one wallet called default and we now have access to new commands.

For instance bt-cli is an alias for bitcoin-cli -regtest that we can use to check that we are indeed running bitcoind on regtest:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ bt-cli -getinfo
Chain: regtest
Blocks: 1
Headers: 1
Verification progress: 100.0000%
Difficulty: 4.656542373906925e-10

Network: in 0, out 0, total 0
Version: 230000
Time offset (s): 0
Proxies: n/a
Min tx relay fee rate (BTC/kvB): 0.00001000

Wallet: default
Keypool size: 4000
Transaction fee rate (-paytxfee) (BTC/kvB): 0.00000000

Balance: 0.00000000

Warnings:

We can see that the default wallet has no coins in its balance.

And since the Lightning nodes don't need to be funded to connect to each other we won't mine any blocks and the balance of the default wallet will remain zero.

To avoid using the l1-cli and l2-cli aliases blindly, let's print out what they are:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ alias | grep 'l[12]'
alias l1-cli='lightning-cli --lightning-dir=/tmp/l1-regtest'
alias l1-log='less /tmp/l1-regtest/log'
alias l2-cli='lightning-cli --lightning-dir=/tmp/l2-regtest'
alias l2-log='less /tmp/l2-regtest/log'

So far we just checked that the command startup_ln started bitcoind.

By running the following command we can also checked that two instances of lightningd have been started, one with the base directory /tmp/l1-regtest/ and the other with /tmp/l2-regtest/:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ ps -ax | grep lightningd
25357 pts/1    S      0:00 lightningd --lightning-dir=/tmp/l1-regtest
25390 pts/1    S      0:00 lightningd --lightning-dir=/tmp/l2-regtest
27326 pts/1    S+     0:00 grep --color=auto lightningd

To connect two nodes, CLN provides the sub-command connect.

We can read in its man page that to connect the node l1 to the node l2 we can passe the argument id@host:port to the command line l1-cli connect where id is the public key of the node l2:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l1-cli help connect
LIGHTNING-CONNECT(7)                                                    LIGHTNING-CONNECT(7)

NAME
       lightning-connect -- Command for connecting to another lightning node

SYNOPSIS
       connect id [host] [port]

DESCRIPTION
       The  connect RPC command establishes a new connection with another node in the Light-
       ning Network.

       id represents the target node's public key. As a convenience, id may be of  the  form
       id@host or id@host:port. In this case, the host and port parameters must be omitted.

       host is the peer's hostname or IP address.

       If  not  specified, the port depends on the current network: - bitcoin mainnet: 9735.
       - bitcoin testnet: 19735.  - bitcoin signet: 39735.  - bitcoin regtest: 19846.

       If host is not specified (or doesn't work), the connection will be attempted to an IP
       belonging to id obtained through gossip with other already connected peers.  This can
       fail if your C-lightning node is a fresh install that has not connected to any  peers

We get those information using the sub-command getinfo, looking at the attribute id and the attributes address and port of the first object in the array binding in the output produced:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l2-cli getinfo
{
  "id": "037493c5b6af6f23bed9d5f5d3d3583c47941bc20ef6c019a21e47d5992da912c4",
  "alias": "UNITEDTOTE",
  "color": "037493",
  "num_peers": 0,
  "num_pending_channels": 0,
  "num_active_channels": 0,
  "num_inactive_channels": 0,
  "address": [],
  "binding": [
    {
      "type": "ipv4",
      "address": "127.0.0.1",
      "port": 7272
    }
  ],
  "version": "0.12.1",
  "blockheight": 1,
  "network": "regtest",
  "fees_collected_msat": 0,
  "lightning-dir": "/tmp/l2-regtest/regtest",
  "our_features": {
    "init": "08a000080269a2",
    "node": "88a000080269a2",
    "channel": "",
    "invoice": "02000000024100"
  }
}

Then we can construct the argument id@host:port for the node l2 and connect the nodes by running the following command:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l1-cli connect 037493c5b6af6f23bed9d5f5d3d3583c47941bc20ef6c019a21e47d5992da912c4@127.0.0.1:7272
{
   "id": "037493c5b6af6f23bed9d5f5d3d3583c47941bc20ef6c019a21e47d5992da912c4",
   "features": "08a000080269a2",
   "direction": "out",
   "address": {
      "type": "ipv4",
      "address": "127.0.0.1",
      "port": 7272
   }
}

In the object returned the value 08a000080269a2 of the features attribute is the BOLT 9 features bitmap offered by the peer (node l2), and the out in the attribute direction signals that we (the node l1) initiated the connection.

To check that we the nodes l1 and l2 are connected we can use the sub-command listpeers either from the perspective of l1 or l2 as follow:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l1-cli listpeers
{
   "peers": [
      {
         "id": "037493c5b6af6f23bed9d5f5d3d3583c47941bc20ef6c019a21e47d5992da912c4",
         "connected": true,
         "netaddr": [
            "127.0.0.1:7272"
         ],
         "features": "08a000080269a2",
         "channels": []
      }
   ]
}
◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l2-cli listpeers
{
   "peers": [
      {
         "id": "02695a3a6ae6ff038407f4bdebe82d7ac7a31d7edab1c4dbaa885e1aa2ba9bacc2",
         "connected": true,
         "netaddr": [
            "127.0.0.1:45348"
         ],
         "features": "08a000080269a2",
         "channels": []
      }
   ]
}

The last thing we can do before we move on to the next part about jq utility, we can disconnect the nodes l1 and l2 with the sub-command disconnect.

As we can read in its man page, we just have to pass the id of the node l2 to the command l1-cli disconnect to disconnect since the node l1 doesn't have an active channel with the node l2:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l1-cli help disconnect
LIGHTNING-DISCONNECT(7)                                              LIGHTNING-DISCONNECT(7)

NAME
       lightning-disconnect -- Command for disconnecting from another lightning node

SYNOPSIS
       disconnect id [force]

DESCRIPTION
       The disconnect RPC command closes an existing connection to a peer, identified by id,
       in the Lightning Network, as long as it doesn't have an active channel. If  force  is
       set then it will disconnect even with an active channel.

       The  id can be discovered in the output of the listpeers command, which returns a set
       of peers:

       {
            "peers": [
                 {
                      "id": "0563aea81...",
                      "connected": true,
                      ...
                 }

We get the id of the node l2 using jq:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l2-cli getinfo | jq .id
"037493c5b6af6f23bed9d5f5d3d3583c47941bc20ef6c019a21e47d5992da912c4"

and disconnect the two nodes by running the following command:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l1-cli disconnect 037493c5b6af6f23bed9d5f5d3d3583c47941bc20ef6c019a21e47d5992da912c4
{}

The two nodes have been disconnected and we can check this out by listing the peers of both nodes like this:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l2-cli listpeers
{
   "peers": []
}
◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l1-cli listpeers
{
   "peers": []
}

We are done with the first part of this video.

In this second part we introduce jq utility and see how we can use it to reproduce one of the command line used in the command connect provided in the script contrib/startup_regtest.sh.

Here is the command connect:

connect() {
    if [ -z "$1" ] || [ -z "$2" ]; then
        printf "usage: connect 1 2\n"
    else
        to=$($LCLI --lightning-dir="/tmp/l$2-$network" -F getinfo \
                 | grep '^\(id\|binding\[0\]\.\(address\|port\)\)' \
                 | cut -d= -f2- \
                 | tr '\n' ' ' \
                 | (read -r ID ADDR PORT; echo "$ID@${ADDR}:$PORT"))
        $LCLI --lightning-dir="/tmp/l$1-$network" connect "$to"
    fi
}

Using it in the terminal like this:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ connect 1 2

has the same effect (connecting the node l1 and l2) as we did before when we ran this command:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l1-cli connect 037493c5b6af6f23bed9d5f5d3d3583c47941bc20ef6c019a21e47d5992da912c4@127.0.0.1:7272

The part in connect we want to play with is to=$(...).

In our case, the variable $LCLI is equal to lightning-cli , $network to regtest and in the call connect 1 2 the variable $2 is 'equal' to 2, so the command line

$LCLI --lightning-dir="/tmp/l$2-$network" -F getinfo

is 'equal' to

lightning-cli --lightning-dir="/tmp/l2-regtest/" -F getinfo

and if we use our alias l2-cli, this command is 'equal' to:

l2-cli -F getinfo

The -F (or --flat) flag returns the JSON result in a one-per-line output as we can see in the following snippet for the node l1:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l1-cli getinfo -F
id=02695a3a6ae6ff038407f4bdebe82d7ac7a31d7edab1c4dbaa885e1aa2ba9bacc2
alias=SLEEPYFEED
color=02695a
num_peers=0
num_pending_channels=0
num_active_channels=0
num_inactive_channels=0
binding[0].type=ipv4
binding[0].address=127.0.0.1
binding[0].port=7171
version=0.12.1
blockheight=1
network=regtest
fees_collected_msat=0
lightning-dir=/tmp/l1-regtest/regtest
our_features.init=08a000080269a2
our_features.node=88a000080269a2
our_features.channel=
our_features.invoice=02000000024100

We can compare that flatten output with the JSON object in the following snippet:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l1-cli getinfo
{
   "id": "02695a3a6ae6ff038407f4bdebe82d7ac7a31d7edab1c4dbaa885e1aa2ba9bacc2",
   "alias": "SLEEPYFEED",
   "color": "02695a",
   "num_peers": 0,
   "num_pending_channels": 0,
   "num_active_channels": 0,
   "num_inactive_channels": 0,
   "address": [],
   "binding": [
      {
         "type": "ipv4",
         "address": "127.0.0.1",
         "port": 7171
      }
   ],
   "version": "0.12.1",
   "blockheight": 1,
   "network": "regtest",
   "fees_collected_msat": 0,
   "lightning-dir": "/tmp/l1-regtest/regtest",
   "our_features": {
      "init": "08a000080269a2",
      "node": "88a000080269a2",
      "channel": "",
      "invoice": "02000000024100"
   }
}

The use of the -F flag is convenient when we want to pipe the output to existing Unix utilities, which are well suited for line processing like grep and that use with field delimiter like cut.

For instance, the grep part in the line to=$(...) to the previous flatten output allows to keep only the information id, address and port of a node. We can see it for the node l1 by running the following command:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l1-cli getinfo -F | grep '^\(id\|binding\[0\]\.\(address\|port\)\)'
id=02695a3a6ae6ff038407f4bdebe82d7ac7a31d7edab1c4dbaa885e1aa2ba9bacc2
binding[0].address=127.0.0.1
binding[0].port=7171

The rest of the line in to=$(...) uses cut, tr, read and echo to produce an output of the form id@host:port that is bound to the variable to.

Let's see how we can do this for the node l2 only using jq:

We can get the id of the node l2 using the filter .id and with the flag -r (or --raw-output) in the case of a string the output is not formatted as JSON string and the double quote are removed from the output:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l2-cli getinfo | jq .id
"037493c5b6af6f23bed9d5f5d3d3583c47941bc20ef6c019a21e47d5992da912c4"
◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l2-cli getinfo | jq .id -r
037493c5b6af6f23bed9d5f5d3d3583c47941bc20ef6c019a21e47d5992da912c4

We can get the the address (i.e., the host) defined in the first object of the array binding with the filter .binding[0].address like this:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l2-cli getinfo | jq .binding[0].address
"127.0.0.1"

In the same vein we can get the port of the node l2 like this:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l2-cli getinfo | jq .binding[0].port
7272

Pay attention to the previous output which is not surrounded by double quote which means that 7272 is a number in jq program. As jq doesn't do implicit cast, we can convert it to a string by passing the output of the filter .binding[0].port to the operator tostring using the pipe | operator like this:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l2-cli getinfo | jq '.binding[0].port | tostring'
"7272"

jq provides the operator + that in the case of two strings join them into a larger string as we can see in the following command where we add @ character to the node's id:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l2-cli getinfo | jq '.id + "@"'
"037493c5b6af6f23bed9d5f5d3d3583c47941bc20ef6c019a21e47d5992da912c4@"

Now, if we put together all we've seen about jq so far we can produce an output of the form id@host:port for the node l2 like this:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l2-cli getinfo | jq '.id + "@" + .binding[0].address + ":" + (.binding[0].port | tostring)'
"037493c5b6af6f23bed9d5f5d3d3583c47941bc20ef6c019a21e47d5992da912c4@127.0.0.1:7272"

We can now connect the node l1 to the node l2 like this:

◉ tony@tony:~/lnroom/lightning:[git»(HEAD detached at v0.12.1)]
$ l1-cli connect 037493c5b6af6f23bed9d5f5d3d3583c47941bc20ef6c019a21e47d5992da912c4@127.0.0.1:7
272
{
   "id": "037493c5b6af6f23bed9d5f5d3d3583c47941bc20ef6c019a21e47d5992da912c4",
   "features": "08a000080269a2",
   "direction": "out",
   "address": {
      "type": "ipv4",
      "address": "127.0.0.1",
      "port": 7272
   }
}

This is what I wanted to do with you in the last part.

jq is an amazing tool that you might want to integrate in your toolkit. Its man page is incredible. Taking 5 to 6 hours to read the man page and try things will pay off in the short and long term. Give it a try.

That's all I have to share with you today.

I hope you get some value from this video.

See you next time in the next episode of LN Room.