Subscribe to lightningd notification topics with a Python plugin

Here is my setup:

◉ tony@tony:~/clnlive:
$ ./setup.sh
Ubuntu 22.04.2 LTS
Python 3.10.6
lightningd v23.02.2

Let's start two Lightning nodes running on the Bitcoin regtest chain by sourcing the script lightning/contrib/startup_regtest.sh provided in CLN repository and by running the command start_ln:

◉ tony@tony:~/clnlive:
$ source lightning/contrib/startup_regtest.sh
lightning-cli is /usr/local/bin/lightning-cli
lightningd is /usr/local/bin/lightningd
Useful commands:
  start_ln 3: start three nodes, l1, l2, l3
  connect 1 2: connect l1 and l2
  fund_nodes: connect all nodes with channels, in a row
  stop_ln: shutdown
  destroy_ln: remove ln directories
◉ tony@tony:~/clnlive:
$ start_ln
Bitcoin Core starting
awaiting bitcoind...
Making "default" bitcoind wallet.
[1] 108917
[2] 108952
WARNING: eatmydata not found: instal it for faster testing
Commands:
        l1-cli, l1-log,
        l2-cli, l2-log,
        bt-cli, stop_ln, fund_nodes

We can check that l1-cli is just an alias for lightning-cli with the base directory being /tmp/l1-regtest:

◉ tony@tony:~/clnlive:
$ alias l1-cli
alias l1-cli='lightning-cli --lightning-dir=/tmp/l1-regtest'

Instead of writing the plugin from scratch we use parts of the code we wrote during the first live which was about registering JSON-RPC methods to lightningd and understanding CLN plugin system.

This way we can focus on how to subscribe to notification topics and how to handle notifications.

So we start with the file myplugin.py containing the following:

#!/usr/bin/env python

import sys
import json

# getmanifest
request = sys.stdin.readline()
sys.stdin.readline() # "\n"
req_id = json.loads(request)["id"]

manifest = {
    "jsonrpc": "2.0",
    "id": req_id,
    "result": {
        "dynamic": True,
        "options": [],
        "rpcmethods": []
    }
}

sys.stdout.write(json.dumps(manifest))
sys.stdout.flush()

# init
request = sys.stdin.readline()
sys.stdin.readline() # "\n"
req_id = json.loads(request)["id"]

init = {
    "jsonrpc": "2.0",
    "id": req_id,
    "result": {}
}

sys.stdout.write(json.dumps(init))
sys.stdout.flush()

# io loop

for request in sys.stdin:
    sys.stdin.readline() # "\n"

In that script, we first receive the getmanifest request from lightningd in our stdin stream, we extract its id and we construct the getmanifest response (the plugin is dynamic, with no startup options and register no JSON-RPC methods) that we send back to lightningd by writing it to our stdout stream:

...
# getmanifest
request = sys.stdin.readline()
sys.stdin.readline() # "\n"
req_id = json.loads(request)["id"]

manifest = {
    "jsonrpc": "2.0",
    "id": req_id,
    "result": {
        "dynamic": True,
        "options": [],
        "rpcmethods": []
    }
}

sys.stdout.write(json.dumps(manifest))
sys.stdout.flush()
...

Then we handle the init request sent by lightningd in our stdin stream:

...
# init
request = sys.stdin.readline()
sys.stdin.readline() # "\n"
req_id = json.loads(request)["id"]

init = {
    "jsonrpc": "2.0",
    "id": req_id,
    "result": {}
}

sys.stdout.write(json.dumps(init))
sys.stdout.flush()
...

And finally we start an I/O loop waiting for incoming request from lightningd:

...
# io loop

for request in sys.stdin:
    sys.stdin.readline() # "\n"

Let's subscribe to connect notification topic by adding it to the array subscriptions part of the result field of the getmanifest response:

...
manifest = {
    "jsonrpc": "2.0",
    "id": req_id,
    "result": {
        "dynamic": True,
        "options": [],
        "rpcmethods": [],
        "subscriptions": ["connect"]
    }
}
...

And when we'll receive notifications for that topic, we'll write them into the file /tmp/myplugin like this:

for request in sys.stdin:
    sys.stdin.readline() # "\n"

    with open("/tmp/myplugin", "a") as myplugin:
        myplugin.write("connect notification: " + request + "\n")

Note that we don't need to write any logic for now because the only notifications we'll ever receive from lightningd are for connect topic (due to our getmanifest response).

Note also that since we are handling notifications, we don't send any responses to lightningd unlike what we did before with getmanifest and init requests.

In our terminal now we can start our plugin and connect l1 and l2 nodes using connect command provided by lightning/contrib/startup_regtest.sh script:

$ l1-cli plugin start $(pwd)/myplugin.py
{
   "command": "start",
   "plugins": [
      {
         "name": "/usr/local/libexec/c-lightning/plugins/autoclean",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/chanbackup",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/bcli",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/commando",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/funder",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/topology",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/keysend",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/offers",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/pay",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/txprepare",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/spenderp",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/sql",
         "active": true,
         "dynamic": true
      },
      {
         "name": "/usr/local/libexec/c-lightning/plugins/bookkeeper",
         "active": true,
         "dynamic": false
      },
      {
         "name": "/home/tony/clnlive/myplugin.py",
         "active": true,
         "dynamic": true
      }
   ]
}
◉ tony@tony:~/clnlive:
$ connect 1 2
{
   "id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
   "features": "08a000080269a2",
   "direction": "out",
   "address": {
      "type": "ipv4",
      "address": "127.0.0.1",
      "port": 7272
   }
}

As the node l1 is running myplugin plugin which subscribes to connect notification topic and write them to the file /tmp/plugin each time l1 connects to another node, and that l1 has just connected to l2, the file /tmp/plugin contains the following connect notification:

connect notification: {"jsonrpc":"2.0","method":"connect","params":{"id":"0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf","direction":"out","address":{"type":"ipv4","address":"127.0.0.1","port":7272}}}

We can prettify that notification like this:

{
  "jsonrpc": "2.0",
  "method": "connect",
  "params": {
    "id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
    "direction": "out",
    "address": {
      "type": "ipv4",
      "address": "127.0.0.1",
      "port": 7272
    }
  }
}

Note that the request from lightningd being a notification has no id.

With that done, now we know how to subscribe to a notification topic in Core Lightning.

To summarize, in the getmanifest response we add the notification topics we want to subscribe too in the array subscriptions of the params field, then each time we receive a notification for one of those topics we handle them without relpying back to lightningd.

Let's subscribe to disconnect notification topic by adding it to the array subscriptions part of the result field of the getmanifest response:

...
manifest = {
    "jsonrpc": "2.0",
    "id": req_id,
    "result": {
        "dynamic": True,
        "options": [],
        "rpcmethods": [],
        "subscriptions": ["connect", "disconnect"]
    }
}
...

Now we add some logic in the I/O loop to dispatch on the notification topics. To do this we extract the method (the notification topic) of the notification we receive from lightningd and we dispatch on it like this:

for request in sys.stdin:
    sys.stdin.readline() # "\n"

    method = json.loads(request)["method"]
    if method == "connect":
        with open("/tmp/myplugin", "a") as myplugin:
            myplugin.write("connect notification: " + request + "\n")

    if method == "disconnect":
        with open("/tmp/myplugin", "a") as myplugin:
            myplugin.write("disconnect notification: " + request + "\n")

In our terminal, first we disconnect the nodes l1 and l2

◉ tony@tony:~/clnlive:
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
{}

then we restart the plugin

◉ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "command": "start",
   "plugins": [...]
}

Note that since the checksum of our plugin script as change, using start subcommand of plugin command stops myplugin.py plugin that was running and restart it with the new modifications.

Then we connect again both nodes and disconnect them immediately:

◉ tony@tony:~/clnlive:
$ connect 1 2
{
   "id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
   "features": "08a000080269a2",
   "direction": "out",
   "address": {
      "type": "ipv4",
      "address": "127.0.0.1",
      "port": 7272
   }
}

The plugin myplugin.py has been notified twice: once for connect notification topic and once for disconnect notification topic.

Therefore, the file /tmp/myplugin is now:

connect notification: {"jsonrpc":"2.0","method":"connect","params":{"id":"0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf","direction":"out","address":{"type":"ipv4","address":"127.0.0.1","port":7272}}}

connect notification: {"jsonrpc":"2.0","method":"connect","params":{"id":"0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf","direction":"out","address":{"type":"ipv4","address":"127.0.0.1","port":7272}}}

disconnect notification: {"jsonrpc":"2.0","method":"disconnect","params":{"id":"0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf"}}

We are done for that first part.