Learn how to subscribe to lightningd event notifications with CLN plugins
In the first part of this live we write a Python plugin without pyln-client package which subscribes to the notification topics connect and disconnect. A benefit of doing it like this is that it allows us to understand how the system works. Another benefit is that the way we write the plugin can be applied to any other languages. In the second part we do it again with Python but this time using pyln-client package.
Transcript with corrections and improvements
What is a JSON-RPC notification
In JSON-RPC 2.0 Specification we can read that
A notification is a Request object without an "id" member.
so that a notification looks like this
{
"jsonrpc": "2.0",
"method": "foo",
"params": {
"bar": "baz"
}
}
and we can also read that
The Server MUST NOT reply to a Notification [...]
Interesting!
What about Core Lightning?
Core Lightning push-based notification mechanism
Core Lightning provides plugins with a push-based notification
mechanism about events from lightningd.
Meaning that plugins can "ask" lightningd to be notified when some
events happened in lightningd by subscribing to the corresponding
notification topics. Once they have subscribed, plugins will be
notified each time those events occur. And when they receive a
notification, plugins handle it without replying back to lightningd.
Ok, but how does a plugin subscribe to a notification topic?
When a plugin is started by lightningd, the plugin receives a
getmanifest request and an init request before starting an I/O loop
waiting for incoming request from lightningd.
A plugin subscribes to notification topics by adding them to the field
subscriptions in the params field of the response to the getmanifest
request from lightningd like this:
{
"jsonrpc": "2.0",
"id": ...,
"result": {
"dynamic": True,
"options": [...],
"rpcmethods": [...],
"subscriptions": ["topic_1", "topic_2"]
}
}
The current notification topics are: channel_opened,
channel_open_failed, channel_state_changed, connect, disconnect,
invoice_payment, invoice_creation, warning, forward_event,
listforwards, sendpay_success, sendpay_failure, coin_movement,
block_added, openchannel_peer_sigs, shutdown.
So for instance, if a plugin wants to be notified when his node connects
or disconnects to another node (and nothing else), the plugin subscribes
to the notification topics connect and disconnect by replying to the
following getmanifest request (sent by lightningd)
{
"jsonrpc": "2.0",
"id": 187,
"method": "getmanifest",
"params": {
"allow-deprecated-apis": false
}
}
with that response:
{
"jsonrpc": "2.0",
"id": 187,
"result": {
"dynamic": True,
"options": [],
"rpcmethods": [],
"subscriptions": ["connect", "disconnect"]
}
}
Once the plugin is started, if the node running the plugin connects to the
node
0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf,
the plugin will be notified with a notification that looks like this:
{
"jsonrpc": "2.0",
"method": "connect",
"params": {
"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
"direction": "out",
"address": {
"type": "ipv4",
"address": "127.0.0.1",
"port": 7272
}
}
}
The plugin can then handle that notification without replying back to
lightningd.
Subscribe to connect and disconnect lightningd notification topics with a Python plugin
Let's write a Python plugin without pyln-client package which
subscribes to the notification topics connect and disconnect.
Setup
Here is my setup:
◉ tony@tony:~/lnroom:
$ ./setup.sh
Ubuntu 22.04.2 LTS
Python 3.10.6
lightningd v23.02.2
Start 2 Lightning nodes running on regtest
Let's start two Lightning nodes running on the Bitcoin regtest chain
by sourcing the script lightning/contrib/startup_regtest.sh provided
in CLN repository and by running the command start_ln:
◉ tony@tony:~/clnlive:
$ source lightning/contrib/startup_regtest.sh
lightning-cli is /usr/local/bin/lightning-cli
lightningd is /usr/local/bin/lightningd
Useful commands:
start_ln 3: start three nodes, l1, l2, l3
connect 1 2: connect l1 and l2
fund_nodes: connect all nodes with channels, in a row
stop_ln: shutdown
destroy_ln: remove ln directories
◉ tony@tony:~/clnlive:
$ start_ln
Bitcoin Core starting
awaiting bitcoind...
Making "default" bitcoind wallet.
[1] 108917
[2] 108952
WARNING: eatmydata not found: instal it for faster testing
Commands:
l1-cli, l1-log,
l2-cli, l2-log,
bt-cli, stop_ln, fund_nodes
We can check that l1-cli is just an alias for lightning-cli with the
base directory being /tmp/l1-regtest:
◉ tony@tony:~/clnlive:
$ alias l1-cli
alias l1-cli='lightning-cli --lightning-dir=/tmp/l1-regtest'
myplugin.py
Instead of writing the plugin from scratch we use parts of the code we
wrote during the first live which was about registering JSON-RPC
methods to lightningd and understanding CLN plugin system.
This way we can focus on how to subscribe to notification topics and how to handle notifications.
So we start with the file myplugin.py containing the following:
#!/usr/bin/env python
import sys
import json
# getmanifest
request = sys.stdin.readline()
sys.stdin.readline() # "\n"
req_id = json.loads(request)["id"]
manifest = {
"jsonrpc": "2.0",
"id": req_id,
"result": {
"dynamic": True,
"options": [],
"rpcmethods": []
}
}
sys.stdout.write(json.dumps(manifest))
sys.stdout.flush()
# init
request = sys.stdin.readline()
sys.stdin.readline() # "\n"
req_id = json.loads(request)["id"]
init = {
"jsonrpc": "2.0",
"id": req_id,
"result": {}
}
sys.stdout.write(json.dumps(init))
sys.stdout.flush()
# io loop
for request in sys.stdin:
sys.stdin.readline() # "\n"
In that script, we first receive the getmanifest request from
lightningd in our stdin stream, we extract its id and we construct the
getmanifest response (the plugin is dynamic, with no startup options and
register no JSON-RPC methods) that we send back to lightningd by
writing it to our stdout stream:
...
# getmanifest
request = sys.stdin.readline()
sys.stdin.readline() # "\n"
req_id = json.loads(request)["id"]
manifest = {
"jsonrpc": "2.0",
"id": req_id,
"result": {
"dynamic": True,
"options": [],
"rpcmethods": []
}
}
sys.stdout.write(json.dumps(manifest))
sys.stdout.flush()
...
Then we handle the init request sent by lightningd in our stdin
stream:
...
# init
request = sys.stdin.readline()
sys.stdin.readline() # "\n"
req_id = json.loads(request)["id"]
init = {
"jsonrpc": "2.0",
"id": req_id,
"result": {}
}
sys.stdout.write(json.dumps(init))
sys.stdout.flush()
...
And finally we start an I/O loop waiting for incoming request from
lightningd:
...
# io loop
for request in sys.stdin:
sys.stdin.readline() # "\n"
subscribing to connect
Let's subscribe to connect notification topic by adding it to the
array subscriptions part of the result field of the getmanifest
response:
...
manifest = {
"jsonrpc": "2.0",
"id": req_id,
"result": {
"dynamic": True,
"options": [],
"rpcmethods": [],
"subscriptions": ["connect"]
}
}
...
And when we'll receive notifications for that topic, we'll write them
into the file /tmp/myplugin like this:
for request in sys.stdin:
sys.stdin.readline() # "\n"
with open("/tmp/myplugin", "a") as myplugin:
myplugin.write("connect notification: " + request + "\n")
Note that we don't need to write any logic for now because the only
notifications we'll ever receive from lightningd are for connect
topic (due to our getmanifest response).
Note also that since we are handling notifications, we don't send any
responses to lightningd unlike what we did before with getmanifest and
init requests.
In our terminal now we can start our plugin and connect l1 and l2 nodes
using connect command provided by lightning/contrib/startup_regtest.sh
script:
$ l1-cli plugin start $(pwd)/myplugin.py
{
"command": "start",
"plugins": [
{
"name": "/usr/local/libexec/c-lightning/plugins/autoclean",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/chanbackup",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/bcli",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/commando",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/funder",
"active": true,
"dynamic": true
},
{
"name": "/usr/local/libexec/c-lightning/plugins/topology",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/keysend",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/offers",
"active": true,
"dynamic": true
},
{
"name": "/usr/local/libexec/c-lightning/plugins/pay",
"active": true,
"dynamic": true
},
{
"name": "/usr/local/libexec/c-lightning/plugins/txprepare",
"active": true,
"dynamic": true
},
{
"name": "/usr/local/libexec/c-lightning/plugins/spenderp",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/sql",
"active": true,
"dynamic": true
},
{
"name": "/usr/local/libexec/c-lightning/plugins/bookkeeper",
"active": true,
"dynamic": false
},
{
"name": "/home/tony/clnlive/myplugin.py",
"active": true,
"dynamic": true
}
]
}
◉ tony@tony:~/clnlive:
$ connect 1 2
{
"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
"features": "08a000080269a2",
"direction": "out",
"address": {
"type": "ipv4",
"address": "127.0.0.1",
"port": 7272
}
}
As the node l1 is running myplugin plugin which subscribes to connect
notification topic and write them to the file /tmp/plugin each time l1
connects to another node, and that l1 has just connected to l2, the
file /tmp/plugin contains the following connect notification:
connect notification: {"jsonrpc":"2.0","method":"connect","params":{"id":"0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf","direction":"out","address":{"type":"ipv4","address":"127.0.0.1","port":7272}}}
We can prettify that notification like this:
{
"jsonrpc": "2.0",
"method": "connect",
"params": {
"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
"direction": "out",
"address": {
"type": "ipv4",
"address": "127.0.0.1",
"port": 7272
}
}
}
Note that the request from lightningd being a notification has no
id.
With that done, now we know how to subscribe to a notification topic in Core Lightning.
To summarize, in the getmanifest response we add the notification
topics we want to subscribe too in the array subscriptions of the
params field, then each time we receive a notification for one of
those topics we handle them without relpying back to lightningd.
subscribing to disconnect
Let's subscribe to disconnect notification topic by adding it to the
array subscriptions part of the result field of the getmanifest
response:
...
manifest = {
"jsonrpc": "2.0",
"id": req_id,
"result": {
"dynamic": True,
"options": [],
"rpcmethods": [],
"subscriptions": ["connect", "disconnect"]
}
}
...
Now we add some logic in the I/O loop to dispatch on the notification
topics. To do this we extract the method (the notification topic) of
the notification we receive from lightningd and we dispatch on it like
this:
for request in sys.stdin:
sys.stdin.readline() # "\n"
method = json.loads(request)["method"]
if method == "connect":
with open("/tmp/myplugin", "a") as myplugin:
myplugin.write("connect notification: " + request + "\n")
if method == "disconnect":
with open("/tmp/myplugin", "a") as myplugin:
myplugin.write("disconnect notification: " + request + "\n")
In our terminal, first we disconnect the nodes l1 and l2
◉ tony@tony:~/clnlive:
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
{}
then we restart the plugin
◉ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin.py
{
"command": "start",
"plugins": [...]
}
Note that since the checksum of our plugin script as change, using
start subcommand of plugin command stops myplugin.py plugin that was
running and restart it with the new modifications.
Then we connect again both nodes and disconnect them immediately:
◉ tony@tony:~/clnlive:
$ connect 1 2
{
"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
"features": "08a000080269a2",
"direction": "out",
"address": {
"type": "ipv4",
"address": "127.0.0.1",
"port": 7272
}
}
The plugin myplugin.py has been notified twice: once for connect
notification topic and once for disconnect notification topic.
Therefore, the file /tmp/myplugin is now:
connect notification: {"jsonrpc":"2.0","method":"connect","params":{"id":"0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf","direction":"out","address":{"type":"ipv4","address":"127.0.0.1","port":7272}}}
connect notification: {"jsonrpc":"2.0","method":"connect","params":{"id":"0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf","direction":"out","address":{"type":"ipv4","address":"127.0.0.1","port":7272}}}
disconnect notification: {"jsonrpc":"2.0","method":"disconnect","params":{"id":"0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf"}}
We are done for that first part.
Subscribe to connect, disconnect and invoice_creation lightningd notification topics with a pyln-client
Let's write a Python plugin with pyln-client package which
subscribes to connect, disconnect and invoice_creation notification
topics.
Install pyln-client
Let's stop the node l1, install pyln-client in .venv python
environment and restart it after activating .venv environment:
◉ tony@tony:~/clnlive:
$ l1-cli stop
"Shutdown complete"
[1]- Done test -f "/tmp/l$i-$network/lightningd-$network.pid" || $EATMYDATA "$LIGHTNINGD" "--lightning-dir=/tmp/l$i-$network"
◉ tony@tony:~/clnlive:
$ python -m venv .venv
◉ tony@tony:~/clnlive:
$ source .venv/bin/activate
(.venv) ◉ tony@tony:~/clnlive:
$ pip install pyln-client
...
(.venv) ◉ tony@tony:~/clnlive:
$ lightningd --lightning-dir=/tmp/l1-regtest --daemon
Subscribe to connect
In the file myplugin-pyln.py we have a working plugin written with
pyln-client (thought it does nothing so far):
#!/usr/bin/env python
from pyln.client import Plugin
import json
plugin = Plugin()
plugin.run()
We use the line @plugin.subscribe("connect") to tell pyln-client that
we want to subscribe to connect notification topic. And the function
defined after that line will be used to handle connect notification
received from lightningd. In our case we write the id of the node we
connect too in the file /tmp/plugin:
So our plugin is now:
#!/usr/bin/env python
from pyln.client import Plugin
import json
plugin = Plugin()
@plugin.subscribe("connect")
def connect_func(plugin, id, direction, address , **kwargs):
with open("/tmp/myplugin-pyln", "a") as myplugin:
myplugin.write("connect notification: " + id + "\n")
plugin.run()
Let's start our pyln-client plugin and connect the node l1 and l2:
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
{
"command": "start",
"plugins": [...]
}
(.venv) ◉ tony@tony:~/clnlive:
$ connect 1 2
{
"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
"features": "08a000080269a2",
"direction": "out",
"address": {
"type": "ipv4",
"address": "127.0.0.1",
"port": 7272
}
}
We check that the file /tmp/plugin-pyln contains
connect notification: 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
indicating that we've successfully subscribed to connect notification
topic.
Let's modify our plugin to return the params object of the connect
notification:
#!/usr/bin/env python
from pyln.client import Plugin
import json
plugin = Plugin()
@plugin.subscribe("connect")
def connect_func(plugin, id, direction, address , **kwargs):
notif = {
"id": id,
"direction": direction,
"address": address
}
with open("/tmp/myplugin-pyln", "a") as myplugin:
myplugin.write("connect notification: " + json.dumps(notif) + "\n")
plugin.run()
Note how pyln-client passes the argument to connect_func. Each first
fields in the params object of a notification request like the
following connect request
{
"jsonrpc": "2.0",
"method": "connect",
"params": {
"id": "...",
"direction": "...",
"address": {...}
}
}
are bound to their value and pass as argument to the function
(connect_func in our case) defined below @plugin.subscribe("...").
Back to our terminal, we disconnect the nodes, restart the plugin and connect them again:
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
{}
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
{
"command": "start",
"plugins": [...]
}
(.venv) ◉ tony@tony:~/clnlive:
$ connect 1 2
{
"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
"features": "08a000080269a2",
"direction": "out",
"address": {
"type": "ipv4",
"address": "127.0.0.1",
"port": 7272
}
}
The file /tmp/plugin-pyln is now:
connect notification: 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
connect notification: {"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf", "direction": "out", "address": {"type": "ipv4", "address": "127.0.0.1", "port": 7272}}
pyln-client logs the Traceback
It is easier to debug our code when we have access to the traceback.
And pyln-client makes it easy by logging in the node's log the
Traceback when an error is caught during the notification handling.
Let's see that by introducing a type error in our plugin like this:
#!/usr/bin/env python
from pyln.client import Plugin
import json
plugin = Plugin()
@plugin.subscribe("connect")
def connect_func(plugin, id, direction, address , **kwargs):
1 + "foo"
plugin.run()
Back to our terminal, we disconnect the nodes, restart the plugin and connect them again:
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
{}
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
{
"command": "start",
"plugins": [...]
}
(.venv) ◉ tony@tony:~/clnlive:
$ connect 1 2
{
"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
"features": "08a000080269a2",
"direction": "out",
"address": {
"type": "ipv4",
"address": "127.0.0.1",
"port": 7272
}
}
If we were running a plugin that return an error due to the expression
1 + "foo", lightningd would have killed the plugin. But, pyln-client
caches errors and logs them in the node's log. And so the plugin is
not killed by lightningd as we can see looking for its process:
(.venv) ◉ tony@tony:~/clnlive:
$ ps -ax | rg myplugin
1207796 ? S 0:00 python /home/tony/clnlive/myplugin-pyln.py
1207907 pts/0 S+ 0:00 rg myplugin
And we can read the traceback in the log file /tmp/l1-regtest/log:
2023-05-11T14:41:22.629Z INFO plugin-myplugin-pyln.py: Traceback (most recent call last):
2023-05-11T14:41:22.629Z DEBUG 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf-connectd: peer_out WIRE_GOSSIP_TIMESTAMP_FILTER
2023-05-11T14:41:22.629Z DEBUG 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf-gossipd: seeker: starting gossip
2023-05-11T14:41:22.629Z INFO plugin-myplugin-pyln.py: File \"/home/tony/clnlive/.venv/lib/python3.10/site-packages/pyln/client/plugin.py\", line 668, in _dispatch_notification
2023-05-11T14:41:22.629Z INFO plugin-myplugin-pyln.py: self._exec_func(func, request)
2023-05-11T14:41:22.629Z INFO plugin-myplugin-pyln.py: File \"/home/tony/clnlive/.venv/lib/python3.10/site-packages/pyln/client/plugin.py\", line 619, in _exec_func
2023-05-11T14:41:22.629Z INFO plugin-myplugin-pyln.py: ret = func(*ba.args, **ba.kwargs)
2023-05-11T14:41:22.629Z INFO plugin-myplugin-pyln.py: File \"/home/tony/clnlive/myplugin-pyln.py\", line 10, in connect_func
2023-05-11T14:41:22.629Z INFO plugin-myplugin-pyln.py: 1 + \"foo\"
2023-05-11T14:41:22.629Z INFO plugin-myplugin-pyln.py: TypeError: unsupported operand type(s) for +: 'int' and 'str'
2023-05-11T14:41:22.629Z INFO plugin-myplugin-pyln.py:
Subscribe to disconnect
Let's modify our plugin to subscribe to disconnect notification topic:
#!/usr/bin/env python
from pyln.client import Plugin
import json
plugin = Plugin()
@plugin.subscribe("connect")
def connect_func(plugin, id, direction, address , **kwargs):
notif = {
"id": id,
"direction": direction,
"address": address
}
with open("/tmp/myplugin-pyln", "a") as myplugin:
myplugin.write("connect notification: " + json.dumps(notif) + "\n")
@plugin.subscribe("disconnect")
def disconnect_func(plugin, id, **kwargs):
with open("/tmp/myplugin-pyln", "a") as myplugin:
myplugin.write("disconnect notification: " + id + "\n")
plugin.run()
Back to our terminal, we disconnect the nodes, restart the plugin, connect them again and disconnect them immediately:
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
{}
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
{
"command": "start",
"plugins": [...]
}
(.venv) ◉ tony@tony:~/clnlive:
$ connect 1 2
{
"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
"features": "08a000080269a2",
"direction": "out",
"address": {
"type": "ipv4",
"address": "127.0.0.1",
"port": 7272
}
}
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
{}
The file /tmp/plugin-pyln is now:
connect notification: 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
connect notification: {"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf", "direction": "out", "address": {"type": "ipv4", "address": "127.0.0.1", "port": 7272}}
connect notification: {"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf", "direction": "out", "address": {"type": "ipv4", "address": "127.0.0.1", "port": 7272}}
disconnect notification: 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
Subscribe to invoice_creation
Let's modify our plugin to subscribe to invoice_creation notification
topic:
#!/usr/bin/env python
from pyln.client import Plugin
import json
plugin = Plugin()
@plugin.subscribe("connect")
def connect_func(plugin, id, direction, address , **kwargs):
notif = {
"id": id,
"direction": direction,
"address": address
}
with open("/tmp/myplugin-pyln", "a") as myplugin:
myplugin.write("connect notification: " + json.dumps(notif) + "\n")
@plugin.subscribe("disconnect")
def disconnect_func(plugin, id, **kwargs):
with open("/tmp/myplugin-pyln", "a") as myplugin:
myplugin.write("disconnect notification: " + id + "\n")
@plugin.subscribe("invoice_creation")
def invoice_creation_func(plugin, invoice_creation, **kwargs):
with open("/tmp/myplugin-pyln", "a") as myplugin:
myplugin.write("invoice_creation notification: " +
json.dumps(invoice_creation) + "\n")
plugin.run()
Back to our terminal, we restart the plugin and create an invoice with
the node l1:
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
{
"command": "start",
"plugins": [...]
}
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli invoice 0.001btc inv pizza
{
"payment_hash": "cdc9e18d9bd7e8c9206f8a1ed085b433ee52c4f669fe2b769b249ea792fb13d0",
"expires_at": 1684421489,
"bolt11": "lnbcrt1m1pj96q83sp59lafyq9v75389rf7vw9naa98wgcjclqlhm0uv078x6x2dl55l3kqpp5ehy7rrvm6l5vjgr03g0dppd5x0h9938kd8lzka5myj020yhmz0gqdqgwp5h57npxqyjw5qcqp29qyysgqnj9sv2jnkvt8ggjwhtz66gqhgakpdux5addva5zpm2u86q7a60rzn8ravhg9wrfkum80sevcpr82gxqs3j7a9nmrwrx4scsdha6jnssq0hs8he",
"payment_secret": "2ffa9200acf522728d3e638b3ef4a772312c7c1fbedfc63fc7368ca6fe94fc6c",
"warning_capacity": "Insufficient incoming channel capacity to pay invoice"
}
The file /tmp/plugin-pyln is now:
connect notification: 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
connect notification: {"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf", "direction": "out", "address": {"type": "ipv4", "address": "127.0.0.1", "port": 7272}}
connect notification: {"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf", "direction": "out", "address": {"type": "ipv4", "address": "127.0.0.1", "port": 7272}}
disconnect notification: 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
invoice_creation notification: {"msat": "100000000msat", "preimage": "88042f007f02283571abbc40aca8b4302643415e85c71413177ef139b4276970", "label": "inv"}
tests/plugins directory
In the directory tests/plugins of lightning repository we can find
more examples of subscription to notification topics:
◉ tony@tony:~/clnlive/lightning/tests/plugins:[git»(HEAD detached at v23.02.2)]
$ rg '@plugin\.subscribe'
misc_notifications.py
17:@plugin.subscribe("channel_opened")
25:@plugin.subscribe("channel_state_changed")
30:@plugin.subscribe("shutdown")
pretend_badlog.py
14:@plugin.subscribe("warning")
forward_payment_status.py
14:@plugin.subscribe("forward_event")
block_added.py
11:@plugin.subscribe("block_added")
coin_movements.py
12:@plugin.subscribe("coin_movement")
sendpay_notifications.py
15:@plugin.subscribe("sendpay_success")
21:@plugin.subscribe("sendpay_failure")
balance_snaps.py
12:@plugin.subscribe("balance_snapshot")
custom_notifications.py
8:@plugin.subscribe("custom")
27:@plugin.subscribe("pay_success")
37:@plugin.subscribe("ididntannouncethis")
Chat
How does connect_func get called when the connect notification is received?
When the plugin is started for each decorator line
@plugin.subscribe("..."), pyln-client adds an entry to the property
subscriptions of plugin instance, where the key is the notification
topic and the value is the function just below each decorator line.
In our case, once the plugin is running, we have plugin.subscriptions
equal to that following object:
{
'connect': <function connect_func at 0x7f801af9be20>,
'disconnect': <function disconnect_func at 0x7f801afc0dc0>,
'invoice_creation': <function invoice_creation_func at 0x7f801afc1870>
}
Then when the plugin receives a notification, let say a connect
notification, in plugin.run() the method plugin._multi_dispatch() with
an array as argument containing the connect notification is called:
class Plugin(object):
...
def run(self) -> None:
# If we are not running inside lightningd we'll print usage
# and some information about the plugin.
if os.environ.get('LIGHTNINGD_PLUGIN', None) != '1':
return self.print_usage()
partial = b""
for l in self.stdin.buffer:
partial += l
msgs = partial.split(b'\n\n')
if len(msgs) < 2:
continue
partial = self._multi_dispatch(msgs)
In plugin._multi_dispatch(), the request being a notification doesn't
have an id and so plugin._dispatch_notification is then called with
the request as argument.
class Plugin(object):
...
def _multi_dispatch(self, msgs: List[bytes]) -> bytes:
"""We received a couple of messages, now try to dispatch them all.
Returns the last partial message that was not complete yet.
"""
for payload in msgs[:-1]:
# Note that we use function annotations to do Millisatoshi
# conversions in _exec_func, so we don't use LightningJSONDecoder
# here.
request = self._parse_request(json.loads(payload.decode('utf8')))
# If this has an 'id'-field, it's a request and returns a
# result. Otherwise it's a notification and it doesn't
# return anything.
if request.id is not None:
self._dispatch_request(request)
else:
self._dispatch_notification(request)
return msgs[-1]
And in plugin._dispatch_notification, the function connect_func is
found in the dictionary plugin.subscriptions and is executed.
class Plugin(object):
...
def _dispatch_notification(self, request: Request) -> None:
if request.method not in self.subscriptions:
raise ValueError("No subscription for {name} found.".format(
name=request.method))
func = self.subscriptions[request.method]
try:
self._exec_func(func, request)
except Exception:
self.log(traceback.format_exc())
Terminal session
We ran the following commands in this order:
$ ./setup.sh
$ source lightning/contrib/startup_regtest.sh
$ start_ln
$ alias l1-cli
$ l1-cli plugin start $(pwd)/myplugin.py
$ connect 1 2
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
$ l1-cli plugin start $(pwd)/myplugin.py
$ connect 1 2
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
$ l1-cli stop
$ python -m venv .venv
$ source .venv/bin/activate
$ pip install pyln-client
$ alias l1-cli
$ lightningd --lightning-dir=/tmp/l1-regtest --daemon
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
$ connect 1 2
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
$ connect 1 2
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
$ connect 1 2
$ ps -ax | rg myplugin
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
$ connect 1 2
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
$ l1-cli invoice 0.001btc inv pizza
And below you can read the terminal session (command lines and outputs):
◉ tony@tony:~/clnlive:
$ ./setup.sh
Ubuntu 22.04.2 LTS
Python 3.10.6
lightningd v23.02.2
◉ tony@tony:~/clnlive:
$ source lightning/contrib/startup_regtest.sh
lightning-cli is /usr/local/bin/lightning-cli
lightningd is /usr/local/bin/lightningd
Useful commands:
start_ln 3: start three nodes, l1, l2, l3
connect 1 2: connect l1 and l2
fund_nodes: connect all nodes with channels, in a row
stop_ln: shutdown
destroy_ln: remove ln directories
◉ tony@tony:~/clnlive:
$ start_ln
Bitcoin Core starting
awaiting bitcoind...
Making "default" bitcoind wallet.
[1] 1205868
[2] 1205902
WARNING: eatmydata not found: instal it for faster testing
Commands:
l1-cli, l1-log,
l2-cli, l2-log,
bt-cli, stop_ln, fund_nodes
◉ tony@tony:~/clnlive:
$ alias l1-cli
alias l1-cli='lightning-cli --lightning-dir=/tmp/l1-regtest'
◉ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin.py
{
"command": "start",
"plugins": [
{
"name": "/usr/local/libexec/c-lightning/plugins/autoclean",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/chanbackup",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/bcli",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/commando",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/funder",
"active": true,
"dynamic": true
},
{
"name": "/usr/local/libexec/c-lightning/plugins/topology",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/keysend",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/offers",
"active": true,
"dynamic": true
},
{
"name": "/usr/local/libexec/c-lightning/plugins/pay",
"active": true,
"dynamic": true
},
{
"name": "/usr/local/libexec/c-lightning/plugins/txprepare",
"active": true,
"dynamic": true
},
{
"name": "/usr/local/libexec/c-lightning/plugins/spenderp",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/sql",
"active": true,
"dynamic": true
},
{
"name": "/usr/local/libexec/c-lightning/plugins/bookkeeper",
"active": true,
"dynamic": false
},
{
"name": "/home/tony/clnlive/myplugin.py",
"active": true,
"dynamic": true
}
]
}
◉ tony@tony:~/clnlive:
$ connect 1 2
{
"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
"features": "08a000080269a2",
"direction": "out",
"address": {
"type": "ipv4",
"address": "127.0.0.1",
"port": 7272
}
}
◉ tony@tony:~/clnlive:
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
{}
◉ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin.py
{
"command": "start",
"plugins": [...]
}
◉ tony@tony:~/clnlive:
$ connect 1 2
{
"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
"features": "08a000080269a2",
"direction": "out",
"address": {
"type": "ipv4",
"address": "127.0.0.1",
"port": 7272
}
}
◉ tony@tony:~/clnlive:
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
{}
◉ tony@tony:~/clnlive:
$ l1-cli stop
"Shutdown complete"
[1]- Done test -f "/tmp/l$i-$network/lightningd-$network.pid" || $EATMYDATA "$LIGHTNINGD" "--lightning-dir=/tmp/l$i-$network"
◉ tony@tony:~/clnlive:
$ python -m venv .venv
◉ tony@tony:~/clnlive:
$ source .venv/bin/activate
(.venv) ◉ tony@tony:~/clnlive:
$ pip install pyln-client
Collecting pyln-client
Using cached pyln_client-23.2-py3-none-any.whl (29 kB)
Collecting pyln-bolt7>=1.0
Using cached pyln_bolt7-1.0.246-py3-none-any.whl (18 kB)
Collecting pyln-proto>=0.12
Using cached pyln_proto-23.2-py3-none-any.whl (31 kB)
Collecting coincurve<18.0.0,>=17.0.0
Using cached coincurve-17.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
Collecting bitstring<4.0.0,>=3.1.9
Using cached bitstring-3.1.9-py3-none-any.whl (38 kB)
Collecting base58<3.0.0,>=2.1.1
Using cached base58-2.1.1-py3-none-any.whl (5.6 kB)
Collecting PySocks<2.0.0,>=1.7.1
Using cached PySocks-1.7.1-py3-none-any.whl (16 kB)
Collecting cryptography<37.0.0,>=36.0.1
Using cached cryptography-36.0.2-cp36-abi3-manylinux_2_24_x86_64.whl (3.6 MB)
Collecting cffi>=1.3.0
Using cached cffi-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
Collecting asn1crypto
Using cached asn1crypto-1.5.1-py2.py3-none-any.whl (105 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Installing collected packages: bitstring, asn1crypto, PySocks, pyln-bolt7, pycparser, base58, cffi, cryptography, coincurve, pyln-proto, pyln-client
Successfully installed PySocks-1.7.1 asn1crypto-1.5.1 base58-2.1.1 bitstring-3.1.9 cffi-1.15.1 coincurve-17.0.0 cryptography-36.0.2 pycparser-2.21 pyln-bolt7-1.0.246 pyln-client-23.2 pyln-proto-23.2
(.venv) ◉ tony@tony:~/clnlive:
$ alias l1-cli
alias l1-cli='lightning-cli --lightning-dir=/tmp/l1-regtest'
(.venv) ◉ tony@tony:~/clnlive:
$ lightningd --lightning-dir=/tmp/l1-regtest --daemon
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
{
"code": -3,
"message": "/home/tony/clnlive/myplugin-pyln.py: exited before replying to getmanifest"
}
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
{
"command": "start",
"plugins": [
{
"name": "/usr/local/libexec/c-lightning/plugins/autoclean",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/chanbackup",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/bcli",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/commando",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/funder",
"active": true,
"dynamic": true
},
{
"name": "/usr/local/libexec/c-lightning/plugins/topology",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/keysend",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/offers",
"active": true,
"dynamic": true
},
{
"name": "/usr/local/libexec/c-lightning/plugins/pay",
"active": true,
"dynamic": true
},
{
"name": "/usr/local/libexec/c-lightning/plugins/txprepare",
"active": true,
"dynamic": true
},
{
"name": "/usr/local/libexec/c-lightning/plugins/spenderp",
"active": true,
"dynamic": false
},
{
"name": "/usr/local/libexec/c-lightning/plugins/sql",
"active": true,
"dynamic": true
},
{
"name": "/usr/local/libexec/c-lightning/plugins/bookkeeper",
"active": true,
"dynamic": false
},
{
"name": "/home/tony/clnlive/myplugin-pyln.py",
"active": true,
"dynamic": true
}
]
}
(.venv) ◉ tony@tony:~/clnlive:
$ connect 1 2
{
"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
"features": "08a000080269a2",
"direction": "out",
"address": {
"type": "ipv4",
"address": "127.0.0.1",
"port": 7272
}
}
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
{}
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
{
"command": "start",
"plugins": [...]
}
(.venv) ◉ tony@tony:~/clnlive:
$ connect 1 2
{
"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
"features": "08a000080269a2",
"direction": "out",
"address": {
"type": "ipv4",
"address": "127.0.0.1",
"port": 7272
}
}
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
{}
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
{
"command": "start",
"plugins": [...]
}
(.venv) ◉ tony@tony:~/clnlive:
$ connect 1 2
{
"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
"features": "08a000080269a2",
"direction": "out",
"address": {
"type": "ipv4",
"address": "127.0.0.1",
"port": 7272
}
}
(.venv) ◉ tony@tony:~/clnlive:
$ ps -ax | rg myplugin
1207796 ? S 0:00 python /home/tony/clnlive/myplugin-pyln.py
1207907 pts/0 S+ 0:00 rg myplugin
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
{}
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
{
"command": "start",
"plugins": [...]
}
(.venv) ◉ tony@tony:~/clnlive:
$ connect 1 2
{
"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
"features": "08a000080269a2",
"direction": "out",
"address": {
"type": "ipv4",
"address": "127.0.0.1",
"port": 7272
}
}
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli disconnect 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
{}
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli plugin start $(pwd)/myplugin-pyln.py
{
"command": "start",
"plugins": [...]
}
(.venv) ◉ tony@tony:~/clnlive:
$ l1-cli invoice 0.001btc inv pizza
{
"payment_hash": "cdc9e18d9bd7e8c9206f8a1ed085b433ee52c4f669fe2b769b249ea792fb13d0",
"expires_at": 1684421489,
"bolt11": "lnbcrt1m1pj96q83sp59lafyq9v75389rf7vw9naa98wgcjclqlhm0uv078x6x2dl55l3kqpp5ehy7rrvm6l5vjgr03g0dppd5x0h9938kd8lzka5myj020yhmz0gqdqgwp5h57npxqyjw5qcqp29qyysgqnj9sv2jnkvt8ggjwhtz66gqhgakpdux5addva5zpm2u86q7a60rzn8ravhg9wrfkum80sevcpr82gxqs3j7a9nmrwrx4scsdha6jnssq0hs8he",
"payment_secret": "2ffa9200acf522728d3e638b3ef4a772312c7c1fbedfc63fc7368ca6fe94fc6c",
"warning_capacity": "Insufficient incoming channel capacity to pay invoice"
}
(.venv) ◉ tony@tony:~/clnlive:
$
Source code
myplugin.py
#!/usr/bin/env python
import sys
import json
# getmanifest
request = sys.stdin.readline()
sys.stdin.readline() # "\n"
req_id = json.loads(request)["id"]
manifest = {
"jsonrpc": "2.0",
"id": req_id,
"result": {
"dynamic": True,
"options": [],
"rpcmethods": [],
"subscriptions": ["connect", "disconnect"]
}
}
sys.stdout.write(json.dumps(manifest))
sys.stdout.flush()
# init
request = sys.stdin.readline()
sys.stdin.readline() # "\n"
req_id = json.loads(request)["id"]
init = {
"jsonrpc": "2.0",
"id": req_id,
"result": {}
}
sys.stdout.write(json.dumps(init))
sys.stdout.flush()
# io loop
for request in sys.stdin:
sys.stdin.readline() # "\n"
method = json.loads(request)["method"]
if method == "connect":
with open("/tmp/myplugin", "a") as myplugin:
myplugin.write("connect notification: " + request + "\n")
if method == "disconnect":
with open("/tmp/myplugin", "a") as myplugin:
myplugin.write("disconnect notification: " + request + "\n")
myplugin-pyln.py
#!/usr/bin/env python
from pyln.client import Plugin
import json
plugin = Plugin()
@plugin.subscribe("connect")
def connect_func(plugin, id, direction, address , **kwargs):
notif = {
"id": id,
"direction": direction,
"address": address
}
with open("/tmp/myplugin-pyln", "a") as myplugin:
myplugin.write("connect notification: " + json.dumps(notif) + "\n")
@plugin.subscribe("disconnect")
def disconnect_func(plugin, id, **kwargs):
with open("/tmp/myplugin-pyln", "a") as myplugin:
myplugin.write("disconnect notification: " + id + "\n")
@plugin.subscribe("invoice_creation")
def invoice_creation_func(plugin, invoice_creation, **kwargs):
with open("/tmp/myplugin-pyln", "a") as myplugin:
myplugin.write("invoice_creation notification: " +
json.dumps(invoice_creation) + "\n")
plugin.run()
setup.sh
#!/usr/bin/env bash
ubuntu=$(lsb_release -ds)
lightningd=$(lightningd --version | xargs printf "lightningd %s\n")
python=$(python --version)
printf "%s\n%s\n%s\n" "$ubuntu" "$python" "$lightningd"
connect
{
"jsonrpc": "2.0",
"method": "connect",
"params": {
"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf",
"direction": "out",
"address": {
"type": "ipv4",
"address": "127.0.0.1",
"port": 7272
}
}
}
myplugin
connect notification: {"jsonrpc":"2.0","method":"connect","params":{"id":"0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf","direction":"out","address":{"type":"ipv4","address":"127.0.0.1","port":7272}}}
connect notification: {"jsonrpc":"2.0","method":"connect","params":{"id":"0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf","direction":"out","address":{"type":"ipv4","address":"127.0.0.1","port":7272}}}
disconnect notification: {"jsonrpc":"2.0","method":"disconnect","params":{"id":"0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf"}}
myplugin-pyln
connect notification: 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
connect notification: {"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf", "direction": "out", "address": {"type": "ipv4", "address": "127.0.0.1", "port": 7272}}
connect notification: {"id": "0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf", "direction": "out", "address": {"type": "ipv4", "address": "127.0.0.1", "port": 7272}}
disconnect notification: 0271aecf3075ce72f2df479b60fc1db98d4d780c66fa495013614054554e1bb2bf
invoice_creation notification: {"msat": "100000000msat", "preimage": "88042f007f02283571abbc40aca8b4302643415e85c71413177ef139b4276970", "label": "inv"}