Overview of pyln-client implementation - @plugin.method() - Part 2

Let's install pyln-client in .venv Python virtual environment:

◉ tony@tony:~/lnroom:
$ python -m venv .venv
◉ tony@tony:~/lnroom:
$ source .venv/bin/activate
(.venv) ◉ tony@tony:~/lnroom:
$ pip install pyln-client
...

Here is my setup

(.venv) ◉ tony@tony:~/lnroom:
$ ./setup.sh
Ubuntu 22.04.2 LTS
Python 3.10.6
lightningd v23.02.2
pyln-client  23.2

Let's start two Lightning nodes running on the Bitcoin regtest chain by sourcing the script lightning/contrib/startup_regtest.sh provided in CLN repository and by running the command start_ln:

(.venv) ◉ tony@tony:~/lnroom:
$ source lightning/contrib/startup_regtest.sh
lightning-cli is /usr/local/bin/lightning-cli
lightningd is /usr/local/bin/lightningd
Useful commands:
  start_ln 3: start three nodes, l1, l2, l3
  connect 1 2: connect l1 and l2
  fund_nodes: connect all nodes with channels, in a row
  stop_ln: shutdown
  destroy_ln: remove ln directories
(.venv) ◉ tony@tony:~/lnroom:
$ start_ln
Bitcoin Core starting
awaiting bitcoind...
Making "default" bitcoind wallet.
[1] 347148
[2] 347186
WARNING: eatmydata not found: instal it for faster testing
Commands:
        l1-cli, l1-log,
        l2-cli, l2-log,
        bt-cli, stop_ln, fund_nodes

We can check that l1-cli is just an alias for lightning-cli with the base directory being /tmp/l1-regtest:

(.venv) ◉ tony@tony:~/lnroom:
$ alias l1-cli
alias l1-cli='lightning-cli --lightning-dir=/tmp/l1-regtest'

The plugin myplugin.py is defined like this:

from pyln.client import Plugin

plugin = Plugin()

@plugin.method("foo")
def foo_func(plugin,x):
    return {"foo":x}

plugin.run()

It register a JSON-RPC method to lightningd that we can try like this:

(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli plugin start $(pwd)/myplugin.py
{
   "command": "start",
   "plugins": [...]
}
(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli foo bar
{
   "foo": "bar"
}
(.venv) ◉ tony@tony:~/lnroom:
$ l1-cli foo baz
{
   "foo": "baz"
}

In Overview of pyln-client implementation - Plugin.run() - Part 1 we saw how Plugin.run is implemented.

Let's describe what happens in plugin.run() in the case of myplugin.py plugin.

When we start our plugin, lightningd sends the getmanifest request to the plugin. To compute the payload of the response, the plugin looks for the function plugin.methods["getmanifest"].func. It executes that function, constructs the response and sends it back to lightningd.

Then lightningd sends the init request to the plugin. To compute the payload of the response, the plugin looks for the function plugin.methods["init"].func. It executes that function, constructs the response and sends it back to lightningd.

Then the plugins waits for incoming requests from lightningd.

Let's say a client sends a foo request to lightningd, hence lightningd forwards that request to the plugin. To compute the payload of the response, the plugin looks for the function plugin.methods["foo"].func. It executes that function, constructs the response and sends it back to lightningd. Finally, lightningd forwards that repsonse to the client.

And now, the plugin waits again for incoming requests from lightningd.

In this episode we are interested in how foo method is added to plugin.methods dictionary.

Let's play with our plugin in a Python interpreter and take a look at plugin.methods property.

First we send those expressions to the Python interpreter:

from pyln.client import Plugin

plugin = Plugin()

Then we can see that plugin.methods is filled with two entries, one for the getmanifest request and the other for the init request:

>>> plugin.methods
{'init': <pyln.client.plugin.Method object at 0x7f21e699c640>,
'getmanifest': <pyln.client.plugin.Method object at 0x7f21e699d360>}
>>> plugin.methods["init"].name
'init'
>>> plugin.methods["init"].func
<bound method Plugin._init of <pyln.client.plugin.Plugin object at 0x7f21e699c5e0>>

Due to the way Python decorators work, sending the following to the Python interpreter

@plugin.method("foo")
def foo_func(plugin,x):
    return {"foo":x}

would have the effect to first define normally the function foo_func and then to set foo_func like this:

foo_func = plugin.method("foo")(foo_func)

To understand what would happened case we can look at the result of applying the method (function) plugin.method to the string "foo":

>>> plugin.method("foo")
<function Plugin.method.<locals>.decorator at 0x7f21e6647a30>
>>> import inspect
>>> print(inspect.getsource(plugin.method("foo")))
        def decorator(f: Callable[..., JSONType]) -> Callable[..., JSONType]:
            self.add_method(method_name,
                            f,
                            background=False,
                            category=category,
                            desc=desc,
                            long_desc=long_desc,
                            deprecated=deprecated)
            return f

We see that plugin.method("foo") returns a function called decorator. That function takes a function f as argument (in our case it is foo_func), uses add_method method to do some stuff (add foo method to plugin.methods, in our case method_name is foo and f is foo_func) and return f (which is foo_func in our case).

Let's try it by sending the following to the Python interpreter:

@plugin.method("foo")
def foo_func(plugin,x):
    return {"foo":x}

This added the entry foo to plugin.methods dictionary as we can see:

>>> plugin.methods
{'init': <pyln.client.plugin.Method object at 0x7f21e699c640>,
'getmanifest': <pyln.client.plugin.Method object at 0x7f21e699d360>,
'foo': <pyln.client.plugin.Method object at 0x7f21e51ab8b0>}
>>> plugin.methods["foo"].func
<function foo_func at 0x7f21e6991750>
>>> plugin.methods["foo"].func(plugin, "bar")
{'foo': 'bar'}

To understand how methods of the class Method are added to the property methods of the class Plugin using Python decorators we implement a simplified and narrowed version of the classes Method and Plugin. We call them MyMethod and MyPlugin.

The class MyMethod has only two properties which are name and func. We define it like this:

class MyMethod:
    def __init__(self,name,func):
        self.name = name
        self.func = func

Let's send it to the Python interpreter to define it and interpret the following expressions:

>>> mymethod = MyMethod("foo", lambda x: x)
>>> mymethod
<__main__.MyMethod object at 0x7f21e51bc970>
>>> mymethod.func
<function <lambda> at 0x7f21e69916c0>
>>> mymethod.func("bar")
'bar'

Now we write MyPlugin class. This class has one property named methods which is a dictionary that we initialize with the entries "init" and "getmanifest". The values of those entries are of the class MyMethod and depend of the values of the methods _init and _getmanifest of the class itself:

class MyPlugin:
    def __init__(self):
        self.methods = {
            "init": MyMethod("init", self._init),
            "getmanifest": MyMethod("getmanifest", self._getmanifest)
        }
    def _init(self):
        return "I'm _init"
    def _getmanifest(self):
        return "I'm _getmanifest"

Let's send it to the Python interpreter to define it and interpret the following expressions:

>>> myplugin = MyPlugin()
>>> myplugin.methods
{'init': <__main__.MyMethod object at 0x7f21e51be290>,
'getmanifest': <__main__.MyMethod object at 0x7f21e51be770>}
>>> myplugin.methods["init"].func()
"I'm _init"
>>>

Let's add add_method methods which given a name name and a function func constructs an object MyMethod and a name entry in methods dictionary. The class MyPlugin is then:

class MyPlugin:
    def __init__(self):
        self.methods = {
            "init": MyMethod("init", self._init),
            "getmanifest": MyMethod("getmanifest", self._getmanifest)
        }
    def _init(self):
        return "I'm _init"
    def _getmanifest(self):
        return "I'm _getmanifest"
    def add_method(self,name,func):
        self.methods[name] = MyMethod(name,func)

We redefine MyPlugin class by sending the previous snippet to the Python interpreter and interpret the following expressions:

>>> myplugin = MyPlugin()
>>> myplugin.methods
{'init': <__main__.MyMethod object at 0x7f21e51ab100>,
'getmanifest': <__main__.MyMethod object at 0x7f21e51ab190>}
>>> myplugin.add_method("foo", lambda x: x)
>>> myplugin.methods
{'init': <__main__.MyMethod object at 0x7f21e51ab100>,
'getmanifest': <__main__.MyMethod object at 0x7f21e51ab190>,
'foo': <__main__.MyMethod object at 0x7f21e6ae35e0>}
>>> myplugin.methods["foo"].func("bar")
'bar'

Finally we can define method method in MyPlugin in a way that we can use Python decorators to add methods to methods property of the class MyPlugin.

The method method of MyPlugin takes name as argument, defines a "local" function decorator which takes a function f, adds the entry name to methods property with its value being Method(name,f). This is done by using add_method method. The function decorator returns f and method method returns the "local" function decorator. Hence the class MyPlugin is now:

class MyPlugin:
    def __init__(self):
        self.methods = {
            "init": MyMethod("init", self._init),
            "getmanifest": MyMethod("getmanifest", self._getmanifest)
        }
    def _init(self):
        return "I'm _init"
    def _getmanifest(self):
        return "I'm _getmanifest"
    def add_method(self,name,func):
        self.methods[name] = MyMethod(name,func)
    def method(self,name):
        def decorator(f):
            self.add_method(name,f)
            return f
        return decorator

We redefine MyPlugin class by sending the previous snippet to the Python interpreter and interpret the following expressions:

>>> myplugin = MyPlugin()
>>> myplugin.methods
{'init': <__main__.MyMethod object at 0x7f21e51abfa0>,
'getmanifest': <__main__.MyMethod object at 0x7f21e51abbe0>}

Then we add foo method to myplugin.methods by sending the following snippet to the Python interpreter:

@myplugin.method("foo")
def foo_func(myplugin,x):
    return {"foo": x}

An we check that myplugin.methods is as expected:

>>> myplugin.methods
{'init': <__main__.MyMethod object at 0x7f21e51abfa0>,
'getmanifest': <__main__.MyMethod object at 0x7f21e51abbe0>,
'foo': <__main__.MyMethod object at 0x7f21e51ab490>}
>>> myplugin.methods["foo"].func
<function foo_func at 0x7f21e51bad40>
>>> myplugin.methods["foo"].func(myplugin,"bar")
{'foo': 'bar'}

We can finally looks at the implementation of Method class and the methods Plugin.__init__, Plugin.add_method and Plugin.method defined in lightning:contrib/pyln-client/pyln/client/plugin.py file keeping only the "meaningful" parts:

class Method(object):
    ...
    def __init__(self, name: str, func: Callable[..., JSONType],
                 mtype: MethodType = MethodType.RPCMETHOD,
                 category: str = None, desc: str = None,
                 long_desc: str = None, deprecated: bool = False):
        self.name = name
        self.func = func
        ...

class Plugin(object):
    ...
    def __init__(self, stdout: Optional[io.TextIOBase] = None,
                 stdin: Optional[io.TextIOBase] = None, autopatch: bool = True,
                 dynamic: bool = True,
                 init_features: Optional[Union[int, str, bytes]] = None,
                 node_features: Optional[Union[int, str, bytes]] = None,
                 invoice_features: Optional[Union[int, str, bytes]] = None):
        self.methods = {
            'init': Method('init', self._init, MethodType.RPCMETHOD)
        }
        ...
        self.add_method("getmanifest", self._getmanifest, background=False)
        ...

    def add_method(self, name: str, func: Callable[..., Any],
                   background: bool = False,
                   category: Optional[str] = None,
                   desc: Optional[str] = None,
                   long_desc: Optional[str] = None,
                   deprecated: bool = False) -> None:
        ...
        if name in self.methods:
            raise ValueError(
                "Name {} is already bound to a method.".format(name)
            )

        # Register the function with the name
        method = Method(
            name, func, MethodType.RPCMETHOD, category, desc, long_desc,
            deprecated
        )

        method.background = background
        self.methods[name] = method

    def method(self, method_name: str, category: Optional[str] = None,
               desc: Optional[str] = None,
               long_desc: Optional[str] = None,
               deprecated: bool = False) -> JsonDecoratorType:
        ...
        def decorator(f: Callable[..., JSONType]) -> Callable[..., JSONType]:
            self.add_method(method_name,
                            f,
                            background=False,
                            category=category,
                            desc=desc,
                            long_desc=long_desc,
                            deprecated=deprecated)
            return f
        return decorator

We are done!

#!/usr/bin/env python

from pyln.client import Plugin

plugin = Plugin()

@plugin.method("foo")
def foo_func(plugin,x):
    return {"foo":x}

plugin.run()

class MyMethod:
    def __init__(self,name,func):
        self.name = name
        self.func = func

class MyPlugin:
    def __init__(self):
        self.methods = {
            "init": MyMethod("init", self._init),
            "getmanifest": MyMethod("getmanifest", self._getmanifest)
        }
    def _init(self):
        return "I'm _init"
    def _getmanifest(self):
        return "I'm _getmanifest"
    def add_method(self,name,func):
        self.methods[name] = MyMethod(name,func)
    def method(self,name):
        def decorator(f):
            self.add_method(name,f)
            return f
        return decorator

#!/usr/bin/env bash

ubuntu=$(lsb_release -ds)
lightningd=$(lightningd --version | xargs printf "lightningd %s\n")
python=$(python --version)
pyln_client=$(pip list | rg pyln-client)

printf "%s\n%s\n%s\n%s\n" "$ubuntu" "$python" "$lightningd" "$pyln_client"