pointers on writing a test client

[sfinnie]

I'm working on a gentle introduction to writing LSP-based tooling, using pygls and targeting VSCode. I've got to the point where I want to write some server tests, i.e. start an instance of the server and simulate requests/responses/notifications between client and server. Given the protocol is symmetric, I think I should be able to take advantage of the LanguageServerProtocol class and e.g. notify() to actually send the message. However, I need to provide a server instance when constructing the LSP instance. And I'm not sure about that - creating a server to test the server seems a bit suspect (or maybe not...?).

So I'm looking for suggestions/guidance. I could go completely independent and use - I think? - the json-rpc package to implement the client. However, I suspect I'd just end up re-implementing much of what's already built in pygls. And, obviously, I'm not trying to test pygls itself - just my feature implementations.

To make that more concrete, here's the skeleton of the test I want to write:

from server.server import greet_server # instance of GreetLanguageServer, a subclass of LanguageServer


def test_invalid_greeting_returns_correct_diagnostic():
    # arrange
    address = "127.0.0.1"
    port = 4242
    server = greet_server.start_tcp(address, port) # TODO: need to start concurrently, this is synchronous
    client = LanguageServerTestClient(address, port) # TODO: implement this class
    did_open_notification = # TODO: construct instance of did_open message

    # act
    client.send_notification(did_open_notification)
    msg = client.await_notification()

    # assert
    #TODO: extract diagnostic and assert range and message as expected.

    server.shutdown()

Thanks in advance.

[tombh]

Alex Carney (I'm sure he'll chime in anyway), is actually working on a dedicated Pygls testing framework https://github.com/alcarney/lsp-devtools/#libpytest-lsp---end-to-end-testing-of-language-servers-with-pytest

[alcarney]

creating a server to test the server seems a bit suspect (or maybe not...?)

This is how pygls tests itself, it spins up a client and a server (which are both instances of LanguageServer) each in a separate thread. The two instances then communicate with each other over a pair of os.pipes() and start_io (rather than start_tcp).
If you're interested, here is the client-server setup and here is an example test case

Alternatively... as @tombh mentioned you might be interested in the pytest-lsp package I've been working on. Under the hood, it's very similar to the approach taken by pygls' test suite. The main difference is that rather than the server sitting in a thread within the pytest Python process, it's run as separate, external process and the client talks to it over stdio similar to how VSCode would.
Also the test client (still a pygls.Server in disguise 😅) has pre-defined methods for each of the lsp requests/notifications.

Anyway, if you're interested an example test case can be seen in the project's readme

[sfinnie]

Thanks both, I'll have a look at your pytest-lsp package @alcarney and the pygls tests (not sure how I missed those).

Appreciate the quick responses too.

[sfinnie]

@alcarney I finally had an opportunity to look at pytest-lsp -- thanks for writing and making available. I have some questions (and hope OK to post here - let me know if not). Summary is: The readme and getting started examples use different approaches, and I'd like to understand if one is preferred over the other. I have a minimally working example that draws from both:

# test_server_readme.py

import sys
import pytest
import pytest_lsp
from pytest_lsp import ClientServerConfig

@pytest_lsp.fixture(
    # scope='session',
    config=ClientServerConfig(
        server_command=[sys.executable, "toy_server.py"],
        root_uri="file:///path/to/test/project/root/"
    ),
)
async def client():
    pass


@pytest.mark.asyncio
async def test_completion(client):
    test_uri="file:///path/to/test/project/root/test_file.rst"
    result = await client.completion_request(test_uri, line=5, character=23)

    assert len(result) == 2

and here's the toy server:

# toy_server.py

from lsprotocol.types import TEXT_DOCUMENT_COMPLETION
from lsprotocol.types import CompletionItem
from lsprotocol.types import CompletionParams
from pygls.server import LanguageServer

server = LanguageServer("hello-world", "v1")


@server.feature(TEXT_DOCUMENT_COMPLETION)
def completion(ls: LanguageServer, params: CompletionParams):
    return [
        CompletionItem(label="hello"),
        CompletionItem(label="world"),
    ]


if __name__ == "__main__":
    server.start_io()

This passes, albeit I had to comment out scope='session' or pytest complained. Pytest also generates an async warning:

test_server_readme.py::test_completion[client0]
  /home/sfinnie/projects/helloLSP/venv/lib/python3.10/site-packages/pytest_asyncio/plugin.py:444: DeprecationWarning: pytest-asyncio detected an unclosed event loop when tearing down the event_loop
  fixture: <_UnixSelectorEventLoop running=False closed=False debug=False>
  pytest-asyncio will close the event loop for you, but future versions of the
  library will no longer do so. In order to ensure compatibility with future
  versions, please make sure that:
      1. Any custom "event_loop" fixture properly closes the loop after yielding it
      2. Your code does not modify the event loop in async fixtures or tests

I notice the getting started example uses a different approach to the fixture (which seems consistent with standard pytest conventions).

# test_server_docs.py

import sys

from lsprotocol.types import ClientCapabilities
from lsprotocol.types import InitializeParams

import pytest_lsp
from pytest_lsp.client import LanguageClient
from pytest_lsp.plugin import ClientServerConfig


@pytest_lsp.fixture(
    config=ClientServerConfig(server_command=[sys.executable, "toy_server.py"]),
)
async def client(lsp_client: LanguageClient):
    # Setup
    params = InitializeParams(capabilities=ClientCapabilities())
    await lsp_client.initialize(params)

    yield

    # Teardown
    await lsp_client.shutdown()


async def test_completions(client: LanguageClient):
    """Ensure that the server implements completions correctly."""

    results = await client.completion_request(
        uri="file:///path/to/file.txt", line=1, character=0
    )

    labels = [item.label for item in results]
    assert labels == ["hello", "world"]

However, pytest fails when I run that:

======================== ERRORS ==============================================
_________________________________ ERROR collecting test_server_docs.py ____________________________________________________
test_server_docs.py:12: in <module>
    config=ClientServerConfig(server_command=[sys.executable, "toy_server.py"]),
E   TypeError: ClientServerConfig.__init__() missing 1 required positional argument: 'root_uri'
=============================== short test summary info =============================
ERROR test_server_docs.py - TypeError: ClientServerConfig.__init__() missing 1 required positional argument: 'root_uri'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=================================== 1 error in 0.05s ===============================

I'm conscious you're in active development, so would be grateful of any pointers on how best to proceed. Running Python 3.10.6 on WSL (Ubuntu 22.04), packages as follows:

$ python3 -m pip freeze
appdirs==1.4.4
attrs==22.2.0
cattrs==22.2.0
exceptiongroup==1.1.1
iniconfig==2.0.0
lsprotocol==2023.0.0a1
packaging==23.0
pluggy==1.0.0
pygls==1.0.1
pytest==7.3.0
pytest-asyncio==0.21.0
pytest-lsp==0.2.1
tomli==2.0.1
tree-sitter==0.20.1
typeguard==2.13.3

Thanks for any suggestions.

[alcarney]

Ah, you've managed to catch me mid-refactor! 😅
I realized that the library was too tightly coupled to the way esbonio currently works, so some breaking changes were required to undo some of the assumptions I had made - I'm also taking the opportunity to simplify the internal architecture a little bit.

If you fancy trying the development version you should be able to try it with the following pip install command

$ pip install "git+https://github.com/swyddfa/lsp-devtools#egg=pytest-lsp&subdirectory=lib/pytest-lsp"

I'm still expecting one more set of breaking changes to land where I align the LSP method signatures on the test client to match the client in #328 since they've drifted a bit, but it should be stable enough to at least get an initial impression of the library. 🤞

The readme and getting started examples use different approaches,

The getting started example will be the correct approach once the new release is out, while the readme is correct for the current version.

albeit I had to comment out scope='session' or pytest complained

If it's the error I'm thinking of, that's due to the default event_loop created by pytest-asyncio not living long enough, there's now a note on the docs site showing how to resolve it.

Pytest also generates an async warning:

test_server_readme.py::test_completion[client0]
  /home/sfinnie/projects/helloLSP/venv/lib/python3.10/site-packages/pytest_asyncio/plugin.py:444: DeprecationWarning: pytest-asyncio detected an unclosed event loop when tearing down the event_loop
  fixture: <_UnixSelectorEventLoop running=False closed=False debug=False>
  pytest-asyncio will close the event loop for you, but future versions of the
  library will no longer do so. In order to ensure compatibility with future
  versions, please make sure that:
      1. Any custom "event_loop" fixture properly closes the loop after yielding it
      2. Your code does not modify the event loop in async fixtures or tests

I am aware of that one, though haven't figured out a fix for all cases just yet :/

Hope that helps! 😄

[sfinnie]

Thanks @alcarney - that makes sense. I've an interim set of tests that work for now. I must say it was very easy to write them - thanks very much for your work. I'll have a go at converting to the dev branch per your suggestion when I get a chance. Thanks again meantime.