--- title: ⚗️ lsp-test date: 2018-08-12 --- My Haskell Summer of Code project, [lsp-test](https://github.com/Bubba/lsp-test), is now available [via Hackage](https://hackage.haskell.org/package/lsp-test-0.1.0.0). It's a framework for writing end-to-end tests for LSP servers, made for testing [haskell-ide-engine](https://github.com/haskell/haskell-ide-engine). But it's not just limited to haskell-ide-engine: It's language agnostic and works with any server that conforms to the [Language Server Protocol](https://microsoft.github.io/language-server-protocol/). In fact lsp-test is basically a client that you can programmatically control. You specify what messages you want to send to the server, and check that the responses you get back are what you expected. # What does it look like? ```haskell runSession :: String -> ClientCapabilities -> FilePath -> Session a -> IO a main = runSession "hie" fullCaps "proj/dir" $ do doc <- openDoc "Foo.hs" "haskell" getDocumentSymbols doc >>= liftIO putStrLn ``` Each test is encapsulated by a `Session`: a client-server connection from start to finish. Here we pass in the command to start the server, the capabilities that the client should declare and the root directory that the session should take place in. (`fullCaps` is a convenience function that declares all the latest features in the LSP specification) Once you're inside a `Session` you're free to poke and talk away to your server. You might have noticed that the example above didn't send an initialize request - `Session` takes care of this and some other laborious parts of the process: - Sending initialize requests with the correct process ID and capabilities - Generating and incrementing message ids - Keeping track of documents through `workspace/applyEdit` requests There's also a [Smörgåsbord of functions](https://hackage.haskell.org/package/lsp-test-0.1.0.0/docs/Language-Haskell-LSP-Test.html#g:6) available for performing common tasks, such as getting a definition or checking the current diagnostics. # The message level Most of the time you'll want to want to write your tests using these helper functions, but in case you're looking to do something more specific, you can drop down to a lower level and work with the individual messages that are sent and received. The venerable [`request`](https://hackage.haskell.org/package/lsp-test-0.1.0.0/docs/Language-Haskell-LSP-Test.html#v:request) allows you to send any request defined in [`haskell-lsp-types`](https://hackage.haskell.org/package/haskell-lsp-types-0.6.0.0/docs/Language-Haskell-LSP-TH-DataTypesJSON.html), and will spit back the response for it. Most of the helper functions are implemented in terms of this. ```haskell request :: (ToJSON params, FromJSON a) => ClientMethod -> params -> Session (ResponseMessage a) runSession "hie" fullCaps "my/dir" = do doc <- openDoc "Foo.hs" "haskell" let params = DocumentSymbolParams doc -- send and wait for the response rsp <- request TextDocumentDocumentSymbol params ``` You can also use the `send` family of functions if you don't want to wait for a response. ```haskell sendRequest :: ToJSON params => ClientMethod -> params -> Session LspId sendNotification :: ToJSON a => ClientMethod -> a -> Session () sendResponse :: ToJSON a => ResponseMessage a -> Session () ``` # Inside Session This is where `lsp-test`'s little breakthrough comes in: The `Session` monad is actually just a [conduit parser](https://hackage.haskell.org/package/conduit-parse) that operates on individual messages. ```haskell type Session = ConduitParser FromServerMessage IO runSession f = runConduit $ source .| runConduitParser f where source = getNextMessage handle >>= yield >> source ``` Incoming messages from the server are parsed in a stream. You specify the messages you expect to receive from the server in order, in between the messages you send to it. ```haskell -- get a specific response msg1 <- message :: Session RspDocumentSymbols -- get any request msg2 <- anyRequest sendRequest TextDocumentDocumentSymbol params -- get a logging notification msg3 <- loggingNotification ``` It has it's own version of `satisfy` that works on messages, which custom parser combinators are built up upon. ```haskell satisfy :: (FromServerMessage -> Bool) -> Session FromServerMessage satisfy pred = do x <- await if pred x then return x else empty loggingNotification = satisfy test where test (NotLogMessage _) = True test (NotShowMessage _) = True test (ReqShowMessage _) = True test _ = False ``` Whenever it's unable to parse a sequence of messages, it throws an exception, which can be used to make assertions about the messages that arrive. But the great part is that it works with backtracking, and all your favourite combinators. So you can be as specific or as general as you'd like when describing the expected interaction with the server. ```haskell skipManyTill loggingNotification publishDiagnosticsNotification count 4 (message :: Session ApplyWorkspaceEditRequest) anyRequest <|> anyResponse ``` # Building up a test suite Other than throwing exceptions when it's unable to parse the incoming sequence of messages, lsp-test doesn't help you make assertions, so you are free to use whatever testing framework you like. In haskell-ide-engine, lsp-test is paired quite nicely with [`HSpec`](https://hackage.haskell.org/package/hspec). Here's an excerpt from some tests for goto definition requests: ```haskell spec :: Spec spec = describe "definitions" $ do it "goto's symbols" $ runSession hieCommand fullCaps "test/testdata" $ do doc <- openDoc "References.hs" "haskell" defs <- getDefinitions doc (Position 7 8) let expRange = Range (Position 4 0) (Position 4 3) liftIO $ defs `shouldBe` [Location (doc ^. uri) expRange] it "goto's imported modules" $ runSession hieCommand fullCaps "test/testdata/definition" $ do doc <- openDoc "Foo.hs" "haskell" defs <- getDefinitions doc (Position 2 8) liftIO $ do fp <- canonicalizePath "test/testdata/definition/Bar.hs" defs `shouldBe` [Location (filePathToUri fp) zeroRange] ``` # Replaying If your language server was built with `haskell-lsp`, you can use its built in capture format to take advantage of [`Language.Haskell.LSP.Test.Replay`](https://hackage.haskell.org/package/lsp-test-0.2.0.0/docs/Language-Haskell-LSP-Test-Replay.html). This module replays captured sessions and ensures that the response from the server matches up with what it received during the original capture. In `haskell-ide-engine` you can capture a session to a file by launching the server with the `--capture` flag. ```bash hie -c session.log ``` To test it with `lsp-test`, place the file as `session.log` inside a directory that mirrors the contents of the project root. ``` projectRoot ├── proj.cabal    ├── src │ └── ... └── session.log ``` And then test it with `replaySession`. ```haskell replaySession "hie" "projectRoot" ``` `lsp-test` is smart enough to swap out the absolute URIs, so if you originally captured the scenario under `/foo/bar/proj` but then replay it at `/test/foo`, `/foo/bar/proj/file.hs` will get swapped as `/test/foo/file.hs`. It also relaxes some "common-sense" checks: - Logging messages are ignored - The order of notifications in between requests and responses doesn't matter - It takes into account [uniqued command IDs](https://github.com/Microsoft/vscode-languageserver-node/issues/333) If the interaction doesn't match up, you'll get nice pretty printed JSON of what was received, what was expected, and the diff between the two[2](footnote:2). This is useful if you want to test for regressions, or if there is some very specific behaviour that would be difficult to describe programmatically. # What's next? ## DSL Currently if you want to use lsp-tests to write tests, you need to write Haskell. There has been [some experimenting](https://github.com/Bubba/haskell-lsp-test/blob/script-fsm/src/Language/Haskell/LSP/Test/Script.hs) in writing a custom DSL for describing tests. ``` "start" { wait for any then open "Test.hs" "haskell" } "get the symbols" { wait for method == "textDocument/publishDiagnostics" then open "Test.hs" "haskell" id1: request "textDocument/documentSymbol" { textDocument: { uri: uri "Test.hs" } } } "check the symbols" { wait for id == 1 then open "Test.hs" "haskell" } ``` However, it may be more attractive to use [dhall](https://github.com/dhall-lang/dhall-lang), a configuration language that ties in nicely with Haskell. The ultimate aim is that lsp-test will be available as an executable binary (via `cabal install`/`stack install`/`apt install`), which can read and run tests from a file via the command line. ```bash lsp-test test myTest.dhall ``` This would mean that servers implemented in languages other than Haskell can easily hook this into their CI without having to have a Haskell environment setup. ## Finite state machine and fuzzy testing My mentor pointed out that the interaction between the client and server during a Session could be described as a [Mealy machine](https://en.wikipedia.org/wiki/Mealy_machine). The state-transition function waits for the next message from the client whilst the output function sends messages to the server. ```haskell data State = Initialize | WaitForDiagnostics | MakeSymbolRequest | Done wait :: FromServerMessage -> State -> State send :: State -> FromServerMessage -> FromClientMessage ``` This led me to create a [proof of concept](https://github.com/Bubba/haskell-lsp-test/blob/ba3255afa89fd1faf4c8ed1a01ba482ec5755264/src/Language/Haskell/LSP/Test/Machine.hs) that integrated with [hedgehog](https://hackage.haskell.org/package/hedgehog-0.6/docs/Hedgehog.html#g:4). Hedgehog is a property based testing system a-la QuickCheck, but it also provides state machine testing. I had to first tweak the `Session` monad to become a `SessionT` transformer and make it an instance of `MonadTest` and `MonadThrow`[1](footnote:1) before it could be used. ```haskell type PropertySession = SessionT (PropertyT IO) instance MonadThrow m => MonadCatch (SessionT m) where catch f h = f instance MonadTest PropertySession where liftTest = lift . liftTest ``` And then the states could be described in terms of its pre-conditions, updates and post-conditions. ```haskell data OpenDoc (v :: * -> *) = OpenDoc deriving (Eq, Show) instance HTraversable OpenDoc where htraverse _ OpenDoc = pure OpenDoc s_openDoc_init :: (Monad n) => Command n PropertySession ModelState s_openDoc_init = let gen TDocClose = Just $ pure OpenDoc gen _ = Nothing execute OpenDoc = openDoc "Foo.hs" "haskell" in Command gen execute [ Require $ \s OpenDoc -> s == TDocClose , Update $ \_s OpenDoc o -> TDocOpen , Ensure $ \before after OpenDoc o -> do before === TDocClose let L.TextDocumentIdentifier uri = o uri === L.Uri "Foo.hs" after === TDocOpen ] ``` Once you describe a bunch of different states, you can then throw them into hedgehog and it will run them in random orders: Giving you fuzzy testing! ![Fuzzy testing](//lukelau.me/hedgehog.mov) # Lessons learnt The best way to get a taste for an API is by using it! By using it to write tests for haskell-ide-engine at an early stage in development, it allowed me to figure out the sticking points and refine the ergonomics. For example, the client capabilities used to be set inside [`SessionConfig`](https://hackage.haskell.org/package/lsp-test-0.1.0.0/docs/Language-Haskell-LSP-Test.html#t:SessionConfig). But when testing for backwards compatibility, it quickly became evident that it needed to be more explicit, hence why it was moved to `runSession`. If I was to start this project again, I would have focused a lot more on the DSL. Although language agnosticism was part the design, not including an easy way for non-Haskell environments to run tests was a major oversight. But with the framework in place, I am confident that we will be able to implement this soon. It's also important to note that this past summer wasn't spent solely on lsp-test, and I invested a large amount of work into haskell-ide-engine as well. I would happily say that lsp-test paid off while doing this work: it helped catch a lot of regressions while doing extensive refactoring and development around the [dispatcher](https://github.com/haskell/haskell-ide-engine/pull/594), [code actions](https://github.com/haskell/haskell-ide-engine/pull/659) and [plugin infrastructure](https://github.com/haskell/haskell-ide-engine/pull/710), not to mention the peace of mind from having a robust test-suite in place. Thanks to my mentor Alan Zimmerman for all his help, and all the community at #haskell-ide-engine and #haskell for putting up with all my stupid questions. If you're developing a language and want to write an LSP server, consider giving `lsp-test` a shot, and let me know! > footnotes 1. As you can see, this is clearly not the right way to implement `MonadThrow`. If you do know how to for a continuation based monad, please let me know! 2. Pro-tip: use the diff to incrementally update your session.log files if your expected output has changed.