6 My Haskell Summer of Code project, [lsp-test](https://github.com/Bubba/lsp-test), is now available [via Hackage](https://hackage.haskell.org/package/lsp-test-0.1.0.0).
7 It's a framework for writing end-to-end tests for LSP servers, made for testing [haskell-ide-engine](https://github.com/haskell/haskell-ide-engine).
9 But it's not just limited to haskell-ide-engine: It's language agnostic and works with any server that conforms to the [Language Server Protocol](https://microsoft.github.io/language-server-protocol/).
10 In fact lsp-test is basically a client that you can programmatically control. You specify what messages you want to send to the server, and check that the responses you get back are what you expected.
12 # What does it look like?
15 runSession :: String -> ClientCapabilities -> FilePath -> Session a -> IO a
17 main = runSession "hie" fullCaps "proj/dir" $ do
18 doc <- openDoc "Foo.hs" "haskell"
19 getDocumentSymbols doc >>= liftIO putStrLn
22 Each test is encapsulated by a `Session`: a client-server connection from start to finish.
23 Here we pass in the command to start the server, the capabilities that the client should declare and the root directory that the session should take place in.
24 (`fullCaps` is a convenience function that declares all the latest features in the LSP specification)
26 Once you're inside a `Session` you're free to poke and talk away to your server.
28 You might have noticed that the example above didn't send an initialize request - `Session` takes care of this and some other laborious parts of the process:
30 - Sending initialize requests with the correct process ID and capabilities
31 - Generating and incrementing message ids
32 - Keeping track of documents through `workspace/applyEdit` requests
34 There's also a [Smörgåsbord of functions](https://hackage.haskell.org/package/lsp-test-0.1.0.0/docs/Language-Haskell-LSP-Test.html#g:6) available for performing common tasks, such as getting a definition or checking the current diagnostics.
38 Most of the time you'll want to want to write your tests using these helper functions, but in case you're looking to do something more specific, you can drop down to a lower level and work with the individual messages that are sent and received.
40 The venerable [`request`](https://hackage.haskell.org/package/lsp-test-0.1.0.0/docs/Language-Haskell-LSP-Test.html#v:request) allows you to send any request defined in [`haskell-lsp-types`](https://hackage.haskell.org/package/haskell-lsp-types-0.6.0.0/docs/Language-Haskell-LSP-TH-DataTypesJSON.html), and will spit back the response for it.
41 Most of the helper functions are implemented in terms of this.
44 request :: (ToJSON params, FromJSON a) => ClientMethod -> params -> Session (ResponseMessage a)
46 runSession "hie" fullCaps "my/dir" = do
47 doc <- openDoc "Foo.hs" "haskell"
48 let params = DocumentSymbolParams doc
49 -- send and wait for the response
50 rsp <- request TextDocumentDocumentSymbol params
53 You can also use the `send` family of functions if you don't want to wait for a response.
56 sendRequest :: ToJSON params => ClientMethod -> params -> Session LspId
57 sendNotification :: ToJSON a => ClientMethod -> a -> Session ()
58 sendResponse :: ToJSON a => ResponseMessage a -> Session ()
63 This is where `lsp-test`'s little breakthrough comes in: The `Session` monad is actually just a [conduit parser](https://hackage.haskell.org/package/conduit-parse) that operates on individual messages.
66 type Session = ConduitParser FromServerMessage IO
68 runSession f = runConduit $ source .| runConduitParser f
70 source = getNextMessage handle >>= yield >> source
73 Incoming messages from the server are parsed in a stream.
74 You specify the messages you expect to receive from the server in order, in between the messages you send to it.
77 -- get a specific response
78 msg1 <- message :: Session RspDocumentSymbols
81 sendRequest TextDocumentDocumentSymbol params
82 -- get a logging notification
83 msg3 <- loggingNotification
86 It has it's own version of `satisfy` that works on messages, which custom parser combinators are built up upon.
89 satisfy :: (FromServerMessage -> Bool) -> Session FromServerMessage
96 loggingNotification = satisfy test
98 test (NotLogMessage _) = True
99 test (NotShowMessage _) = True
100 test (ReqShowMessage _) = True
104 Whenever it's unable to parse a sequence of messages, it throws an exception, which can be used to make assertions about the messages that arrive.
106 But the great part is that it works with backtracking, and all your favourite combinators. So you can be as specific or as general as you'd like when describing the expected interaction with the server.
109 skipManyTill loggingNotification publishDiagnosticsNotification
110 count 4 (message :: Session ApplyWorkspaceEditRequest)
111 anyRequest <|> anyResponse
114 # Building up a test suite
116 Other than throwing exceptions when it's unable to parse the incoming sequence of messages, lsp-test doesn't help you make assertions, so you are free to use whatever testing framework you like.
117 In haskell-ide-engine, lsp-test is paired quite nicely with [`HSpec`](https://hackage.haskell.org/package/hspec).
118 Here's an excerpt from some tests for goto definition requests:
122 spec = describe "definitions" $ do
123 it "goto's symbols" $ runSession hieCommand fullCaps "test/testdata" $ do
124 doc <- openDoc "References.hs" "haskell"
125 defs <- getDefinitions doc (Position 7 8)
126 let expRange = Range (Position 4 0) (Position 4 3)
127 liftIO $ defs `shouldBe` [Location (doc ^. uri) expRange]
129 it "goto's imported modules" $ runSession hieCommand fullCaps "test/testdata/definition" $ do
130 doc <- openDoc "Foo.hs" "haskell"
131 defs <- getDefinitions doc (Position 2 8)
133 fp <- canonicalizePath "test/testdata/definition/Bar.hs"
134 defs `shouldBe` [Location (filePathToUri fp) zeroRange]
139 If your language server was built with `haskell-lsp`, you can use its built in capture format to take advantage of [`Language.Haskell.LSP.Test.Replay`](https://hackage.haskell.org/package/lsp-test-0.2.0.0/docs/Language-Haskell-LSP-Test-Replay.html).
140 This module replays captured sessions and ensures that the response from the server matches up with what it received during the original capture.
141 In `haskell-ide-engine` you can capture a session to a file by launching the server with the `--capture` flag.
147 To test it with `lsp-test`, place the file as `session.log` inside a directory that mirrors the contents of the project root.
157 And then test it with `replaySession`.
160 replaySession "hie" "projectRoot"
163 `lsp-test` is smart enough to swap out the absolute URIs, so if you originally captured the scenario under `/foo/bar/proj` but then replay it at `/test/foo`, `/foo/bar/proj/file.hs` will get swapped as `/test/foo/file.hs`.
165 It also relaxes some "common-sense" checks:
166 - Logging messages are ignored
167 - The order of notifications in between requests and responses doesn't matter
168 - It takes into account [uniqued command IDs](https://github.com/Microsoft/vscode-languageserver-node/issues/333)
170 If the interaction doesn't match up, you'll get nice pretty printed JSON of what was received, what was expected, and the diff between the two[2](footnote:2).
172 This is useful if you want to test for regressions, or if there is some very specific behaviour that would be difficult to describe programmatically.
177 Currently if you want to use lsp-tests to write tests, you need to write Haskell.
178 There has been [some experimenting](https://github.com/Bubba/haskell-lsp-test/blob/script-fsm/src/Language/Haskell/LSP/Test/Script.hs) in writing a custom DSL for describing tests.
181 "start" { wait for any then open "Test.hs" "haskell" }
184 method == "textDocument/publishDiagnostics"
186 open "Test.hs" "haskell"
187 id1: request "textDocument/documentSymbol" {
193 "check the symbols" {
197 open "Test.hs" "haskell"
201 However, it may be more attractive to use [dhall](https://github.com/dhall-lang/dhall-lang), a configuration language that ties in nicely with Haskell.
203 The ultimate aim is that lsp-test will be available as an executable binary (via `cabal install`/`stack install`/`apt install`), which can read and run tests from a file via the command line.
206 lsp-test test myTest.dhall
209 This would mean that servers implemented in languages other than Haskell can easily hook this into their CI without having to have a Haskell environment setup.
211 ## Finite state machine and fuzzy testing
213 My mentor pointed out that the interaction between the client and server during a Session could be described as a [Mealy machine](https://en.wikipedia.org/wiki/Mealy_machine).
214 The state-transition function waits for the next message from the client whilst the output function sends messages to the server.
217 data State = Initialize
221 wait :: FromServerMessage -> State -> State
222 send :: State -> FromServerMessage -> FromClientMessage
225 This led me to create a [proof of concept](https://github.com/Bubba/haskell-lsp-test/blob/ba3255afa89fd1faf4c8ed1a01ba482ec5755264/src/Language/Haskell/LSP/Test/Machine.hs) that integrated with [hedgehog](https://hackage.haskell.org/package/hedgehog-0.6/docs/Hedgehog.html#g:4).
226 Hedgehog is a property based testing system a-la QuickCheck, but it also provides state machine testing.
228 I had to first tweak the `Session` monad to become a `SessionT` transformer and make it an instance of `MonadTest` and `MonadThrow`[1](footnote:1) before it could be used.
231 type PropertySession = SessionT (PropertyT IO)
233 instance MonadThrow m => MonadCatch (SessionT m) where
236 instance MonadTest PropertySession where
237 liftTest = lift . liftTest
240 And then the states could be described in terms of its pre-conditions, updates and post-conditions.
243 data OpenDoc (v :: * -> *) = OpenDoc
246 instance HTraversable OpenDoc where
247 htraverse _ OpenDoc = pure OpenDoc
249 s_openDoc_init :: (Monad n) => Command n PropertySession ModelState
251 let gen TDocClose = Just $ pure OpenDoc
253 execute OpenDoc = openDoc "Foo.hs" "haskell"
254 in Command gen execute [
255 Require $ \s OpenDoc -> s == TDocClose
256 , Update $ \_s OpenDoc o -> TDocOpen
257 , Ensure $ \before after OpenDoc o -> do
259 let L.TextDocumentIdentifier uri = o
260 uri === L.Uri "Foo.hs"
265 Once you describe a bunch of different states, you can then throw them into hedgehog and it will run them in random orders: Giving you fuzzy testing!
267 ![Fuzzy testing](//lukelau.me/hedgehog.mov)
271 The best way to get a taste for an API is by using it!
273 By using it to write tests for haskell-ide-engine at an early stage in development, it allowed me to figure out the sticking points and refine the ergonomics.
274 For example, the client capabilities used to be set inside [`SessionConfig`](https://hackage.haskell.org/package/lsp-test-0.1.0.0/docs/Language-Haskell-LSP-Test.html#t:SessionConfig).
275 But when testing for backwards compatibility, it quickly became evident that it needed to be more explicit, hence why it was moved to `runSession`.
277 If I was to start this project again, I would have focused a lot more on the DSL.
278 Although language agnosticism was part the design, not including an easy way for non-Haskell environments to run tests was a major oversight.
279 But with the framework in place, I am confident that we will be able to implement this soon.
281 It's also important to note that this past summer wasn't spent solely on lsp-test, and I invested a large amount of work into haskell-ide-engine as well.
282 I would happily say that lsp-test paid off while doing this work: it helped catch a lot of regressions while doing extensive refactoring and development around the [dispatcher](https://github.com/haskell/haskell-ide-engine/pull/594), [code actions](https://github.com/haskell/haskell-ide-engine/pull/659) and [plugin infrastructure](https://github.com/haskell/haskell-ide-engine/pull/710), not to mention the peace of mind from having a robust test-suite in place.
284 Thanks to my mentor Alan Zimmerman for all his help, and all the community at #haskell-ide-engine and #haskell for putting up with all my stupid questions.
285 If you're developing a language and want to write an LSP server, consider giving `lsp-test` a shot, and let me know!
288 1. As you can see, this is clearly not the right way to implement `MonadThrow`. If you do know how to for a continuation based monad, please let me know!
289 2. Pro-tip: use the diff to incrementally update your session.log files if your expected output has changed.