--- /dev/null
+_cache
+_site
+dist
+dist-newstyle
+.DS_Store
+.ghc.environment*
+*.swp
--- /dev/null
+---
+title: About
+---
+Nullam imperdiet sodales orci vitae molestie. Nunc quam orci, pharetra a
+rhoncus vitae, eleifend id felis. Suspendisse potenti. Etiam vitae urna orci.
+Quisque pellentesque dignissim felis, egestas tempus urna luctus vitae. In hac
+habitasse platea dictumst. Morbi fringilla mattis odio, et mattis tellus
+accumsan vitae.
+
+1. Amamus Unicode 碁
+2. Interdum nex magna.
+
+Vivamus eget mauris sit amet nulla laoreet lobortis. Nulla in diam elementum
+risus convallis commodo. Cras vehicula varius dui vitae facilisis. Proin
+elementum libero eget leo aliquet quis euismod orci vestibulum. Duis rhoncus
+lorem consequat tellus vestibulum aliquam. Quisque orci orci, malesuada porta
+blandit et, interdum nec magna.
--- /dev/null
+packages:
+ ./
+constraints: hakyll +watchserver +previewserver
--- /dev/null
+---
+title: Contact
+---
+
+I live in a small hut in the mountains of Kumano Kodō on Kii Hantō and would not
+like to be contacted.
--- /dev/null
+html {
+ font-size: 0.6em;
+ font-family: -apple-system, verdana, sans-serif;
+}
+
+body {
+ font-size: 1.6rem;
+ color: #333;
+}
+
+header {
+ margin: 1rem 0;
+}
+
+nav {
+ text-align: left;
+}
+
+nav a {
+ color: inherit;
+ font-size: 1.8rem;
+ font-weight: bold;
+ text-decoration: none;
+ margin: 0 0 0 1.2rem;
+ display: inline;
+}
+
+.logo {
+ margin: 0;
+ text-align: left;
+}
+
+.logo a {
+ float: left;
+ font-size: 1.8rem;
+}
+
+pre {
+ overflow-x: scroll;
+}
+
+footer {
+ margin-top: 3rem;
+ padding: 1.2rem 0;
+ font-size: 1.2rem;
+ color: #555;
+ text-align: right;
+}
+
+h1 {
+ font-size: 2.4rem;
+}
+
+h2 {
+ font-size: 2rem;
+}
+
+article .header {
+ font-size: 1.4rem;
+ font-style: italic;
+ color: #555;
+}
+
+article img {
+ max-width: 100%;
+}
+
+.logo a {
+ font-weight: bold;
+ color: #000;
+ text-decoration: none;
+}
+
+/* on mobile prevent stupidly long inline code snippets */
+p code {
+ word-break: break-word;
+}
+
+@media (max-width: 319px) {
+ body {
+ width: 90%;
+ margin: 0;
+ padding: 0 5%;
+ }
+}
+
+@media (min-width: 320px) {
+ body {
+ width: 90%;
+ margin: 0;
+ padding: 0 5%;
+ }
+}
+
+@media (min-width: 640px) {
+ body {
+ width: 60rem;
+ margin: 0 auto;
+ padding: 0;
+ }
+}
--- /dev/null
+.source-code .dt {
+ color: #6f42c1
+}
+
+.source-code .dv {
+ color: #fd7e14
+}
+
+.source-code .cf {
+ font-weight: 700;
+ color: #2727af
+}
+
+.source-code .bn {
+ color: #17a2b8
+}
+
+.source-code .fu {
+ color: #007bff
+}
+
+.source-code .kw {
+ font-weight: 700;
+ color: #2727af
+}
+
+.source-code .pr {
+ color: #dc3545
+}
+
+.source-code .sy {
+ color: #e83e8c
+}
+
+.source-code .va {
+ color: #007bff
+}
+
+.source-code .cr {
+ color: #6f42c1
+}
+
+.source-code .op {
+ color: #2727af
+}
+
+.source-code .ch {
+ color: #17a2b8
+}
+
+.source-code .st {
+ color: #20c997
+}
+
+.source-code .it {
+ color: #fd7e14
+}
+
+.source-code .ra {
+ color: #fd7e14
+}
+
+.source-code .co {
+ color: #6c757d
+}
--- /dev/null
+---
+title: 🔌 Making a HIE plugin
+date: 2018-09-24
+---
+
+haskell-ide-engine provides a lot of different language features, from giving diagnostics, to refactoring names and suggesting imports.
+How does one server manage to do so much? Well the truth is that haskell-ide-engine is actually several plugins stacked on top of each other in a trenchcoat.
+Most of the functionality comes from existing tools already in the ecosystem:
+
+- ghc-mod provides the diagnostics
+- HaRe provides refactoring
+- hlint provides quick fixes
+- hsimport provides import suggestions
+- hoogle provides documentation
+
+HIE provides endpoints for these plugins to tap into various LSP features:
+
+```haskell
+data PluginDescriptor =
+ PluginDescriptor { ...
+ , pluginCodeActionProvider :: Maybe CodeActionProvider
+ , pluginDiagnosticProvider :: Maybe DiagnosticProvider
+ , pluginHoverProvider :: Maybe HoverProvider
+ , pluginSymbolProvider :: Maybe SymbolProvider
+ }
+```
+
+# Code actions
+
+```haskell
+type CodeActionProvider = PluginId
+ -> VersionedTextDocumentIdentifier
+ -> Maybe FilePath -- ^ Project root directory
+ -> Range
+ -> CodeActionContext
+ -> IdeM (IdeResult [CodeAction])
+```
+
+
+
+Code actions are the main method of exposing functionality to the client within LSP.
+Whenever the client requests the code actions for the document, HIE will query your plugin with information about the current document and position, and you can return back any relevant code actions.
+Where your code actions will appear (e.g. in the refactor menu or as quick-fix) is determined by the `CodeActionKind` of the `CodeAction`, so make sure to choose a type that is suitable for the type of action that you are providing:
+
+```haskell
+data CodeActionKind = CodeActionQuickFix
+ | CodeActionRefactor
+ | CodeActionRefactorExtract
+ | CodeActionRefactorInline
+ | CodeActionRefactorRewrite
+ | CodeActionSource
+ | CodeActionSourceOrganizeImports
+```
+
--- /dev/null
+name: haskell-blog
+version: 0.1.0.0
+build-type: Simple
+cabal-version: >= 1.10
+
+executable site
+ main-is: site.hs
+ build-depends: base == 4.*
+ , hakyll >= 4.12
+ , mmark
+ , mmark-ext
+ , megaparsec
+ , filepath
+ , text
+ , lucid
+ , eventlog2html
+ , blaze-html
+ , directory
+ , containers
+ ghc-options: -threaded
+ default-language: Haskell2010
--- /dev/null
+---
+title: Posts
+---
+
+$partial("templates/post-list.html")$
--- /dev/null
+---
+title: ⌨️ MMark and Hakyll
+date: 2018-07-29
+---
+
+I've moved over the blog to Hakyll, so things are a bit less makeshfit.
+
+I really the syntax highlighting, but one problem I noticed was that the code blocks didn't show up on Safari Reader.
+After doing a bit of digging it turned out that some time around Pandoc 2.0.4, skylighting
+started adding line numbers that had an `<a>` tag, which seemed to throw Safari off.
+
+```html
+<code class="sourceCode haskell">
+ <a class="sourceLine" id="cb1-1" data-line-number="1">
+ <span class="kw">data</span>
+ <span class="dt">RequestMessage</span> m req resp <span class="fu">=</span>
+ <span class="fu">...</span>
+ </a>
+</code>
+```
+
+Unfortunately there doesn't seem to be a way to turn this off, and rather than try to strip off those tags myself,
+I decided just to use another compiler other than Pandoc.
+
+Since I was only going to be compiling markdown, I landed on [MMark](https://hackage.haskell.org/package/mmark).
+Unlike other compilers, MMark is strict - I plugged it into hakyll:
+
+```haskell
+mmarkCompiler :: Compiler (Item String)
+mmarkCompiler = do
+ fp <- getResourceFilePath
+ getResourceLBS >>= withItemBody (\lbs ->
+ let text = T.toStrict $ T.decodeUtf8 lbs
+ in case MMark.parse fp text of
+ Left e -> error (MMark.parseErrorsPretty text e)
+ Right doc -> return $ T.unpack $ Lucid.renderText $ MMark.render doc
+```
+
+And right off the bat it picked up a bunch of mistakes in my existing blog posts:
+
+```
+Initialising...
+ Creating store...
+ Creating provider...
+ Running rules...
+Checking for out-of-date items
+Compiling
+ updated templates/default.html
+ updated about.rst
+ updated templates/post.html
+ updated posts/hakyll.md
+ updated posts/haskellMake.md
+ updated posts/hello.md
+ [ERROR] ./posts/lens.md:177:66:
+ |
+177 | In `haskell-lsp` these are also generated via (Template Haskell)[https://artyom.me/lens-over-tea-6].
+ | ^
+could not find a matching reference definition for "https://artyom.me/lens-over-tea-6"
+```
+
+It also comes with an extension for [ghc-syntax-highlighter](https://markkarpov.com/post/announcing-ghc-syntax-highlighter.html),
+which was what I came for in the first place.
+Setting it up was super easy:
+
+```haskell
+MMark.render $ MMark.useExtensions extensions doc
+ where extensions = [ MMark.ghcSyntaxHighlighter
+ , MMark.skylighting
+ ]
+```
+
+And now I have clean, Safari-readable tags:
+
+```html
+<code class="sourceCode haskell">
+ <span class="kw">data</span>
+ <span class="dt">RequestMessage</span> m req resp <span class="fu">=</span>
+ <span class="fu">...</span>
+</code>
+```
+
--- /dev/null
+---
+title: 📖 Blog script with Haskell
+date: 2018-05-25
+---
+
+```haskell
+ import Prelude hiding (readFile, writeFile, concat)
+ import System.Directory
+ import System.FilePath
+ import Data.Text.Lazy hiding (filter, map, reverse)
+ import Data.Text.Lazy.IO
+ import Text.Markdown
+ import Text.Blaze.Html.Renderer.Text
+
+ main = do
+ files <- getDirectoryContents "."
+ let mds = filter ((== ".md") . takeExtension) files
+ content <- mapM (readFile) mds
+ let htmls = reverse $ map (renderHtml . markdown def) content
+
+ header <- readFile "header.html"
+ footer <- readFile "footer.html"
+
+ writeFile "index.html" $ concat $ header:htmls ++ [footer]
+```
+
+I'd eventually like to serve this from some Haskell webserver rather than manually generating static pages.
+Maybe also use [this?](https://github.com/mrkkrp/ghc-syntax-highlighter)
--- /dev/null
+---
+title: 🚰 Hunting leaks with GDB
+date: 2019-06-16
+---
+
+There's currently a nasty memory leak in
+[haskell-ide-engine](https://github.com/haskell/haskell-ide-engine) which leaks
+all the cached module information about from GHC. It seems to only occur on
+certain platforms, but on platforms unfortunate to be affected by it, a sizeable
+portion memory is leaked every time the user types or edits their document. For
+a small module of about 60 lines, this was 30-ishMB per edit. (Yikes)
+
+During this year's ZuriHac, Matthew Pickering, Daniel Gröber and I tried to get
+this sorted out once and for all: It ended up taking the entire weekend, and I
+learnt more about the RTS than I ever intended to.
+
+# Profiling
+
+The first step in the investigation began with figuring out what exactly was
+leaking. To do this we built `hie` with [profiling
+enabled](https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/profiling.html):
+This injects *call site* annotations into your code, and causes your executable
+to be linked with one of the versions of the RTS that was built with profiling.
+
+```bash
+$ ghc -prof Main.hs
+# or in our case, for cabal projects
+$ cabal v2-install :hie --enable-profiling
+```
+
+GHC comes with a bunch of different RTS libraries, each built with a
+different combination of features.
+
+```bash
+$ ls `ghc --print-libdir`/rts | grep rts
+libHSrts-ghc8.6.5.dylib
+libHSrts.a
+libHSrts_debug-ghc8.6.5.dylib
+libHSrts_debug.a
+libHSrts_l-ghc8.6.5.dylib
+libHSrts_l.a
+libHSrts_p.a
+libHSrts_thr-ghc8.6.5.dylib
+libHSrts_thr.a
+libHSrts_thr_debug-ghc8.6.5.dylib
+libHSrts_thr_debug.a
+libHSrts_thr_l-ghc8.6.5.dylib
+libHSrts_thr_l.a
+libHSrts_thr_p.a
+```
+
+`thr` stands for threading, `debug` for debug, `l` for [event
+logging](#bonus-event-logs) and `p` for profiling. If you pass GHC `-v`, you should
+see `-lHSrts_thr_p` or equivalent when linking: We're bringing in the RTS built
+with threading and profiling enabled.
+
+Now we could profile our executable by running it with specific flags passed to
+the RTS. To see what's available, we ran `hie +RTS --help`
+```
+...
+hie: -h<break-down> Heap residency profile (hp2ps) (output file <program>.hp)
+hie: break-down: c = cost centre stack (default)
+hie: m = module
+hie: d = closure description
+hie: y = type description
+hie: r = retainer
+hie: b = biography (LAG,DRAG,VOID,USE)
+...
+```
+
+Here we used use the retainer breakdown, `-hr`, to help us find what exactly is
+being retained i.e. leaking.
+We originally just configured the language client to launch `hie` with the flag,
+but we found ourselves doing it so often that we created a test scenario with
+[lsp-test](https://hackage.haskell.org/package/lsp-test) and automated the
+process:
+
+```haskell
+#!/usr/bin/env cabal
+{- cabal:
+build-depends: base, lsp-test
+-}
+import Language.Haskell.LSP.Test
+import Language.Haskell.LSP.Types
+import Control.Monad
+main = do
+ runSession ("hie +RTS -hr") fullCaps "../test" $ do
+ doc <- openDoc "TestFile.hs" "haskell"
+ waitForDiagnostics
+ replicateM_ 20 $ do
+ sendNotification TextDocumentDidSave (DidSaveTextDocumentParams doc)
+ waitForDiagnostics -- wait before sending it again
+```
+
+Then we profiled and got some data:
+```bash
+$ ./Leak.hs
+$ hp2ps -c hie.ps # c for colour
+```
+
+
+
+You can see `tcRnImports` and `zonkEvBinds` gradually eating up more memory over
+time: These are parts of the GHC API and so I narrowed it down to most likely
+coming from our `UriCaches`, where we store the `TypecheckedModule` and
+`ParsedModule`.
+
+```haskell
+type UriCaches = Map.Map FilePath UriCacheResult
+
+data UriCacheResult = UriCacheSuccess UriCache
+ | UriCacheFailed
+
+data UriCache = UriCache
+ { cachedInfo :: !CachedInfo
+ , cachedPsMod :: !ParsedModule
+ , cachedTcMod :: !(Maybe TypecheckedModule)
+ -- | Data pertaining to the typechecked module,
+ -- not the parsed module
+ , cachedData :: !(Map.Map TypeRep Dynamic)
+ }
+```
+
+# The weak pointer trap
+
+Now that we knew *what* was leaking, the question turned to *where* was it leaking.
+The next steps are taken from [Simon Marlow's excellent blog
+post](http://simonmar.github.io/posts/2018-06-20-Finding-fixing-space-leaks.html),
+in which he details how he tracked down and fixed multiple memory leaks in GHCi.
+
+The trick is to create a weak pointer -- a pointer that the garbage collector
+ignores when looking for retainers. Point it to the offending type and store it
+alongside the original, strongly referenced one.
+Whenever you know the object should have been deallocated, force garbage
+collection by calling `performGC` and dereference the weak pointer. If the
+pointer is null, it was released, otherwise something is still holding onto it:
+the object is being leaked.
+
+We modified `UriCache` to look like the following:
+
+```haskell
+data UriCacheResult = UriCacheSuccess (Weak UriCache) UriCache
+ | UriCacheFailed
+
+mkUriCacheSuccess :: UriCache -> IO UriCacheResult
+mkUriCacheSuccess uc = do
+ -- second arg is callback when GC is run
+ ptr <- mkWeakPtr uc Nothing
+ return (UriCacheSucces ptr uc)
+```
+
+We knew that when caching a module in the `cacheModule` function, it should
+replace the old cached module, so that seemed like a sensible place to add in
+the `performGC` trap.
+
+```haskell
+cacheModule = do
+ -- ...
+ res <- liftIO $ mkUriCacheSuccess newUc
+
+ maybeOldUc <- Map.lookup uri' . uriCaches <$> getModuleCache
+
+ modifyCache $ \gmc ->
+ gmc { uriCaches = Map.insert uri' res (uriCaches gmc) }
+
+ -- check leaks
+ case maybeOldUc of
+ Just (UriCacheSuccess wkPtr _) -> liftIO $ do
+ liftIO performGC
+ derefed <- deRefWeak wkPtr
+ case derefed of
+ Just _ -> error "Leaking!"
+ Nothing -> hPutStrLn stderr "Not leaking"
+ _ -> return ()
+```
+
+With our trap set up in place, we ran the test again.
+
+```
+$ ./Leak.hs
+...
+2019-06-20 00:01:32.650578 [ThreadId 4] - Using stack GHC version
+2019-06-20 00:01:33.826226 [ThreadId 4] - Run entered for HIE(hie) Version 0.10.0.0, Git revision d679eb595a6a1583a8b63a3fdd5964f84a8d00f6 (dirty) (2801 commits) x86_64 ghc-8.6.5
+2019-06-20 00:01:33.826997 [ThreadId 4] - Current directory:/Users/luke/Source/haskell-ide-engine
+2019-06-20 00:01:33.836594 [ThreadId 9] - Using stack GHC version
+hie: Leaking!
+CallStack (from HasCallStack):
+ error, called at ./Haskell/Ide/Engine/ModuleCache.hs:255:19 in hie-plugin-api-0.10.0.0-inplace:Haskell.Ide.Engine.ModuleCache
+CallStack (from -prof):
+ Haskell.Ide.Engine.ModuleCache.cacheModule (Haskell/Ide/Engine/ModuleCache.hs:(221,1)-(260,29))
+ Haskell.Ide.Engine.ModuleCache.CAF (<entire-module>)
+```
+
+Nice. At this point we knew the *what* and the *where*, all we needed to know
+was the *how*.
+
+# Poking about in GDB
+
+For this there is the convenient `findPtr` function included in
+the debug RTS. It's a C function that we can call during runtime from a
+debugger to find out what is referencing a closure. To access it we first needed
+to pass two flags to GHC[1](footnote:1) (or in our case set them in Cabal):
+
+```bash
+$ ghc -debug -fwhole-archive-hs-libs Main.hs
+```
+
+`-debug` brings in the version of RTS with debug stuff in it, whilst
+`-fwhole-archive-hs-libs` is needed to prevent the linker from optimising out
+the `findPtr` function: Nothing actually calls it, but we still want it
+available in our final binary.
+
+After rebuilding `hie`, it was time to hijack:
+
+```bash
+# after hie bombs
+$ gdb -pid `pidof hie`
+# alternatively, beforehand:
+$ gdb -ex "set follow-fork-mode child" ./Leak.hs
+```
+
+Now that we were in, we needed to find out the address of the [info
+table](https://gitlab.haskell.org/ghc/ghc/wikis/commentary/rts/storage/heap-objects#info-tables):
+We're looking for a symbol that ends in `UriCacheSuccess_con_info`.
+
+```
+(gdb) info functions UriCacheSuccess_con_info
+All functions matching regular expression "UriCacheSuccess_con_info":
+
+Non-debugging symbols:
+0x0000000111956650 hiezmpluginzmapizm0zi10zi0zi0zminplace_HaskellziIdeziEngineziGhcModuleCache_UriCacheSuccess_con_info_dsp
+0x0000000111956668 hiezmpluginzmapizm0zi10zi0zi0zminplace_HaskellziIdeziEngineziGhcModuleCache_UriCacheSuccess_con_info
+```
+
+We plugged in the rather gnarly
+`hiezmpluginzmapizm0zi10zi0zi0zminplace_HaskellziIdeziEngineziGhcModuleCache_UriCacheSuccess_con_info`
+into `findPtr`, which will go ahead and find all closures in the heap referring
+to a `UriCacheSuccess`:
+
+```
+(gdb) p (void)findPtr(...UriCacheSuccess_con_info, 1)
+0x4205499088 = hie-plugin-api-0.10.0.0-inplace:Haskell.Ide.Engine.GhcModuleCache.UriCacheSuccess(0x4205499139, 0x4205499110)
+0x42154a88a8 = hie-plugin-api-0.10.0.0-inplace:Haskell.Ide.Engine.GhcModuleCache.UriCacheSuccess(0x42154be8d1, 0x42154a8a30)
+```
+
+And now we have the culprit retainers lined up right before us. We can go deeper
+with another call to `findPtr`: here the 1 passed to it means to follow the
+trail.
+
+```
+(gdb) p (void)findPtr(0x4205499088, 1)
+
+== in ./Leak.hs stderr ==
+
+0x4205499088 = hie-plugin-api-0.10.0.0-inplace:Haskell.Ide.Engine.GhcModuleCache.UriCacheSuccess(0x4205499139, 0x4205499110)
+0x42154a88a8 = hie-plugin-api-0.10.0.0-inplace:Haskell.Ide.Engine.GhcModuleCache.UriCacheSuccess(0x42154be8d1, 0x42154a8a30)
+0x4205499148 = THUNK(0x43b9600, 0x4205499072, 0x4205499089, 0x42154a93c1)
+-->
+0x42054990a0 = hie-plugin-api-0.10.0.0-inplace:Haskell.Ide.Engine.PluginsIdeMonads.IdeState(0x4205499148, 0x42154a9370, 0xc19ff3a, 0x42154a9390)
+-->
+0x42154be928 = TVAR(value=0x42054990a1, wq=0xc353d90, num_updates=0)
+-->
+0x42154be918 = base:GHC.Conc.Sync.TVar(0x42154be928)
+-->
+0x42154bebf8 = STACK
+0x42154bf548 = FUN/1(0x43f76b0, 0x42154be919)
+0x42154bf5c8 = FUN/1(0x43f76b0, 0x42154be919)
+0x42001cb000 = STACK
+0x42001cb000 = STACK
+0x42001cb000 = STACK
+0x42001cb000 = STACK
+```
+
+Is that an unevaluated thunk? Let's take a closer look at it.
+
+```
+0x4205499148 = THUNK(0x43b9600, 0x4205499072, 0x4205499089, 0x42154a93c1)
+```
+
+The first address we see on the right is the address of the thunk on the heap.
+To the right of the `=` is its contents. We can see the layout of such a thunk
+in GHC's source, at `rts/storage/Closures.h`:
+
+```c
+typedef struct {
+ StgThunkHeader header;
+ struct StgClosure_ *payload[];
+} StgThunk;
+
+typedef struct {
+ const StgInfoTable* info;
+#if defined(PROFILING)
+ StgProfHeader prof;
+#endif
+ StgSMPThunkHeader smp;
+} StgThunkHeader;
+
+typedef struct {
+ StgWord pad;
+} StgSMPThunkHeader;
+```
+
+So the first address we're seeing, `0x43b9600`, is the `const StgInfoTable* info`
+pointer. It's then followed by a word of padding, which `findPtr` leaves out for
+us. (The profiling enabled version of the RTS also passes about an extra word of
+profiling info)
+
+The remaining values are the free variables of the function. There's a lot more
+to talk about [the runtime representation of
+thunks](https://gitlab.haskell.org/ghc/ghc/wikis/commentary/rts/storage/heap-objects#thunks),
+, more than I could ever fit into a blog post, so I'll return to our leak.
+
+# Debugging info
+
+We now knew the thunk that was retaining the `UriCache`s, but we didn't know
+where it was being created. We could figure out where it's coming from by
+compiling `hie` with debug information: not to be confused with linking with the
+debug version of the RTS.
+
+```bash
+$ ghc -g3 -debug -fwhole-archive-hs-libs Main.hs
+# or via cabal
+# (make sure to stop cabal from just stripping it back out)
+$ cabal v2-build hie --enable-debug-info --disable-executable-stripping --disable-library-stripping
+# or in your cabal.project
+$ cat cabal.project
+packages:
+ ./
+debug-info: 3
+executable-stripping: False
+library-stripping: False
+```
+
+This attached source locations to the info tables, so after another rebuild and
+another GDB session, we could simply print out the source code listing for the
+closure:
+
+```
+0x42104e1148 = THUNK(0x43c6280, 0x42104e1072, 0x42104e1089, 0x4217654a31)
+
+# meanwhile in gdb
+(gdb) list *0x43c6280
+56 -- ---------------------------------------------------------------------
+57
+58 modifyCache :: (HasGhcModuleCache m) => (GhcModuleCache -> GhcModuleCache) -> m ()
+59 modifyCache f = do
+60 mc <- getModuleCache
+61 setModuleCache (f mc)
+62
+63 -- ---------------------------------------------------------------------
+64 -- | Runs an IdeM action with the given Cradle
+65 withCradle :: (GM.GmEnv m) => GM.Cradle -> m a -> m a
+```
+
+And there we had it: unevaluated thunks of `f` piling up and eating memory.
+
+[The solution for
+this](https://github.com/mpickering/haskell-ide-engine/commit/c3bdae0f7df63c8bdb4f360285feeb7eadff8966)
+is usually [making sure to strictly
+evaluate](https://github.com/haskell/haskell-ide-engine/pull/1292/files) the
+[offending thunk](https://github.com/haskell/haskell-ide-engine/pull/1305/files).
+
+We used this technique to hunt down several retainers, but be warned: stifling
+one retainer isn't necessarily guaranteed to fix the memory leak, as it is
+possible (and likely) that another retainer out exists and is still holding on.
+
+But now we know what we're doing. We'll be using this to squash more memory
+leaks down the line, and hopefully you can too.
+
+# Bonus: Event logs
+
+In addition to profiling via `-hr`, it's possible to capture event logs. They
+can be thought of as a more modern version of the heap profiles.
+If your executable was compiled with `-debug`, all you need to do is pass `-l`
+alongside your regular profiling flags:
+
+```bash
+$ hie +RTS -r -l
+```
+
+This will produce a little `exe.eventlog` file which you can use with
+profiling tools like ThreadScope.
+
+Matthew Pickering has made a handy tool,
+[eventlog2html](https://mpickering.github.io/eventlog2html/), to convert these
+into interactive graphs. I'll leave you with the trace of our memory leak
+earlier:
+
+```eventlog
+hie.eventlog
+```
+
+> footnotes
+ 1. GHC 8.6.5 and earlier don't come with an RTS with both debugging and
+ profiling enabled, so we've disable profiling here. This has
+ [been fixed in GHC 8.8](https://github.com/ghc/ghc/commit/d36b1ffac9960db70043aaab43c931ce217912ba#diff-21dcbd7f6f00e9a602377d3f69bff4ba).
--- /dev/null
+---
+title: 🚰 Hunting Leaks
+date: 2019-06-16
+---
+
+There's currently a nasty memory leak in
+[haskell-ide-engine](https://github.com/haskell/haskell-ide-engine) which leaks
+all the cached information about a module from GHC. It seems to only occur on
+some platforms, but on platforms unfortunate to be affected by it, it means that
+a sizeable portion memory is leaked every time the user types.
+For a small module of about 60 lines, this was 30-ishMB.
+
+During this year's ZuriHac, Matthew Pickering, Daniel Gröber and I tried to get
+this sorted out once and for all: It ended up taking the entire weekend!
+
+# Profiling
+The first step in the investigation began with figuring out what exactly was
+leaking: Prior to ZuriHac, I had heap profiled HIE with a sample session. To do
+this I had to build the executable with profiling enabled: This adds some extra
+information regarding closures and "call sites", and causes your executable to
+be linked with one of the versions of the RTS that was built with
+profiling.
+
+```bash
+ghc -prof Main.hs
+# or in our case, for cabal projects
+cabal v2-install :hie --enable-profiling
+
+```
+GHC comes with a bunch of different RTS libraries, each built with a
+different combination of features.
+
+```bash
+ls `ghc --print-libdir`/rts | grep rts
+libHSrts-ghc8.6.5.dylib
+libHSrts.a
+libHSrts_debug-ghc8.6.5.dylib
+libHSrts_debug.a
+libHSrts_l-ghc8.6.5.dylib
+libHSrts_l.a
+libHSrts_p.a
+libHSrts_thr-ghc8.6.5.dylib
+libHSrts_thr.a
+libHSrts_thr_debug-ghc8.6.5.dylib
+libHSrts_thr_debug.a
+libHSrts_thr_l-ghc8.6.5.dylib
+libHSrts_thr_l.a
+libHSrts_thr_p.a
+```
+
+`thr` stands for threading, `debug` for debug, `l` for ??? and `p` for
+profiling. If you pass GHC `-v`, you should see
+`-lHSrts_thr_p` or equivalent when linking: We're bringing in the RTS built with
+threading and profiling enabled.
+
+Now we can profile our executable by running it with specific flags passed to
+the RTS. To see what's available, we ran `hie +RTS --help`
+```
+...
+hie: -h<break-down> Heap residency profile (hp2ps) (output file <program>.hp)
+hie: break-down: c = cost centre stack (default)
+hie: m = module
+hie: d = closure description
+hie: y = type description
+hie: r = retainer
+hie: b = biography (LAG,DRAG,VOID,USE)
+...
+```
+
+# Weak pointers
+
+Now that we know what is leaking, the question turns to where is it leaking.
+The next steps are taken from [Simon Marlow's excellent blog
+post](http://simonmar.github.io/posts/2018-06-20-Finding-fixing-space-leaks.html),
+in which he details how he tracked down and fixed multiple memory leaks in GHCi.
+
+The trick is to create a weak pointer -- a pointer that the garbage collector
+ignores when looking for retainers. Point it to the offending type and store it
+alongside the original, strongly referenced one.
+Whenever you know the object should have been dellocated, force garbage
+collection by calling `performGC` and dereference the weak pointer. If the
+pointer is null, it was released, otherwise something is still holding onto it:
+the object is being leaked!
+
+In
+
+
+```
+(lldb) e findPtr((P_)myobjptr & ~0b111, 1)
+0x420bbdec50 = hie-plugin-api-0.10.0.0-inplace:Haskell.Ide.Engine.GhcModuleCache.UriCache(0x420bbc04f9, 0x420fdff691, 0x420fdfd39a, 0x10cca094a, 0x420bbc0539)
+0x420fdfd150 = WEAK(key=0x420fdfd39a value=0x420fdfd39a finalizer=0x10c56ba41)
+0x420fdfd150 = WEAK(key=0x420fdfd39a value=0x420fdfd39a finalizer=0x10c56ba41)
+```
+
+```
+-debug -g -fwhole-archive-hs-libs
+```
--- /dev/null
+---
+title: 🔭 Lenses and extensions
+date: 2018-06-12
+---
+
+Recently I found myself needing to change a field in a record nested a couple of layers deep.
+I was working with `haskell-lsp` messages, of which there are several flavours:
+
+```haskell
+data RequestMessage m req resp = ...
+data ResponseMessage a = ...
+data NotificationMessage m a =
+ NotificationMessage { jsonrpc :: Text, method :: m, params :: a }
+```
+
+I wanted to change the `params` of the message. `params` is polymorphic/generic, so we don't know for certain what it's going to be. You can do this with plain old records like so:
+
+```haskell
+foo :: NotificationMessage m a -> NotificationMessage m a
+foo x = x { params = spiceUp (params x) }
+```
+
+However we've already hit our first problem. We would need to write this method for each type of message:
+
+```haskell
+
+fooNotification :: NotificationMessage m a -> NotificationMessage m a
+fooNotification x = x { params = spiceUp (params x) }
+
+fooRequest :: RequestMessage m req resp -> RequestMessage m req resp
+fooRequest x = x { params = spiceUp (params x) }
+
+...
+```
+
+The implementations are entirely identical which is a painful waste and we need to declare all these functions with different names.
+Thankfully, as you might expect, this common functionality of grabbing some `params` is abstracted to a class, the `HasParams` class:
+
+```haskell
+class HasParams s a where
+ getParams :: s -> a
+ setParams :: a -> s -> s
+
+instance HasParams (RequestMessage m req resp) req where
+ getParams = params
+ setParams x p = x { params = p }
+
+instance HasParams (NotificationMessage m a) req where
+ getParams = params
+ setParams x p = x { params = p }
+```
+
+You may be wondering how this works when there's the duplicate record field params, but this can be remedied with the [DuplicateRecordFields](https://ghc.haskell.org/trac/ghc/wiki/Records/OverloadedRecordFields/DuplicateRecordFields) extension.
+We also need [MultiParamTypeClasses](https://ghc.haskell.org/trac/haskell-prime/wiki/MultiParamTypeClasses).
+But now, we can rewrite foo to be nice and polymorphic:
+
+```haskell
+foo :: HasParams s a => s -> s
+foo x = setParams x (spiceUp (getParams x))
+```
+
+Looks good. I didn't really want the `params` field though, what I actually wanted was a URI field, which is nested several layers deep:
+
+```
+message -> params -> document -> uri
+```
+
+So we just need to create another class to represent types that have documents right?
+
+```haskell
+instance HasDocument s where
+ getDocument :: s -> Document
+ setDocument :: s -> Document -> Document
+
+instance HasDocument MyParams where
+ getDocument = document
+ setDocument x d = x { document = d }
+```
+
+Now we can chain these type class requirements together:
+
+```haskell
+foo :: (HasParams a b, HasDocument b) => s -> s
+foo x = setParams x newParams
+ where oldParams = getParams x
+ oldDoc = getDocument oldParams
+ newDoc = spiceUp oldDoc
+ newParams = setDocument oldParams newDoc
+```
+
+But as it turns out, you can have different types of documents, like versioned documents or file paths to documents, so `document` is polymorphic just like `params`. And we still need to go another layer deep to get the URI (which is also polymorphic).
+
+```haskell
+instance HasDocument s d where
+ getDocument :: s -> d
+ setDocument :: s -> d -> d
+
+instance HasUri s u where
+ getUri :: s -> u
+ setUri :: s -> u -> u
+
+foo :: (HasParams a p, HasDocument p d, HasUri d u) => a -> a
+foo x = setParams x newParams
+ where oldParams = getParams x
+ oldDoc = getDocument oldParams
+ oldUri = getUri oldDoc
+ newUri = spiceUp newUri
+ newDoc = setUri oldDoc newUri
+ newParams = setDoc oldParams newDoc
+```
+
+Things are starting to look a bit sad again. Not only do we have to write tons of repeated instances for each class, we also have this weird list where we change all the record fields at each level. This is where lenses come in.
+
+## Lenses
+
+```haskell
+foo :: (HasParams a p, HasDocument p d, HasUri d u) => a -> a
+foo x = (params . document . uri) .~ newUri $ x
+ where newUri = spiceUp (x ^. params . textDocument . uri)
+```
+
+Lenses give a natural way of accessing record fields kind of in an object-orientated dot-notation style.
+The gist is this: I lied earlier, the actual definition of the messages in `haskell-lsp` is more like this:
+
+```haskell
+{-# LANGUAGE TemplateHaskell #-}
+data NotificationMessage m a =
+ NotificationMessage { _jsonrpc :: Text, _method :: m, _params :: a }
+makeLenses ''NotificationMessage
+```
+
+Template Haskell provides metaprogramming (think macros on steroids - you can write code that codes code), which we `makeLenses` uses to automatically synthesize lenses for each field:
+
+```haskell
+jsonrpc :: Simple Lens NotificationMessage Text
+method :: Simple Lens NotificationMessage m
+params :: Simple Lens NotificationMessage a
+```
+
+`Simple Lens a b` says that it can access type `b` from `a`. It's common enough that there's a type synonym for it, `Lens'`.
+You can now use one of the `Lens` with `^.` to access fields:
+
+```haskell
+x ^. params
+```
+
+Set fields with `~.`:
+
+```haskell
+params ~. newParams $ x
+```
+
+And even chain them magically with regular function composition:
+
+```haskell
+params.document.uri .~ spiceUp (x ^. params . textDocument . uri) $ x
+```
+
+There's a good tutorial on how [the magic actually works](https://hackage.haskell.org/package/lens-tutorial-1.0.1/docs/Control-Lens-Tutorial.html). For now though, lets focus on making these work with our classes.
+
+```haskell
+class HasParams s a where
+ params :: Lens' s a
+class HasTextDocument s a where
+ textDocument :: Lens' s a
+class HasUri s a where
+ uri :: Lens' s a
+```
+
+And then our `data`s simply conform to these types:
+
+```haskell
+instance HasParams (NotificationMessage m a) a where ...
+instance HasParams (RequestMessage m a) a where ...
+```
+
+In `haskell-lsp` these are also generated via [Template Haskell](https://artyom.me/lens-over-tea-6).
+But there's a slight difference, that adds an extra sprinkle of type safety.
+
+## Functional dependencies
+
+Say we had this normal implementation of `HasParams` for a specific type of `NotificationMessage`:
+
+```haskell
+class HasParams a p where
+ params :: a -> p
+
+instance HasParams (NotificationMessage String Int) Int where
+ params (NotificationMessage _ _ p) = p
+```
+
+Nothing is stopping us from writing this on top of this:
+
+```haskell
+instance HasParams (NotificationMessage String Int) String where
+ params (NotificationMessage _ m _) = m
+```
+
+And now we have two definitions for params. This makes no sense. Each concrete type of `NotificationMessage a b` should have exactly one type for `params`. The implementation doesn't matter, it could return something that isn't the params, just as long as its consistent.
+
+To prevent this happening with lenses, the actual definition ends up looking like this:
+
+```haskell
+{-# LANGUAGE FunctionalDependencies #-}
+class HasParams s a | s -> a where
+ params :: Lens' s a
+class HasTextDocument s a | s -> a where
+ textDocument :: Lens' s a
+class HasUri s a | s -> a where
+ uri :: Lens' s a
+```
+
+What are these arrows doing here? These are functional dependencies. Here they say that `a` is determined by `s`. In other words, if we have an instance of `HasParams`:
+
+```haskell
+instance HasParams (NotificationMessage String Int) Int
+```
+
+Then `(NotificationMessage String Int)` is guaranteed to always return an `Int` in `params`.
+
+## Flexible contexts
+
+But I digress. Back to our `foo` function, which is actually meant to swap file URIs, we had:
+
+```haskell
+swapFile :: (HasParams a b, HasTextDocument b c, HasUri c d) => a -> a
+swapFile x = (params.textDocument.uri) .~ (swapUri oldUri) $ x
+ where oldUri = x ^. params . textDocument . uri
+```
+
+What could the signature of `swapUri` look like? Well, it has to take in the type variable `d` which doesn't make it very useful. `swapUri` should be `Uri -> Uri`.
+
+Ideally `HasUri` should only take one type argument, and have its `uri` lens always return a `Uri`, but we're left with the type variable `d` instead because of how it was synthesized.
+
+What we want to do is force this type variable to, well, not be a variable. Like this:
+
+```haskell
+swapFile :: (HasParams a b, HasTextDocument b c, HasUri c Uri) => a -> a
+```
+
+Doing this gives us some error:
+
+> Non type-variable argument in the constraint: HasUri c Uri (Use FlexibleContexts to permit this)
+> In the type signature:
+> swapFile :: (HasParams a b, HasTextDocument b c, HasUri c Uri) => a -> a
+
+But the error also gives us the solution - enable FlexibleContexts.
+
+```haskell
+{-# LANGUAGE FlexibleContexts #-}
+```
+
+And just like that, we now have a safe, succint function that will work on any type that contains a uri in this configuration.
+
+```haskell
+swapFile :: (HasParams a b, HasTextDocument b c, HasUri c Uri) => a -> a
+swapFile x = (params.textDocument.uri) .~ (swapUri oldUri) $ x
+ where oldUri = x ^. params . textDocument . uri
+ swapUri uri = uri ++ "/extra/path"
+```
--- /dev/null
+---
+title: ⚗️ lsp-test
+date: 2018-08-12
+---
+
+My Haskell Summer of Code project, [lsp-test](https://github.com/Bubba/lsp-test), is now available [via Hackage](https://hackage.haskell.org/package/lsp-test-0.1.0.0).
+It's a framework for writing end-to-end tests for LSP servers, made for testing [haskell-ide-engine](https://github.com/haskell/haskell-ide-engine).
+
+But it's not just limited to haskell-ide-engine: It's language agnostic and works with any server that conforms to the [Language Server Protocol](https://microsoft.github.io/language-server-protocol/).
+In fact lsp-test is basically a client that you can programmatically control. You specify what messages you want to send to the server, and check that the responses you get back are what you expected.
+
+# What does it look like?
+
+```haskell
+runSession :: String -> ClientCapabilities -> FilePath -> Session a -> IO a
+
+main = runSession "hie" fullCaps "proj/dir" $ do
+ doc <- openDoc "Foo.hs" "haskell"
+ getDocumentSymbols doc >>= liftIO putStrLn
+```
+
+Each test is encapsulated by a `Session`: a client-server connection from start to finish.
+Here we pass in the command to start the server, the capabilities that the client should declare and the root directory that the session should take place in.
+(`fullCaps` is a convenience function that declares all the latest features in the LSP specification)
+
+Once you're inside a `Session` you're free to poke and talk away to your server.
+
+You might have noticed that the example above didn't send an initialize request - `Session` takes care of this and some other laborious parts of the process:
+
+- Sending initialize requests with the correct process ID and capabilities
+- Generating and incrementing message ids
+- Keeping track of documents through `workspace/applyEdit` requests
+
+There's also a [Smörgåsbord of functions](https://hackage.haskell.org/package/lsp-test-0.1.0.0/docs/Language-Haskell-LSP-Test.html#g:6) available for performing common tasks, such as getting a definition or checking the current diagnostics.
+
+# The message level
+
+Most of the time you'll want to want to write your tests using these helper functions, but in case you're looking to do something more specific, you can drop down to a lower level and work with the individual messages that are sent and received.
+
+The venerable [`request`](https://hackage.haskell.org/package/lsp-test-0.1.0.0/docs/Language-Haskell-LSP-Test.html#v:request) allows you to send any request defined in [`haskell-lsp-types`](https://hackage.haskell.org/package/haskell-lsp-types-0.6.0.0/docs/Language-Haskell-LSP-TH-DataTypesJSON.html), and will spit back the response for it.
+Most of the helper functions are implemented in terms of this.
+
+```haskell
+request :: (ToJSON params, FromJSON a) => ClientMethod -> params -> Session (ResponseMessage a)
+
+runSession "hie" fullCaps "my/dir" = do
+ doc <- openDoc "Foo.hs" "haskell"
+ let params = DocumentSymbolParams doc
+ -- send and wait for the response
+ rsp <- request TextDocumentDocumentSymbol params
+```
+
+You can also use the `send` family of functions if you don't want to wait for a response.
+
+```haskell
+sendRequest :: ToJSON params => ClientMethod -> params -> Session LspId
+sendNotification :: ToJSON a => ClientMethod -> a -> Session ()
+sendResponse :: ToJSON a => ResponseMessage a -> Session ()
+```
+
+# Inside Session
+
+This is where `lsp-test`'s little breakthrough comes in: The `Session` monad is actually just a [conduit parser](https://hackage.haskell.org/package/conduit-parse) that operates on individual messages.
+
+```haskell
+type Session = ConduitParser FromServerMessage IO
+
+runSession f = runConduit $ source .| runConduitParser f
+ where
+ source = getNextMessage handle >>= yield >> source
+```
+
+Incoming messages from the server are parsed in a stream.
+You specify the messages you expect to receive from the server in order, in between the messages you send to it.
+
+```haskell
+-- get a specific response
+msg1 <- message :: Session RspDocumentSymbols
+-- get any request
+msg2 <- anyRequest
+sendRequest TextDocumentDocumentSymbol params
+-- get a logging notification
+msg3 <- loggingNotification
+```
+
+It has it's own version of `satisfy` that works on messages, which custom parser combinators are built up upon.
+
+```haskell
+satisfy :: (FromServerMessage -> Bool) -> Session FromServerMessage
+satisfy pred = do
+ x <- await
+ if pred x
+ then return x
+ else empty
+
+loggingNotification = satisfy test
+ where
+ test (NotLogMessage _) = True
+ test (NotShowMessage _) = True
+ test (ReqShowMessage _) = True
+ test _ = False
+```
+
+Whenever it's unable to parse a sequence of messages, it throws an exception, which can be used to make assertions about the messages that arrive.
+
+But the great part is that it works with backtracking, and all your favourite combinators. So you can be as specific or as general as you'd like when describing the expected interaction with the server.
+
+```haskell
+skipManyTill loggingNotification publishDiagnosticsNotification
+count 4 (message :: Session ApplyWorkspaceEditRequest)
+anyRequest <|> anyResponse
+```
+
+# Building up a test suite
+
+Other than throwing exceptions when it's unable to parse the incoming sequence of messages, lsp-test doesn't help you make assertions, so you are free to use whatever testing framework you like.
+In haskell-ide-engine, lsp-test is paired quite nicely with [`HSpec`](https://hackage.haskell.org/package/hspec).
+Here's an excerpt from some tests for goto definition requests:
+
+```haskell
+spec :: Spec
+spec = describe "definitions" $ do
+ it "goto's symbols" $ runSession hieCommand fullCaps "test/testdata" $ do
+ doc <- openDoc "References.hs" "haskell"
+ defs <- getDefinitions doc (Position 7 8)
+ let expRange = Range (Position 4 0) (Position 4 3)
+ liftIO $ defs `shouldBe` [Location (doc ^. uri) expRange]
+
+ it "goto's imported modules" $ runSession hieCommand fullCaps "test/testdata/definition" $ do
+ doc <- openDoc "Foo.hs" "haskell"
+ defs <- getDefinitions doc (Position 2 8)
+ liftIO $ do
+ fp <- canonicalizePath "test/testdata/definition/Bar.hs"
+ defs `shouldBe` [Location (filePathToUri fp) zeroRange]
+```
+
+# Replaying
+
+If your language server was built with `haskell-lsp`, you can use its built in capture format to take advantage of [`Language.Haskell.LSP.Test.Replay`](https://hackage.haskell.org/package/lsp-test-0.2.0.0/docs/Language-Haskell-LSP-Test-Replay.html).
+This module replays captured sessions and ensures that the response from the server matches up with what it received during the original capture.
+In `haskell-ide-engine` you can capture a session to a file by launching the server with the `--capture` flag.
+
+```bash
+hie -c session.log
+```
+
+To test it with `lsp-test`, place the file as `session.log` inside a directory that mirrors the contents of the project root.
+
+```
+projectRoot
+ ├── proj.cabal
+ ├── src
+ │ └── ...
+ └── session.log
+```
+
+And then test it with `replaySession`.
+
+```haskell
+replaySession "hie" "projectRoot"
+```
+
+`lsp-test` is smart enough to swap out the absolute URIs, so if you originally captured the scenario under `/foo/bar/proj` but then replay it at `/test/foo`, `/foo/bar/proj/file.hs` will get swapped as `/test/foo/file.hs`.
+
+It also relaxes some "common-sense" checks:
+- Logging messages are ignored
+- The order of notifications in between requests and responses doesn't matter
+- It takes into account [uniqued command IDs](https://github.com/Microsoft/vscode-languageserver-node/issues/333)
+
+If the interaction doesn't match up, you'll get nice pretty printed JSON of what was received, what was expected, and the diff between the two[2](footnote:2).
+
+This is useful if you want to test for regressions, or if there is some very specific behaviour that would be difficult to describe programmatically.
+
+# What's next?
+
+## DSL
+Currently if you want to use lsp-tests to write tests, you need to write Haskell.
+There has been [some experimenting](https://github.com/Bubba/haskell-lsp-test/blob/script-fsm/src/Language/Haskell/LSP/Test/Script.hs) in writing a custom DSL for describing tests.
+
+```
+"start" { wait for any then open "Test.hs" "haskell" }
+"get the symbols" {
+ wait for
+ method == "textDocument/publishDiagnostics"
+ then
+ open "Test.hs" "haskell"
+ id1: request "textDocument/documentSymbol" {
+ textDocument: {
+ uri: uri "Test.hs"
+ }
+ }
+}
+"check the symbols" {
+ wait for
+ id == 1
+ then
+ open "Test.hs" "haskell"
+}
+```
+
+However, it may be more attractive to use [dhall](https://github.com/dhall-lang/dhall-lang), a configuration language that ties in nicely with Haskell.
+
+The ultimate aim is that lsp-test will be available as an executable binary (via `cabal install`/`stack install`/`apt install`), which can read and run tests from a file via the command line.
+
+```bash
+lsp-test test myTest.dhall
+```
+
+This would mean that servers implemented in languages other than Haskell can easily hook this into their CI without having to have a Haskell environment setup.
+
+## Finite state machine and fuzzy testing
+
+My mentor pointed out that the interaction between the client and server during a Session could be described as a [Mealy machine](https://en.wikipedia.org/wiki/Mealy_machine).
+The state-transition function waits for the next message from the client whilst the output function sends messages to the server.
+
+```haskell
+data State = Initialize
+ | WaitForDiagnostics
+ | MakeSymbolRequest
+ | Done
+wait :: FromServerMessage -> State -> State
+send :: State -> FromServerMessage -> FromClientMessage
+```
+
+This led me to create a [proof of concept](https://github.com/Bubba/haskell-lsp-test/blob/ba3255afa89fd1faf4c8ed1a01ba482ec5755264/src/Language/Haskell/LSP/Test/Machine.hs) that integrated with [hedgehog](https://hackage.haskell.org/package/hedgehog-0.6/docs/Hedgehog.html#g:4).
+Hedgehog is a property based testing system a-la QuickCheck, but it also provides state machine testing.
+
+I had to first tweak the `Session` monad to become a `SessionT` transformer and make it an instance of `MonadTest` and `MonadThrow`[1](footnote:1) before it could be used.
+
+```haskell
+type PropertySession = SessionT (PropertyT IO)
+
+instance MonadThrow m => MonadCatch (SessionT m) where
+ catch f h = f
+
+instance MonadTest PropertySession where
+ liftTest = lift . liftTest
+```
+
+And then the states could be described in terms of its pre-conditions, updates and post-conditions.
+
+```haskell
+data OpenDoc (v :: * -> *) = OpenDoc
+ deriving (Eq, Show)
+
+instance HTraversable OpenDoc where
+ htraverse _ OpenDoc = pure OpenDoc
+
+s_openDoc_init :: (Monad n) => Command n PropertySession ModelState
+s_openDoc_init =
+ let gen TDocClose = Just $ pure OpenDoc
+ gen _ = Nothing
+ execute OpenDoc = openDoc "Foo.hs" "haskell"
+ in Command gen execute [
+ Require $ \s OpenDoc -> s == TDocClose
+ , Update $ \_s OpenDoc o -> TDocOpen
+ , Ensure $ \before after OpenDoc o -> do
+ before === TDocClose
+ let L.TextDocumentIdentifier uri = o
+ uri === L.Uri "Foo.hs"
+ after === TDocOpen
+ ]
+```
+
+Once you describe a bunch of different states, you can then throw them into hedgehog and it will run them in random orders: Giving you fuzzy testing!
+
+
+
+# Lessons learnt
+
+The best way to get a taste for an API is by using it!
+
+By using it to write tests for haskell-ide-engine at an early stage in development, it allowed me to figure out the sticking points and refine the ergonomics.
+For example, the client capabilities used to be set inside [`SessionConfig`](https://hackage.haskell.org/package/lsp-test-0.1.0.0/docs/Language-Haskell-LSP-Test.html#t:SessionConfig).
+But when testing for backwards compatibility, it quickly became evident that it needed to be more explicit, hence why it was moved to `runSession`.
+
+If I was to start this project again, I would have focused a lot more on the DSL.
+Although language agnosticism was part the design, not including an easy way for non-Haskell environments to run tests was a major oversight.
+But with the framework in place, I am confident that we will be able to implement this soon.
+
+It's also important to note that this past summer wasn't spent solely on lsp-test, and I invested a large amount of work into haskell-ide-engine as well.
+I would happily say that lsp-test paid off while doing this work: it helped catch a lot of regressions while doing extensive refactoring and development around the [dispatcher](https://github.com/haskell/haskell-ide-engine/pull/594), [code actions](https://github.com/haskell/haskell-ide-engine/pull/659) and [plugin infrastructure](https://github.com/haskell/haskell-ide-engine/pull/710), not to mention the peace of mind from having a robust test-suite in place.
+
+Thanks to my mentor Alan Zimmerman for all his help, and all the community at #haskell-ide-engine and #haskell for putting up with all my stupid questions.
+If you're developing a language and want to write an LSP server, consider giving `lsp-test` a shot, and let me know!
+
+> footnotes
+ 1. As you can see, this is clearly not the right way to implement `MonadThrow`. If you do know how to for a continuation based monad, please let me know!
+ 2. Pro-tip: use the diff to incrementally update your session.log files if your expected output has changed.
--- /dev/null
+---
+title: 📦 Intro
+date: 2018-05-24
+---
+
+I was fortunate enough to be accepted for the Haskell foundation for this year's Summer of Code, and the project I will be working on is with the [Haskell IDE Engine](https://github.com/haskell/haskell-ide-engine).
+From the repository description, the Haskell IDE Engine (hie) is the engine for Haskell IDE integration.
+Most interestingly, it acts as a server for the Language Server Protocol so it can provide rich Haskell support for any IDE or text editor that supports the protocol.
+It can give diagnostics, refactor code, search document symbols and tons of other neat stuff.
+I'll be working on a test framework that can simulate a session interacting with a client. [You can read more about it here.](https://summerofcode.withgoogle.com/projects/#6041896627994624)
+
+## haskell-ide-engine
+The project kind of acts as the glue between LSP and the various coding tools in the Haskell ecosystem.
+It has a plugin API that's used for integrating fan favourites such as `ghc-mod`, `hlint` and `HaRE`.
+
+It's spread across multiple repositories:
+
+1. **haskell-ide-engine** provides the I/O and communicates between the client and ghc-mod, as well as any plugins such as HaRE or brittany
+2. **haskell-lsp** (and **haskell-lsp-types**) contain definitions for functions and types according to the [LSP specification](https://microsoft.github.io/language-server-protocol/specification)
+3. **ghc-mod** is a separate tool that provides most of the analysis and diagnostics, but its tightly coupled with HIE.
+4. **haskell-lsp-client** is a library for LSP clients that my mentor Alan pointed out, we plan to use it as a starting point for #5.
+5. **haskell-lsp-test** will soon be the testing framework!
+
+
+## IRC
+I've mostly been communicating in the `#haskell-ide-engine` room on freenode.
+It's been a while since I've used IRC, but I got round to setting up and running an IRC bouncer on my server (ZNC).
+Launching [Colloquy](http://colloquy.info) was a blast from the past, complete with pre-retina icons.
+But as it turns out it's [still actively developed on GitHub!](https://github.com/colloquy/colloquy).
+It's no longer distributed on the website, so I cloned the repository, created an archive with Xcode and then moved the `.app` to `/Applications`.
+
+## Blog
+
+At the moment I'm using this makeshift bash script to blog:
+
+```bash
+cat header.html > index.html
+for md in $(ls -tr *.md); do
+ markdown $md >> index.html
+done
+cat footer.html >> index.html
+```
+
+## Setup
+
+It took me a while to get the development environment set up.
+
+I started off by using VSCode and [vscode-hie-server](//github.com/alanz/vscode-hie-server) for the client.
+There was an issue on master that caused the LSP parser to fail with VSCode, so I created a [pull request](https://github.com/alanz/haskell-lsp/pull/81) for it.
+
+But I'm mainly a Vim user, so I tried out [vim-lsc](//github.com/natebosch/vim-lsc) and then eventually settled for [LanguageClient-neovim](https://github.com/autozimu/LanguageClient-neovim) which seems to support more LSP features.
+Here's my current `~/.vimrc` bindings for it
+
+```vim
+let g:LanguageClient_serverCommands = {
+ 'haskell': ['hie', '--lsp', '--debug', '-l', '/tmp/hie.log', '--vomit']
+ \ }
+
+nnoremap <silent> K :call LanguageClient#textDocument_hover()<CR>
+nnoremap <silent> gd :call LanguageClient#textDocument_definition()<CR>
+nnoremap <silent> gr :call LanguageClient#textDocument_rename()<CR>
+nnoremap <silent> ga :call LanguageClient#textDocument_codeAction()<CR>
+nnoremap <silent> gs :call LanguageClient#textDocument_documentSymbol()<CR>
+set completefunc=LanguageClient#complete
+```
+
+At one point I ended up getting strange errors from HIE when running it on haskell-lsp and haskell-ide-engine (very meta):
+```
+hie: <command line>: cannot satisfy -package-id HaRe-0.8.4.1-inplace:
+ HaRe-0.8.4.1-inplace is unusable due to shadowed dependencies:
+ base-4.11.1.0 Strfnsk-StrtgyLb-5.0.1.0-e162e946 cabal-helper-0.8.0.3-inplace containers-0.5.11.0 directory-1.3.1.5 ghc-8.4.2 ghc-xctprnt-0.5.6.1-3f0c080b ghc-mod-core-5.9.0.0-inplace hslggr-1.2.10-e253fcf2 mnd-cntrl-1.0.2.3-90183ebd syb-0.7-652252ef syz-0.2.0.0-ae4391c5
+ (use -v for more information)
+```
+
+After two days of repeated `rm -r ~/.stack .stack-work`, I learnt two important lessons when working with stack projects:
+
+1. Double check with `ls -a` to make sure there are no `dist`, `dist-newstyle`, `.cabal.project.local` or `.ghc-environment`s lying around. These will fool `ghc-mod` into thinking into using cabal instead of stack.
+2. You need to use a `hie` that was compiled with the same GHC version as your stack resolver. Turning on the hie wrapper in VSCode can automatically find it for you, otherwise you will need to specify it yourself.
+
+But once I got hie working, it was amazing. Being able to rename variables with two keystrokes, apply quick-fixes straight fresh from `hlint` and jump to symbols with fuzzy finding all from vim was a glorious feeling.
+
+Unfortunately this did not last for long as the second time I launched vim I got this [delightful bug](https://github.com/haskell/haskell-ide-engine/issues/562).
+It's an issue that lies all the way within `ghc` and only affects macOS on 8.4.2.
+It looks like it won't get fixed till the next 8.6 release either, which is due around August.
+It only affects modules that use the PatternSynonyms language extension, which `haskell-ide-engine` uses.
+I'm still trying various linker flags to see if there is a workaround, but for the meantime it means that I can't use `hie` on `hie` without swapping out the ghc-8.4.2 resolver for ghc-8.2.
+
+## Starter PRs
+
+Here are some PRs so far:
+
+- [Preventing hie from showing quickfixes for hlint suggestions with no possible refactorings](https://github.com/haskell/haskell-ide-engine/pull/548)
+- [Fixes for](https://github.com/haskell/haskell-ide-engine/pull/525) [extraneous newlines](https://github.com/haskell/haskell-ide-engine/pull/563) being added with Brittany
+- [Restructuring the response and cache architecture](https://github.com/haskell/haskell-ide-engine/pull/568) to allow for deferred IDE responses
+
+The last one is still a work in progress, but here's a summary of how it came about and what's going down.
+
+- When trying out `haskell-lsp-client`, the document symbols request returned an empty list when run immediately after starting `hie`, unless a delay was added so that `hie` had time to load the module.
+- I tried submitting a PR to move the symbol request from the `IdeM` monad to the `IdeGhcM` monad
+- These two monads determine what thread they run on: `IdeM` is for internal requests and stuff, but `IdeGhcM` is for anything that goes through `ghc-mod`, which may take some time to run
+- This turned out to be a bad idea™ since putting it on `IdeGhcM` would cause the request to block whenever a large module was being compiled in the background. We want it to only wait for the module to load whenever there was no cache available, but serve the cache whenever possible.
+
+I owe all of my thanks to `wz1000` for noticing this and pointing me in the right direction.
+The agreed plan is this:
+
+Change the function that returns cached modules, `getCachedModule` to return not just a `Maybe` but a more descriptive ADT:
+
+```haskell
+data CachedModuleResult = ModuleLoading
+ | ModuleFailed String
+ | ModuleCached CachedModule IsStale
+```
+
+Add a queue of actions that get executed whenever a module is finished loading:
+
+```haskell
+data IdeState = IdeState
+ { moduleCache :: GhcModuleCache
+ -- | A queue of actions to be performed once a module is loaded
+ , actionQueue :: Map.Map FilePath [CachedModule -> IdeM ()]
+ ...
+ }
+```
+
+Change `IdeResponse` to be a full blown ADT instead of a pattern, and add a new deferred type that the dispatcher can distinguish between and handle the queueing for:
+
+```haskell
+-- | The IDE response, with the type of response it contains
+data IdeResponse a = IdeResponseOk a
+ | IdeResponseDeferred FilePath (CachedModule -> IdeGhcM (IdeResponse a))
+ | IdeResponseFail IdeError
+```
+
+This is where I'm currently at.
+My head hurts and this is turning out to be a huge undertaking for such a small edge case.
+
+But as `wz1000` said:
+
+> there really is no better way to get to know a codebase than to tear it up
+
+Tomorrow the community bonding period ends, and the real coding begins.
+I've a lot to learn about pracitcal Haskell: I've lived a very sheltered life in university, far away from the IO monad.
+But I'm looking forward to working with my mentor Alan and the other lovely members of the Haskell community, and hopefully making the future of Haskell tooling a little better.
--- /dev/null
+---
+title: 🔨 The all in one stack command
+date: 2018-05-25
+---
+
+```haskell
+stack build --fast --test --haddock --file-watch
+```
--- /dev/null
+--------------------------------------------------------------------------------
+{-# LANGUAGE OverloadedStrings #-}
+import Control.Monad
+import qualified Data.Map.Lazy as Map
+import qualified Data.Text
+import qualified Data.Text.Lazy as T
+import qualified Data.Text.Lazy.Encoding as T
+import Data.Monoid (mappend)
+import Hakyll
+import qualified Text.MMark as MMark
+import qualified Text.MMark.Extension as MMark
+import qualified Text.MMark.Extension.Common as MMark
+import qualified Text.Megaparsec.Error as Mega
+import System.Directory
+import System.FilePath
+import Lucid
+import Eventlog.Data
+import Eventlog.Args
+import Eventlog.VegaTemplate
+import Eventlog.HtmlTemplate
+import qualified Text.Blaze.Html.Renderer.Text as Blaze
+
+--------------------------------------------------------------------------------
+main :: IO ()
+main = do
+ eventLogs <- renderEventLogs
+ hakyllWith conf $ do
+ match "images/*" $ do
+ route idRoute
+ compile copyFileCompiler
+
+ match "css/*" $ do
+ route idRoute
+ compile compressCssCompiler
+
+ match (fromList ["about.rst", "contact.markdown"]) $ do
+ route $ setExtension "html"
+ compile $ pandocCompiler
+ >>= loadAndApplyTemplate "templates/default.html" defaultContext
+ >>= relativizeUrls
+
+ match "posts/*" $ do
+ route indexHtml
+ compile $ mmarkCompiler eventLogs
+ >>= loadAndApplyTemplate "templates/post.html" postCtx
+ >>= loadAndApplyTemplate "templates/default.html" postCtx
+ >>= relativizeUrls
+
+ create ["archive.html"] $ do
+ route $ setExtension "html"
+ compile $ do
+ posts <- recentFirst =<< loadAll "posts/*"
+ let archiveCtx =
+ listField "posts" postCtx (return posts) `mappend`
+ constField "title" "Archives" `mappend`
+ defaultContext
+
+ makeItem ""
+ >>= loadAndApplyTemplate "templates/archive.html" archiveCtx
+ >>= loadAndApplyTemplate "templates/default.html" archiveCtx
+ >>= relativizeUrls
+ >>= removeIndexes
+
+
+ match "index.html" $ do
+ route idRoute
+ compile $ do
+ posts <- recentFirst =<< loadAll "posts/*"
+ let indexCtx =
+ listField "posts" postCtx (return posts) `mappend`
+ defaultContext
+
+ getResourceBody
+ >>= applyAsTemplate indexCtx
+ >>= loadAndApplyTemplate "templates/default.html" indexCtx
+ >>= relativizeUrls
+ >>= removeIndexes
+
+ match "templates/*" $ compile templateBodyCompiler
+
+
+--------------------------------------------------------------------------------
+postCtx :: Context String
+postCtx =
+ dateField "date" "%B %e, %Y" `mappend`
+ defaultContext
+
+indexHtml :: Routes
+indexHtml = customRoute go
+ where
+ go i = let fp = toFilePath i
+ in takeDirectory fp </> takeBaseName fp </> "index.html"
+
+removeIndexes :: Item FilePath -> Compiler (Item FilePath)
+removeIndexes = return . fmap (withUrls removeIndex)
+ where
+ removeIndex fp
+ | takeFileName fp == "index.html" = takeDirectory fp
+ | otherwise = fp
+
+conf = defaultConfiguration
+ { deployCommand = "rsync -a _site/ luke@lukelau.me:/var/www/luke/haskell" }
+
+mmarkCompiler :: Map.Map FilePath (Html ()) -> Compiler (Item String)
+mmarkCompiler eventlogs = do
+ fp <- getResourceFilePath
+ getResourceLBS >>= withItemBody (\lbs ->
+ let text = T.toStrict $ T.decodeUtf8 lbs
+ in case MMark.parse fp text of
+ Left e -> error (Mega.errorBundlePretty e)
+ Right doc ->
+ let html = MMark.render (MMark.useExtensions extensions doc)
+ in return $ T.unpack $ renderText html)
+
+ where extensions = [ MMark.ghcSyntaxHighlighter
+ , MMark.skylighting
+ , MMark.footnotes
+ , eventLogsExt eventlogs
+ ]
+
+renderEventLogs :: IO (Map.Map FilePath (Html ()))
+renderEventLogs = foldM render mempty =<< listDirectory "eventlogs"
+ where
+ render acc fp = do
+ let args = Args { sorting = Size
+ , reversing = False
+ , tracePercent = 1
+ , nBands = 15
+ , heapProfile = False
+ , noIncludejs = False
+ , json = True
+ , noTraces = True
+ , userColourScheme = ""
+ , files = []
+ }
+ (_, dat) <- generateJson ("eventlogs" </> fp) args
+ let conf = ChartConfig 450 500 True "accent" (AreaChart Stacked)
+ html :: Html ()
+ html = div_ [style_ "overflow: scroll"] $ do
+ forM_ vegaScripts $ \url ->
+ script_ [src_ url] ("" :: T.Text) :: Html ()
+ toHtmlRaw $ Blaze.renderHtml $ renderChartWithJson 1 dat (vegaJsonText conf)
+ return $ Map.insert fp html acc
+
+ vegaScripts = [ "https://cdn.jsdelivr.net/npm/vega@5.4.0"
+ , "https://cdn.jsdelivr.net/npm/vega-lite@3.3.0"
+ , "https://cdn.jsdelivr.net/npm/vega-embed@4.2.0"
+ ]
+
+eventLogsExt :: Map.Map FilePath (Html ()) -> MMark.Extension
+eventLogsExt eventLogs = MMark.blockRender go
+ where
+ go :: (MMark.Block (MMark.Ois, Html ()) -> Html ()) -> MMark.Block (MMark.Ois, Html ()) -> Html ()
+ go f (MMark.CodeBlock (Just "eventlog") url) = eventLogs Map.! (Data.Text.unpack (Data.Text.strip url))
+ go f x = f x
--- /dev/null
+Here you can find all my previous posts:
+$partial("templates/post-list.html")$
--- /dev/null
+<!doctype html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="x-ua-compatible" content="ie=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>$title$</title>
+ <link rel="stylesheet" href="/css/default.css" />
+ <link rel="stylesheet" href="/css/syntax.css" />
+ </head>
+ <body>
+ <header>
+ <div class="logo">
+ <a href="/"><img src="/images/haskell.png" height="24px"/></a>
+ </div>
+ <nav>
+ <a href="/">Luke's Haskell Blog</a>
+ </nav>
+ </header>
+
+ <main role="main">
+ <h1>$title$</h1>
+ $body$
+ </main>
+
+ <footer>
+ Site proudly generated by
+ <a href="//jaspervdj.be/hakyll">Hakyll</a>
+ and
+ <a href="//hackage.haskell.org/package/mmark">MMark</a>
+ </footer>
+ </body>
+</html>
--- /dev/null
+$for(posts)$
+ <a href="$url$">$title$</a> - $date$
+ </br>
+$endfor$
--- /dev/null
+<article>
+ <section class="header">
+ Posted on $date$
+ $if(author)$
+ by $author$
+ $endif$
+ </section>
+ <section>
+ $body$
+ </section>
+</article>
projects / haskell-blog.git / commitdiff
