Manipulating Canvas Pixels with Haskell and Haste

Posted on July 31, 2016 by Brian Jaress
Tags: ,

Sometimes you just want to set pixels, even in a browser. If you don’t need the sophisticated features of something like WebGL, there’s much wider browser support for the HTML5 canvas element’s getImageData and putImageData.

Reload the page to randomly generate another image.

Unfortunately, the API is conceptually low-level and a bit awkward. Probably because of that, it’s not widely used and isn’t exposed by many of the systems that wrap JavaScript in another language. The Haste Haskell-to-JavaScript compiler, for example, doesn’t come with a wrapper providing easy use of putImageData. It does come with a foreign function interface and an API for extending the wrapper, so here’s a little extension I wrote.

Working with ImageData

All I really want to do is get the ImageData, change it, and put it back. Haste’s withContext is a backdoor for extending the canvas API in just that sort of way by giving low-level access to the canvas context.

fillPixels :: (Fold.Foldable container) =>
    Int -> Int -> container Graphics.Color -> Graphics.Picture ()
fillPixels width height pixelColors =
    Graphics.withContext $ \ctx -> do
        imageData <- getImageData ctx 0 0 width height
        updateImage imageData $ pixelColors
        putImageData ctx imageData 0 0 0 0 width height

The fillPixels function expects a foldable container, which is a one-dimensional interface, rather than two dimensional. Haskell has plenty of two dimensional containers, but they don’t share a common typeclass, so I’d have to choose one to force on the caller. As you’ll see in a moment, the ImageData itself is natively one dimensional, so I decided to just go with the flow.

Manipulating the Data

The interesting part of this is altering the image data in between getting and putting it. What you get from getImageData (and put with putImageData) is a one dimensional array-like object of bytes representing color components: the first element is the amount of red in the upper left pixel, while the second element is the amount of green in that same pixel. Only when you get to the fifth element are you talking about the next pixel to the right of the first. (The fourth element is the alpha transparency of the first pixel.)

updateImagePixel :: ImageData -> Int -> Graphics.Color -> IO ()
updateImagePixel imageData index (Graphics.RGB r g b) =
    updateImagePixel imageData index (Graphics.RGBA r g b 1.0)
updateImagePixel imageData index (Graphics.RGBA r g b a) = do
    jsUpdateImageComponent imageData (rawIndex+0) r
    jsUpdateImageComponent imageData (rawIndex+1) g
    jsUpdateImageComponent imageData (rawIndex+2) b
    -- Unlike everywhere else, alpha is a byte here
    jsUpdateImageComponent imageData (rawIndex+3) $ round (a * 255)
    where
    rawIndex = index * 4

jsUpdateImageComponent :: ImageData -> Int -> Int -> IO ()
jsUpdateImageComponent = FFI.ffi "(function(d,i,v){d.data[i]=v;})"

The above code is a low-level helper that translates between indexes that refer to pixels and “raw” indexes that refer to color components. It’s just a four-to-one correspondence, nothing fancy.

To assign all the pixels at once, we take a foldable container of colors and fold over it with foldlM. The basic idea is that the mondadic fold lets you carry over an accumulated monad as well as the regular fold accumulator, so the index is the accumulator, and the update actions go in the IO monad.

updateImage :: (Fold.Foldable f) => ImageData -> f Graphics.Color -> IO (Int)
updateImage imageData colors = Fold.foldlM visitPixel 0 colors
    where
    visitPixel index color = do
        updateImagePixel imageData index color
        return (index + 1)

Getting From and Putting To the Canvas

All of that depends on some straightforward Haskell equivalents for the JavaScript getImageData and putImageData methods, defined using the foreign function interface.

newtype ImageData = ImageData Haste.JSAny
  deriving (FFI.ToAny, FFI.FromAny)

getImageData = jsGetImageData
putImageData = jsPutImageData

jsGetImageData :: Graphics.Ctx -> Int -> Int -> Int -> Int -> IO ImageData
jsGetImageData = FFI.ffi
    "(function(c, x, y, w, h){return c.getImageData(x, y, w, h);})"
jsPutImageData :: Graphics.Ctx -> ImageData -> Int -> Int -> Int ->
    Int -> Int -> Int -> IO ()
jsPutImageData = FFI.ffi
    "(function(c, d, x, y, dx, dy, dw, dh){return c.putImageData(d, x, y, dx, dy, dw, dh);})"

The getImageData = jsGetImageData pattern is just a convention from inside Haste itself. As you can see, the FFI takes a pretty simple approach: inline JavaScript attached to Haskell type signatures.

Demo Code

If all this code works in your browser1 it should fill the the canvas at the top of this post with randomly generated image data on each page load. Although it shouldn’t be too hard to define an order for folding on a genuinely two-dimensional structure, I’ve taken advantage of the fact that I want something random to just generate an infinite list of pixel colors and take what I need.

main = do
    Just canvas <- Graphics.getCanvasById "canvas"
    (width, height) <- normalizedSize canvas
    colors <- selectColors
    arrangement <- assignColors colors
    Graphics.render canvas $ fillPixels width height $
        take (width * height) $ arrangement

Random Generation

Aside from setting the dimensions, the main function is built around two helper functions, selectColors and assignColors, that both use randomness. The foreground and background colors are selected randomly, and then the pixels are randomly assigned to be foreground or background pixels. The assignment is weighted in favor of the background color, with the fixed weighting being randomly chosen before the assignment begins.2

selectColors :: IO (Graphics.Color, Graphics.Color)
selectColors = do
    seed <- App.newSeed
    return (pick foreChoices seed, pick backChoices $ App.next seed)
    where
    pick choices seed =
        choices !! (fst $ App.randomR (0, (length choices)-1) seed)
assignColors :: (Graphics.Color, Graphics.Color) -> IO [Graphics.Color]
assignColors (foreground, background) = do
    seed <- App.newSeed
    backgroundWeight <- return $ fst $ App.randomR (2, 30) seed
    seed <- App.newSeed
    return $ map zeroToForeground $
        App.randomRs (0, backgroundWeight) seed
    where
    zeroToForeground :: Int -> Graphics.Color
    zeroToForeground randomInt =
        if randomInt == 0
        then foreground
        else background

Both of those functions use the random helpers in Haste.App, which have been completely replaced in the upcoming release.3

It’s good that those functions are being removed, because they contain a very annoying bug. The documentation clearly says that the range is inclusive below and exclusive above, but the implementation is inclusive at both ends.

You can check this by passing integer bounds of zero and one. The half-open interval should give you all zeros, but the fully closed interval gives you a mix of ones and zeros.

bugInHaste = do
    seed <- App.newSeed
    print $ take 100 $ App.randomRs (0::Int, 1::Int) seed

Dimensions

Canvases have two sets of dimensions: one width and height for the space they actually take up, and one “logical” width and height for all the drawing functions. They have the same defaults, and some ways of setting the width and height set both at once (but others don’t). Any difference between the actual and logical dimensions is bridged by automatically scaling, which can make the whole image look distorted and blurry.

I’m sure there’s a good reason for that, but I like to ignore it by resetting the logical dimensions to match the actual dimensions, then passing them as parameters to the rest of the code:

normalizedSize canvas = do
    DOM.getProp canvas "clientWidth" >>= DOM.setAttr canvas "width"
    DOM.getProp canvas "clientHeight" >>= DOM.setAttr canvas "height"
    width <- DOM.getAttr canvas "width" >>= asInt
    height <- DOM.getAttr canvas "height" >>= asInt
    return (width, height)
    where
    asInt :: String -> IO(Int)
    asInt = return . fromIntegral . toInteger . read

Colors

The foreground colors are from the XKCD Color Survey, which is very cool and useful. These are the top five colors, skipping pink, which was a bit too faint compared to the others.

foreChoices =
    [ Graphics.RGB 0x7e 0x1e 0x9c -- purple
    , Graphics.RGB 0x15 0xb0 0x1a -- green
    , Graphics.RGB 0x03 0x43 0xdf -- blue
    , Graphics.RGB 0x65 0x37 0x00 -- brown
    , Graphics.RGB 0xe5 0x00 0x00 -- red
    ]

I was all set to do something similar for the background colors, but it actually ended up looking best with a fully transparent background, so the background colors are a list of one.

backChoices =
    [ Graphics.RGBA 255 255 255 0.0
    ]

Thoughts on Haste and Haskell

I’m still undecided on Haskell and the ecosystem around it. There are some powerful tools here that have been used to create great software, but it also feels like there’s been a failure to learn from certain historical sources.

For example, integration is a major pain point in software. Libraries should ease that pain by producing and consuming data in general formats defined in the core of the language and commonly used in non-library code, like lists, maps, or structs. Third-party Haskell libraries fall down pretty hard in that area. They bristle with custom data types, trying to have type signatures so “rich” they can replace documentation, and often ignoring types that represent essentially the same thing a different way in the standard libraries.

I realize the type system is powerful and Haskellers are justly proud of it, but there’s a time and place for everything. APIs are not the place for type signatures complex enough to replace English sentences.

Try to use several third-party Haskell libraries in the same or related domains – a random number generator, a stats package, and a simulation package, for example – and you’ll discover that just schlepping numbers around can be more painful than you ever imagined. It’s pain that has nothing to do with type theory, and everything to do with type practice.

  1. You’ll need JavaScript and support for ImageData.↩︎

  2. The range when randomly choosing the backgroundWeight is something I came up with by trial and error to produce decent images.↩︎

  3. The replacement seems to be Haste support for the standard Haskell System.Random, which currently crashes when I try to use it with Haste.↩︎