Stachu.NET

reMarkable.fs

Stachu Korick -
  • This post is my submission to F# Advent 2021.
  • It's part 3 in a series of posts regarding hacking the reMarkable eInk tablet.

Summary:

  • The reMarkable 2 is an 18.8*24.6cm (7.4*9.6in) eInk Linux tablet that I use as a daily driver for taking notes, reading books, drawing diagrams, doodling, etc.
  • I'd like to extend my use of the hardware by writing custom software for it.
  • In part 1, I covered the device itself, my use cases and motivations, a primer on getting familiar with the software+hardware.
  • In part 2, I covered the the existing reMarkable hacking ecosystem as far as I know it.
  • This post covers a hobby project of mine - a WIP framework for building custom reMarkable apps, with F#

Building and running your first "app" with reMarkable.fs

(note: I've only tested this on my reMarkable 2, with a Linux host machine.)

Let's say you have a reMarkable and you'd like to play with reMarkable.fs. Here are some instructions (unfortunately long-winded...) to get to "hello world":

  • SSH into your reMarkable per instructions found here. Be sure to set up the "Passwordless Login with SSH Keys" instructions within so you don't need to re-enter your credentials many times over
  • Ensure you have .NET SDK installed onto your "host" machine. I'm targeting .NET 5 at this point.
  • Clone the reMarkable.fs repository from github. The code may have changed significantly since time of writing, so it may be best to download the zip file here: https://github.com/StachuDotNet/reMarkable.fs/releases/tag/v0.1
  • Open the code in your editor of choice; take a look around
  • Take note of the remarkable.fs.Demo/Program.fs file - this is the "app" that will be built. Edit it if you'd like.
  • Build the Demo project (`dotnet build` via CLI is fine)
  • Install the .NET runtime on your reMarkable tablet. Download links are here. The reMarkable has an ARM processor, so under the ".NET Runtime" portion of this page, I chose "Arm64". The simplest way to 'install' is to copy the extracted .tar.gz to the rM (using scp or rclone), into the /home/root directory of your rM
  • To clarify the above: after downloading, I extracted the contents (via UI), cd'd to where the resultant folder is, and then ran scp -r dotnet-runtime-5.0.13-linux-arm64 10.11.99.1:/home/root
  • Copy the build demo project
    Similarly to above, but with rsync this time (just to show it off),
    `rsync -avh ./reMarkable.fs.Demo/bin/Debug/net5.0/ rM:/home/root/reMarkable.fs.Demo`
  • Install toltec and rm2fb (after installing toltec, opkg install rm2fb). Toltec is an rM-specific package manager, and rm2fb is some sort of thing we need to write
  • Run the demo project on your rM
  • systemctl stop xochitl to stop the main UI application
  • /opt/bin/rm2fb-client /home/root/dotnet/dotnet /home/root/reMarkable.fs.Demo/reMarkable.fs.Demo.dll

Yes, it's a lot of work! Definitely need to streamline this.

A quick tour of the reMarkable.fs repository

Currently, there are 3 .NET projects involved:

  • reMarkable.fs - this is the actual 'framework', and includes everything related to event-listening, framebuffer-drawing, etc.
  • reMarkable.fs.UI - this is a thin (for now) layer for displaying common UI elements such as "labels" and "battery indicators"
  • remarkable.fs.Demo - this is the project that I compile and run on the rM itself. Once I pull rM.fs into a NuGet package, this will be in a separate repo. (and, effectively, this is what all that an "end user" would use to build an app)

Beyond those 3 projects, there's not much - a .sln to wrap them together, the source code in .fs files, some resources (fonts, images), and .sh scripts that I use during local development.

The next few sections will look deeper into the "framework-level" reMarkable.fs project. Here's its .fsproj, for reference

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
      <TargetFramework>netstandard2.1</TargetFramework>
      <GenerateDocumentationFile>true</GenerateDocumentationFile>
      <WarnOn>3390;$(WarnOn)</WarnOn>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="SixLabors.Fonts" Version="1.0.0-beta0013" />
    <PackageReference Include="SixLabors.ImageSharp" Version="1.0.0-rc0003" />
    <PackageReference Include="SixLabors.ImageSharp.Drawing" Version="1.0.0-beta0010" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="NLog" Version="4.7.2" />
  </ItemGroup>

  <ItemGroup>
    <None Include="Resources\lifebook.bin" CopyToOutputDirectory="PreserveNewest" />
    <None Include="Resources\marker.bin" CopyToOutputDirectory="PreserveNewest" />
    <None Include="Resources\x220.bin" CopyToOutputDirectory="PreserveNewest" />
  </ItemGroup>

  <ItemGroup>
    <Compile Include="Util\ResourceHelpers.fs" />
    <Compile Include="Util\UnixExceptions.fs" />
    <Compile Include="Util\BufferExtensions.fs" />
    <Compile Include="Util\StreamWatcher.fs" />
    <Compile Include="Util\UnixInputDriver.fs" />
    <Compile Include="Util\UnixFileMode.fs" />
    <Compile Include="Util\Stream.fs" />
    <Compile Include="PhysicalButtons.fs" />
    <Compile Include="Digitizer.fs" />
    <Compile Include="TouchScreen.fs" />
    <Compile Include="Display\RGB565.fs" />
    <Compile Include="Display\Framebuffer.fs" />
    <Compile Include="Display\IDisplayDriver.fs" />
    <Compile Include="Display\HardwareDisplayDriver.fs" />
    <Compile Include="Display\RM2ShimDisplayDriver.fs" />
    <Compile Include="Performance.fs" />
    <Compile Include="Keyboard\KeyboardKeys.fs" />
    <Compile Include="Keyboard\Keyboard.fs" />
    <Compile Include="PowerSupply.fs" />
    <Compile Include="Wireless.fs" />
    <Compile Include="ReMarkable.fs" />
  </ItemGroup>
</Project>

I'd recommend following along here: https://github.com/StachuDotNet/reMarkable.fs/tree/v0.1/reMarkable.fs or locally

A few things of note:

  • we use some NuGet packages - NLog for logging (though I'm not doing much w/ this), and SixLabors' ImageSharp for image buffer manipulation
  • a ResourceHelpers.fs is handy to fetch resource files from disk
  • since we're working in a Unix system, we'll need to expect and handle various Unix exceptions, hence UnixExceptions.fs
  • we'll be listening to a few streams of Unix events, waiting for user input events to come through. We have a base UnixInputDriver.fs that listens for streams of events at known locations; this base driver powers our ability to listen to events regarding the digitizer (pen/stylus), physical buttons, a USB-connected keyboard, and the multi-touch touchscreen
  • there's a bag of magic in the Display folder related to our ability to draw pixels onto the screen. Honestly a lot of this is still magic to me :)

Fetching metadata from reMarkable

There are a number of metrics available as text files in the reMarkable file system. rM.fs provides an easy ability to fetch information about WiFi connectivity, power supply and consumption, and various performance metrics such as CPU/RAM usage.

Listening to user inputs

I've never listened for user input events outside the context of an existing framework (Web, VB6, etc.), so listening for events on a Linux eInk tablet was a fun adventure for me!

The main take-away that I have is that evtest is awesome! If you run evtest on a linux machine (may require an install), it'll display something like this:

Returned back is a list of user input devices, with a path of the relevant stream (e.g. /dev/input/event1) and some human-readable title.

From this point, you can choose the event device number (let's say I wanted to listen to whenever hit the power button on this laptop - I would chose '1', and then start playing with that input:

This is a recording of me clicking the button twice. You can see we have two records for each press - one for click  and one for release.

It turns out, you can do this for all sorts of user input events. Specific to the rM, you can do this for the digitizer, buttons, touch screen, or usb-input keyboard (in case of rM1 where that's possible...).

I'd recommend researching evtest further, but the summary here is that we can use the tool to understand the raw events that we can listen for - and what types of values and ranges those events might be in. (for example, there's a ton of cool data around the digitizer movement, including tilt and pressure).

Based on this research and our base UnixInputDriver, our input device listeners are built - the simplest example to read through is likely the physical button event listener.

Drawing to the eInk display

Unfortunately as mentioned before, the process of drawing pixels to this eInk screen is still a bit of mystery to me - for now, please read the code. I hope to follow up and provide more explanation when possible.

Motivation to build reMarkable.fs

I want to build some custom reMarkable apps for my life

  • Cooking - I've yet to find a software to manage nutrition, recipe organization, meal-prep, and grocery shopping in a way that makes sense to me. I'm building a bag of software to do such for myself, and several components would be ideal UX on such a piece of hardware
  • Exercise - Similarly, I'd like to build an app to manage various workout routines (e.g. track my barbell lifting data - barbells are in my basement, and I'd love to only bring down my rM for data collection)
  • Misc. other apps - ideally, this framework matures to a point where creating a little app for life-management and data-collection becomes trivial, in which case I can make additional apps for other areas of life (personal calendar, music apps, cat health monitor, etc.). I love to make little apps to use around the house, but dedicating my eyes to more non-eInk screens daily is something I'd like to avoid.

I love F#, and would be more motivated to build such apps if it were in F#

Most of my software experience has been in the world of full-stack web development (from data to front-end). While I'm interested in the full spectrum of computing, my view has been somewhat limited by the jobs I've chosen, and I see this project as an opportunity to play in additional areas: Linux, cross-compilation, hardware, display, listening for (button, digitizer, etc.) events, design, streams, package publishing, perf. optimization.

How it was made

I've wanted to build little UI apps for my reMarkable for years, but many times got blocked by knowledge gaps. Specifically, cross-platform development, the rM linux ecosystem, etc.

Finally, I made some progress a few months ago based on three resources:

So, many thanks to Scott and Colby, and to the community at large!

Plans for the future

  • Release reMarkable.fs as a NuGet package for easier use
  • Streamline and better document the process of getting development+testing set up
  • Implement many UI helpers+components
  • Write "reMarkable-Elmish" to quickly build rM apps
  • Create several demo apps to showcase different sorts of I/O
  • Consider/switch image libraries
  • Move font stuff to consumer (but provide helpers)
  • Double-check display and digitizer coordinates
  • rM1 compatibility testing and adjustments
  • build myself some custom apps, finally!