A gentle intro to assembly with Rust

Liam Falconer — Mon, 03 Aug 2020 15:00:00 BST

One of the things I’ve wanted to do for a while is really dig into assembly and get into the weeds of how programs actually run. A rework of the asm macro has recently landed in nightly rust so it seemed like a good time.

And compared to some other ways I’ve tried to approach this there’s a lot less setup we need to do if we just use the rust playground to do all the heavy lifting.

My process for figuring things out has been pretty simple. I write a tiny bit of rust code, look at the assembly output and try to figure out what’s going on (with lots of googling). I’m going to walk you through what I did, and what I figured out.

Let’s start with the simplest possible thing I can think of:

fn main() {
    1 + 2;
}

playground link

You can get the assembly output for this by clicking the three dots next to run and selecting asm from the dropdown. You will probably also want to change the flavour (often referred to as syntax elsewhere) of assembly to intel (rather than at&t) ¹ if it isn’t already, by clicking the toggle under the config menu.

The assembly output from this in debug mode is far more massive than you’d expect - I get 157 lines. And most of it isn’t our program. The code we’ve written should be fairly easy to find though, as the compiler helpfully labels all of the functions with their crate and function names. In this case since we’re in the playground, the create is implicitly playground, so we can find our code by searching with ctrl-f for playground::main. Doing this gets me to:

playground::main: # @playground::main
# %bb.0:
	ret
                                        # -- End function

So even though this is a debug build, evidently there’s still some optimization going on, since there’s no numbers or anything that looks like it’s adding them together. All that’s happening here is we’re returning (ret) back to the function that called playground::main. Everything prefixed with # is a comment, and therefore ignored when we run this code.

The only other point of interest is the label playground::main: - anything suffixed with : is a label we can jump to with various commands, and indeed if we continue searching for playground::main we can find a rather indirected call to it in main. Hopefully by the end of this we’ll be understand that!

Avoiding optimizations

For now, let’s try and evade whatever’s doing the optimization:

fn add() -> usize {
    1 + 2
}

fn main() {
    add();
}

playground link

Again, searching for playground::main get us to:

playground::add: # @playground::add
# %bb.0:
	mov	eax, 3
	ret
                                        # -- End function

playground::main: # @playground::main
# %bb.0:
	push	rax
	call	playground::add
# %bb.1:
	pop	rax
	ret
                                        # -- End function

So we’ve got a bit more progress here. Still some optimization going on, since we don’t see 1 or 2 in the code, just 3. We can see that being moved (mov) into the eax register in playground::add. This must be how we’re returning the value back up to main.

And indeed, inside main we can see push rax - saving the value in the register rax to the stack, then a call to our add function, then we pop rax off the stack. The push call pop sequence is to preserve whatever values are in the registers used in add. It also just throws away the value we saved in eax in add, because eax and rax are the same register. The table here shows how ‘skinnier’ registers overlap with their ‘wider’ counterparts.

Avoiding optimizations, take 2

So how can we make this actually do some math? Let’s try again:

fn add(i: usize) -> usize {
    1 + i
}

fn main() {
    add(2);
}

playground link

So we’ve got a lot more going on this time:

playground::add: # @playground::add
# %bb.0:
	sub	rsp, 24
	mov	qword ptr [rsp + 16], rdi
	add	rdi, 1
	setb al
	test al, 1
	mov	qword ptr [rsp + 8], rdi # 8-byte Spill
	jne	.LBB8_2
# %bb.1:
	mov	rax, qword ptr [rsp + 8] # 8-byte Reload
	add	rsp, 24
	ret

.LBB8_2:
	lea	rdi, [rip + str.0]
	lea	rdx, [rip + .L__unnamed_2]
	mov	rax, qword ptr [rip + core::panicking::panic@GOTPCREL]
	mov	esi, 28
	call rax
	ud2
                                        # -- End function

playground::main: # @playground::main
# %bb.0:
	push rax
	mov	edi, 2
	call playground::add
# %bb.1:
	pop	rax
	ret
                                        # -- End function

The thing we were actually trying to produce is finally in there! We can see add rdi, 1 in the output, surrounded by a pile of other stuff. So what is all this other code?

Let’s start from the top of the call stack in main. First we can see 2 is stored in the edi register before we call playground::add, so we know our argument must be in the edi register. Again, we can see the push, pop on rax, so that must be the return value.

Looking inside the function

Now, looking into playground::add we first see sub rsp, 24. rsp is the register that holds the stack pointer, so this is growing the stack (since the stack grows downwards in x86²). Further down we can see we shrink the stack by the corresponding amount with add rsp, 24.

Then we have mov qword ptr [rsp + 16], rdi. This is copying the value from rdi onto the stack at rsp + 16 - the top of the region we just grew the stack by. The qword ptr (quadword (i.e. 64bit) pointer) bit is a hint to disambiguate the argument. Why is that pushed that onto the stack? I think this is just to make it easier to debug, since we don’t ever access that value again.

In any case, we then proceed on to actually adding 1 to rdi. The value is stored back in rdi, and importantly for what comes next, we may set some of the flags.

Then it gets complicated again - we’ve got setb al. All of the set* instructions deal with the flag register. The flag register is possibly the most magical of registers, since it’s manipulated by a bunch of instructions as a side effect.

The last instruction we ran was add, which sets 6 of the the flags: carry, parity, adjust (aka auxiliary carry), zero, sign and overflow

In this case we’re checking if the carry bit is set, and then setting the al register to 1 if that’s the case. What is this actually doing though? The carry bit gets set to 1 if there is a carry from the two numbers we add, meaning the resulting number is too big to be stored in the register. What should we do in that case? Let’s read on to find out.

Then in the next line (test al, 1) we’re checking if the value in al is equal to one. (test does a a bitwise and operation on the two arguments - like & in rust.) This sets some more flags, notably the zero flag, which is then read by the following jne instruction.

jne stands for jump if not equal (and again there’s a series of other j* instructions). Since it uses flags, it just takes a single argument: where to jump to.

Looking at where that jumps to gives us a big hint about the intent of the logic above: core::panicking::panic@GOTPCREL really gives it away. Basically all of this chunk of assembly from setb to jne is checking if we’ve overflowed the register and panicking if we have.

The one bit we didn’t discuss is mov qword ptr [rsp + 8], rdi # 8-byte Spill. As the comment implies this is “spilling” the value from the rdi register onto the stack, since the code we’re possibly about to jump to might overwrite that register - immediately after the jne we load the value back off the stack.

Finally we shuffle the stack pointer back to it’s starting point, and ret back to the caller. ret uses the last value on the stack (which is pushed by call) to figure out where to jump back to, so moving the stack pointer back is very important.

So maybe at this point we’ve seen enough to take a stab at replacing the guts of the add function with the asm! macro. Since we’re interested in performance, we’ll ignore those pesky overflow checks, and just assume that we’re within the bounds of u64.

The biggest new thing we’ll have to deal with here is specifying the in and out registers. The rfc has a very approachable explaination of these, so I’d recommend reading that. There’s a skeleton you can start with here, if you want to have a go yourself.

The version I’ve cooked up looks like this. This is probably the “fanciest” possible version of this, since we’re using as many features of the asm macro as possible:

we’re letting the rust compiler pick the register we use, and then writing it in using the format string behaviour of the asm macro.
we’re also using inlateout to hint that we can just use a single register.

This seems like a reasonable point at which to break. We’ve covered a reasonable chunk of the instruction set in x64 assembly, and seen examples of most of the classes of instructions. There’s tons more we can explore, like:

How do loops work?
What happens when we use values that don’t just fit in registers?
How do we make a syscall?

Hopefully the resources I’ve linked to from here are sufficent for you to continue digging in if you want, and maybe I’ll manage to follow this up.

This is one of the things that I find most confusing about assembly. There’s (at least) two different major kinds of syntax, and they are only different in syntax. So the instructions are the same (add, mov etc), but they take their arguments in different order. And the AT&T style assembly also has a pile of random symbols in it. Since rusts asm defaults to taking intel style assembly, we’re going to stick to that. If you do start googling stuff, it’s a roll of the dice as to which kind of assembly you’ll get. AT&T has a lot of % symbols in it, so that’s usually a giveaway. ↩︎
https://stackoverflow.com/questions/4560720/why-does-the-stack-address-grow-towards-decreasing-memory-addresses ↩︎

The Power of Names

Liam Falconer — Sun, 07 Apr 2019 10:45:40 BST

So I’ve just started working with a new team at BlueCove. I’m really enjoying it, especially since we have a lot of decision making space to work with. The company has just turned one (there was cake, it was delicious) and the process of building up the code we need is still in full swing.

Something interesting happened to me the other day. I was talking with one of my co-workers about a problem we were trying to solve, and was about to reach for one of the terms from the Domain Driven Design (DDD)¹ book, namely Bounded Context I asked him if he’d read the book, and he hadn’t. I didn’t think much about it until I was on the tube home, and I was going over my day.

Before we go any further, I should give you a hefty disclaimer - I’m not any sort of expert in what I’m about to write about. When I was explaining why I was excited about writing this to my wife, she told me it sounded like I was writing about Semantics, which prompted an educational excursion through wikipedia. I am, however very willing to be wrong on the internet, so let’s dig in.

So while I was commuting home, I realised that a chunk of the language I use when talking about design with other programmers comes from the DDD book. More broadly, I’ve picked up quite a few bits of language from various places that I think help drag conversations up the “ladder of abstraction” - even that’s a term behind which there’s a bunch of related ideas. I think this is the obvious superpower² that humans have. Once we know something, name it, and share it we gain a form of limited telepathy that lets use one or a few words to describe something, and often much more evocatively than using a whole sentence or more.

Since I called it a superpower I obviously reckon it’s pretty neat, but it has some limitations. The obvious part is that it requires both parties to a conversation to have an understanding of whatever terms you’re using. And you probably want it to be the same understanding. For instance TDD is overloaded - while it might be safe to assume programmers don’t think it stands for Telecommunications device for the deaf, it’s not like we know what paths our fellow coders have taken through life. They may have just not run into a term before. Or it can be coloured by their experiences with it. Something you view as unambigously excellent like Dependency Injection might be a jungle someone else had to cut through. And I’m sure there are plenty of us that have been scoffed at for not knowing a term.

But of course, we all start out that way. Writing this has made me realise how much I like learning new bits of language that I can use - not only with my peers but also to help my clarify my own thoughts. Some of my favourite conference talks have introduced me to new words: Peter Goodman told me what Connasence is. Rich Hickey has given me Complect and Hammocking. I’m pretty sure a big part of the reason why I love Clojure is because almost every talk expands my vocabulary or even just sharpens my knowledge on some bit of language, like for instance the distinction between bugs and errors.

What’s interesting to me is that a lot of these terms haven’t really escaped the Clojure community. While I can be relatively sure that at a Clojure meetup I can throw Complect around with reckless abandon, I’m less sure of that during my day job. To bring it back to where this started - there’s a bounded context within which a name is valid, or at least has a given meaning. DDD also offers us a related term: Ubiquitous Language I think this describes one of the smallest contexts within which language is valid³ - DDD’s author, Eric Evans, talks about it within the scope of a project, and being formed and used between the authors and expert users of a system. Hauling it up to a more general level I think yields jargon

Jargon is specialized terminology used to define specific words and phrases used in a particular profession, trade, or group.

It’s tempting to think of this like an onion - with layers of more general language expanding outwards from some core of highly specialized language. If we drew on the examples above:

You might notice the little question marked section there. There’s obviously a unit of language knoweledge smaller than a team - you!

There’s a problem here though, your command of language covers more than one domain, so I think if we try and capture that we end up with something a bit messier:

I don’t think this tell us much in isolation. Almost anything you measure needs a point of comparison to make sense of. So let’s zoom in a bit on the programming chunk, and add another person’s language blob:

Note this is just an illustrative example. I don’t think you could or should actually produce something like this. But it does put into pictures what started this whole thing off - there’s a difference in the language my colleagues and I know and use to talk about programming.

Some of the gaps here look scary. And even more scarily, it’s not just about language, since words are a mechanism for encoding concepts and ideas, there’s a gap in knowledge in those shaded chunks.

Is this actually a problem? Absolutely not. In fact I’d say having a team where everyone knows exactly the same things is a terrific waste. You’re far better off having a wide range of experiences and the resulting different ways to talk about those experiences. And if you’re in a healthy organisation, saying “I don’t know this” will be seen as a chance to show you something neat, rather something to make fun of.

I’m sure there’s more I could say about the relationship between progammers and language⁴, but this post is closing in on 1000 words and at risk of losing coherence. So what point was I trying to make with all of this? I think the shared language teams, communities, and programmers use and have built up over time has made us more effective. Personally I find learning new bits of specialized language very interesting⁵, and I think we should try and spread the knowledge of these words more widely. Finally, different knowledge helps us be more effective as teams, as long as we trust each other and feel safe not knowing something.

I hope you got something good out of this. If you reckon I’ve got something wrong here, or just want to have a bit of a yarn, the best way to reach me is on twitter.

This is an Amazon affiliate link. If you object to that sort of thing, use this link instead. ↩︎
Many ancient cultures and religions held that knowing the “True Name” of something or someone was to have absolute power over them. ↩︎
There are tighter scopes - Cryptophasia is the language shared by twins, and generally unintelligble by anyone apart from them. ↩︎
I’m so sure I’ve already pulled the kernel of another blog post out of this one. ↩︎
If you’re a massive nerd like me and you’ve got a favourite bit a jargon, please tweet me a mini-explaination, I’d love to hear about it. ↩︎

Reducible Systems

Liam Falconer — Wed, 06 Feb 2019 21:22:46 GMT

A while ago I had a minor revelation about why I’m ok with working in the LMAX codebase. I’ve got a pretty strong interest in functional programming by way of Clojure, so you’d think working in a system where pervasive mutation is the rule rather than the exception would slowly drive me nuts.

I think there’s a few reasons I’ve retained my sanity, but the most important one is the way the system is designed it’s easy to keep a mental model of, and that model isn’t often broken. Why is that the case, though?

LMAX’s production system

In order to explain I’m first going to have to give you a bit of background. The production exchanges at LMAX have around 70 services at last count. There’s a few major categories however, and we’re only going to concern ourselves with what we call the “core” services: the Execution Venue (EV), also known in industry parlance as a Multilateral Trading Facility or MTF. And it gets called “the exchange” relatively often as well. The other service that’s similarly designed is the Execution Management Service (EMS), also known as “The Broker”. Again, the wider industry would probably refer to it as an Order Management System

What is it that makes these services different? Our core services do most of the heavy lifting in our system. They’re where most of the “business logic” actually resides so we have high speed and resiliency requirements for them. These are largely met by running the core services in hot-hot pairs, which in turn means we need the services to be deterministic.

So our requirements have led us down a path that results in a design something like this:

What’s important to note about this is that the ordered, replicable journal that is the disruptor in front of the service is what enables so many of the desirable characteristics of the system.

As I mentioned, mutability is fairly pervasive in these systems. Especially in the EMS, there’s a lot of account level state that can change over the lifecycle of an order. You’d think this would be a cause for concern for me, and it was initially, but I don’t really worry about it now.

Why? Well, let me redraw that diagram above.

Basically, I think our core services are modellable as big reduce style operations:

class Ev {
	public Pair<State, Collection<Message>> processEvMessage(
		State s, 
		Message m) {
		//... do some actual work, produce messages etc ...
	}
}

This isn’t strictly speaking a reduce operation since the accumulator (state) isn’t the only result of the function, but we could just as easily stuff the output messages in there.

Anyway, that was the revelation that I had. Our system can be viewed as a giant reduce operation over a stream of messages. Not only that, but I think this idea generalizes a bit.

The Duality of Message Passing

I think this idea of a system based around message passing being “trivially” convertible to a reducer can tell us a lot about the gaps in the message passing toolkit.

We have a wide variety of primitives for composing functions that operate on values together, but far fewer corresponding way of composing objects or actors in systems that operate on messages.

For instance the problem of fan-out is trivially handled in Clojure:

(->> messages 
	 (map (juxt do-thing-one do-thing-two))
	 (reduce combine-thing-results))

(Juxt creates a function that calls the provided functions, returning their results in a collection)

When we have to do this in the LMAX codebase it invariably involves a new class called SomethingFanOut. Admittedly this may be due to our choice of Java, which is somewhat notorious for the amount of boilerplate you’re required to lay at it’s altar.

Looking at the advantages that message passing has, I think the primary one is to allow us to encapsulate things we’d regard as distasteful or downright wrong and still present a clean interface to the outside world.

This allows us to take up more complexity into our code than a reducer function does, especially since within any given method we can spread our state out still further across more sub-components, while a reduce function needs to hold all of it’s state inside the accumulator value.

Again, flipping it over to the message passing side, I think this tells us that we need to tightly control any possible changes to our state. Anything that isn’t fed through a serialized message stream is a cause for concern, since we could find ourselves with state that isn’t reproducible.

This rules out querying a database directly from any service that we want to maintain this reducibility around, and indeed this is a constraint we enforce in the LMAX code base - all database interactions have to be routed via a “general” service that pushes the response back to us over the message bus. The re-frame docs also strongly advise us to to avoid any impurity in our application and to instead push that to the edges of the system.

I think there’s also lots of benefits to be gained from this duality, along with quite a few we’ve already realised. Let’s have a look at what this effort has yielded at LMAX.

The usefulness of reducefulness

There’s many occasions where we’ve wired up a core service to a journal from production and replayed all the messages from it, so we can examine the state of the application as a particular message or series of messages pass through it.

It’s also useful to replay production traffic through a system for performance testing, allowing us to run production traffic with assertions enabled, etc. The reason we have these journals in the first instance is for our recovery and failover strategy. We want to ensure we can bring our system back up to the same state it was in if anything goes wrong. That we can ship them around and use them for many other purposes is a great side effect of this choice.

If there’s one thing you take away from this post, if you can record an executable journal of your system’s state, I would try to do so. Being able to replay a bug repeatedly until you’re sure what’s happened, fix it, and then watch it not happen is one of the best workflows I’ve experienced.

The accumulative approach also suggests another way to design our system, one that I’ve seen before, in fact. For a while the “one big atom” model has been relatively popular in the Clojurescript ecosystem - several popular libraries such as re-frame and om both use this pattern for state management.

I think there could be several benefits to this pattern for LMAX’s core services. It could allow us to shed a lot of the persistence services we’ve written, by using our secondary and tertiary instances as read only slaves. That would also eliminate a ton of the boilerplate we write to persist configuration changes to a database and generally ensure the database and core services remain in a consistent state.

It would also allow us to roll back a failed transaction, mark a message as dead and continue operating - something we cannot presently do without manual intervention and the risk of state corruption in the core services. We probably won’t be able to do this - the performance impact would probably be too high. This should not dissuade you from considering it, many environments do not have our performance constraints.

Finally it would allow us to more easily snapshot the state of the core services, since we’d already have a handle on the root of the state tree. Having easily snapshottable event streams is an advantage in other situations as well. For many of our analytics workload we work of a stream of orders, and want to coalesce these into a image of the state of the order book at a given moment. Having systems where this is baked into the design would make our lives a lot easier in this case.

Other reducible systems

I don’t think LMAX is the only place where these kinds of systems are in use. In fact, I’ve got a much stronger claim: I think almost every system relies on something like our core applications in order to actually make their system understandable.

Most multitier-architecture web applications involve some sort of concurrency, either involving threads or some other sort of way of getting more work out of the multiple cores available in your server.

The problem comes when you’re trying to get something involving state done. The interaction between concurrency and state is an area where a lot of things can go very wrong. Having to deal with it in your application is something we generally try to avoid, and instead make them stateless if possible. This means we push all of that mess down to the workhorses of web applications: databases.

The key thing is that every relational database uses a query log in order to ensure queries are executed in a well known order. There’s a lot of optimizations in most databases to get them to actually execute concurrently, but as application programmers, we can assume that serializability holds.

Databases also, of course, serve as big repositories of application state. Here’s a (rough!) sketch of the design - contrast it to the “EV” diagram above.

Databases offer us the same constraints as LMAX’s core applications. There’s a very clear sequence of events. We can get a very good idea of the application’s state (which we’ve outsourced to the database) and what allowed transitions and mutations could, should and did actually occur. Without this I think it gets much, much harder for us to reason about the way our programs really operate, especially when something goes wrong. It becomes much easier to become trapped in quagmires of FUD when trying to add new features to your codebase.

There are other examples in other domains: React and the tools built on top of it ( flux and the aforementioned re-frame and om) have dramatically reduced the cognitive burden of working on large front end applications, by using the same idea of serialized messages altering state. (and then deriving data from it)

The thing that these systems make more clear is the usefulness of having your state in one place and being able to easily query it. I think that’s a thing that many systems built around message passing get wrong. You don’t need to protect your application state from your other the outside world if you control the ways they’re allowed to update it.

Conclusions

While maintaining the invariants needed to build a reducible system does involve quite a bit of hard work, in our case we’ve found the payoff to be worth it. Both in terms of performance and productivity it’s a terrific aid once your system has grown to a reasonable size. Anecdotally, the productivity boost I felt when moving to Clojure is roughly the same amount as I found when I started working with and in LMAX’s codebase.

Unfortunately I’m not sure how broadly applicable this is. I know it works at LMAX, but I think generalizing “reducible messaging” to other environments (without doing all the heavy lifting we do at LMAX! Our process might be overkill for many other lines of business) needs something along the lines of “re-frame for the backend” which I don’t think is a small amount of work!

All that aside, I think there’s some interesting ideas we can gather from studying the differences between these kinds of systems that should be the same, but aren’t quite.

If you’ve got any thoughts or feedback, I’d love to hear from you. The best way to reach me is on twitter - dm’s are open.

Big thanks to my colleague James Byatt, for seeding and helping me to refine this post, and also to to Nick Palmer for reviewing it.

re-frame and the Back Button

Liam Falconer — Mon, 07 Jan 2019 20:31:00 GMT

A few things I’ve been working on over the christmas break have involved a fair bit of front end work. I’ve been using re-frame to do most of it, which has been a very satisfying experience. It’s made me quite a bit happier than the month or so I spent working with Vue.js for a project at LMAX. But all of that’s probably a topic for another post.

One of the things that I (along with a good chunk of the internet, apparently) loathe is when a web site or app renders the back button non-functional, or otherwise stops it from doing what I expect. So of course as soon as I added a second panel to my application (I managed to hack away on a single page for a while.) I had to fix the back button.

Re-frame’s (extensive, witty, and generally awesome) documentation already has a page on how to handle navigation. The approach it takes is totally reasonable in terms of the web-app itself, and if you’re using re-frame outside of a web page, it’s probably just fine.

It will not, result in the browser knowing a transition has happened, so the url will not change, and the back button will drag you back to the last site you were on, rather than undoing the last thing you clicked on (as I would expect in this case).

Fortunately there’s a widely supported browser history api that’ll let us sort this out.

There’s three things we need to do:

pushState when a link is clicked
Handle the popstate event (triggered when someone hits the forward or back button)
Make sure our initial state is correct

Let’s get started.

Using `pushState`

As of the time of writing, the re-frame docs have suggested the following snippet for navigating between panels in an app:

(re-frame/reg-event-db
  :set-active-panel
  (fn [db [_ value]]
    (assoc db :active-panel value)))

(re-frame/dispatch 
  [:set-active-panel :panel1])


(re-frame/reg-sub
  :active-panel
  (fn [db _]
    (:active-panel db)))

(defn panel1
 []
 [:div  {:on-click #(re-frame/dispatch [:set-active-panel :panel2])}
        "Here" ])

(defn panel2
 []
 [:div "There"])

(defn high-level-view 
  []
  (let [active  (re-frame/subscribe [:active-panel])]
    (fn []
      [:div
       [:div.title   "Heading"]
       (condp = @active                ;; or you could look up in a map
         :panel1   [panel1]
         :panel2   [panel2])])))

There’s a couple of different ways we can handle this, but what seems to be the most correct method (at least to me) is using re-frame’s effects api.

So what do we need to get this working? We’ll need to use reg-event-fx to create something very similar to the existing :set-active-panel event handler, have that emit a new effect, and finally reg-fx to handle that effect. I suggest you leave the existing :set-active-panel event handler in place, we’ll need it again in a minute. But let’s get started with the new event handler:

(re-frame/reg-event-fx :navigate-to-panel
  (fn [{:keys [db]} [_ url panel]]
    ;Same as :set-active-panel
    {:db         (assoc db :active-panel panel)
    ;New push state effect
     :push-state [url panel]}))

Let’s add a effect handler using reg-fx to deal with our newly added :push-state effect:

(re-frame/reg-fx :push-state
  (fn [[url state :as value]]
  ;Grab the history object off the top level window
    (-> (.-history js/window)
        ;Invoke the `pushState` function with...
        (.pushState                     
          ;state object this is stored by the browser
          ;"serialize" some edn by `pr-str`ing it
          (pr-str state)
          ;"title" - currently ignored.
          ""        
          ;the new url we want displayed
          url))))

We’ll also have to update the existing on-click handler:

(defn panel1
  []
                                       ;change the key and add the url
  [:div {:on-click #(re-frame/dispatch [:navigate-to-panel "/panel2" :panel2])}
   "Here"])

This is only a part of the wiring we need to do, but it should trigger some of the changes we’re looking for. For starters, you should see the url change to “/panel2” when you click on “Here”. If you open up the browser console and type in window.history.state you should see an object with the value you just passed as the first argument to pushState - in this case ":panel2" Clicking the back button won’t work yet, we still have to update the application state on popstate.

Handling the `popstate` event

This bit’s pretty simple, we just need to make a function that’ll consume the event object we’re sent by the browser when the back button is hit.

In this case we can do something like:

(defn handle-pop-state [evt]
  ;`.-state` has what we passed as the first arg to `pushState`
  (let [state (edn/read-string (.-state evt))]      
    (re-frame.core/dispatch [:set-active-panel state])))

Hopefully that’s understandable. We’re just trying to update the application state back to what it was before the link was clicked. In this case that just means setting the :active-panel key back to :panel1, which we can use the existing :set-active-panel event for.

Obviously if your application is more complicated you might need to add more logic than what’s in the sample :set-active-panel handler. In fact you could send the whole db state. There’s a limitation on the size of the serialized state value (640k characters in Firefox, at least), but you can probably get quite a bit in there before running into trouble.

In any case, we then need to bind that pop-state handler in the right place:

(set! (.-onpopstate js/window) handle-pop-state)

I’ve got this inside my main method in my core.cljs file. There shouldn’t be any issues with rebinding it if it gets reloaded by figwheel or anything.

So what happens now? If you try to go back from :panel2 you should see nothing, and if you look in the console you’ll probably see an error pointing towards the edn/read-string expression in handle-pop-state.

The problem here is the fact that we didn’t push any state for the initial page we were on, so when we go back (triggering a popstate) we get nil for .-state which read-string rightly barfs on.

Thankfully, the fix is simple.

Making sure our initial state is correct

Open up a new tab and navigate to the page again, then open the console and enter window.history.state again. You’ll get back null. This is the cause of the problem we’re having.

So what we want to do is add a snippet like so:

(.replaceState (.-history js/window)
                   (pr-str :panel1)     ; state, will replace the initial null state
                   "")                  ; title

This will replace the initial null state with the provided state. You can put this at the top level, or inside your main function. This should make the back button work perfectly. Only problem you might run into is if you’re using something like figwheel to hot reload the page. On this case this chunk of code will execute every time the page reloads, smashing the actual current state. To fix this we can simply wrap it with a conditional statement:

(when (nil? (-> js/window (.-history) (.-state)))
    (.replaceState (.-history js/window)
                   (pr-str {:handler :home})
                   ""))

Wrapping up

So that should be enough to get you started at least. When used as part of a larger application, there’s going to be some more wrinkles of course. You’ll probably have more state that needs to be rolled back as part of heading back - this should just mean that the panel argument becomes a map that’s merged with the re-frame db. There’s also concerns around urls, parsing values out of them, and what to do when the user first lands on a page that isn’t the root page. I haven’t got all the answers for those just yet, but I’ve started grappling with them and I’m hopeful I’ll be able to write another post with some of the answers soon.

If you liked this, or if you’ve just got any questions, please let me know. The best way to reach me is probably on Twitter - DMs are open.

Where the GC Fears to Tread

Liam Falconer — Sat, 22 Sep 2018 12:30:00 BST

A while ago at LMAX Exchange our staging environment was having services killed frequently by the OOM killer. This wasn’t that unusual, since our staging vms are relatively under-provisioned compared to our production environments

(We have separate performance testing environments, with much beefier hardware for making sure we go fast, whereas the staging environments exist more for testing our deployment and configuration.)

But it was irritating, since it led to a reasonable amount of toil on the part of both the dev and systems teams. We eventually ended up figuring it out, but it was a bit of a journey. All of the dev team was interested in it, so we begged, bothered and cajoled the whole story out of Nick, who did most of the digging. I thought it was good enough to share with the world at large, so I’ve reproduced his presentation in blog form here.

So one of our systems team was having a bit of a poke around for services that were using more memory than they perhaps should have been:

#Search for java services...
ps -eo pid,comm,rss,cmd | grep java | grep -v grep |
#...pull the Xmx and display name paramters out of the command line...
awk '{ match($0,/Xmx[^ ]+/,a) ; match($0,/display.name=[^ ]+/,b) ; 
       printf "%s PID=%s RSS=%s %s %s\n", $2, $1, $3 , a[0], b[0] }'
#... then print them

#...
#java PID=18168 RSS=627804 Xmx128m display.name=payment-svc
#java PID=18169 RSS=1147628 Xmx256m display.name=account-svc
#...

I’ve elided most of the output above to highlight the two interesting services. These have their maximum heap set fairly low, much lower than the ~600m and ~1.1g they’re using.

So what’s going on, and how can we figure that out? At this point I’m going to pull a bit of a bait and switch on you, so you can follow along at home if you want. Nick, also made a little demo app that showcases a lot of the ways you can use up memory in the JVM, which you can find on Github.

The demo app is subject to the same leak we have had in production, as well as showing off a few other ways you can leak memory. It’s also a lot smaller than the services we were dealing with, so you won’t have as much “fun” searching through the codebase for the culprit. I’d encourage you not to look through the code in the demo app, and instead follow the investigation in this post.

If you’re not particularly interested in that, and just want to hear a good story about memory usage gone wild, feel free to skip ahead

If you do feel like following along, clone the source from the repo using git clone https://github.com/Palmr/java-off-heap-leak-example.git and use gradle assemble twice to build it. (Apparently there’s something wrong with the subproject dependency ordering or something…

Once that’s done you can extract the built artifacts: cd build/distributions && unzip OffHeapLeakExample-1.0-SNAPSHOT.zip Then head in and run the app: cd OffHeapLeakExample-1.0-SNAPSHOT && ./bin/OffHeapLeakExample.

If that gives you any issues, have a look at the Readme, or raise an issue. I’m sure Nick will be happy to help.

Sifting through the heap

So first order of business how much memory does it reckon it’s got in the heap?

That’s fairly easy to figure out using jmap -heap $PID:

$ jmap -heap 22974
Attaching to process ID 22974, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.172-b11

using thread-local object allocation.
Parallel GC with 20 thread(s)

Heap Configuration:
   ...Elided, for being uninteresting...
   
Heap Usage:
PS Young Generation
Eden Space:
   capacity = 183500800 (175.0MB)
   used     = 123278096 (117.56715393066406MB)
   free     = 60222704 (57.43284606933594MB)
   67.18123081752232% used
From Space:
   capacity = 30408704 (29.0MB)
   used     = 0 (0.0MB)
   free     = 30408704 (29.0MB)
   0.0% used
To Space:
   capacity = 30408704 (29.0MB)
   used     = 0 (0.0MB)
   free     = 30408704 (29.0MB)
   0.0% used
PS Old Generation
   capacity = 489684992 (467.0MB)
   used     = 0 (0.0MB)
   free     = 489684992 (467.0MB)
   0.0% used

In our case we got numbers nowhere near the size of the resident set. So what else does a JVM allocate? Well, we’re about to enter the strange and terrifying world of Off heap memory. It’s a dark and twisted place where the normal rules of garbage collection are suspended in favour of the freewheeling rules of user managed allocation. Thankfully, most of the users in this case are still under the control of the JVM itself: Metaspace and Direct Buffers.

Both of these we can inspect the using jVisualVM, which is bundled with the Oracle JDK. If $JAVA_HOME is set to point to an Oracle JDK you can just do $JAVA_HOME/bin/jvisualvm and it should start up, and if we go have a look at the monitor page we should see something like:

Nothing here seems particularly amiss, right? The heap usage is fairly low, there’s a few threads running, but nothing too crazy going on here. So at this point we’re pretty sure that heap allocation isn’t the cause of our problems. What else can we dig into using jVisualVM? Well I’ve already given that away a bit - we can examine the Buffer Pools tab:

As you can see, this shows the memory committed to Direct and Mapped buffers. In the case of this demo app the allocation is flat - nothing to see here. The other bit we should have a look into is the Metaspace, the size of which we can see in the VisualGC tab:

(Note that this is an optional plugin to jVisualVM, and might not be included with your distribution. If that’s the case you can download it inside jVisualVM under Tools | Plugins | Available Plugins)

That’s the orange bit right down the bottom there: we can see it’s not really changing. Strike off another one. We’ve pretty much exhausted what jVisualVM is able to tell us, but while we’re here I should mention the way you’d normally find a leak in Java - with a Heap Dump. If you go back to the monitor page (or just scroll up a bit), you’ll see a Heap Dump button on the top right. Whacking that should take you to a page something like this:

Given the heap size looked as we expected it according to jmap and the other tabs, there’s not much to see here. There’s 1000 objects the example makes which you can trace GC roots and generally have a bit of a poke about, but they aren’t the smoking gun we’re looking for. There’s also one other leak that you’ll be able to find from here. That’s left as an exercise for the reader, if you want the answer it’s in the slides that are linked to in the example app’s readme. There’s another war story behind why we know about that particular leak being a thing, but that’s for another time.

So. Now we have to get a bit deeper into the weeds. What about the memory allocated and used JVM itself? Fortunately, there’s a tool for investigating that: Native Memory Tracking (NMT). This is accessed via a command line flag: -XX:NativeMemoryTracking, with two useful values: -XX:NativeMemoryTracking=summary and -XX:NativeMemoryTracking=detail. This does of course come with a hit in terms of performance, between 5 and 10 percent. In order to actually make use of it, you should use jcmd to get snapshots: jcmd ${PID} VM.native_memory baseline and then diff them against baselines: jcmd ${PID} VM.native_memory summary.diff. For example when I run them against the example app:

#This is included in the baseline_NMT_and_log_memory.sh
#script Nick created.

$ jcmd 24055 VM.native_memory baseline

#... some time later

$ jcmd 24055 VM.native_memory summary.diff
24055:

Native Memory Tracking:

Total: reserved=1473463KB +41KB, committed=181895KB +105KB

             Java Heap (reserved=32768KB, committed=32768KB)
                       (mmap: reserved=32768KB, committed=32768KB)
 
                 Class (reserved=1080633KB, committed=33465KB)
                       (classes #1947)
                       (malloc=21817KB #1580 +15)
                       (mmap: reserved=1058816KB, committed=11648KB)
 
                Thread (reserved=52514KB, committed=52514KB)
                       (thread #52)
                       (stack: reserved=52288KB, committed=52288KB)
                       (malloc=166KB #277)
                       (arena=60KB #90)
 
                  Code (reserved=250105KB +11KB, committed=5705KB +75KB)
                       (malloc=505KB +11KB #1414 +43)
                       (mmap: reserved=249600KB, committed=5200KB +64KB)
 
                    GC (reserved=25423KB, committed=25423KB)
                       (malloc=24219KB #172)
                       (mmap: reserved=1204KB, committed=1204KB)
 
              Compiler (reserved=137KB, committed=137KB)
                       (malloc=6KB #75)
                       (arena=131KB #15)
 
              Internal (reserved=27748KB, committed=27748KB)
                       (malloc=27716KB #3628)
                       (mmap: reserved=32KB, committed=32KB)
                       
                Symbol (reserved=3469KB, committed=3469KB)
                       (malloc=2215KB #9816)
                       (arena=1255KB #1)
 
Native Memory Tracking (reserved=459KB +30KB, committed=459KB +30KB)
                       (malloc=157KB +24KB #2210 +329)
                       (tracking overhead=302KB +6KB)
 
           Arena Chunk (reserved=206KB, committed=206KB)
                       (malloc=206KB)

Basically, there’s not a lot going on here - looks like the JVM isn’t responsible for allocating the memory we’re looking for. What next?

Outside the JVM

We’ve still got one more place to look: in memory that’s allocated from native code which we invoke via Java Native Access (JNA) or Java Native Interface (JNI). The memory that’s allocated by native code is still tied to the JVM process - since it’s still responsible for invoking the code in question, but the JVM doesn’t have any ability to see or control what the native code mallocs.

There’s plenty of tools for examining memory allocation, for instance Valgrind, but we’re just humble Java programmers and Valgrind seems hard. Looking around a bit we found simpler tools, specifically jemalloc.

Jemalloc is a malloc implementation that includes the ability to profile all of the allocations that are requested, and will very helpfully show you the full stack trace leading to that allocation. It can be fairly easily added when running a process via an environment variable: LD_PRELOAD=${JEMALLOC_PATH}/lib/libjemalloc.so.2 <COMMAND_TO_RUN> In our case since the application we’re profiling doesn’t exit, you probably want to also add some environment variables so it emits information during the run: export MALLOC_CONF=prof:true,lg_prof_interval:25,lg_prof_sample:17 - that’ll cause it to memory profile dump every 30 mB, allocation sample every 128 kB. (The configuration options are in log base 2)

Since Nick’s a real good guy, jemalloc is used by default, and there’s a script that’ll automagically produce a pretty chart in the bin directory.

If you’re doing it the hard way, you should end up with some files kicking around - something like jeprof.19678.0.f.heap. That can be fed into jeprof to get some pretty pictures using jeprof --show_bytes --pdf jeprof.19678.0.f.heap > w.pdf, which should emit something like:

You can also go via dot, there’s an example of that in Nick’s script

Which does seem to contain something rather like a smoking gun… and indeed, if we look for that we find:

void do_stuff_leaky(char* string)
{
    char* str = (char*) malloc(allocation_amount);
    memset(str, 0xff, allocation_amount);
    strcpy(str, string);
    printf("String = %s, Address = %p\n", str, (void *) str);
    fflush(stdout);

    //Whoops!
    //free(str);
}

If you followed all that along, good work, you solved the mystery, and definitely not with my help! Hopefully next time you Java program OOM’s, you can just stop at like step two though. That said, the actual issue was a touch more interesting, so let’s have a quick run-through of what was happening there.

The real leak

So the services we were dealing with (account service and payment service) weren’t particularly heavily loaded - they’re definitely off the hot path for our trading flows. Also our staging environments are fairly under-utilized, they probably burn 90% of their cpu cycles during deployment and startup. One would expect that an environment that didn’t get much traffic wouldn’t have much in the way of performance or memory issues, but in this case it was exactly because the environment was under utilized that we got into trouble. But first, a quick detour to talk about reliable messaging over UDP.

So much like developers of video games, those of us writing high performance financial software prefer UDP to TCP for it’s performance.

However, we have strict requirements around reliability, that is we can’t afford for our services to miss messages. Because of this, we have a scheme of Negative Acknowledgement or NAKing for short. The reason we choose to use NAKs rather than the ACKs traditionally seen in TCP is because in the normal, no packet loss environment, we don’t have to wait for the receiver of a message to acknowledge it before we can send them more data. When using this kind of scheme, every message gets a sequence number, again, similar to TCP.

When we do lose a message somehow, a service will NAK to the sender of message when it sees a gap in the sequence numbers, and the service will (hopefully) resend a copy of that message by reading it back out of it’s memory. This does introduce a significant “hiccup” in the sender, since it has to scan back through memory, but in the normal case we expect to be operating in it’s significantly more efficent than the alternative.

Why is this important? Well back in the mists of time, many of our services upon starting up would send a NAK to other services that they listened to, in order to ensure they were up to date with what had been sent. We alert on NAKs in our monitoring system (since normally they’re indicative of something going wrong) so on startup our monitoring would light up like an angry christmas tree, typically spooking whoever was doing work on the system.

In order to prevent this, we added a heartbeat mechanism, that would send a message with the latest sequence number, once a second if there was no other traffic between services. We could use this to determine the starting point for a freshly booted service, without firing off NAKs. This also helped us reduce the potential impact of some cases of message loss. Obviously this message was fairly small, since it only had a single 64 bit value and some headers on it.

The other thing that’s worth mentioning, is that our application level NAKs aren’t the only NAKs in town. We’ve built on top of a lower level library we buy which provides a fairly fast UDP networking layer. This also maintains it’s own store in memory that it is able to service NAKs from, at it’s layer.

This storage is set to a fairly small amount - 25mb by default. However, this library also maintains a header alongside the message body (this header isn’t send across the wire - we’re not really sure what it’s for.). These headers are about 200 bytes in size, and are not counted towards the 25mb limit. In our case with the heartbeat messages, these were only about 12 bytes in size, so in order to fill the 25mb buffer, we’d need around 2 million messages to be sent, each allocating a 200 byte header, meaning we’d end up using around 400mb for the headers. That’s quite a bit more memory usage than we expect, especially given these services were running with heaps around half that size.

At this point you might be asking me about the numbers we came up with right at the beginning of this, since they were bigger than that 400mb, and surely we didn’t leave our staging environment alone for the 24 days or so these buffers would take to fill up in any case? You’re right on both counts, however the underlying messaging library, and our heartbeating scheme above it both work on a per topic basis - so we were sending these messages significantly more than once a second, and giving them more than just one 25mb buffer to fill.

We have since fixed the issue (we don’t need the lower level NAKs when we already have application level ones, so we’ve smashed that buffer) and now quite happily spend much less time resurrecting dead services on our staging hosts!

Summary

That was quite a lot of words, so good work if you read the whole thing. Hopefully you’ve learned a thing or two! If you’re looking for more resources, there’s quite a few links in Nick’s slides or I’m happy to try and answer any questions you’ve got on twitter.

Bitcoin crash course

Liam Falconer — Wed, 01 Aug 2018 10:40:00 BST

I’ve just moved to London, and been working on LMAX Digital and as a result been given a bit of a crash course on how exactly Bitcoin (and other cryptocurrency) transactions actually work. I think there’s a near zero chance I’m going to remember it as I’ve moved on to another area of work, so I figured I’d better write it all down before I forget it all.

There’s quite a few concepts associated with the bitcoin blockchain, but probably the most important one, and one of the few that exists as a concrete object is the transaction. This is what actually goes into any given block on a blockchain.

So let’s talk a little about the structure of the actual payments or transactions. There’s some header information, but mostly they consist of inputs and outputs. The inputs are the unspent outputs in a previous transaction that you have the ability to spend, and the outputs are the payments that you want to send to other addresses. Of course it’s a little more complicated than that, since we have to have some verification - you have to form a transaction that proves you can actually spend the inputs, and create some sort of condition that needs to be met for the recipient to spend your outputs.

This is done by building up a script from the inputs and outputs of transactions. The script is run by all the nodes participating in the bitcoin blockchain to verify that the inputs are allowed to consume the outputs from older transactions they refer to. The structure is fairly recursive, and probably best described in pictures, rather than words:

There are other parts I’ve omitted from this diagram: the outputs also contain amounts, and the block itself has a header with various information including a value that whoever is mining the block can tweak to try and produce the desired hash. There’s many other parts that are even more ancillary to the process, if you’re interested refer to the bitcoin wiki.

It’s easiest if we consider the output first, as these form the input to the next transaction. Each output specifies a script, written in a forth like language. This specifies what scriptSig is needed to unlock the output and spend it. When it comes time to spend that output, we prepend the scriptSig that’s provided with the input, and check that the script leaves true on top of the stack.

Let’s work through an example of that. The most common script is Pay-to-PubkeyHash which looks something like: OP_DUP OP_HASH160 <pubKeyHash> OP_EQUALVERIFY OP_CHECKSIG. <pubKeyHash> is a placeholder for the hash of an actual public key, provided by whoever wants to receive the payment. This is often what’s referred to as an address. The corresponding scriptSig that’s used to spend this transaction will look like: <sig> <pubKey>. The reason for this is that public keys can be relatively widely shared or unsecured, so we include a signature generated from the private key which can be verified by using the corresponding public key. So we join the two together and get: <sig> <pubKey> OP_DUP OP_HASH160 <pubKeyHash> OP_EQUALVERIFY OP_CHECKSIG

So anything that isn’t an operation get’s copied straight onto the stack, so the first thing that happens is the
<sig> and <pubKey> are pushed onto the stack:

Stack:
<sig> <pubkey> 

Remaining Script:
OP_DUP OP_HASH160 <pubKeyHash> OP_EQUALVERIFY OP_CHECKSIG

OP_DUP copies whatever’s on the top of stack and pushes it onto the stack, yielding:

Stack:
<sig> <pubkey> <pubkey> 

Remaining Script:
OP_HASH160 <pubKeyHash> OP_EQUALVERIFY OP_CHECKSIG

OP_HASH160 hashes the value on the top of the stack, then pushes it back onto the stack, giving us:

Stack:
<sig> <pubkey> <pubKeyHash>

Remaining Script:
<pubKeyHash> OP_EQUALVERIFY OP_CHECKSIG

The pubKeyHash from the script then gets pushed onto the stack, and we then compare it to the hash we generated in the previous step. This is the first point at which we actually check the input is allowed to be spent, by verifying that the pubkey corresponds to the hash in the script. If this step fails, the transaction will get rejected by miners and not included in a block. Assuming this succeeds, we then move on to the final step:

Stack:
<sig> <pubkey>

Remaining Script:
OP_CHECKSIG

OP_CHECKSIG does what it says on the can, checking the signature on the stack was produced using the private key corresponding to the public key on the stack, and that the signature is for the contents of the entire transaction. This leaves either true or false on the stack, which indicates if the transaction is valid or not.

At this point the transaction can be broadcast to the blockchain, where miners will run the same set of checks before deciding if they want to include it in the block they’re mining or not. Their choice is mostly based on how many bitcoins you’re willing to give them: during the process of forming a transaction you should leave an amount from your inputs that isn’t allocated to an output for the miner to claim, to incentivize them to pick your transaction to be included in the block.

As each block is of a fixed size (1mb at the time of writing) miners will want to pack the block with the transactions offering the highest payout to them. For this reason transaction fees are often measured in Satoshis per byte (a Satoshi is the smallest useable increment of a bitcoin: one hundred millionth of a bitcoin).

What I think is most interesting about this is what we left out: we didn’t talk about wallets or addresses, since they’re abstractions we’ve built on top of transactions. A wallet is, at a bare minimum, the keys required to produce the signatures for a given output. Often we want other features like knowing the balance we have available (that’s the sum of all the unspent transactions we have the ability to produce signatures for).

Similarly, an address is really just a output script that you know you can fulfill the requirements for. Typically you’ll only supply the pubKeyHash in the script we worked through above, but of course if you’re using a multisig wallet or something else more exotic, you can ask whoever is paying you to use a more complex script.

This about sums up what I think are the essential elements of the bitcoin blockchain, which is to say, transactions and their constituent parts. Of those parts the most complex and crucial to the operation of the bitcoin payment infrastructure is the script mechanism, used to verify that you’re actually allowed to spend any coins sent your way. If you’ve got any questions or got corrections, message me on twitter.

Roasters log 1

Liam Falconer — Tue, 06 Feb 2018 17:40:00 +1300

So I’m up to roast number 7 at this point (probably 8 or 9 by the time I’ve finished this blog post…). I’m beginning to think maybe I like the process of roasting coffee more than drinking it - my biggest issue at this point is that I can’t iterate quickly enough on the roasts because I don’t drink coffee quickly enough. I only want to roast enough coffee for about a 5 days worth of drinking, i.e. ~5 batches in my wee popcorn machine, since apparently the “freshness” really starts to fall off after 5 or so days. I’m tempted to try and foist it on my friends, but I think I need to try and dial it in a bit first.

Yesterday I picked up a copy of Home Coffee Roasting: Romance and Revival from the library. I’ve been jumping all over the book, but so far it’s been great. I’ve already locked in my roaster, so a lot of advice about what the options are is somewhat wasted on me, but probably the most important thing in there is keeping a log.

Which I wish I’d been doing from my first roast, but I was a bit excited to get started. As a result my first couple of roasts were all over the place in terms of quantity, darkness etc. I was messing around, rather than experimenting. Exhibit A being this “innovation”:

While it definitely sped up my roasts I don’t think it added much to the flavour. The thing I have stuck with, however, is using my wife’s hairdryer on a ‘cool’ setting to cool the beans after I dump them out of the roaster. This is to try and stop the roasting process as fast as possible once the beans have left the roaster. Due to the residual heat inside the beans it would otherwise continue for a while after they’re dumped out of the roaster. It also helps get rid of bits of chaff still stuck to the beans.

Anyway, what I’ve noticed so far is there’s a pretty reasonable amount of variability. Between my sixth and seventh roasts the first crack (when the coffee beans make a cracking sound as they start to undergo pyrolysis) happened at 2 minutes, 15 seconds and 3 minutes, 7 seconds respectively. That’s 52 seconds, or about a fifth of the total time I spent roasting the beans (I targeted 5 & 6 minutes for those roasts). As you can see from the picture below, there’s only a slight difference in colour as a result.

I’m not sure what’s causing that variability. I don’t have any effective means of measuring temperature, yet. I tried some digital thermometers that are rated up to 300°C, but although the probe might have been fine at that temperature, the display most definitely was not. I’m planning on drilling a hole through the side of the tin can I’ve put over the top of the metal “cup” and pushing the probe in through there to stop the display from overheating. Need to pinch a drill from my dad’s place first though.

I’ve also ordered a thermocouple from China by way of eBay, but of course that’ll probably take a month or so to show up, but when it does the amount of science I’m gonna be able to do is gonna go through the roof (I’m talking like continuous profiling and pretty graphs and stuff). Really looking forward to that.

So that means only variable I can effectively vary at this point is roasting time. (I need to get a scale as well) So I’m gonna totally try and science the hell out of that in the meantime. Based on discussions online it seems like I should be trying to target longer rather than shorter roasting times at lower (average) temperatures. Unfortunately I don’t have any way of controlling that, and I’m in a country where 240v is the standard, so the machine I’m using is a 2400 watt one. This probably means it’s heating up way to fast (I was seeing 200°C on the thermometer within a couple of minutes before the display gave out on me.)

The other thing I’m somewhat intentionally sacrificing control over is the amount of time the coffee is resting for between roasting and drinking. At the moment I’m laying in stocks until the weekend, when I should be able to get a scale to further refine the process. At that point I’m gonna try and consistently roast every day so we’re always drinking the coffee the same number of days after the roast.

Lots of variables to keep track of. At least the coffee I’ve produced so far is drinkable, and it’s definitely cheaper than buying it from the supermarket. The ideas that are bubbling out of my head in terms of potential improvements are numerous enough to keep me busy for a while, let alone the process of refining what I’m already working with. So I’ve hit at least a couple of my goals, and so far I’m definitely enjoying geeking out on the process.

Roasting coffee with a popcorn maker

Liam Falconer — Mon, 05 Feb 2018 15:16:00 +1300

I’m a big coffee fan. I’m also a massive cheapskate. Normally the first one wins, but the other day I found this post about how you can roast your own coffee. Within about an hour I’d ordered the necessary bits, and now less than a week later, I’ve roasted my 5th artisanal microbatch of coffee.

As you might be able to tell, I haven’t dialed in the actual roasting part of it just yet, but I can definitely tell you how to pull apart a popcorn maker.

So what I’ve worked with is one of these:

Which seems to be the only one available in at NZ retail stores. Thankfully, the design of these things doesn’t differ that much from model to model. Unfortunately I took all these photos after I’d pulled the thing apart, so they’re more representative than anything else.

We are going to be doing some damage to the plastic casing, and circumventing the safety measures. Obviously these actions are warranty voiding, and do definitely increase the fire risk posed by the altered device. I would not leave it unattended, and keep an eye on the machine while it’s operating. Maybe keep a fire extingusher on hand as well.

Alright so step zero, crack it open. There’s screws in the spots marked with arrows. The four outside ones are covered with little rubber feet you can just pull off. The other two are deep inside the casing, you’ll need a screwdriver with a reasonably long shaft to get at them.

After doing that, you should be able to yank the base off your soon to be coffee maker, and you should be greeted by something like this:

The bit that we need to deal with immediately in order to free the important parts from their feeble plastic shell is the switch:

You should be able to pull the cables off the switch by hand or with a pair of pliers. We have to do this since the switch will only come out if we push it towards the outside of the plastic shell. Once you’ve pulled it out you should reattach it to the cables which should just press fit back on.

At this point you should be able to pull the whole assembly out of the shell, along with the power cable and the still attached base:

It’s a bit of a choose you own adventure at this point. We’ve got to remove the overheat protection measures, and although you don’t have to remove the base, but it’s probably a good idea since otherwise it’ll just get in your way.

Lets start with the base, which is just a case of removing two of the screws retaining the power cable, (indicated with the arrows) and then cutting through the plastic base to free the cable. A big pair of side cutters will do the job just fine, and will make sure the warranty is voided.

Once that’s done we’re onto the increasing fire risk part of the operation. There’s a large metal band around the “cup”, holding a thermal fuse:

The fuse is rated for 196°C, which is a bit lower than what we want. We’re targeting around 200°C. So that’s got to go, along with the other bit that the metal band is holding on:

This is a termostat. Unlike the fuse it’ll reset itself once it’s cooled off, but it’s got a slightly lower cutoff at 180°C.

The band is held together with a bolt which can just be undone, and then you’ll have to cut through a part of it to free the thermocouple. Once that was done I just duct taped them together to keep them off to the side while the roaster is running.

The only other thing you need to do at this point is figure out how to handle the thing when it’s hot. You could probably fashion a handle from some stiff wire, but being an apartment dweller I didn’t have any handy. What I did have was a pipe wrench:

Just make sure it’s not making contact between the body of the motor and any of the components on the board, and you should be able to safely flip the whole thing upside down to extract your roasted coffee.

One piece of advice I am going to give, is don’t use the switch to turn it on or off. I’ve managed to electrocute myself since it’s not that easy to hold in place while operating. I’m probably going to replace it with something a bit more fit for purpose later.

Oh and do your roasts outside. It’ll spit out a ton of “chaff” from the beans which you don’t really want all over the stuff you own. I’m thinking of getting some aluminum ducting so I can control where the chaff ends up going.

I’m going to refrain from offering any further roasting advice, since I definitely don’t have anything resembling a process dialed in yet. You should go have a look at other people’s advice on this one, at least until I’ve had a chance to “dial in” my process.

The unreasonable effectiveness of checklists

Liam Falconer — Tue, 09 Jan 2018 08:23:00 +1300

One of the things we did recently was start using a checklist once we think we’ve ‘completed’ a story. The checklist is basically just a big list of features that already exist in our codebase, and some notable gotchas that have caught us out before. All we do is go through it and ask ourselves if what we’ve just written interacts with that feature or gotcha, and if so, have we written a test for it?

The checklist is pretty long as a result, and constantly changing, but still really useful, especially considering the amount of effort that goes into it is ~5 minutes per story. In spite of us knowing we’re going to go through the checklist, we still pretty much always turn up something, either a test that needs writing or some unexpected, broken interaction.

All you have to do is sit down and spent 15 minutes writing a list in markdown of all the major things your codebase does. Every time you think you’re done with a bit of work, take a copy, delete everything that’s irrelevant, and tick everything you’ve got a test for. Anything that’s left, well there’s something that’s at least untested, and maybe if you think about it, buggy. The only other thing we do is try to add to the template when we introduce a bug that is due to unexpected interaction between what we’re writing, and some other feature.

For us, the return on investment for this has been exceptional. We’ve caught enough issues that the tiny amount of time we’ve spent on this has more than paid for itself. And we’re not the only ones.

Why bother with property tests

Liam Falconer — Mon, 08 Jan 2018 09:00:00 +1300

So something I’ve been trying to do lately is write more property or generative tests. I’ve been spending a bit of time thinking about why we’re actually doing this. Obviously I think it’s a good idea, but what actual benefits do we get from these, apart from slightly more arcane tests?

What is a property test anyway?

Just in case you’re not sure what a property based test is, it’s a test where rather than specifying the input and output, e.g. 1 + 1 = 2, instead get the computer to generate the input for you, and validate a more abstract property of the thing that you’re testing. For example: x < x + y. They’re also known as generative tests.

Hopefully that shows they’re a little more complicated to write (the above example isn’t even correct unless we ignore negative numbers), but I’m going to spend the rest of this post explaining why I think they’re worth writing despite the extra effort you have to put in.

Increases your confidence in your code

One of the things I’ve learned over the few years I’ve spent programming is that I’m wrong a lot. Sometimes in obvious ways that a compiler will catch and tell me about nearly instantly, other times in ways that I have to go and dig at or write tests to figure out. Even when I write tests to cover whatever code I just came up with, I’m never actually sure that I’ve written all of the tests I need to. Property based tests go some way towards alleviating this fear, since I no longer have to worry about covering every individual case, but rather just each class of test.

Unless you can encode the logic you’re testing in a different way for the test (that’s obviously correct!), or you have an oracle of some kind to check against in your property test, you’re still going to have to write at least one example based test. Otherwise you can’t really be sure that the code does exactly what you expect it to (and that it will continue to do so).

If you do have an oracle to check against, even one that’s only good for a subset of the inputs, that’s great! You’re probably in a position to be super lazy, and might be able to get away with writing only a property single test for the function you’re testing. Which brings me nicely to my next point:

You’re lazy, and that’s not a bad thing

I don’t really love writing tests. For me they’re very much a means to an end, in that they help me write code more correctly, faster. Unit tests, in particular, I question the lasting value of quite a bit. More often than not all they tell you when they break is that you changed some code. I think property tests give you more bang for you buck, since usually you have to make the assertions general enough that they reflect the essence of the function you’re testing.

I’m also a big fan of making the computer do as much work as possible or at least practical. If you use a good property testing library, it’ll automatically pick out commonly problematic test cases, e.g. -1 and 0 if you’re using numbers, or "" and “í” for strings. It should also hit the edges of any boundaries you specify fairly quickly, so if you request a number in the range 25 - 75, you should see the library emit the values 25 and 75 during any given test run. This means you don’t need to worry about the boundary conditions, since they’ll already be covered.

In addition, there’s some kinds of tests where you don’t really care too much about the mechanics of what happens, you’re just looking to check some invariant is preserved. For example, serialization and deserialization. In this case, the actual values we pass in to be serialized are not really important to the test, so we can use a property testing library to reflect over the method, and feed it arbitrary generated values.

They go great with contracts

I’m of the opinion that one of the best things you can do to make your system more robust is adding contracts to check the data that flows through your code. Even if you don’t enable them at runtime (or only enable them at the borders of your system) they greatly help you confirm the consistency of your system in a way that less advanced type systems can otherwise struggle with. Little things like being able to say that a particular value with never be negative, throughout your entire system are great. They let you focus on the actual problems you have to solve, rather than worrying about the providence of a particular bit of data.

If you then write some simple generative tests at the borders of your system (they don’t have to assert anything, apart from the fact they don’t get an error as their response) you can be relatively certain that you don’t violate the constraints you’ve specified inside your codebase.

Helps you find the (actual) boundaries of your system

The first property test you write for any bit of code doesn’t even have to have assertions, just calling your code and making sure it doesn’t explode is enough. The great thing about this test is it’s always good. At most, you just need to adjust the ranges in your generator when you change the code under test.

When it does explode on some input that you didn’t count on getting, you’ll adjust the generator to not emit those values. At the same time you should step further up the stack, towards where your customers interact with the system, and write a test to make sure that your function can’t be invoked with these values.

This isn’t actually much different from the point above about contracts, it’s just that the contracts here are implicit rather than explicit.

You have to think more

This is the reason I got into programming. Sometimes it’s hard, and you have to spend a while noodling on a problem before you can make a reasonable attack on it. Often, figuring out the properties to write is one of those hard things. I don’t think that’s a bad thing. Personally I’m far more prone to under thinking a particular bit of code than over thinking it, and anything that makes me slow down a bit and worry about what the code is actually supposed to be doing is probably a good thing.

I think that’s one of the unrealized or underrated advantages of TDD, that you spend more time thinking about what the code does and how it does it. Often having read the code I just wrote is enough to make me discover there’s something in need of correction. Property testing has the same effect, just more of it.

Make impractical tests possible.

Would you write a test that performs 17 different sequenced actions? Probably not unless you already knew there was a bug there, right? If you dive into writing stateful tests, you won’t have to write the 17 action test, your property testing library will do it for you. In systems where correctness is important and there’s lots of state to cover this is one of the only ways to get this sort of coverage, short of using production workloads.

These sorts of tests really amplify the “have to think more” aspect, since you have to manipulate your system through a sequence of valid states, and find some properties to actually assert on that hold over the lifecycle of your systen. Other people have explained how to do this, so if it’s something you’re interested in I’d suggest you take a look at their efforts.

tl;dr

Property based tests are harder to write than unit tests. Sometimes, that doesn’t mean you have to spend more time writing them than you would for an equivalent amount of coverage with unit tests. Other times, that extra effort can let you test things that wouldn’t be practical or possible without property tests. Finally, while writing these tests, you often discover that the edges of valid input aren’t where you thought they would be, either because of the contracts you wrote, or the implicit contracts of the language or code you’re using. If correctness is important to you and your application’s users, I would highly recommend you put some effort into writing property tests for your system.

Flamegraphs and benchmarks

Liam Falconer — Thu, 14 Dec 2017 08:08:12 +1300

lein jmh recently turned up on the Clojure scene. I’ve been using it to debug some performance issues with a library I’m about to open source called undertaker. I’m gonna tell you a little bit about the why and the how of getting benchmarks to produce Flamegraphs like this:

Why use flamegraphs?

Why did I go to the effort of generating these things if I could just use Java Microbenchmark Harness (JMH) to figure out where it’s slow? In order to do that I’d have to write quite a lot of benchmarks to drill down into where exactly the problem is. That’s tricky and arduous, since you’ve got to make sure that each benchmark is realistic. In my case there’s quite a bit of complexity buried inside the library as well - there’s a reasonably compact public api with quite a bit of code underneath.

The other thing is anyone using the library doesn’t really care about the performance of the internals of the library. Measuring the performance of the internals doesn’t really tell me much about the user experience. Ideally what I want is comprehensive benchmarks of the public APIs, and some sort of profiling to help me figure out what code internally is slowing them down.

So what profiling options do we have? There’s stuff like Java Mission Control (JMC) that comes bundled with the Java Development Kit (JDK), but I didn’t really know how to start with scripting that. The way JMH works, (with multiple forked processes) also makes it pretty challenging to use the JMC UI to profile.

Mostly though, I’d wanted to use flamegraphs for a while, and this project was a simple one to get started on.

How do I flamegraph?

Most talks or blog posts about flamegraphs focus on using them to diagnose production issues by producing them from live workloads. What we want to do is a little different, and somewhat complicated by JMH’s way of running benchmarks. JMH runs a single benchmark repeatedly by forking a process. This makes sure that the benchmark is running in a clean, isolated environment, i.e. not impacted by other optimizations that the Java compiler has already performed.

So I started out with the crudest possible method, just using jstack, which is again part of JDK. It’s definitely not the greatest way of doing this, but I wanted to get it working without having to get in between the JMH runner process and the forked process. jstack takes a pid and returns a snapshot of the current stack. So by using jps (also included in the JDK.) to find the forked JMH process, and running jstack against the pid:

jstack $(jps | grep ForkedMain | awk '{ print $1 }') >> /tmp/jstack

This assumes that jstack is on your path, otherwise you can find it next to your java executable (try which java, or echo $JAVA_HOME)

Anyway, running that we get a single stack trace. We can run this in a loop to capture the output of an entire jmh run:

while true;
do
  jstack $(jps | grep ForkedMain | awk '{ print $1 }') >> /tmp/jstack
done

There’s not much smarts to that, you’ll have to manually kill it once JMH is done running. It’ll also print a bunch of stuff to stderr if a JMH forked process isn’t currently running. That aside, it’ll give you a file to feed Brendan Gregg’s flamegraph tools:

stackcollapse-jstack.pl < /tmp/jstack | flamegraph.pl > /tmp/flame.svg

And that’ll give you the beautiful picture you saw above. Or something like it. I could probably write another entire blog post about interpreting these in the context of Clojure, but the essence of it is finding your code in the graph, and then looking up the ‘flame’ to find what it’s spending it’s time doing.

Problems

So there’s some issues with this. The most obvious is the collection part. Ideally I’d use a java agent of some description to handle this, but the solutions that exist and are described mostly target linux. You’ll probably have a much better time if you’re running on linux using perf as described by Brendan. Probably the best answer on OS X would be honest profiler. I’m not sure if the output format for that is parseable by the existing flamegraph tools, or if I can get it to start outputting on program startup.

There’s obviously some overhead to whatever technique you use to capture the stack. This means the benchmark results you get are almost certainly suspect while you’re profiling. When I’m using this technique I’m usually looking at a specific benchmark so I don’t record the results anyway.

I’d recommend only using this against a single benchmark, otherwise you’ll mix the profiles of several benchmarks together. In terms of feedback it’s a lot faster when you do it that was as well.

In terms of ease, however, this is second to none. If you’re writing java, you’ve already got jstack, and jmh and the FlameGraph scripts are both easy to use and cross platform.

These issues aside, the insight I’ve gotten from this profiling and visualization has been invaluable. I’ve gotten a 4x speedup (enough for my case, for now) and it’s given me a good idea of what to go after next.

So if you’re having perf issues, and you’re a bit too lazy to write benchmarks for all the things, I’d highly recommend it.

Clojure fspec surprise

Liam Falconer — Sat, 22 Jul 2017 23:03:00 +1300

Recently when writing Clojure I’ve been trying to cover everything I can in specs. This led me to a bit of a surprise when I used fspec. It evaluates the fspec’d function when it’s passed to a fdef’d function, not when you invoke the function. That’s probably super unclear, so look below the fold for an example.

So let’s say we’ve got a function, that’s specced:

(defn applier [f x y]
  (f x y))

(s/fdef applier
        :args (s/cat :f (s/fspec :args (s/cat :x int? :y int?)
                                 :ret int?)
                     :x int?
                     :y int?)
        :ret int?)

Obviously this example may be slightly contrived. So since we’re all conscientious and stuff, we run around with all our vars instrumented all the time. At some point, we put the following into the repl:

(applier (fn [x y]
           (prn x)
           x) 12345 3)

Surprisingly, this results in:

being printed. What’s happening here is that the function you pass in is getting checked to see if it matches the fspec. That was a little unexpected to me. Especially since in my case, my fspec’d function threw an exception in some cases, leading to a bit of a rabbit hole.

Which leads to one more bit of advice: “specs cover non-exceptional use”. There isn’t a throws spec. You want to write your :args specs such that they can’t cause exceptions, or use with-gen to make sure you don’t generate args that can cause exceptions to be thrown.

Note all of this is as of clojure.spec 0.1.123, and clojure 1.9.0-alpha17, and given those version numbers, likely to change.

Writing IntelliJ plugins

Liam Falconer — Sat, 27 May 2017 10:21:00 +1200

I’ve spent some time working on an IntelliJ plugin during our free time at work. There’s quite a bit of stuff I’ve learned from either reading source or messing with things until they worked, and I thought I’d record some of that.

You’ll need some knowledge of how IntelliJ plugins work, mostly about the Psi model and it’s API. If you want to learn about that, you should probably start with this or this. Anyway, tip #1 - PsiTrees are sensitive about what elements you put where.

The problem I was having was that after inserting an annotation I’d generated on a variable, the code analysis was freaking out about it being in the wrong place.

This was inside a quickfix, and the code in question looked something like this:

PsiElementFactory factory = JavaPsiFacade.getElementFactory(project);

PsiAnnotation annotation = factory.createAnnotationFromText(
  "@" + annotationToApply.getSubtypeFQN(),
  variableToAnnotate.getContainingFile());

if (variableToAnnotate != null) {
    variableToAnnotate.addBefore(annotation,
                                 variableToAnnotate.getTypeElement());
}

The code it was generating looked correct after it’d been output, but I was getting complaints that “annotations are not allowed here”.

Along with it not showing the inspection I’d fixed as being fixed. (You can see the quickfix is still available). I had a look at the PsiTree and it looked correct.

You can see the annotation element is there, inside the modifier list. There’s one thing that’s important to know about the Psi Viewer: It reparses the code in question, it doesn’t use the same PsiTree as the editor does for file you’re looking at.

Hopefully that should give away the issue: I was putting the annotation directly under the variable PsiElement, rather than inside a ModifierList. The fix was pretty simple, what I ended up doing was this:

PsiElementFactory factory = JavaPsiFacade.getElementFactory(project);

PsiAnnotation annotation = factory.createAnnotationFromText(
  "@" + annotationToApply.getSubtypeFQN(),
  variableToAnnotate.getContainingFile());

if (variableToAnnotate != null &&
    variableToAnnotate.getModifierList() != null) {
  variableToAnnotate.getModifierList().add(annotation);
}

Rather than just putting the annotation on the variable, I pull out the modifier list and put it on the end.

It seems like every variable has a modifier list regardless of if it actually has any modifiers, i.e. at worst there should be an empty list. So the null check there is probably a little overly defensive.

I’ve got a bunch more similar stories tucked away in my brain, so hopefully this can be a series that I might actually write semi-regularly, unlike the scattered posts I otherwise make…

Property testing tooling in Java

Liam Falconer — Sun, 07 May 2017 11:43:00 +1200

The NZ contingent of LMAX went to Codemania at the end of last week. And it was awesome, I always come out of there excited to make things. Then usually I give myself a hangover which puts an end to that. Anyway, the two themes we picked out from the conference were automate more, and property test all the things.

I’m already a huge fan of test.check for my Clojure code, and have been feeling some friction working with the tooling we had for property testing in Java. So I spent some time digging into the options we had, and this post is the result of that.

This post assumes you have some knowledge of what property based testing is, and how it works. I’m just going to compare and contrast Java libraries that enable it.

So we already had two libraries in our codebase for doing property testing: junit-quickcheck and quicktheories. I leaned slightly more towards quicktheories to start with, since it’s got a more functional API that I’m a bit of a sucker for.

Junit, and by extension, junit-quickcheck, doesn’t have a very functional api. But that’s actually a good thing when you’re writing Java. Coming from a Clojure background, the functional programming options available in Java feel a little clunky to be honest. Junit has been around for a while, and has an api that doesn’t really try to hide that. Which is fine! It’s easy to read and reason about in the context of the language it lives in.

The junit-quickcheck library pretty logically follows the path that junit laid out, using annotations and test methods that take arguments to do it’s thing:

@Property
public void simple(String s1, String s2)
{
    assertEquals(s1.length() + s2.length(), (s1 + s2).length());
}

The api for “I want to use this particular example” is pretty obvious:

@Test
public void example()
{
    simple("hello", "world!");
}

The thing I like most about this library is it’s simple and approachable for people who are at least a little familiar with Java and Junit. Even the way you write a generator is comparatively simple:

public class MatrixGenerator extends Generator<Matrix> {
    public MatrixGenerator() {
        super(Matrix.class);
    }

    @Override
    public Matrix generate(SourceOfRandomness sourceOfRandomness,
                           GenerationStatus generationStatus) {
        int width = sourceOfRandomness.nextInt(0, 1000);
        int height = sourceOfRandomness.nextInt(0, 1000);

        int[][] matrix = new int[width][height];

        for (int i = 0; i < width; i++) {
            for (int j = 0; j < height; j++) {
                matrix[i][j] = sourceOfRandomness.nextInt();
            }
        }

        return new Matrix(matrix);
    }
}

Yes, it’s verbose, but shrug it’s Java. Go figure.

So that should have given you a vague idea of how junit-quickcheck works, so I’m going to contrast that with quicktheories. The first example of checking string length looks like this:

@Test
public void simple() {
    qt().forAll(strings().allPossible().ofLengthBetween(0, 100),
                strings().allPossible().ofLengthBetween(0, 100))
        .checkAssert((s1, s2) ->
          assertEquals(s1.length() + s2.length(), (s1 + s2).length()));
}

Clearly, it’s a lot more functional, right? The generators are configured using builders, we can write our function as a lambda… but the test body is 4 times the size of the junit-quickcheck example, and the actual bit we care about, assertEquals... is a lot more… buried than in the junit-quickcheck example. This is a problem that gets worse, as you write more complex tests.

Another issue I have with quicktheories is that it doesn’t integrate amazingly well with the rest of Junit. The @Before annotation doesn’t work as expected, you have to explicitly call it in the middle of your assertion.

I guess this is the general problem with quicktheories. It’s just not Java-y enough, and that sticks out. The process for producing new generators suffers terribly from this in my opinion:

Source<Matrix> matrixSource =
  Source.of(
    integers().between(1, 100)
              .combine(
                integers().between(1, 100),
                //Can't size generator based on prior generators?
                lists().arrayListsOf(integers().all())
                       .ofSizeBetween(1, 100),
                (w, h, vals) -> {
                  int[][] arr = new int[w][h];
                  for (int i = 0; i < w; i++) {
                    for (int j = 0; j < h; j++) {
                      arr[i][j] = vals.get((i + j) % vals.size());
                    }
                  }
                  return new Matrix(arr);
                }));

Note this doesn’t include functionality to allow shrinking. Compare to the junit-quickcheck generator above. I think the junit-quickcheck generator is considerably easier to read, and you get shrinking out of the source of randomness, rather than having to supply it yourself.

There is one problem both these libraries share, in that they require you to specify all of the input you require to a test up front, either in the args to a @Property annotated method, or in the qt().forAll() call. I was hopeful that the “dark horse” entrant hypothesis would let me write tests that read a little better.

This was not the case. It’s not ready for prime time, which the author openly admits. It’s a real shame, since I the programming model looks very appealing.

So the conclusion I came to was that junit-quickcheck was the best option I could find at the moment. It integrates the best with Junit, and has a simple, idiomatic api. Quicktheories could be improved to make it more competitive, for example by making generators have sane defaults that are more succinct than the current examples. To a certain extent it’s hamstrung by the fact that it is just harder to use lambdas in Java than in more functional languages.

All the code that I posted here, and used to evaluate these libraries is available on GitHub.

Performance checks on metrics

Liam Falconer — Mon, 24 Apr 2017 21:31:22 +1200

At LMAX we do [a lot of automated testing] (https://www.symphonious.net/testing-at-lmax/), including running several dedicated environments for performance testing. We’ve got a lot of specific numbers that come out of these environments that we surface on dashboards like this:

But we also collect a bunch of other metrics from our servers and services that, until recently, we didn’t have much visibility over.

Rediscovering these lost numbers started with us deploying a service with debug logging we were using to investigate intermittency to production. This gave some of our web servers a hard time, since they were logging lots more on certain types of requests. It didn’t have a huge impact, but it was definitely something we noticed, and didn’t really want to repeat. Especially since it took longer than we would have liked to trace the performance problems back to the additional logging.

This is the kind of thing you can’t really catch with the approach we use to cover our really performance sensitive code paths, which is a combination of replaying events and JMH micro-benchmarking. This is not because you wouldn’t actually be able to catch a performance regression with these tools, but because the time and effort we would expend. Covering everything that might possibly cause a performance problem at some point in the future with a JMH benchmark would probably double our already significant number of tests. Not to mention that it would massively increase how long our tests take to run.

So we decided needed something to cover more ground. Turns out that we already had the data we needed, we just weren’t actually making decisions based on it. We collect a lot of information from all of our environments, for example one of our CI environments has around 600,000 metrics that have had numbers against them across all of it’s servers. There’s a ton of potentially useful information in there: GC pause times, open file handles, as well as more application specific stuff.

We do surface all of that in some pretty (basic) graphs: These do get used when we’re troubleshooting, and can generate alerts in our production monitoring. But we haven’t really exploited them fully for our testing cycle.

The open file handles we’d already used in a check on some of our reporting services, since we’d had issues with those not cleaning up after themselves. We thought we’d take that and riff on it a little for other numbers.

Our goal was to “catch regressions in unexpected places”, so we took a bunch of system and jvm level metrics we thought might be applicable: GC Pause times, running proc count, and load average. We picked these since they’re some of the more general performance indicators we could think of. That’s good since we’re looking to catch any and all regressions, rather than target anything application specific.

Essentially all we did was pick the average or maximum for these values out of our metric store, and compare it to a number in our checker. We set some overrides for specific values on specific services, since we expect some of our services like the exchange to use a lot more resources than say our web app servers.

Of course we wanted to prove this would have actually caught the problem that kicked all this off, so we ran these tests over the metrics we collected from the dodgy build to verify that it would have failed if they were in place. In that case, the load average check did fail due to all the additional string processing we were doing.

These checks are pretty crude at the moment. We’re hoping they’ll be effective in the long run, but if they aren’t, that’s fine. Crude measures can be refined, but things you aren’t looking at are never gonna be improved. Of course if we figure out at some point these checks are causing us more pain than they’re saving us, we’ll look at finding another solution.

We did try some mildly fancy stuff involving doing a regression over the data to check if we were leaking resources over time, but our acceptance style (involving hardware and topology pretty close to our production environments) perf test runs take on the order of half an hour, which isn’t really long enough to produce an alertable trend.

This process also delivered some value in another way. In the process of setting overrides for our services, we discovered that some of them were experiencing longer GC pauses than we would like. Again, these are off the path we have pretty well covered. None of them are at panic inducing levels, but some are definitely at moderate concern inducing levels, and something we’re digging into.

Hopefully this showed you a little bit of our process for dealing with issues we encounter. Not so much the actual finding and fixing part, but more the making sure it doesn’t happen again part. Also you should go take a look at some of the numbers you’re collecting from your testing environments, or if you’re not collecting any, start. Since if you aren’t measuring a thing, it’s near impossible to improve it.

Coupling, Connascence, CSP and Actors

Liam Falconer — Mon, 03 Oct 2016 23:10:00 +1300

In the last 6 months, I’ve been lucky enough to be exposed to two very different ways of dealing with asynchrony in one of our production systems at LiveOps Cloud. We converted one of our core backend services from Clojure’s core.async to Pulsar. Having been a part of that transformation, I now, of course, have opinions that I’m going to subject you to. Those opinions boil down to CSP (Communicating Sequential Processes) as implemented in core.async is better than the actor model, at least as implemented in Pulsar.

A crash course on Connascence

First of all, a little description about what connascence is, since as far as I know it’s a relatively uncommon term. It’s used to describe the degree of dependency between components in a OO (object oriented) system. Two components are connascent if a change to one would require a change to another. We can also talk about the strength of a connascence, where stronger connascences imply the change is more difficult to apply. There is more detail on wikipedia, which I recommend you read if you want to follow along.

I’ll give an example of two connascences first, however:

(defn validate [data]
  (and (= (:type data) :person)
       (:name data)
       (:age data)))

(defn has-parent? [data]
  (and (:name data)
       (:parent data)
       (= (:type data) :person)))

This code is intentionally pretty bad, but you can imagine the case where this might be spread over multiple code bases somehow. We can describe how bad it is as a connascence of algorithm: both validate and has-parent? depend on similar logic. This can be relatively trivially fixed:

(defn is-person? [data]
  (and (= (:type data) :person)
       (:name data)))

(defn validate [data]
  (and (is-person? data)
       (:age data)))

(defn has-parent? [data]
  (and (is-person? data)
       (:parent data))

We’ve reduced what was previously a connascence of algorithm to a connascence of name - they both depend on the same extracted function. Hopefully it makes sense that we prefer weaker forms of connascence, and less of it where possible.

Actors vs CSP

I’ve got a few reasons for preferring CSP style to actor style. One of the big ones is that actors have introduced pervasive, name based dependencies throughout our code base, via the actor registry mechanism. Normally connascence of name isn’t really a problem, but I would argue in the case of actors it is, since these names are both common and smeared across the code base. This also introduces a dependence on the actor registry itself.

This means that our registry is global state, that all our actors depend on - a connascence of identity. I don’t think this is necessarily a bad thing, after all a database is a giant blob of global state, and ‘using a database’ has been a pervasive and effective idea in software engineering. The problem here is the semantics of the ‘database’. Actor registries intentionally prevent registering multiple actors under the same name. This isn’t a bad thing, until you want to write tests using actors, since you now have to write setup and tear down code for each test. This is not a fun experience.

The mechanism of communication in core.async is the channel, which doesn’t have the same global scope as an actor registration (you can of course make them have global scope, but I’ll talk about that a bit later). You don’t even have to close channels once you’ve completed your tests, as typically go blocks won’t continue executing without you placing additional work on the channel that has now fallen out of scope. Of course some memory will be consumed by the parked blocks, but usually this isn’t a problem.

Another issue that arises during testing is the promiscuous communication that happens between actors. Since any actor can invoke behaviour on another just by looking it up in the registry and sending it a message, you typically have to do a lot of actor mocking. This is that connascence of name again. Macros can help with the mocking, but it’s still a lot more verbose than supplying channels to a function and pulling or pushing results to them. You could claim that it’s easier for actors to communicate actors when you don’t have to pass around channels, however if you really want to (and it’s not something I’d normally recommend) you can simply place a channel in a def or even a derefable data structure (probably an atom or a promise) and get access to it where you need, but with a clearer link between it and it’s producers or consumers.

This is a recurring pattern I’ve found - it’s a lot easier to implement actor’s functionality with core.async than the other way around. For instance a registry can be easily implemented with compare-and-swap! on a global atom containing a map. You can express a more options here when you encounter an existing registration, such as transparently using the one in place, replacing it, or going the Pulsar route of throwing an exception. The important thing here is that you have a choice about it. For a case of a feature that’s more difficult to implement in Pulsar, there is customized queues. In Pulsar you have to implement an entire Actor, and some sort of feedback mechanism from consumer to queue (coupling the consumer to it’s queue), otherwise it will just saturate the consumer’s mailbox, rather than retaining the items in the queue. In core.async, all one has to do is reify the take and put protocols, which can be done trivially by closing over a channel or two. This queue can be used anywhere, rather than just where a consumer is designed for it.

There are stylistic differences, inherited from the linage of these two systems that introduce their own connascences. Pulsar of course follows the example of Erlang, a language where pattern matching on tuples is pervasive. While potentially powerful, it introduces a reliance on the ordering of parameters that is a stronger connascence than the idiomatic way of conveying data in Clojure. Normally if you want to pass a message around in Clojure, you will use a map, where lookup is done by name rather than by position. This isn’t a hard or fast rule, and it’s not normally something that really concerns me to be honest. Again, normally a connascence of position isn’t a big deal - every function call has it to some degree, but the fact it exists among supposedly decoupled actors is what I find troubling.

Too many masks

On the topic of coupling, there is the actor itself. Personally I think actors have conflated several different concepts together, state being the most egregious one. Which of course I would say, being a Clojure programmer. What I mean by that is that whenever you’re working with actors, state is right there, ready for you to reach out and grab it. One thing that makes me very happy about Clojure is that you have to jump through a few hoops to get mutability, and in doing so, you have to think about it. Being forced to have that moment is a good thing.

What else is in an actor? A queue, in the form of a mailbox. Actor systems are designed to be ‘fire and forget’. This is great in terms of decoupling, at least for the senders. Pulsars implementation, however, is a little less inspiring. In some trials, we’ve had messages in a mailbox disappear. There’s also the problem of dead actors disappearing from the registry. Since looking up an actor is a blocking operation, in practice it isn’t actually fire and forget. Contrast this with core.async, where a ‘actor’ (go-loop) and it’s channels are distinct entities. At most, you’ll be able to disappear the item a go-loop is currently processing if you manage to blow it up.

Another property that actors provide is fault isolation. This isn’t a feature that’s missing from other programming languages. The difference in actor based systems is it’s pervasive nature - you can’t not have it. That is definitely a good thing. Actors aren’t the only way to get it, of course - Java threads and core.async go blocks are both perfectly capable of exploding without affecting things around them, but you have to choose to apply these, rather than getting it by default. I would say that this and the final property are the two important parts of actors, and they are cohesive, rather than coupled.

So finally, there’s the ability to supervise other actors. I would argue this is probably the most useful part of the actor model, the firm separation of the ‘happy path’ from the sad path, allowing us to write our “business logic” without a care (or try-catch) in the world. This is the one aspect of actor systems I think should be exported to the wider programming world. At least in Clojure, it has been, in the form of dire. There’s some blog posts linked there that explain better than I can why this is such a good thing. Between this and go blocks, I would argue that CSP can provide the most important benefits of actor systems, without what I view as the costs.

That is not to say I’m going to give up entirely on actors, however. I would like to take a closer look at Erlang or Elixir, and I’m quite interested in what the Elixir community produces in terms of patterns and architecture. The pure and ideal form of actors is appealing, and certainly offers a more perfect model than other OO languages. That said, we probably will not be using Pulsar again, or attempting to port another system. Most of the benefit we gained was in terms of a cleaner code base, which I would argue was in spite of Pulsar, rather than due to it. A concerted clean up effort could have had much the same effect as the rewrite we attempted, taken less time, and delivered more value. Next time we’ll just have to do it in Elixir.

Dependency Injection in Clojure

Liam Falconer — Mon, 19 Sep 2016 21:57:44 +1200

So this is the follow up to a follow up. I’ve been writing Clojure more or less professionally for about a year now, and I just re-read my old thoughts on Dependency Injection. Once again, it having been more than 6 months, my opinions have changed. The reasons I used dependency injection in C# don’t affect me nearly as badly in Clojure, so the amount I use it has vastly decreased, and the way and reasons I use it have changed.

To recap the previous post, there were three main reasons I advocated using dependency injection (DI) in mainstream OO languages:

Made it easier (or in some cases, possible) to write unit tests
Enforced more separation of concerns in design
Allowed development without external dependencies

The first reason is still somewhat valid in Clojure, but to a much lesser extent. The last two, not so much. There is one facet of DI that is widely used in Clojure - life-cycle management, which I will discuss at the end.

So the initial reason I adopted DI was to make it easier to write tests. I didn’t realise this at the time, but this is because it introduces seams into a program. Seams are places where you can swap out behavior without affecting the rest of the program. Consciously inserting seams isn’t required nearly as much in Clojure, since with-redefs allows you to use every single function as a seam. Since most of your program should be functions, this makes it trivial to mock out side-effecting or non-deterministic operations. For example:

(defn slurp-and-split [file-name]
  (->> (slurp file-name) ;Read in file as a string
       (str/split-lines)
       (map str/trim)))

(deftest slurp-and-split-splits
  (with-redefs [slurp (constantly "1
                                   2")]
    (is (= (list "1" "2") (slurp-and-split "")))))

As you can see using with-redefs is also much, much more simple and succinct than any other mocking or stubbing library I’ve dealt with, but that’s mostly since Clojure has stuff like constantly.

I think it’s worth mentioning, I can’t really see a good way of making the above function more amenable to DI. Passing in the slurp function is possible, but then I want to wrap that up inside another function in any case. If we were using a database or something similar it would make more sense:

(defn get-user [db user-id]
  (query db "SELECT name, email FROM users WHERE id = ?" user-id))

But this function itself would become the target of mocking using with-redefs, presuming query returns a map or vector. One issue with this function is that we have to manually thread db through to the call sites. I know from experience that can be painful, it’s very similar to the problems you have in OO code bases with threading values through constructors and objects that don’t need them. But by using some other Clojure libraries I mention at the end, this could be rewritten to omit the db parameter, eliminating that issue.

That aside, we still don’t have to inject test seams, which alleviates a lot of design burden. You don’t have to inflict ‘test induced design damage’ on your code base. I used to view the hyper-abstracted, hollowed out designs that resulted from widespread use of DI as good, but now I just see this as needlessly complex. In Clojure, the simple act of extracting functions creates a lot of room for adding tests. As a functional language Clojure has great tools for putting functions together, which makes chopping up and recombining your code relatively painless, even without editor support.

This means that separation of concerns is a lot easier to achieve in Clojure, since it’s easier to pull apart functions. Part of this is also that functions aren’t coupled to the data they operate on by a class definition - you just have the arguments they’re passed rather than some ambient state. You still have to be conscious of functions that involve external resources but this is the case in OO languages as well. Ideally these functions can be identified by the namespace they live in, because it has something scary like the name of a database in it.

Development without those kinds of external deps is also a lot easier. A well designed Clojure program is, again, mostly functions. And definitely the ‘business logic’ should be almost entirely pure functions - you should be trying to push all the side-effects to the edges of your system. So hopefully (as long as you have some sample data to work with) it should be trivial for you to try out new code in the REPL without having to rely on the presence of databases and the like.

There is one place where something of DI is still of value in Clojure, and that’s when dealing with stuff that has a life-cycle. Most good DI libs try to deal with this as well. Clojure has two well known solutions to life-cycle management, Component and Mount. They’re both much, much more simple than the OO options, probably since they aren’t complected with other requirements. They don’t tackle of problems that OO DI libraries have to, since many of these problems are alleviated by providing first class functions (That’s an entire other post though).

Since the only problem these Clojure libraries have to solve is ‘start these things’ they boil down to ’tell me about all of the things’. The two libraries take very different approaches to this, Component using an API that was more familiar to me, coming from OO, while Mount’s usage feels to me as if it embraces the design of Clojure more. Other people have written more about the differences between these libraries. Personally I don’t have strong opinions about which one is better (or even if one of them is better), they both solve the problem and it’s great we have good options in this area.

Looping back around to our get-user function from the beginning, I’ll demonstrate how we would eliminate the db parameter using mount (there’s less code involved than component, so of course it’s more impressive.)

(defstate db :start (create-db-connection connection-string))

(defn get-user [user-id]
  (query db "SELECT name, email FROM users WHERE id = ?" user-id))

If you invoke (mount.core/start) somewhere near your -main function, this will result in the :start function in all the defstate mount can find being called (There’s a bit more to it than that, of course, you’ll want to see the readme to actually get started using it).

Doing something like this does make the db related functions difficult to unit test. In my opinion, that’s perfectly ok. Integration tests should be what you’re using to validate your database interaction is working correctly, and they don’t need to be exhaustive. Unless significant portions of your applications logic live in your database, and then I’d argue you’ve got a bigger problem.

I’m basically out of DI related things to talk about for now, so to wrap up: Most aspects of DI aren’t amazingly useful in Clojure. It can still be used to make testing easier, but isn’t really necessary. Using lots of simple functions gives you plenty of seams to exploit. Life-cycle management is still important, and Clojure has good libraries for dealing with it that are much more painless to use than the OO solutions I’ve seen.

Leaving the M$alt Mines

Liam Falconer — Thu, 08 Oct 2015 22:38:13 +1300

So a little while ago (ok, a couple of months ago now.) I quit my job at Ubquity in favour of one at LiveOps. There were a bunch of reasons for that, but you’ll have to buy me a beer to get those out of me. The main draw though, was being able to work with Clojure instead of C# (Hence the punny title).

There’s a lot of posts about why and how companies moved to Clojure but not so many about individuals - I guess we don’t really make good marketing pieces? (I’m going to have to eat my hat on this one: Owen Rees-Hayward totally beat me to the punch.)

So I’m going to tell you a little about why I made the jump to Clojure.

One of the main reasons I want to use Clojure is selfish - there are terrific productivity gains promised with any lisp, along with “magic mind expanding powers”. By that I mean the claim that it’ll make you think about solving problems differently. I’m not certain how much that’s happened yet. I know I do code differently, but it’s difficult to separate thought patterns from the languages that they’re encoded in.

But the point is, anything that makes me a better programmer, or at the very least broadens my skill set, is something I want to try. I’ve already done that, of course, but there’s a big degree of difference between using something for, well, fun basically, and using it in anger. Spending 8 hours a day doing something is probably a pretty good way to get used to wielding it. I already feel just as productive as I was after two years working in C# - I’ve got a lot of theories about why that is, but that’s something I’ll save for a later post.

So I’ve used Clojure in the past, but it’s always been dabbling rather than full on commitment - I’d mess with something once every couple of months and then drift off again and forget everything just in time to try and pick it all up again. Clojure wasn’t the only programming language I had repeated flings with - I’ve been bellyaching about my choice of programming language for a while. C# is too verbose, corporate and bland, Go doesn’t have enough abstractive power, F# is undersupported (I mean both in the sense of it’s corporate master and it’s not so super array of libraries), Rust involves too much compiler fighting, ditto Haskell. Yeah. I’m fussy. Even Clojure makes compromises, but at least the choices it makes I can understand (and agree with) the reasoning behind.

Case in point, basing off the JVM. You trade startup time and memory use for a huge ecosystem that’s been built up over many, many years, and a stellar virtual machine. (Yes, that’s other people’s words, I’m not enough of an expert on the JVM to trust my own commentary.) To me, that seems like a pretty good deal. That said, I’m massively hopeful that Clojurescript will prove viable as a serverside lanuguage under node, since it doesn’t have the memory footprint or the startup time of the JVM hosted version of Clojure. (This is probably more related to the agressive optimizations that the Closure compiler does, which incurs a hit at compile time as a result.)

That’s another thing that’s worth mentioning. Don’t read this as me bashing on Javascript, because I’d probably pick it as my favourite of any C styled langauge, for a bunch of reasons I won’t get into here. It has it’s warts (if you haven’t seen it yet Gary Bernhardt’s “Javascript Wat” talk is a great way to witness a few of them in a few minutes.) and when you throw in the general terribleness of a few implementations, it becomes a little unpalatable. Thankfully, there’s clojurescript.

Being able to write your front and back end code in the same language with the same semantics is a godsend. I don’t get to do a lot of it at work, but not having to make a language context switch every time something needs to get tweaked on the front end is amazing, and having exactly the same data structures shared between the client and the server is really useful. These a whole bunch of pretty amazing libraries and tooling, like figwheel, devcards, and om.

So now it’s probably worth discussing the elephant in the room. You know, what normally drives people away from lisps. The parens (also known by their proper name, the parenthesis. But that makes my brain kinda go thunk every time I write it.). All of the parens. Yeah, Clojure has that ‘problem’. There’s ways to mitigate it (the -> and -» threading macros, or as I like to call them, the spear and pointy spear.) but you’ll still see more ()’s than in say C# or Java.

On the other hand, C# and Java have their own sea of brackets problems, it’s just that they tend to be curlier than in Clojure. You also get some chunks in your bracket soup in the form of a ton of keywords. The average Clojure namespace (probably the closest equivalent to a class, and they do compile down to a class, but not as you know it.) doesn’t have half the amount of stuff that litters your average java/C#/blub class.

But if you’ve been working in a language for any length of time, you’ve probably stopped conciously noticing a whole bunch of things - you don’t have to actually look for stuff like the class keyword or the indications that something is a method - you only actually see the names and just kind of know about the surrounding fluff.

The same thing essentially happens in clojure. It’s only occasionally that you actually need to go hunting for a stray paren. Rainbow parens and a little bit of bracket match-y highlight-y stuff makes that go away. Or makes the parens you’ve got too many of go away. There’s other tricks as well - Structual editing (aka paredit) seems promising, but also kinda scary. My first encounter with it reminded me of the first time I used vim - mostly an experience of “what the fcsk is going on”. That’s something I’m going to try and fix, however.

Ok, so that being dealt with, there’s one more big draw to clojure (in my opinion, of course. There’s lots of other reasons, but I’m just trying to hit the high notes here.) and it’s not so much about the language itself as the ecosystem around it, the way it was created, and how that affects the Clojure community. First of all, Rich Hickey gives a lot of very good talks, (you should check them out. It’ll make your brain think all different and stuff) and uses a lot of interesting vocabulary. I don’t think I’ve ever heard the term complect uttered by someone who isn’t at least a little bit of a functional programming nerd.

It’s not the fancy choice of words that really matters, but the stuff behind that - the thinking, and the amount of it that gets done. In spite of the REPL, and the opportunity that gives you to really quickly iterate on your code, the ‘best practice’ for a working clojurist is to think a lot about problems before you tackle them - Hammock Driven Development (as coined by, you guessed it, Rich.)

It’s something that shows elsewhere - in my admittedly thus far limited interaction with the wider Clojure community, I’ve found people to be a lot more thoughtful and considerate than I’ve encountered in other communities. That might just be a side-effect of the relatively small size of the community, but I’d like to hope it’s something that will persist as more people come into the fold. It’s helped make me a lot more comfortable with saying “I don’t know” than I was in the past. That probably has something do to with me knowing the response won’t be ridicule. And I think it’s something that’s rubbed off on me a little, which is great. So much of coding is about how you interact with other people, not just in person but through what you produce. Having a little compassion for the people who are going to be suffering through your code later (even if it is going to be you) can only be a good thing.

I know the “Clojure journey” is only really just beginning (doing it for work is completely different to doing it for fun, but I’ve already said that), but so far it’s all promising. It’s different to pretty much any other programming language I’ve worked with, which is making it both interesting and challenging. There’s a pragmatic, considered design that infects anything that touches it. In a good way, mind you. It’s gotten into the community, who in my dealings and observations (read: lurking!) are all nice in a way that’s pretty uncharateristic of the internet. So yeah. I think I’ll be happy here, hopefully for quite some time to come.

Error Handling in ASP.NET MVC

Liam Falconer — Sun, 08 Mar 2015 23:06:00 +1300

So today we’re gonna talk about how to make sure you don’t show your users something like this:

And then making sure that you know that they would have been shown that, and hopefully give you some more infomation to boot. Originally this post was going to be a bit of a monster, but I’ve chosen to chop up the hydra a little bit.

This post just covers how to deal with making IIS catch all the errors that might not get caught inside of your MVC application. I’ll dig into the levels closer to MVC in the next post, and then handling error logging in another one after that. They’re both already semi-written, so they should follow along soonish. If you aren’t really too worried about the reasoning, and trust me enough to just want to get to the money, I’ll have a quick summary “just do this” post up after the long version is finished.

So. Let’s cover the territory of possible ways to show custom error pages to your users, because this being something from Microsoft, there’s at least six ways to do a thing, and none of them are exactly what you want. Well, ok. There’s only like 4 that I know of:

You can use an exception handling filter, like the HandleErrorAttribute that’s included by default, or you can use a module (basically the same idea as a filter, but further up the chain.), the most promienent example of which would probably be ELMAH. Further torwards the edges of MVC, you can turn to the customErrors thing in your web.config, which as far as I can tell is implemented with a module as well, or finally you can look to the httpErrors option, which lives inside of IIS rather than MVC, but is also in your web.config.

Basically I’m of the opinion you should always pick #4, the IIS option, since it’s the highest level of coverage. Anything that goes wrong with all of the other options will end up with whatever is in the httpErrors tag getting shown.

You can mix and match any of the other ones along with that to suit your preference, I usually also use the customErrors option, since this is the point most outside of the MVC pipeline but still inside it you can catch exceptions, so it gives you the least exposure to potential issues with your MVC site. At least that’s my reasoning, you may come up with a different calculus.

The reason I don’t use filters is because an error inside of your filter config or somewhere inside your global.asax or whatever will cause them to fail completely. The config option does leave you open to issues in your web.config, but if that’s hosed you have to rely on your base IIS config. Which hopefully won’t show anything ugly to your users. You should probably check that.

The other issue with filters is they won’t fire for what I call the “deathmurder exceptions” - stuff like stack overflows and out of memory exceptions that just totally kill your application. The stuff you define under that httpErrors tag is, once again, what will save your ass.

So let’s cover that first.

So in order to deal with this, you have to add another section to your web.config, under the system.webserver tag:

<httpErrors errorMode="Custom">
  <remove statusCode="404"/>
  <error statusCode="404" path="/Views/Errors/404.html" responseMode="File" />
  <remove statusCode="500"/>
  <error statusCode="500" path="/Views/Errors/500.html" responseMode="File" />
</httpErrors>

So what’s going on here is we’re telling IIS to use pretty error pages using the errorMode="Custom" attribute. You can set that to DetailedLocalOnly for general use, but while we’re messing with this stuff, we want to make sure you’ll actually see the custom error pages, so leave it as Custom for the moment.

The rest of it is basically removing the default IIS error pages (otherwise it’ll spew at you when you start up the app) and replacing them with our own static HTML files. You can use .aspx files here, but if you get to this point, something has probably gone terribly wrong inside your app, so I would try and avoid relying on any code actually doing anything.

There are a couple of additonal attributes you might be considering adding, like defaultPath and defaultResponseMode however defaultPath seems to cause IIS express to throw it’s own exception when running on my machine which would seem to make defaultResponseMode somewhat pointless. But if you manage to get it working, please let me know @lfln3.

There is another option if you want to use some custom code, where you can change the responseMode to ExecuteUrl and then point the path to an .aspx file. This does mean it will rewrite the response code to a 200, however, which is probably not what you want. To cover that, you can add this snippet to the top of your .aspx file.

<%@ Page Language="C#" %>
<% Response.StatusCode = 404; %>

Ideally there’d be some way of jacking the error status code when getting directed from the custom error definition, but I haven’t figured out a way of doing that yet, or if it’s even possible. If anyone’s got any ideas, once again hit me up at @lfln3.

I’d not hugely comfortable personally with having code execute in my error handlers, but if you’re ok with it, go nuts. In any case, this should basically iron-clad your app against the possiblity of showing any ugly yellow pages to your users.

I have created a nuget package to speed this thing up a little, which you can find here, or just run Install-Package MVCErrorPages. It does include the MVC customErrors stuff, which I’m gonna cover in a later installment. But in the meantime, google should be able to help you out. Until then… good luck, I guess?

Fresh Thoughts on Dependency Injection

Liam Falconer — Sun, 15 Feb 2015 12:10:22 +1300

This is essentially a follow up to an earlier post Dependency Injection - A necessary evil?, from back when I had more reservations about using DI, and the benefits it gave me. I’ve since been fully converted to a ‘believer’. This post is bascially about why and how I think that happened.

I’m still toiling in the C# mines, but I’ve moved from using Ninject to a more… static library, Autofac. Ninject suffers from a little bit of stringly typing, while Autofac uses stuff like lambdas and generics to make errors happen at compile time rather than run time, which I’m a huge fan of. There’s heaps of other neat stuff too, so if you’re using C#, it’s almost certainly worth checking out if you haven’t already used it.

So the reason I was initally led to DI was due to the need to get tests across a fragile part of a long running project. This was basically a calendar you could making bookings on (the process of making a booking was fairly time consuming - the booking form has something like 30 fields on it or something.), and depending on what options you picked, it would block out more or less time around the date you chose. There were a whole bunch of other rules that could also apply, just to make it more fun.

In any case, it was painful to manually test, and we ran into some weird bugs around public holidays and that sort of thing that we very luckily caught before they made it into production. It took that to get management to sign off on me adding tests to this area of the code. Since doing that, it’s been (unsurprisingly) bug free.

This experience was enough to sell me on the complexity trade offs of DI. At this point I still haven’t fully embraced TDD or anything - I tend to mostly apply tests to areas I’m nervous about rather than everything. Not that I would have the time to put them on everything at my day job in any case.

The other thing that I came to realize was that DI really does enforce cleaner separation between your code - you’re constantly asking if this bit of code really belongs in your controllers (and the answer is usually no.) so tons of stuff gets pushed further out, where it’s easier to abstract and extract for reuse. Presently I’m in the process of using T4 templates to generate a whole bunch of boilerplate I was writing for a lot of projects at work, something I probably never would have realised was an option without using DI.

Being able to draw clean lines around parts of your system also means that you can reason about them in isolation. Anything that means you don’t have to juggle a whole bunch of infomation at once is great, and by splitting everything up for the purposes of injection it becomes much more explict where exactly any bad data is coming from, as long as you let your injected objects maintain responsiblity for their output. That is, output from anything injected should be modified as little as possible - project it into a new form, rather than mutating it’s existing one if necessary.

And since you’ve got these independent objects, if you can’t pin down what exactly is causing an error, it’s a lot easier to write code to help you find out. Put the code into a test harness, through a whole bunch of stuff at it, and make sure your invariants hold. Just make sure you print out the inputs that cause a faliure when an assertion fails.

The last benefit I got is being able to use my test objects for development. I’m not sure if this is something that anyone ever touts, but whenever someone needs to make changes to a database, or some service you’re dependent on isn’t working, I fall back on the objects I’ve written for my tests. They’re also great early on in development when you’re not yet worried about integrating with actual data or services - you just want to prove the UI/UX works. Obviously you have to eventually switch back to the real thing, but avoiding that block, and hopefully maintaining flow is certainly worth the distance from reality.

So I guess the main reason I embraced DI was because of the confidence that unit tests gave me - but once I was there, it turns out there’s other tangible benefits you get - a better awareness of where and why to decouple your code, an easier debugging experience as a result of that. Once you’ve got the test objects for testing, they’re also perfect for when something external would otherwise stop you from making forward progress. I’m sure I’m preaching to the choir here, but if you’re working in a mainstream OO language, and you aren’t using dependency injection, you’re missing out.

Moving to Hugo and Lanyon

Liam Falconer — Sat, 14 Feb 2015 18:52:49 +1300

So if you’ve been here before (highly unlikely, I know), you might have noticed this place might have previously looked almost entirely different. There’s reasons for that. I got somewhat frustrated with my existing cobbled together solution of python scripts, so I had a bit of a look around and found this nice new-ish project (in Go, which is an important factor for me, I like to be able to stare into the guts of the things I use.) called Hugo. It seems to support pretty much everything I want it to, which is super. And it’s also relatively simple in terms of what it expects from the user, which is also great.

Even better, a whole bunch of people have already made some pretty themes for it. Like this one, Lanyon. I’ve messed with it a little (I’m not a huge fan of pagination, amongst other things.), and I still want to add that lightswitch I mentioned when I first wrote anything here, but hopefully Hugo’s ease of use will mean I actually write some stuff in future, rather than despair at the code I’d need to write to get my scripts to do a thing and then go do something else instead.

Dependency Injection - A necessary evil?

Liam Falconer — Thu, 18 Sep 2014 07:27:25 +1200

Basically my position on DI has since changed - go take a look at Fresh Thoughts on Dependency Injection

So first of all, to preface this and so you get a little bit of insight from where I’m coming from, I spend most of my days toiling in the great C# mine, so lovingly provided by Microsoft. I used to be a python guy, and I’ve gotten dangerous with Go, and am trying to do so with Clojure. The clojure thing is recent, so this might be a bit more cargo culty than I’d like. Anyway.

So the shop where I currently get paid is a little behind the times. Legacy code base (in the Micheal Feathers “Legacy code is code without tests” sense) and an enterprise-y monolithic feel. There’s some seemingly arbitrary layering and attempts to divide responsibility, but (to me at least) they seem to all follow their own internal logic rather than any cohesive structure.

Luckily, I’ve managed to largely dodge that particular bullet, and mostly work on stuff external to the mothership.

One of the things I’ve been trying to adopt is unit testing (or just testing in general). It’s proven to be far more difficult to implement in ASP.net MVC than I was expecting (hoping?). One of the tools you are essentially required to lean on (which is not entirely a bad thing, admittedly.) is dependency injection (hereafter referred to as DI), which, as the title implies, I view as almost as much of a pain as it is useful.

It does, very usefully, impose a need to split apart unrelated functionality. There seems to be a very real love of gigantic libaries, sprouting tendrils to handle every possible eventuality, in the C# world. I’ve seen my co-workers build them, repeatedly, fuelled by some sort of steriodially inflamed Not Invented Here syndrome. By using DI you stand a far better chance of producing something that’s at least slightly reusable.

I can sympathize with their distrust of DI a little, at least. DHH has recently spoken of “test induced design damage” and I think it’s certainly possible (ok, I know it’s possible, I’ve done it.) to contort your code into strange and impossible shapes as a result of desire to test everything that’s not nailed down.

The primary thing that spooks me about DI is the explosion of complexity necessary in its usage. Layer upon layer of factories, facades and interfaces (Seriously. So many interfaces.) follow in it’s wake, each seemingly necessary, but all clouding the actual purpose of your software in arcane scribblings. And for what? Sure, you might be able to test it, but if it takes you half an hour to trace through the code to figure out that an error found in production was caused by something 6 layers deep, and would have been found by a test that would only fail very intermittently…

I guess that you have to believe that the safety net given to you by the tests is worth the time it takes to write them and maintain them and the additional complexity that DI causes. Not to mention the cost of refactoring an existing system (presumably without the benefit of tests to make sure you didn’t break anything. Or at least much.) I don’t know that I’m at that point. Yet.

Fun with Icon Fonts

Liam Falconer — Tue, 12 Mar 2013 18:32:40 +1200

So I spent a good chunk of today messing around with making a few icons for use on this site. Which was… interesting. There isn’t really a heck of a lot of information out there about how to do this, or at least nothing that google turned up. I managed to cobble something together though. And now hopefully reading this will save you all that effort.

First of all, why would you want to do this anyway? Well personally, I like the fact that I stop serving as many images, save on bandwidth, and get all the css and html text attributes for free. Basically the only drawback as far as I can see is the extra work involved in producing vector art and the messing around with turning it into a font. The only other issue is that your icon has to be mono coloured. You can’t have some multicoloured Bob Ross shit. If you want to do something like that, you’ll have to use actual images.

Got some pretty picture in your head you want to make into reality? Read on.

There’s two tools I ended up using: Inkscape and FontForge, both of which are free and cross platform. Inkscape includes some basic tools that allow you to produce a SVG font, however that isn’t amazingly useful in order to actually produce something the people looking at your site can see. I then used FontForge to edit the attributes, make sure I didn’t completely mangle everything and export all the formats I wanted.

If you’re new to vector imaging tools, InkScape is probably going to be kinda confusing. I might throw together a quick tutorial on it later, however until then, try taking a look round here, and the official docs.

Ok. So you’ve opened up Inkscape, and you’re looking at a blank page. Probably the wrong page. Hit File -> New -> fontforge_glyph. The devs of inkscape are ever so thoughtful, aren’t they? This will give you the 1000px by 1000px canvas you need. That line blue about 3/4 of the way down the canvas is the baseline, or where most fonts put the bottom of their characters. You’ll want to treat that as the bottom of whatever you’ve chosen to make.

At this point you’ll probably want to pull open the sidebar, since the Fill and Stroke pane is really, really useful If you head to Object -> Fill and Stroke, you should get something like this: - Also while we’re at it, the SVG Font Editor Pane will also see some heavy use. That can be found under Text -> SVG Font Editor

So now comes the fun part. Draw something pretty! Keep in mind the colour constraint. Also be aware that you’re basically going to flatten the image later, so anything you do with layering will also be ignored. I’ve chosen to excercise my artistic talent with this beautiful ball… thing. Note that the redness of the outline and the fill will be lost when we turn this into a glyph. You’ll see that in a moment.

The font specific stuff can be found under the Text menu. You’ll want to open up the SVG Font Editor Pane if you haven’t already.

If you haven’t already made one, you’ll need to make a new font in the Font Editor pane. You can ignore the name, (we can change it later in fontforge) but not the width. Make sure to set that to 1000. I forgot this time, and it will make fontforge angry at you later.

Head over to Glyphs, and add a new glyph. Name it something informative by double clicking on the name and changing it to whatever. Do the same for the matching string.

Again, this can be altered later in FontForge (and will be in the case of the string). Then select your path in the main window, and hit Get curves from selection. This should set the chosen glyph to use your current path. If you type the matching string into the Sample Text field it should show up just above it.

Ok. So at this point we can skip out of inkscape and make a font with our one glyph. If you want to do that, skip ahead a bit. There are a few more little things that you should learn with regard to inkscape however: Layers, Squishing and Converting paths.

If you want to add more glyphs (which you should. One glyph per font will drive you insane with the CSS and be horrifically inefficent.), you’ll probably want to use layers to hide the current path. Open up the Layers Pane: You can just hide and lock the current layer, but you’ll probably want to name it something vaguely informative first. Usually I use the same name as in the SVG Font Editor bit.

What ever you do, don’t delete it. You’ll tempt Murphy and his law - if later on you have the wrong glyph selected when you hit Get Curves from Selection, it’ll be overwritten. And no, it doesn’t ask for confirmation or anything like that. As far as I know, you can’t just use undo on it. Yeah. I made that mistake. Only the once. (Ok, like four times.)

So anyway, lets go ahead and create a new layer and deal with Path Conversion.

This time, I’ve used the pen tool to make some fancy curve. You can adjust the thickness of the stroke and whatever in the Fill and Stroke Pane. So let’s add another glyph to our font in the SVG Font Pane, and hit the magic button, Get Curves from Selection… and we (ok, I) screwed it up. Not only did I overwrite the beauty of my circle, but also the glyph looks nothing like what I expected it to. Ok.

Fixing the glyph overwriting is easy thanks to layers, but now there’s still the issue of the malformed glyph over there. First a slight detour to the aforementioned Stroke and Fill Pane to make my curve more curvy, then this is where the whole path conversion thing comes in. If we just hit Path -> Stroke to Path and then Get Curves from Selection we should get exactly what we’re looking for:

The last thing that might be important if you’re making something complex (like the globe logo you’ve seen plastered all over this website.) is what I crudely termed Squishing paths. I don’t have any images for this, but basically you can use the tools under the Path menu (Union, Difference, Intersection etc.) to combine multiple paths into one for conversion into a glyph.

Combine and Break Apart look tempting, however as far as I know, they only work on objects, and therefore aren’t that useful to us “font-artists”.

So, now you can go ahead and save a copy of the SVG, and get cracking in fontforge.

Presuming you managed to open up the SVG in FontForge, you’ll be presented with a grid of your glyphs, hopefully. So. Now we have to assign all of the glyphs unicode code points. I recommend using the ones from the Private Use Areas, since they won’t get picked up by screen readers or anything, and it seems like the logical place to put whatever weird logo you’ve come up with. Since we’re defining our own font, you could use whatever value you want, however.

So, right click on one of your glyphs, then hit Glyph Info. You’ll get a window where you can give the glyph a name (Pick something logical. You’ll probably want to keep this consistent if you do the whole CSS @font-face and i tag thing.) and set the unicode value for the character. Basically, do that for all your glyphs.

Once you’ve got that sussed, hit Element -> Font Properties. Here you can define all the details for your font. It should all be fairly self explanatory.

Once that’s done, save your font again (preferably in the FontForge format, .sfd) and start exporting in all the different formats you want. I’d recommend at least doing ttf and woff. You’ll probably want to get it into eot, which font-forge unfortunately can’t natively produce (It’s a format created by Microsoft for IE, so go figure.). This website should allow you to convert it to .eot, along with a bunch of other random formats if you feel like that kind of thing.

You might run into some errors during export. Most of these can be fixed with the tools in fontforge, right clicking on the appropriate glyphs should give you options to convert to integer variables, correct direction and AutoHint. You can find the option to fix the extrema error in Element -> Add Extrema.

So now that that’s done, we need to make some CSS. Here’s what I’ve used, it’s basically taken directly from FontAwesome:

@font-face {
  font-family: 'byatlas-iconfont';
  src: url('/assets/font/byatlas-iconfont.eot');
  src: url('/assets/font/byatlas-iconfont.eot') format('embedded-opentype'),
    url('/assets/font/byatlas-iconfont.woff') format('woff'),
    url('/assets/font/byatlas-iconfont.ttf') format('truetype');
  font-weight: normal;
  font-style: normal;
}
[class^="icon-"],
[class*=" icon-"] {
  font-family: byatlas-iconfont;
  font-weight: normal;
  font-style: normal;
  text-decoration: inherit;
  -webkit-font-smoothing: antialiased;

  /* sprites.less reset */
  display: inline;
  width: auto;
  height: auto;
  line-height: normal;
  vertical-align: baseline;
  background-image: none;
  background-position: 0% 0%;
  background-repeat: repeat;
  margin-top: 0;
}
.icon-globe:before    { content: "\f0000";}
.icon-half-globe:before    { content: "\f0001";}
.icon-twitter:before		{ content: "\f0002"; }
.icon-github:before		{ content: "\f0003"; }

Hopefully all the parameters there are, again, self explanatory. The bits right down the bottom are used to produce the i tags that everyone loves so much. If you don’t want to do that, just so long as you enclose your text with tags that have the correct font-family set, you can use this type of thing:

&#983040;
&#983041;

And so on. Those are just the hex values converted to decimal, prefixed with &# and ending with ;. Much like how you use &nsbp and the like.

So hopefully you’ve made your shiny new startup or whatever even shinier now with svelte, lightweight icon fonts. You are going to cut me in on a slice of the profits, right?

The Making of

Liam Falconer — Mon, 11 Mar 2013 18:32:40 +1200

This post is hideously out of date. Go take a look at Moving to Hugo and Lanyon for the lowdown on way this site works now

So. I’d written two posts in anticipation of getting this up and running, however they proved to be glouriously incorrect. I’ve been through about three or four changes in terms of how I actually decided to do this, so… yeah.

Anyway, at this point, it’s a site that’s built from Jinja2 templates. Mostly. In case you haven’t noticed, some bits of it are still not yet entirely working. Mostly the contact page. And all the examples I need to throw on the page that’s supposed to convince you to pay me. Which is just a little bit awkward.

So initally I just planned to generate the blog using Pelican, but that seemed like a bit of overkill for what I wanted to do. Plus you know, I can’t call myself a real hacker if I don’t DIY this bad boy. So although it works at the moment, it’ll probably be something I polish for quite some time to come.

But at least so far the site is still almost entirely static. I’ll have to do something server side for the contact page, since I don’t really want to get spammed into the ground, but apart from that… it should be all slick and fast and stuff. Even more so once I eventually get around to doing some Ajax stuff.

Oh, so to summarize the two posts I’d originally written, my thanks to danneu for darkstrap, from which I stole quite a bit of css. And to the twitter bootstrap guys, Mark Otto and fat for more css, and making darkstrap possible in the first place.

I also owe fontawesome and subtle patterns for art cribbed from them. Chchur.

Oh, and Mr. doob for Three.js, even though I haven’t quite got it doing what I want it to yet. I’m planning on making this place at least 200% more shiny… soon…

Well, congratulations on surviving my aimless rambling for this long.

This reads way too much like one those tracks giving shoutouts

A gentle intro to assembly with Rust

Avoiding optimizations

Avoiding optimizations, take 2

Looking inside the function

The Power of Names

Reducible Systems

LMAX’s production system

The Duality of Message Passing

The usefulness of reducefulness

Other reducible systems

Conclusions

re-frame and the Back Button

Using pushState

Handling the popstate event

Making sure our initial state is correct

Wrapping up

Where the GC Fears to Tread

Sifting through the heap

Outside the JVM

The real leak

Summary

Bitcoin crash course

Roasters log 1

Roasting coffee with a popcorn maker

The unreasonable effectiveness of checklists

Why bother with property tests

What is a property test anyway?

Increases your confidence in your code

You’re lazy, and that’s not a bad thing

They go great with contracts

Helps you find the (actual) boundaries of your system

You have to think more

Make impractical tests possible.

tl;dr

Flamegraphs and benchmarks

Why use flamegraphs?

How do I flamegraph?

Problems

Clojure fspec surprise

Writing IntelliJ plugins

Property testing tooling in Java

Performance checks on metrics

Coupling, Connascence, CSP and Actors

A crash course on Connascence

Actors vs CSP

Too many masks

Dependency Injection in Clojure

Leaving the M$alt Mines

Error Handling in ASP.NET MVC

Fresh Thoughts on Dependency Injection

Moving to Hugo and Lanyon

Dependency Injection - A necessary evil?

Fun with Icon Fonts

The Making of

Using `pushState`

Handling the `popstate` event