Nick Desaulniers

The enemy's gate is down

Let’s Write Some X86-64

| Comments

…“‘Our speech interposes itself between apprehension and truth like a dusty pane or warped mirror. The tongue of Eden was like a flawless glass; a light of total understanding streamed through it. Thus Babel was a second Fall.’ And Isaac the Blind, an early Kabbalist, said that, to quote Gershom Scholem’s translation, ‘The speech of men is connected with divine speech and all language whether heavenly or human derives from one source: the Divine Name.’ The practical kabbalists, the sorcerers, bore the title Ba’al Shem, meaning ‘master of the divine name.’”

“The machine language of the world,” Hiro says.

“Is this another analogy?”

“Computers speak machine language,” Hiro says. “It’s written in ones and zeroes - binary code. At the lowest level, all computers are programmed with strings of ones and zeroes. When you program in machine language, you are controlling the computer at its brainstem, the root of its existence. It’s the tongue of Eden. But it’s very difficult to work in machine language because you go crazy after a while, working at such a minute level. So a whole Babel of computer languages has been created for programmers: FORTRAN, BASIC, COBOL, LISP, Pascal, C, PROLOG, FORTH. You talk to the computer in one of these languages, and a piece of software called a compiler converts it into machine language. But you never can tell exactly what the compiler is doing. It doesn’t always come out the way you want. Like a dusty pane or warped mirror. A really advanced hacker comes to understand the true inner workings of the machine – he sees through the language he’s working in and glimpses the secret functioning of the binary code – becomes a Ba’al Shem of sorts.”

Hiro Protagonist and The Librarian Snow Crash by Neal Stephenson

This a beautiful quote, one that I think truly captures the relationship between higher level languages and the Instruction Set Architecture (ISA)’s machine code, though this is from the angle of controlling the machine with its implementation specific quirks which can detract from what you’re actually trying to do.

This blog is meant for those who don’t know x86-64 assembly, but maybe know a little C, and are curious about code generation. Or maybe if you’ve ever tried to hand write x86-64 assembly, and got stuck trying to understand the tooling or seemingly random segfaults from what appears to be valid instructions.

I really enjoy writing code in CoffeeScript and C, so I have a quick anecdote about CoffeeScript though you don’t need to know the language. When writing CoffeeScript, I find myself frequently using a vim plugin to view the emitted JavaScript. I know when CoffeeScript emits less than optimal JavaScript. For example in the code:

1
nick = -> console.log x for x in [0..100]

I know that CoffeeScript is going to push the results of the call to console.log into an array and return that, because of the implicit return of the final expression in a function body, which in this case happens to be a for loop (array comprehension being treated itself as an expression). The emitted JavaScript looks like:

1
2
3
4
5
6
7
8
9
10
var nick;

nick = function() {
  var x, _i, _results;
  _results = [];
  for (x = _i = 0; _i <= 100; x = ++_i) {
    _results.push(console.log(x));
  }
  return _results;
};

By putting a seemingly meaningless undefined statement as the final statement in the function body, we can significantly reduce what the function is doing and decrease the number of allocations:

1
2
3
nick = ->
  console.log x for x in [0..100]
  undefined

emits:

1
2
3
4
5
6
7
nick = function() {
  var x, _i;
  for (x = _i = 0; _i <= 100; x = ++_i) {
    console.log(x);
  }
  return void 0;
};

That return void 0 may seem odd, but functions in JavaScript without an explicit return value return undefined, but since the undefined identifier can be reassigned to, the expression void 0 evaluates to the value undefined.

You can see that making the CoffeeScript function body slightly longer and adding a seemingly meaningless lone undefined statement at the end of the function body, the emitted JavaScript does not allocate an array or waste time pushing the results of console.log, which would be undefined, into that array a hundred times. This reminds me of how seemingly meaningless noop instructions can keep a processor’s pipeline full by preventing stalls, though a pipeline stall doesn’t change the correctness of a program, so it’s an imperfect analogy.

Now I’m not saying that you should be thinking about these kinds of optimizations when programming at such a high level, as they might be premature. I shared with you this example because while writing C code, and reading Michael Abrash’s Graphics Programming Black Book, I wondered to myself if hardcore C programmers also would know the equivalent assembly instructions that would be emitted from their higher level C code (before optimizing compilers even existed).

In college, I was taught 68k and MIPS ISAs. To understand x86-64 we need to be able to write and run it. Unfortunately, I did not have the training to know how to do so. My 68k code was run on a MCU from a FreeScale IDE in Windows, so the process might as well have been indistinguishable from magic to me. I understood that you’d start with low level source, in (somewhat) human readable instructions that would be converted to binary representing op codes. The assembler would then translate the assembly into non-executable object files that contained binary code that had placeholders for sections of code defined in other object files. The linker would then be used to replace the placeholders with the now combined binary code’s relative positions and then converted into an executable. But how do I do this from my x86-64 machine itself? The goto book I’ve been recommended many times is Professional Assembly Language by Richard Blum, but this book only covers x86, not x86-64. There’s been some very big changes to the ABI between x86 and x86-64. You may be familiar with Application Programmer Interfaces (APIs), but what is an Application Binary Interface? I think of an ABI as how two pieces of native code interact with one another, such as calling convention (how arguments are passed to functions at the ISA level).

I’m very lucky to have the privilege to work with a compiler engineer, Dan Gohman, who has worked on a variety of compilers. I was citing a particular email of Dan’s for some time before he came to work with us, when I would talk about how the naming of LLVM gives the imagery of a virtual machine, though it’s more so a compiler intermediate representation. Dan is an amazing and patient resource who has helped me learn more about the subtleties of the x86-64 ABI. Throughout this blog, I’ll copy some responses to questions I’ve had answered by Dan.

Our first goal is to write an x86-64 program that does nothing, but that we can build. Assembly files typically have the .s file extension, so let’s fire up our text editor and get started. I’ll be doing my coding from OSX 10.8.5, but most examples will work from Linux. All of my symbol names, like _main, _exit, and _printf, are prefixed with underscores, as Darwin requires. Most Linux systems don’t require this, so Linux users should omit the leading underscores from all such names. Unfortunately, I cannot figure out how to link with ld in Linux, so I recommend trying to understand what gcc -v your_obj_file.o is doing, and this might help. Let me know in the comments if there’s an easy way to use ld when linking your object files from linux and I’ll be happy to post an edit.

Let’s start with this fragment and get it building, then I’ll cover what it’s doing.

1
2
3
4
5
6
.text
.globl _main
_main:
  subq $8, %rsp
  movq $0, %rdi
  call _exit

Let’s use OSX’s built in assembler (as) and linker (ld).

1
2
as nothing.s -o nothing.o
ld -lc -macosx_version_min 10.8.5 nothing.o -o nothing

We should now be able to run ./nothing without any segfaults. Without -macosx_version_min 10.8.5 I get a warning and my executable segfaults.

Now let’s create a basic generic Makefile to help us automate these steps. Watch your step; archaic syntax ahead.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
SOURCES = $(wildcard *.s)
OBJECTS = $(SOURCES:.s=.o)
EXECUTABLES = $(OBJECTS:.o=)

# Generic rule
# $< is the first dependency name
# $@ is the target filename
%.o: %.s
  as $< -o $@

default: $(OBJECTS)
  for exe in $(EXECUTABLES) ; do \
    ld -lc -macosx_version_min 10.8.5 $$exe.o -o $$exe ; \
  done

.PHONY: clean
clean:
  rm *.o
  for exe in $(EXECUTABLES) ; do \
    rm $$exe ; \
  done

By the time you read this, I’ve already forgotten how half of that code works. But, this will allow us to run make to assemble and link all of our .s files individually, and make clean to remove our object and executable files. Surely you can whip up a better build script? Let me know in the comments below.

So now let’s go back to our assembly file and go over it line by line. Again, it looks like:

1
2
3
4
5
6
.text
.globl _main
_main:
  subq $8, %rsp
  movq $0, %rdi
  call _exit

.text is the text section. This section defines the instructions that the processor will execute. There can be other sections as well. We’ll see more later, but the “data” section typically has static variables that have been initialized with non null (and non zero) values, where as “bss” will have static but non initialized values. Also, there will be a heap and a stack although you don’t declare them as you would for text or data.

Next up is the global directive. The global directive tells the linker that there will be a section named _main that it may call into, making the _main section visible to other sections. You may be wondering why directives and sections both begin with a dot.

’.” isn’t a valid identifier character in C, so way back when it became common to use ‘.” as a prefix in the assembler and linker in order to avoid clashing with C symbol names. This unfortunately was used for both section names and directives, because both can appear in contexts where a simple parser wouldn’t be able to disambiguate otherwise.

I don’t know why they used the same convention for directives and section names, but they did, and it’s not practical to change it now.

Dan Gohman

Ok now the subtraction instruction. We’ve got a bit to go over with just this one line. The first is the instruction itself. The sub instruction has numerous suffixes that specify how many bytes to operate on. The typical convention for numerous instructions is to have a suffix of b for 1 byte (8 bits), w for a word (2 bytes, 16 bits), l for a long or double word (4 bytes, 32 bits), and q for a quad word (8 bytes, 64 bits). Leaving off the suffix, the assembler will try and guess based off of the operands, which can lead to obscure bugs. So subq operates on 64 bits. Extending this we should be able to recognize that subb operates on 8 bits, subw operates on 16 bits, subl operates on 32 bits, and subq operates on 64 bits. What’s important to understand is that instruction suffix is dictated by the inputs and destination size. See Figure 3-3 of the AMD64 ABI.

Ok now let’s look at the full instruction subq $8, %rsp. The current order of the operands is known as the AT&T syntax, where the destination is specified last (as opposed to the Intel syntax, where the destination follows the instruction name ex. subq rsp, 8).

I’m biased towards AT&T-syntax because GCC, LLVM, and icc (at least on Unix-like platforms) all use it, so it’s what I’m used to by necessity. People familiar with assembly languages on other platforms sometimes find it feels backwards from what they’re used to, but it is learnable.

Dan Gohman

I’m writing my examples in AT&T syntax simply because when I compile my C code from clang with the -S flag, or run my object files through gobjdump, I get AT&T syntax by default (though I’m sure there are flags for either AT&T or Intel syntaxes). Also, the ABI docs are in AT&T. What are your thoughts on the two different syntaxes? Let me know in the comments below.

So when we say subq $8, %rsp, we’re subtracting the immediate value of 8 from the stack pointer (the register %rsp contains our stack pointer). But why are we doing this? This is something that is left out from some of the basic hello world assembly programs I’ve seen. This is the first ABI point I want to make:

x86-64 ABI point 1: function calls need the stack pointer to be aligned by a multiple of 16 bytes.

By default, they are off by 8 on function entry. See Section 3.2.2 page 16 of the ABI.

Why is the stack pointer misaligned by 8 bytes on function entry? I’m going to punt on the answer to that for a bit, but I promise I’ll come back to it. The most important thing is that that call instruction later on will fail unless we align our stack pointer, which started out misaligned. If we comment it out (pound signs, #, comment out the rest of the line) and make our executable, we’ll get a segfault. You could even add 8 bytes to the stack pointer and our basic example would work (we just need a multiple of 16 remember), but when we learn later (I promise) about how the stack works in x86-64, we’ll see we can mess things up by adding rather than subtracting.

Next up we’re moving the immediate value 0x0 into %rdi. You may have heard that arguments to functions are pushed on the stack in reverse order, but that’s an old x86 convention. With the addition of 8 more general purpose registers, we now pass up to the first 6 arguments in registers (then push the rest, if any, on the stack in reverse order). The convention (in OSX and Linux) is our second ABI point:

x86-64 ABI point 2: The calling conventions for function invocations require passing integer arguments in the following sequence of registers: %rdi, %rsi, %rdx, %rcx, %r8, %r9, then pushing the rest on the stack in reverse order.

See section 3.2.3 under “Passing”. Warning: Microsoft has a different calling convention. This is quite troubling to me, because I assumed that Instruction Set Architectures were created so that the same code could run on two different machines with the same microarchitecture, but because the ISA does not define how arguments would be passed, this ambiguity is left up to the OS implementor to decide. Thus the same code may not run on two different machines with the same microarchitecture if their operating systems are incompatible at the ABI layer.

UPDATE: Further, I just learned that Go, and C code compiled by 6c don’t use the “normal” SysV ABI and calling convention, but have their own.

What our goal is is to call exit(0); where exit is defined in libc, which we link against during the linking phase with with flag -lc. This is another punt on system calls. So to invoke exit with the first integer argument of 0, we first need to move the immediate value of 0x0 into %rdi. Now if you run your executable from your shell, then echo $?, you should see that the previous command’s exit code was 0. Try changing the exit code to 42 and verify that it works successfully.

Ok, well a program that does nothing is more boring than hello world. Now that we have our build setup out of the way, let’s make a hello world program. If you’re familiar with ASCII tables, we can use putchar from libc since we’re already linking to it. Use man putchar to look at its signature and this ASCII table to move immediate values into a certain register (remember the calling convention, point #2) and make sure you setup the stack pointer before any calls and exit after all other calls.

I’ll leave that up to an exercise for the reader. Let’s use a string and printf.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
.data
_hello:
  .asciz "hello world\n"

.text
.globl _main
_main:
  subq $8, %rsp

  movb $0, %al
  leaq _hello(%rip), %rdi
  call _printf

  movq $0, %rdi
  call _exit

First up is our data section .data. I previously mentioned the data section contains global non null and non 0 variables. You can see here that the string itself becomes part of the binary by using the unix command strings and passing your executable as the first argument. Further, if you pass your executable to hexdump you can even see the ASCII codes in hex:

1
0001020 68 65 6c 6c 6f 20 77 6f 72 6c 64 0a 00 00 00 00

Also, we can run our binary through objdump as well and see the string:

1
2
3
4
gobjdump -j .data -s hello_world
...
Contents of section .data:
 2020 68656c6c 6f20776f 726c640a 00        hello world..

Ok so now we’re moving an immediate value of 0x0 to %al. %al is 1 byte wide, so we use the b suffix on the mov instruction. The next important point of the ABI has to do with functions that use a variable number of arguments (varargs), like printf does:

x86-64 ABI point 3: Variadic functions need to have the number of vector arguments specified in %al.

This will make printf debugging hard without. Also in section 3.2.3 under passing.

If you don’t know what vector arguments are, no worries! I’m not going to cover them. Just know that without this, the contents of %al may work in a basic example, where we haven’t touched %al, %ax, %eax, or %rax yet, but we shouldn’t bank on it being 0x0. In fact we shouldn’t bank on most registers being preserved after a function call. Now’s a good time to talk about volatility:

x86-64 ABI point 4: Most registers are not preserved across function calls.

Only %rbx, %rsp, %rbp, and %r12-%r15 (and some others) are. These are called “call saved” or “non volatile” registers. The rest should be considered “call clobbered” or “volatile.” That means every time we invoke a call like printf, we need to reset %al, since it is the lower 8 bits of %rax which is the 1st return register, so it is always clobbered.

The next instruction loads the effective address of the string relative to the current instruction pointer into %rdi, the first argument for printf. The .asciz directive appends the null byte for us, since C strings are null terminated.

With this knowledge, can you modify hello world to print “hello world 42”, without putting 42 into the string in the data section? Hint: you’ll need a placeholder in your string and need to know the x86-64 calling convention to pass an additional argument to printf.

Finally, let’s talk about the stack. When we create automatic variables in C, they are created in the segment called the stack. On x86-64 the stack starts at some arbitrary address (virtual memory backed by physical memory) and “grows” downwards. That is why we subtracted 8 bytes, rather than add 8 bytes to the stack for alignment earlier. The metaphor of a stack of plates is kinda upside-down as additional plates (variables) are going underneath the current bottom plate if you can imagine, in this case. The stack grows towards the heap, and it is possible for them to collide if you don’t ask the OS to expand your data segment (sbrk).

credit

Let’s say we want to call something like memset, which from man memset we can see takes an address, a value to fill, and a number of bytes to fill. The equivalent of say this C code:

1
2
3
4
5
6
7
8
9
10
11
12
13
#include <stdio.h>
#include <string.h>
#include <inttypes.h>
int main () {
  int8_t array [16];
  memset(&array, 42, 16);
  int8_t* ptr = array;
  printf("Current byte: %" PRId8 "\n", *ptr);
  ++(*ptr);
  printf("Current byte: %" PRId8 "\n", *ptr);
  ++ptr;
  printf("Current byte: %" PRId8 "\n", *ptr);
}

Well, that might look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
.data
_answer:
  .asciz "Current byte: %d\n"

.text
.globl _main
_main:
  subq $8, %rsp

  subq $16, %rsp # allocate 16B
  leaq (%rsp), %rdi # first arg, &array
  movq $42, %rsi # second arg, 42
  movq $16, %rdx, # third arg, 16B
  call _memset

  leaq _answer(%rip), %rdi
  movq $0, %rsi
  movb (%rsp), %sil # these two are equavlent to movzql (%rsp), %esi
  movb $0, %al
  call _printf

  incq (%rsp)

  leaq _answer(%rip), %rdi
  movq $0, %rsi
  movb (%rsp), %sil
  movb $0, %al
  call _printf

  leaq _answer(%rip), %rdi
  movq $0, %rsi
  movb 1(%rsp), %sil
  movb $0, %al
  call _printf

  addq $16, %rsp # clean up stack

  movq $0, %rdi
  call _exit

This isn’t a perfect example because I’m not allocating space for the ptr on the stack. Instead, I’m using the %rsp register to keep track of the address I’m working with.

What we’re doing is allocating 16B on the stack. Remember we need to keep %rsp aligned on 16B boundaries, making it a multiple of 16B. If we needed a non 16B multiple, we could allocate more than needed on the stack, and then do some arithmetic later when access our automatic variables.

For memset, we need to pass the address of our first argument. In x86-64, the stack grows downwards, but our variables “point” upwards, so %rsp and the higher 16B is the memory addresses of our array, with %rsp currently pointing to the front. The rest you should recognize by now as part of the calling convention.

In the next grouping of instructions, we want to verify that memset set every byte to 42 (0x2A). So what we’ll do is copy the first byte from our array, currently pointed to by %rsp, to the lower 8b of %rsi which is named %sil. It’s important to zero out the 64b contents of %rsi first, since it may have been clobbered by our previous call to memset.

Then we dereference and increment the value pointed to by our array pointer, ++(*ptr) or ++array[0]. Now array[0] is 43, not 42.

In the next grouping of instructions, we print the second byte of our array, array[1], and get 42 from memset. Now we could try to increment the stack pointer itself by one, but then the call to printf will fail, so instead when we load the value of array[1], we do some pointer arithmetic movb 1(%rsp), %sil. This is relative addressing, though you’ve already seen this with loading the strings. You might wonder why I’m not loading the byte in the other “direction,” say movb -1(%rsp), %sil. Well, that goes back to my point that while the stack pointer moves down as we allocate automatic variables, their address and memory they take up “points up.”

Finally, we clean up our automatic variable’s allocated space on the stack. Note that we do not zero out that memory. A preceding function call might overwrite that data on the stack, but until it does or unless we explicitly zero it out, a buffer overrun could accidentally read that data a la Heartbleed.

Now I did promise I would talk about why the stack pointer is misaligned by 8 bytes on function entry. That is because unoptimized functions typically have a function prolog and epilog. Typically, besides creating room on the stack for automatic variables at the beginning of a function, we typically want to save the frame AKA base pointer, %rbp, on the stack. Since %rbp is 64b or 8B and the push instruction will decrement the stack pointer by 8b, this will align the misaligned stack to a 16B multiple. So in function bodies, you’ll typically see:

1
2
3
4
5
6
my_func:
  push %rbp
  movq %rsp, %rbp
  # your code here...
  popq %rbp
  ret

This great article explains that you may want to profile your running application, or look at the call stack in a debugger, and by having a linked list of stack frames of the function that invoked yours and it’s caller and so on in a dedicated register makes it trivial to know the call stack at any given point in runtime. Since we’re always pushing %rbp immediately thereby saving it on the stack and putting our stack pointer (%rsp) in the base pointer (%rbp) (later restoring it, %rbp is call saved), we can keep popping %rbp then moving our stack pointer to that value to see that quux was called by bar was called by foo (foo(bar(quux()));). Now you saw that I was able to write code that clearly worked without the three additonal instructions in the prolog and epilog, and indeed that’s what happens with optimized code emitted from your compiler. And since GDB uses something called DWARF (adds symbols to your objects) anyways, it isn’t a huge issue to remove the prolog and epilog.

So, I think I’ve shown you enough to get started hand writing assembly. To learn more, you should write the higher level C code for what you’re trying to do and then study the emitted assembly by compiling with the -S flag. With clang, you’ll probably see a bunch of stack check guards for each frame, but those just prevent stack buffer overflows. Try compiling simple conditionals (jump forwards), then simple loops (jump backwards) without optimizations. Jumping to sections and calling your own functions should be pretty easy to figure out. Hint: don’t duplicate section names, but the assembler will catch this and warn you pretty explicitly.

Don’t let people discourage from learning assembly because “compilers will always beat you.” “Just use LLVM or libjit or whatever for codegen.” Well, existing solutions aren’t perfect solutions in every scenario. Someday you might be tasked with doing codegen because LLVM is not optimal under certain constraints. You’ll never know if you can beat them unless you try; those most comfortable are those most vulnerable. I’m afraid that if enough people are turned away from learning the lower levels of programming because the higher level is unquestionably better, then assembly will ultimately be forgotten and the computer becomes a black box again. This is something that troubles me and that I see occurring around me frequently; a lot of devs new to web development conflate jquery with JavaScript, and Three.js with WebGL. If you’re around the Bay Area, I’ll be giving a talk at HTML5DevConf on May 22 demystifying Raw WebGL. You should come and check it out.

In summary, remember:

  • The stack pointer needs to be aligned by 16B multiples when calling another function.
  • Calling convention dictates passing arguments in %rdi, %rsi, %rdx, %rcx, %r8, %r9, then stack.
  • %al needs the number of vector arguments for variadic functions.
  • Know which registers are call saved (%rbx, %rsp, %rbp, and %r12-%r15 (and some others)) and call clobbered.

Closing thoughts by Dan:

To me, machine code doesn’t feel analogous to this concept of the divine name. I certainly wouldn’t describe it as “like a flawless glass; a light of total understanding streamed through it”. Even if you strip away all the accidental complexity, machine code is still permeated by mundane implementation-oriented concerns. Also, the abstractions which we use to organize and “understand” the program are gone; information is lost.

My imagination of a divine language looks a lot more like an idealized and cleaned-up LISP. It’d be capable of representing all known means of abstraction and combination (in the SICP sense), it might be thought of as a kind of super-language of which all of our real-world languages are just messy subsets.

Dan Gohman

Write a Test Case

| Comments

Your application just broke, oh no! It couldn’t have been your code, right?

I’ve always had trouble spotting mistakes in my own work such as spelling, grammar, mathematical, or even in programming. With spelling or grammar, office applications quickly pick up on my mistakes and underline them for me, but most of my mistakes come from my own hubris. I’m confident in what I do, and that gets me in trouble when I make little mistakes. I’m confident that I solved this problem correctly and didn’t make any small mistakes. As a kid competing in timed math competitions, I quickly learned that reviewing your work cost time, so it was important to recognize where common mistakes would crop up on certain problems and spend slightly extra time the first time through those problem areas, rather than double checking the work in its entirety, unless I had the time to spare. Measure twice, cut once.

The same is true for programming. Writing test cases is time consuming, and if time is money, then it could be said that writing tests is costly. There’s probably a logical fallacy in there. In general, we hope that the time-cost of writing test cases will be recuperated by the time-cost of bugs caught, though it’s not easy to measure the time-cost of averted bugs. I’m all for test cases. I think that SQLite having 100% branch coverage is incredible, truly an lofty achievement. People get bogged down in arguments over type systems where testing is more of a must for languages without a compiler to catch certain bugs.

Ok, so going back to your code, something is wrong. But we’re confident it couldn’t be our code. Maybe it’s one of those open source modules I’m using not being tested enough, or that pesky browser implementation, or this damned OS. It couldn’t be my code.

Prove it.

I see bug reports all the time where people say your X breaks my Y, but when asked to provide Y they can’t for whatever software licensing reason. The bug resolver (person who is enabled to fix said bug in X), doesn’t know at this point whether the bug reporter (developer of Y) did their homework; the bug is unconfirmed. Both reporter and resolver are suspicious of each others’ code. I don’t make silly mistakes, right?

So it’s up to the resolver to work out a test case. Being able to reproduce said issue is the first goal to resolving a bug. Just like scientific statements aren’t taken as fact until reproducible, so too will this bug be merely conjecture at this point. It kills me when the bug resolver closes an issue because it works for me. Nothing boils my blood more; I took the time to try and help you and wasn’t rewarded on some level so I feel that I wasted my time. But from where you stand the resolver is not correct, so again I say…

Prove it.

I think that it’s super hard to get meaningful feedback from users. It’s hard for them to express why they’re frustrated. We as software developers don’t always point them in the right direction. As users can have wide ranges of technical capabilities, sometimes we as more technical oriented people have to do a little hand holding. For instance, from Firefox, how many hops does it take to get from the application’s menus to their open bug tracker, bugzilla? Where can I report an issue? How do I report an issue? Do I annoy the hell out of the user until they rate my app? Users might be complaining on Twitter, but that’s sure as hell not where devlopers are looking for bug reports. Enabling the user to provide feedback could be viewed as a double edged sword. Can I make meaningful conclusions from the torrent of feedback I’m now getting? Did I structure my form in such a way to ask the right questions? Should we make our issue tracker public? Should we make security issues that could harm users, if more widely understood, public? Public to fellow employees, even? What did you expect to happen, and is that what actually happened? This rabbit hole goes deep.

The other day, I was writing a patch for a large application that was developed by someone else and that I still don’t fully comprehend in its entirety. My patch should have worked, why wasn’t it working? Probably something in this person’s code right? Sure enough, I wrote a simple test case to eliminate everything but the control variables, and it turns out I was wrong.

It’s hard to get users to do more than the bare minimum to report an issue, even at all, but to prevent the issue from being ignored or closed because it works for the developer in their environment, take the time to report it correctly the first time. As someone who understands technology more than the average person, take the time to write a cut down test case that exposes the issue. The developer who looks at the issue will be grateful; instead of wasting their time with something that may not even be a bug, you’ve just done something that would have to be done anyways if it is indeed a bug. You might find that a reduced code size shows that it’s in fact not an issue with someone else’s code, but in fact a simple mistake you made somewhere in your monolithic application. Or it might be exactly the evidence you need to get someone’s attention to fix said problem. That test case might even be included in the latest test suite. Either way, you have now proved beyond a reasonable doubt that the issue does not lie in your fault.

Write a Test Case for Your Next Bug Report or Issue.

Function.prototype.bind Edge Cases

| Comments

ECMAScript 5’s Function.prototype.bind is a great tool that’s implemented in all modern browser JavaScript engines. It allows you to modify the context, this, of a function when it is evaluated in the future. Knowing what this refers to in various contexts is key to being a professional JavaScript developer; don’t show up to an interview without knowing all about it.

Here’s a common use case that developers need to watch for. Can you spot the mistake?

1
2
3
4
5
6
7
8
9
10
var person = "Bill";

var obj = {
  person: "Nick",
  hi: function () {
    console.log("Hi " + this.person);
  }
};

window.addEventListener("DOMContentLoaded", obj.hi);

Ooops!!! Turns out that since we added the event listener to the window object, this in the event handler or callback refers to window. So this code prints "Hi Bill" instead of "Hi Nick". We could wrap obj.hi in an anonymous function:

1
2
3
window.addEventListener("DOMContentLoaded", function () {
  obj.hi();
});

But that is so needlessly verbose and what we were trying to avoid in the first place. The three functions you should know for modifying this (a question I ask all my interview candidates) are Function.prototype.call, Function.prototype.apply, and Function.prototype.bind. call is variadic, while apply takes an array of arguments, but the two both immediately invoke the function. We don’t want to do that just yet. The fix we need is Function.prototype.bind.

1
window.addEventListener("DOMContentLoaded", obj.hi.bind(obj));

There, now isn’t that nice and short? Instead of saving this as another variable then closing over it, you can instead use bind!

1
2
3
4
5
6
7
8
9
var obj = {
  person: "Nick",
  wait: function () {
    var self = this;
    someButton.onclick = function () {
      console.log(self.person + " clicked!");
    };
  },
};

becomes

1
2
3
4
5
6
7
8
var obj = {
  person: "Nick",
  wait: function () {
    someButton.onclick = function () {
      console.log(this.person + " clicked!");
    }.bind(this);
  },
};

No need to store this into self, then close over it. One great shortcut I use all the time is creating an alias for document.getElementById.

1
2
3
4
5
var $ = document.getElementById.bind(document);
$('someElementsId').doSomething();
$('anotherElement').doSomethingElse();
$('aThirdElement').doSomethingDifferent();
$('theFifthElementOops').doSomethingFun();

Why did I bind getElementById back to document? Try it without the call to bind. Any luck?

bind can also be great for partially applying functions, too.

1
2
3
4
5
6
7
function add (a, b) {
  console.log("a: " + a);
  console.log("b: " + b);
  return a + b;
};
var todo = add.bind(null, 4);
console.log(todo(7));

will print

1
2
3
a: 4
b: 7
11

What Function.prototype.bind is essentially doing is wrapping add in a function that essentially looks like:

1
2
3
var todo = function () {
  add.apply(null, [4].concat(Array.prototype.slice.call(arguments)));
};

The array has the captured arguments (just 4), and is converting todo’s arguments into an array (a common idiom for converting “Array-like” objects into Arrays), then joining (concat) them and invoking the bound function (apply) with the value for this (in this case, null).

In fact, if you look at the compatibility section of the MDN page for bind, you’ll see a function that returns a function that is essentially the above. One caveat is that this approach only allows you to partially apply variables in order.

So bind is a great addition to the language. Now to the point I wanted to make; there are edge cases when bind doesn’t work or might trip you up. The first is that bind evaluates its arguments when bound, not when invoked. The other is that bind returns a new function, always. And the final is to be careful binding to variadic functions when you don’t intend to use all of the passed in variables. Um, duh right? Well, let me show you three examples that have bitten me (recently). The first is with ajax calls.

1
2
3
4
5
6
7
8
function crunch (data) {
  // operate on data
};

var xhr = new XMLHttpRequest;
xhr.open("GET", "data.json");
xhr.onload = crunch.bind(this.response);
xhr.send();

Oops, while I do want to operate on this.result within crunch with this referring to xhr, this at the time of biding was referring to window! Let’s hope window.results is undefined! What if we changed this.result with xhr.result? Well, we’re no longer referring to the window object, but xhr.result is evaluated at bind time (and for an unsent XMLHttpRequest object, is null), so we’ve bound null as the first argument. We must delay the handling of xhr.onload; either use an anonymous function inline or named function to control nesting depth.

1
2
3
xhr.onload = function () {
  crunch(this.result);
};

The next is that bind always returns a new function. Dude, it says that in the docs, RTFM. Yeah I know, but this case still caught me. When removing an event listener, you need to supply the same handler function. Example, a once function:

1
2
3
4
5
6
function todo () {
  document.removeEventListener("myCustomEvent", todo);
  console.log(this.person);
});

document.addEventListener("myCustomEvent", todo.bind({ person: "Nick" }));

Try firing myCustomEvent twice, see what happens! "Nick" is logged twice. A once function that handles two separate events is not very good. In fact, it will continue to handle events, since document does not have todo as an event handler for myCustomEvent events. The event listener you bound was a new function; bind always returns a new function. The solution:

1
2
3
4
5
var todo = function () {
  console.log(this.person);
  document.removeEventListener("myCustomEvent", todo);
}.bind({ person: "Nick" });
document.addEventListener("myCustomEvent", todo);

That would be a good interview question. The final gotcha is with functions that are variadic. Extending one of my earlier examples:

1
2
3
4
5
6
7
8
9
10
11
var obj = {
  person: "Nick",
  wait: function () {
    var someButton = document.createElement("button");
    someButton.onclick = function () {
      console.log(this.person + " clicked!");
    }.bind(this);
    someButton.click();
  },
};
obj.wait();

Let’s say I thought I could use bind to simplify the onclick using the trick I did with document.getElementById:

1
2
3
4
5
6
7
8
9
var obj = {
  person: "Nick",
  wait: function () {
    var someButton = document.createElement("button");
    someButton.onclick = console.log.bind(console, this.person + " clicked!");
    someButton.click();
  },
};
obj.wait();

Can you guess what this prints? It does prints the expected, but with an unexpected addition. Think about what I said about variadic functions. What might be wrong here?

Turns out this prints "Nick clicked! [object MouseEvent]" This one took me a while to think through, but luckily I had other experiences with bind that helped me understand why this occurred.

console.log is variadic, so it prints all of its arguments. When we called bind on console.log, we set the onclick handler to be a new function that applied that expected output with any additional arguments. Well, onclick handlers are passed a MouseEvent object (think e.target), which winds up being passed as the second argument to console.log. If this was the example with add from earlier, this.person + " clicked!" would be the 4 and the MouseEvent would be the 7:

1
2
3
someButton.onclick = function (e) {
  console.log.apply(console, ["Nick clicked!"].concat([e]));
};

I love bind, but sometimes, it will get you. What are some examples of times when you’ve been bitten by bind?

Making Great Node.js Modules With CoffeeScript

| Comments

Node.js is a great runtime for writing applications in JavaScript, the language I primarily develop in. CoffeeScript is a programming language that compiles to JavaScript. Why would we write a reusable piece of code, a module , in CoffeeScript? CoffeeScript is a very high level language and beautifully brings together my favorite aspects of JavaScript, Ruby, and Python. In this tutorial, I’ll show you how I create reusable open source modules for Node.js from CoffeeScript, which is something I recently discovered while creating a playlist parser module. The point is to focus on how to turn a quick hack into a nicely laid out Node.js module.

The steps are as follows:

  1. Turn an idea into a git repo.
  2. Add directory structure.
  3. Split library functions from testing.
  4. Add a Build script.
  5. Create node module.
  6. Add LICENSE and README.
  7. Publish.

First thing’s first, we have to have an idea. It doesn’t have to be revolutionary, just do one thing and do it well. That is the first rule of UNIX fightclub philosophy, which resonates well within the Node.js community. When I’m hacking on something, I start out with a single file to test something out. Then I progressively refine the example until it’s something reusable. That way, I can reuse it, others can reuse it, others can learn from it, and the world can be a better place.

For this tutorial, I’ll show you my process for creating a binding for nanomsg, the latest scalability protocol library from the creator of ZeroMQ, Martin Sústrik. I had played with ZeroMQ in the past and thought that it was really awesome, and I was excited to see a new library from it’s creator, based on C, since I also really enjoyed his post on why he shouldn’t have written it in C++.

So messing around real quick, let’s make sure we have node up to date. I like to use nvm and the latest stable minor version of node (stable versions have even minor patch numbers where versions are in the format major.minor.patch, so v0.11.0 is unstable). node -v –> v0.10.17

Then I need to download and install the library that I’ll be dynamically linking to, build, and install it.

1
2
3
4
5
6
7
8
curl -O http://download.nanomsg.org/nanomsg-0.1-alpha.zip && \
unzip nanomsg-0.1-alpha.zip && \
cd nanomsg-0.1-alpha && \
mkdir build && \
cd build && \
../configure && \
make && \
make install

We’ll use node’s FFI module to interface with the dynamically linked library, because it’s easier to write bindings than using native addons, and v8’s API has recently changed causing some headaches for native extensions.

npm install ffi

We’ll be writing the example in CoffeeScript.

npm install -g coffee-script

Now to mess around we can create main.coffee based on the C++ binding’s example:

main.coffee
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
ffi = require 'ffi'
assert = require 'assert'

AF_SP = 1
NN_PAIR = 16

nanomsg = ffi.Library 'libnanomsg',
  nn_socket: [ 'int', [ 'int', 'int' ]]
  nn_bind: [ 'int', [ 'int', 'string' ]]
  nn_connect: [ 'int', ['int', 'string' ]]
  nn_send: [ 'int', ['int', 'pointer', 'int', 'int']]
  nn_recv: [ 'int', ['int', 'pointer', 'int', 'int']]
  nn_errno: [ 'int', []]

# test
s1 = nanomsg.nn_socket AF_SP, NN_PAIR
assert s1 >= 0, 's1: ' + nanomsg.nn_errno()

ret = nanomsg.nn_bind s1, 'inproc://a'
assert ret > 0, 'bind'

s2 = nanomsg.nn_socket AF_SP, NN_PAIR
assert s2 >= 0, 's2: ' + nanomsg.nn_errno()

ret = nanomsg.nn_connect s2, 'inproc://a'
assert ret > 0, 'connect'

msg = new Buffer 'hello'
ret = nanomsg.nn_send s2, msg, msg.length, 0
assert ret > 0, 'send'

recv = new Buffer msg.length
ret = nanomsg.nn_recv s1, recv, recv.length, 0
assert ret > 0, 'recv'

console.log recv.toString()
assert msg.toString() is recv.toString(), 'received message did not match sent'

coffee main.coffee –> hello

This quick example shows that we have something working. Currently our working directory should look like:

1
2
3
4
5
6
7
tree -L 2
.
├── main.coffee
└── node_modules
    └── ffi

2 directories, 1 file

Turn an idea into a git repo

Next up is to create a repository using a version control system like git and start saving our work. Check in early, check in often.

Let’s add a .gitignore so that were not adding files that really don’t need to be committed. The node_modules folder is unnecessary because when this node module is installed, its dependencies will be recursively installed, so there’s no need to commit them to source control. The swap files are because I use vim and I accidentally commit the swap files from open buffers all the time like a noob.

.gitignore
1
2
node_modules/
*.swp

Let’s turn this into a git repo:

1
2
3
git init && \
git add . && \
git commit -am “initial commit”

Up on github, let’s create an uninitialized repo and push to it:

1
2
git remote add origin git@github.com:nickdesaulniers/node-nanomsg.git && \
git push -u origin master

So we have:

1
2
3
4
5
6
7
8
tree -L 2 -a
.
├── .gitignore
├── main.coffee
└── node_modules
    └── ffi

2 directories, 2 files

Add directory structure

Now that we have our repo under version control, let’s start adding some structure. Let’s create src/, lib/, and test/ directories. Our CoffeeScript will live in src/, compiled JavaScript will be in lib/, and our test code will be in test/.

mkdir src lib test

Split library functions from testing

Now let’s move a copy of main.coffee into src/ and one into test/. We are going to split the library definition away from the testing logic.

1
2
3
cp main.coffee test/test.coffee && \
git add test/test.coffee && \
git mv main.coffee src/nanomsg.coffee

This way git status tells us:

1
2
3
4
5
6
7
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
# renamed:    main.coffee -> src/nanomsg.coffee
# new file:   test/test.coffee
#

Let’s edit src/main.coffee to look like:

src/main.coffee
1
2
3
4
5
6
7
8
9
10
11
12
ffi = require 'ffi'

exports = module.exports = ffi.Library 'libnanomsg',
  nn_socket: [ 'int', [ 'int', 'int' ]]
  nn_bind: [ 'int', [ 'int', 'string' ]]
  nn_connect: [ 'int', ['int', 'string' ]]
  nn_send: [ 'int', ['int', 'pointer', 'int', 'int']]
  nn_recv: [ 'int', ['int', 'pointer', 'int', 'int']]
  nn_errno: [ 'int', []]

exports.AF_SP = 1
exports.NN_PAIR = 16

and edit the tests to:

test/test.coffee
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
assert = require 'assert'
nanomsg = require '../lib/nanomsg.js'

{ AF_SP, NN_PAIR } = nanomsg

s1 = nanomsg.nn_socket AF_SP, NN_PAIR
assert s1 >= 0, 's1: ' + nanomsg.nn_errno()

ret = nanomsg.nn_bind s1, 'inproc://a'
assert ret > 0, 'bind'

s2 = nanomsg.nn_socket AF_SP, NN_PAIR
assert s2 >= 0, 's2: ' + nanomsg.nn_errno()

ret = nanomsg.nn_connect s2, 'inproc://a'
assert ret > 0, 'connect'

msg = new Buffer 'hello'
ret = nanomsg.nn_send s2, msg, msg.length, 0
assert ret > 0, 'send'

recv = new Buffer msg.length
ret = nanomsg.nn_recv s1, recv, recv.length, 0
assert ret > 0, 'recv'

assert msg.toString() is recv.toString(), 'received message did not match sent'

Notice how in the test we’re including the compiled javascript from lib/ which doesn’t exist yet? If you try running coffee test/test.coffee it should crash. Let’s make the compiled version. coffee -o lib -c src/nanomsg.coffee

Once the compiled lib exists, we can run our tests with coffee test/test.coffee and shouldn’t see any errors.

Now we should have a little more order, let’s commit. Hold off on adding lib/ to version control, I’ll explain why in a bit.

1
2
3
4
5
6
7
8
9
10
11
12
13
tree -L 2 -C -a -I '.git'
.
├── .gitignore
├── lib
│   └── nanomsg.js
├── node_modules
│   └── ffi
├── src
│   └── nanomsg.coffee
└── test
    └── test.coffee

5 directories, 4 files

At this point, if we add features and want to rerun our tests, we need to execute:

coffee -o lib -c src/nanomsg.coffee && coffee test/test.coffee

While this command is simple now and easy to reverse search, anyone else contributing to you project is going to have to know the commands to run the tests. Let’s use Grunt, the JavaScript task runner, to automate our build and test process.

Add a Build script

1
2
npm install -g grunt-cli && \
npm install grunt-contrib-coffee

Create a simple Gruntfile which can also be written in CoffeeScript:

Gruntfile.coffee
1
2
3
4
5
6
7
8
module.exports = (grunt) ->
  grunt.initConfig
    coffee:
      compile:
        files:
          'lib/nanomsg.js': ['src/*.coffee']
  grunt.loadNpmTasks 'grunt-contrib-coffee'
  grunt.registerTask 'default', ['coffee']

Running grunt builds our lib which is a start, so let’s commit that.

But grunt is not running our tests. And our tests don’t have nice output. Let’s change that:

1
2
npm install -g mocha && \
npm install chai grunt-mocha-test

edit test/test.coffee to:

test/test.coffee
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
assert = require 'assert'
should = require('chai').should()
nanomsg = require '../lib/nanomsg.js'

describe 'nanomsg', ->
  it 'should at least work', ->
    { AF_SP, NN_PAIR } = nanomsg

    s1 = nanomsg.nn_socket AF_SP, NN_PAIR
    s1.should.be.at.least 0

    ret = nanomsg.nn_bind s1, 'inproc://a'
    ret.should.be.above 0

    s2 = nanomsg.nn_socket AF_SP, NN_PAIR
    s2.should.be.at.least 0

    ret = nanomsg.nn_connect s2, 'inproc://a'
    ret.should.be.above 0

    msg = new Buffer 'hello'
    ret = nanomsg.nn_send s2, msg, msg.length, 0
    ret.should.be.above 0

    recv = new Buffer msg.length
    ret = nanomsg.nn_recv s1, recv, recv.length, 0
    ret.should.be.above 0

    msg.toString().should.equal recv.toString()

and modify your gruntfile to add a testing step:

Gruntfile.coffee
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
module.exports = (grunt) ->
  grunt.initConfig
    coffee:
      compile:
        files:
          'lib/nanomsg.js': ['src/*.coffee']
    mochaTest:
      options:
        reporter: 'nyan'
      src: ['test/test.coffee']

  grunt.loadNpmTasks 'grunt-contrib-coffee'
  grunt.loadNpmTasks 'grunt-mocha-test'

  grunt.registerTask 'default', ['coffee', 'mochaTest']

Now when we run grunt, our build process will run, then our test process, then we should see one incredibly happy nyan cat. The nyan cat mocha test reporter is basically the pinnacle of human intellectual achievement.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
grunt
Running "coffee:compile" (coffee) task
File lib/nanomsg.js created.

Running "mochaTest:src" (mochaTest) task
 1   -__,------,
 0   -__|  /\_/\
 0   -_~|_( ^ .^)
     -_ ""  ""

  1 passing (5 ms)


Done, without errors.

Commit time.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
tree -L 2 -C -a -I '.git'
.
├── .gitignore
├── Gruntfile.coffee
├── lib
│   └── nanomsg.js
├── node_modules
│   ├── ffi
│   ├── grunt
│   └── grunt-contrib-coffee
├── src
│   └── nanomsg.coffee
└── test
    └── test.coffee

7 directories, 5 files

Create node module

Now that we have a more modular design with build and test logic built in, let’s make this module redistributable. First, let’s talk about ignoring files. Create a .npmignore file that will specify what not to include in the module that is downloaded. Node Package Manager, npm, will ignore a bunch of files by default for us.

.npmignore
1
2
3
Gruntfile.coffee
src/
test/

Here we’re ignoring the src/ dir, where in our .gitignore we are going to ignore lib/.

.gitignore
1
2
3
node_modules/
lib/
*.swp

Why are we doing this? Admittedly, none of this is strictly necessary, but here’s why I think it is useful. When someone is checking out the source, they don’t need the results of the compilation step, as they can make modifications and would need to recompile anyways. Adding lib/nanomsg.js would just be another thing to download (though its size is relatively insignificant). Likewise, when someone downloads the module, they most likely just want the results of the compilation step, not the source, build script, or test suite. If I was planned on making the compiled JavaScript accessible to a web browser, I would not add lib/ to .gitignore, that way it could be referenced from the github raw URL. Again, these are generalizations that are not always true. To make up for not having the entire source when installed as a module, we’ll make up for it by adding a link to the repo from of manifest, but first let’s commit!

Time to create a manifest file that has some basic info about our app. It’s a pretty good idea to run npm search <packagename> before hand to make sure your planned package name is not taken. Since we have all of our dependencies in a row, let’s run npm init.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sane defaults.

See `npm help json` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg> --save` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
name: (nanomsg)
version: (0.0.0)
description: nanomsg bindings
entry point: (index.js) lib/nanomsg.js
test command: grunt
git repository: (git://github.com/nickdesaulniers/node-nanomsg.git)
keywords: nanomsg
author: Nick Desaulniers
license: (BSD-2-Clause) Beerware
About to write to /Users/Nicholas/code/c/nanomsg/package.json:

{
  "name": "nanomsg",
  "version": "0.0.0",
  "description": "nanomsg bindings",
  "main": "lib/nanomsg.js",
  "directories": {
    "test": "test"
  },
  "dependencies": {
    "chai": "~1.7.2",
    "ffi": "~1.2.5",
    "grunt": "~0.4.1",
    "grunt-mocha-test": "~0.6.3",
    "grunt-contrib-coffee": "~0.7.0"
  },
  "devDependencies": {},
  "scripts": {
    "test": "grunt"
  },
  "repository": {
    "type": "git",
    "url": "git://github.com/nickdesaulniers/node-nanomsg.git"
  },
  "keywords": [
    "nanomsg"
  ],
  "author": "Nick Desaulniers",
  "license": "Beerware",
  "bugs": {
    "url": "https://github.com/nickdesaulniers/node-nanomsg/issues"
  }
}


Is this ok? (yes)

That should create for us a nice package.json manifest file for npm.

We can now run our tests with the command npm test in addition to grunt. Let’s hold off on publishing just yet, committing instead.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
tree -L 2 -C -a -I '.git'
.
├── .gitignore
├── .npmignore
├── Gruntfile.coffee
├── lib
│   └── nanomsg.js
├── node_modules
│   ├── .bin
│   ├── chai
│   ├── ffi
│   ├── grunt
│   ├── grunt-contrib-coffee
│   └── grunt-mocha-test
├── package.json
├── src
│   └── nanomsg.coffee
└── test
    └── test.coffee

10 directories, 7 files

Add LICENSE and README

So we have a module that’s almost ready to go. But how will developers know how to reuse this code? As much as I like to view the source, Luke, npm will complain without a readme. The readme also looks nice on the github repo.

# Node-NanoMSG
Node.js binding for [nanomsg](http://nanomsg.org/index.html).

## Usage

`npm install nanomsg`

```javascript
var nanomsg = require('nanomsg');
var assert = require('assert');
var AF_SP = nanomsg.AF_SP;
var NN_PAIR = nanomsg.NN_PAIR;
var msg = new Buffer('hello');
var recv = new Buffer(msg.length);
var s1, s2, ret;

s1 = nanomsg.nn_socket(AF_SP, NN_PAIR);
assert(s1 >= 0, 's1: ' + nanomsg.errno());

ret = nanomsg.nn_bind(s1, 'inproc://a');
assert(ret > 0, 'bind');

s2 = nanomsg.nn_socket(AF_SP, NN_PAIR);
assert(s2 >= 0, 's2: ' + nanomsg.errno());

ret = nanomsg.nn_connect(s2, 'inproc://a');
assert(ret > 0, 'connect');

ret = nanomsg.nn_send(s2, msg, msg.length, 0);
assert(ret > 0, 'send');

ret = nanomsg.recv(s1, recv, recv.length, 0);
assert(ret > 0, 'recv');

assert(msg.toString() === recv.toString(), "didn't receive sent message");
console.log(recv.toString());

Before we publish, let’s create a license file, because though we are making our code publicly viewable, public source code without an explicit license is still under copyright and cannot be reused.

LICENSE
1
2
3
4
5
6
7
8
/*
 * ----------------------------------------------------------------------------
 * "THE BEER-WARE LICENSE" (Revision 42):
 * <nick@mozilla.com> wrote this file. As long as you retain this notice you
 * can do whatever you want with this stuff. If we meet some day, and you think
 * this stuff is worth it, you can buy me a beer in return. Nick Desaulniers
 * ----------------------------------------------------------------------------
 */

If you want to be more serious, maybe instead shoot for an MIT or BSD style license if you don’t care what your repo gets used for or GPL style if you do. TLDRLegal has a great breakdown on common licenses.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
tree -L 2 -C -a -I '.git'
.
├── .gitignore
├── .npmignore
├── Gruntfile.coffee
├── LICENSE
├── README.md
├── lib
│   └── nanomsg.js
├── node_modules
│   ├── .bin
│   ├── chai
│   ├── ffi
│   ├── grunt
│   ├── grunt-contrib-coffee
│   └── grunt-mocha-test
├── package.json
├── src
│   └── nanomsg.coffee
└── test
    └── test.coffee

10 directories, 9 files

Publish

npm publish

1
2
3
4
5
6
7
8
9
npm http PUT https://registry.npmjs.org/nanomsg
npm http 201 https://registry.npmjs.org/nanomsg
npm http GET https://registry.npmjs.org/nanomsg
npm http 200 https://registry.npmjs.org/nanomsg
npm http PUT https://registry.npmjs.org/nanomsg/-/nanomsg-0.0.0.tgz/-rev/1-20f1ec5ca2eed51e840feff22479bb5d
npm http 201 https://registry.npmjs.org/nanomsg/-/nanomsg-0.0.0.tgz/-rev/1-20f1ec5ca2eed51e840feff22479bb5d
npm http PUT https://registry.npmjs.org/nanomsg/0.0.0/-tag/latest
npm http 201 https://registry.npmjs.org/nanomsg/0.0.0/-tag/latest
+ nanomsg@0.0.0

Finally as a sanity check, I like to make a new folder elsewhere, and run through the steps in the readme manually to make sure the package is reuseable. Which is good, since in the readme I accidentally forgot the nn_ prefix in front of errno and recv!

After updating the example in the readme, let’s bump the version number and republish. Use npm version without arguments to find the current version, then npm version patch to bump it. You have to commit the readme changes before bumping the version. Finally don’t forget to rerun npm publish.

Our final directory structure ends up looking like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
tree -L 2 -C -a -I '.git'
.
├── .gitignore
├── .npmignore
├── Gruntfile.coffee
├── LICENSE
├── README.md
├── lib
│   └── nanomsg.js
├── node_modules
│   ├── .bin
│   ├── chai
│   ├── ffi
│   ├── grunt
│   ├── grunt-contrib-coffee
│   └── grunt-mocha-test
├── package.json
├── src
│   └── nanomsg.coffee
└── test
    └── test.coffee

10 directories, 9 files

Lastly, I’ll reach out to Martin Sústrik and let him know that nanomsg has a new binding.

The bindings are far from complete, the test coverage could be better, and the API is very C like and could use some OO syntactic sugar, but we’re at a great starting point and ready to rock and roll. If you’d like to help out, fork https://github.com/nickdesaulniers/node-nanomsg.git.

What are some of your thoughts on build steps, testing, and directory layout of node module? This tutorial was definitely not meant to be an authoritarian guide. I look forward to your comments on your experiences!

Designated Initialization With Compound Literals in C

| Comments

Just a quick post on something I just discovered and found neat (I always find obscure C syntax interesting). I was trying to figure out how to use a C designated initializer, where a member was a pointer to another designated initializer. At this point, you need a compound literal. Just a quick background on C initialization:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
// verbosely create an array with a known size
int arr [3];
arr[0] = 1;
arr[1] = 2;
arr[2] = 3;
// => [1, 2, 3]

// concisely create an array with a known size
int arr [3] = { 1, 2, 3 }; // => [1, 2, 3]

// creates an array with unspecified values initialized to 0
int arr [4] = { 1, 2, 3 }; // => [1, 2, 3, 0]

// truncates declaration
int arr [1] = { 1, 2, 3 }; // => [1]

// based on number of initializers
int arr [] = { 1, 2, 3 }; // => [1, 2, 3]

Let’s look at how we might have initialized a struct in C89. In C89, you are required to declare local variables at the top of a block. A previous initialization of a point struct might have looked like:

1
2
3
4
5
6
7
8
9
struct point {
  int x, y;
};

{
  struct point a;
  a.x = 2;
  a.y = 3;
}

Just as we can define array literals in C using the initializer list syntax, we can use the same concise syntax for initializing structs!

1
2
// point a is located at (2, 3)
struct point a = { 2, 3 };

Well, this can be bad. Where would point a be located if say a fellow team mate came along and modified the definition of the point struct to:

1
2
3
struct point {
  int y, x; // used to be `int x, y;`
};

Suddenly point a points to (3, 2), not (2, 3). It’s better if you use designated initializers to declare the values for members of your struct. It’s up to the compiler to decide on the order of initialization, but it wont mess up where the data is intended to go.

1
2
// point b is located at (2, 3)
struct point b = { .y = 3, .x = 2 };

So now we have designated initializers, cool. What about if we want to use the same syntax to reassign point b?

1
2
3
b = { .x = 5, .y = 6 };
//    ^
// error: expected expression

While you are being explicit about the shape of the struct that you are trying to assign to b, the compiler cannot figure out that you’re trying to assign a point struct to another point struct. A C cast would probably help here and that’s what the concept of compound literals are.

1
b = (struct point) { .x = 5, .y = 6 }; // works!

Notice: I just combined a compound literal with a designated initializer. A compound literal on its own would look like:

1
b = (struct point) { 5, 6 }; // works!

To recap we can define points like so:

1
2
3
4
5
6
struct point a;
a.x = 1;
a.y = 2; // C89 (too verbose)
struct point b = { 3, 4 }; // initializer list (struct member order specific)
struct point c = { .x = 5, .y = 6 }; // designated initializer (non struct member order specific)
struct point d = (struct point) { .x = 7, .y = 8 }; // compound literal (cast + designated initialization)

My favorite part of compound literals is that you can define values inline of an argument list. Say you have a function prototype like:

1
2
3
4
5
6
7
8
9
10
int distance (struct point, struct point);

// instead of calling it like this
// (creating temp objects just to pass in)
struct point a = { .x = 1, .y = 2 };
struct point b = { .x = 5, .y = 6 };
distance(a, b);

// we can use compound literals
distance((struct point) { .x = 1, .y = 2 }, (struct point) { .x = 5, .y = 6 });

So compound literals help with reassignment of structs, and not storing temporary variables just to pass as function arguments. What happens though when one of your members is a pointer? C strings are easy because they already have a literal value:

1
2
3
4
5
6
7
8
9
10
struct node {
  char *value;
  struct node *next;
};

// just using designated initialization
struct node a = {
  .value = hello world,
  .next = NULL
};

But what happens if we want to initialize node.next? We could do:

1
2
3
4
5
struct node b = {
  .value = foo,
  .next = NULL
};
a.next = &b;

Again, we have to define b before assigning it to a.next. That’s worthwhile if you need to reference b later in that scope, but sometimes you don’t (just like how compound literals can help with function arguments)! But that’s where I was stumped. How do you nest designated initializers when the a member is a pointer to another designated initializer? A first naïve attempt was:

1
2
3
4
5
6
7
8
9
struct node c = {
  .value = bar,
  .next = {
    .value = baz,
//  ^
// error: designator in initializer for scalar type 'struct node *'
    .next = NULL
  }
};

WTF? Well, if you go back to the example with nodes a and b, we don’t assign the value of b to a.next, we assign it a pointer to b. So how can we use designated initializers to define, say, the first two nodes of a linked list? Compound literals. Remember, a compound literal is essentially a designated initialization + cast.

1
2
3
4
5
6
7
struct node d = {
  .value = qux,
  .next = &((struct node) {
    .value = fred,
    .next = NULL
  })
};

And that works, but why? d.next is assigned an address of a compound literal. Granted, you probably don’t want to be declaring your entire linked list like this, as nesting gets out of control fast. I really like this style because it reminds me of JavaScript’s syntax for declaring object literals. It would look nicer if all of your nested structs were values and not references though; then you could just use designated initializers and wouldn’t need compound literals or address of operators.

What’s your favorite or most interesting part of C syntax?

Acknowledgements:

Why I’ll Be Marching This 4th

| Comments

If you’ve done nothing wrong, then you’ve got nothing to hide.

Wrong. Nothing ever justifies giving up your human rights , especially to prove lack of wrong doing, and any government that asks you to do so is not your friend.

Terrorism has become a weapon used against us by those elected to lead to keep us compliant, like blinders you’d put on a horse. The threat just keeps growing and growing instead diminishing, despite the money and lives we throw at it. Terrorism is the new Communism, and the NSA PRISM program and the Patriot Act are the modern day overreactions equivalent to McCarthyism and the Red Scare.

A captain and crew of a boat would reflect upon the current state of their vessel and make corrections. The cycle of reflection and correction is what keeps the boat from running ashore, hitting other boats, or making other costly mistakes. Unfortunately, I feel that NSA, enabled by section 215 of the Patriot Act, have become so powerful that they feel they don’t need to reflect upon their actions anymore.

If your neighbor was peeking in your window at night, that would probably make you very angry, and you would probably never trust that neighbor again. Think about how all of our fellow countries feel about us right now. It’s really unfortunate that Germany and our allies now severely distrust us. I apologize to all non US citizens that my government doesn’t respect you enough to extend the human right to privacy to you, as it supposedly does to its own citizens, and I’m ashamed I have to apologize on behalf of them. It’s troubling that the NSA has trouble telling the truth. It’s awesome but ultimately embarrassing that Ecuador offers to fund Human Rights training to us.

It’s hypocritical that we frown upon countries like China spying on their citizens, like the East German Stasi, and yet we are doing it to our own people! While this administration may try its best to be benevolent with this power, who’s to say all future administrations will use it for good? Even without content you can glean an awful (unacceptable) amount of information about somebody.

So how did we get here? The justification I keep hearing is that this is all legal. Regardless of whether or not something is legal, we should be asking ourselves “but is this right?” and “what were the motivations of those that created this law? Was it to entrench the privileged or represent the constituents?” Good people not standing up for the right thing is what got us here. Good people not standing up at the Tech companies accused of handing over data to the NSA. Good people at the NSA not speaking out about their surveillance programs. Good people in Congress not speaking out about programs they’ve been briefed on. Everyone just keeps saying how they’re just complying with the law while looking the other way. Sounds to me like the crew hasn’t been doing enough reflection. What about the captain?

I think it’s important to recognize that you can’t have 100 percent security
and also then have 100 per cent privacy and zero inconvenience.

Hearing the President of our country say that, the incompetent media’s focus on Snowden, and the American people’s fundamental misunderstand of privacy, literally reminds me of the part in Star Wars where in order to stop a growing threat and protect the people, a democracy is turned into an authoritarian empire.

So this is how liberty dies…with thunderous applause?

Not on my watch. And not on yours either. Join us this 4th of July for a peaceful rally against infringements on our 4th Amendment rights as part of the national Restore the Fourth movement. Let’s ask the NSA to go back to the drawing board and find a way to uncompromisingly provide security and privacy.

8 Months in Mozilla

| Comments

Besides 3 months interrupted to finish degree requirements, and including an internship, I’ve been at Mozilla for about 8 months now. After reading a blog post of another software engineer’s experience at Microsoft, I count my blessings. Reading that article set off too many alarms in my head. It was well written, and I sugguest you go read it, but my takeaway was that that any big name corporation people dream about working at is actually quite dystopian, and I do not feel that that is the case here.


Expect no documentation in corporations

Expect documentation to be sparse or bit rotted

When developers are coding at an unhindered pace, writing documentation does suck. But once a project reaches a certain size, it cannot sustain growth without additional support from additional developers. When a project reaches this point, it’s worthwhile to step back from unbridled coding to write some docs for your new devs. This is really important for open source projects, to get help from volunteers. I’ve heard roughly 50% of Mozilla’s code comes from volunteers; so making their lives easier is a must. Also, writing documentation is a great way to start contributing to an open source code base. Documentation is a gift that keeps on giving. By keeping information to yourself, you have power over others, but that doesn’t help your organization move forward as a whole. Depending on the pace of code development, documentation tends to bit rot. I’m all for code that’s clear enough to be self documenting, but sometimes comment blocks that describe high level overviews of the code base’s architecture can really help newbies. When you update code, you should update the relevant tests and documentation.


It is not what you do, it is what you sell

It’s all about maintainability

I’ve had nightmare code reviews. Literally reviews so long I’ve had to print them out and cross them out one by one as I fixed them to help me keep track of what I had left to fix. But by working through them, I learned an awful lot about professional code development. It literally taught me to write some bulletproof code. While it was embarrassing for me at the time, and even a little now to admit, I’m glad because it helped me grow professionally at an accelerated pace and I can appreciate that. People come and go, but their code lives on. Think about this, some day you will die, but code you’ve written will probably outlast you. You wont be around to defend your coding decisions, so commit it right while you can. Imagine the poor bastard who has to maintain your code, why not try and make their life a little easier? As an engineer, ethics and morality are the Hippocratic Oath you answer to, not someone with a business title. If you think this is wrong, if you’re not allowed to offer feedback or criticize the decisions made by those above you, then you work for the wrong company.


Not everbody is passionate for engineering

Everyone here is super passionate about what they do

While a lot of people here have active family lives, they are all active Mozillians. I’ve yet to meet anyone who didn’t have a common ground in web based software (back end, front end, browser, OS, graphics, etc.) to speak the same language as me, but also acute knowledge in his or her specialty. By being surrounded be so many smart, enthusiastic people I can tell you all about JavaScript performance, about compiler front ends and back ends, about codecs, about layout and rendering, about handling billions of hits a day, etc. If you’re the smartest in the group, the group is dragging you down. I frequently feel like I’m not the smartest when surrounded by most of my peers, and I’m cool with that. I am young and naïve and have so much to learn, and Mozilla is an awesome place to grow.


2-3 hours of coding a day is great

Contribute at your pace

There’s no pressure here about time spent coding. Time spent or number of lines written is not a benchmark to live by, code quality is. I’m a five o’clock developer, but when I go home I’m reading up on various software topics and expanding my range of knowledge. I have a big bookshelf I pride myself in filling with various programming and hardware design related books. Mozilla provides all kinds of distractions (tech talks, food, pool table, video games), because comfortable developers are better developers. I went to a talk today where the senior employee talking told me that we should not waste our time in meetings if we feel they are meaningless. “Just walk out!”


Not giving back to the public domain is the norm

Giving back to the public domain is the norm

My very first week as an intern, I had just finished my first project. I approached my manager and said “Hey I’m going to publish this now on Github. That’s cool right?” There was an uncomfortably long pause from my manager, then “yeah man go for!” Literally everything I’ve ever worked on here has been published as open source. Writing software is about expression. To be able to publish my code freely, literally the reason the World Wide Web was created, allows me to be a part of someone else’s professional development; my expressions and code mannerisms living on in theirs. Probably the coolest story I have about working here on open source software happened a few weeks ago. It was a Friday afternoon and I was assessing issues in our issue tracker, deciding which I was going to tackle first on Monday. Monday morning I came in to a pull request with my name on it. Literally some person who had never spoken to me, nor any other core devs. Someone who didn’t live in the same country as me or possibly speak my language natively, they just picked up an issue and wrote a patch in a mutually common language: code. It didn’t matter that the volunteer was or wasn’t employed by Mozilla; they understood the problem they faced enough to fix it for themselves and everyone else. Anyone can partake in the development process and earn the priviledge and trust to become a core volunteer.


The world outside is not known here a lot

Everyone here is aware of the outside world

Seeing as we tend to wind up on HN et al almost daily, many of us frequent HN, Reddit, Twitter, and /.. It’s always interesting to see coworkers show up in comment threads. When you say something hurtful about Mozilla or Mozilla products in these forums, more than a few people here are going to see this and get their feelings hurt. It’s really unfortunate especially since our issue tracker is public, and it takes a few minutes to fill out a simple bug (and a few more for steps to reproduce) where we actually look for issues (and not on Twitter, for instance). Anyways, being aware of the new hotness has benefits again in terms of expression. So much of Rust is based on ideas from various languages. So much of ECMAScript 6 comes from CoffeeScript. So many libraries have interesting and useful abstractions. Saying “bah humbug new hotness bullshit” is naïve; new languages and frameworks became popular because people found that they made software development marginally better, and by closing your mind to them you loose out on being able to take those ideas and incorporate them in your current or favorite language/framework/project. If anything, being more informed about your enemy makes your argument stronger. I can’t tell you how many times I’ve heard someone talk shit about Ruby, yet when pressed didn’t have the slightest knowledge of the language. Yet I’ve had truly great debates with those well versed in Ruby. Try to only remark on positives, and lay off negatives unless they have actually bitten you.


Copy-pasting code can be okay

Measure once, cut twice [is wrong!]

Stack overflow is great. There’s no way to know every little bit of C or C++. JavaScript has some bad parts. But by copying and pasting code that was written from another context, you’re probably introducing bugs into your code. Don’t repeat yourself, and don’t cut corners just to get shit done.


Code reviews can be skipped

Code reviews are nothing but beneficial

If they are taking too long to the point where you’re considering cutting them out, your team is probably understaffed and overworked. Code reviews allow others to spot bugs that you may have overlooked. Seeing as they didn’t write the code, they can look at the code objectively without the same assumptions you made that may or may not actually be there. It also allows you to grow as a developer, assuming you take the lessons learned to heart. The reviewer may learn a thing or two. This should obviously occur less frequently (otherwise you have the lesser experienced coder reviewing the more experienced coder’s code), but you can still teach an old dog new tricks. If you get to a point where you’re getting too bogged down in code reviews, you probably haven’t been delegating enough or haven’t communicated well enough with your manager or the rest of the team.


Latest software, meh

Latest software, duh

Old, unpatched versions of popular software are susceptible to security flaws. Java especially. It may involve work to update your code for new environments, but it’s easier to bite the bullet in small increments frequently than it is to wait until it’s too late.


Your specialties usually do not matter

You were hired because of the need for someone on your team to have a certain expertise

If this is not the case, what the hell is your recruiting department doing? You know the part where start-ups say speed is everything? It’s because of the big established players not optimally allocating resources. Even if you’re not happy when you get here, you’re able to move or at least work on something else. I’ve had the freedom to work on Rust compiler, something I never would have dreamed of doing before and not something I’m an expert in but something that is mentally fulfilling and thought provoking. Many of the people here are famous for one thing or another and many of them work on things similar to what they’re known for, while others are free to start new or tackle existing projects. We frequently borrow people from other teams who are able to lend a hand in various aspects they specialize in.

Besides these responses, there’s just so much about my job that I love. The Mozilla Manifesto is my holy book. It makes me so happy to be a part of such an incredible organization. We have the transparency and accountability the government could only dream of. Instead of running from wrongdoing, we fight! We have so many interesting projects that anyone can get involved with! We have so many employees well known for their contributions. We have so much competition that keeps us humble. We have so many experienced and friendly senior software engineers who make great mentors to the new guard. We have so many people dedicated to users, who don’t compromise privacy for security (as those who do deserve neither). We have such an amazing community of staff, volunteers, reps, advocates, evangelists, users, localizers, testers, etc. who understand that they have the power to affect the trajectory of their favorite open source browser, mobile operating system, service, or library.

Again, I’m not making any of these points with your experiences in mind. I’m talking about what’s the norm here at my place of employment.

I’ve been told that working at Mozilla out of college spoils you. If that’s the case, if life at M$ is the norm, then my next job will have to be one of my own creation. I can only hope that a company I [will] create be in the image of Mozilla. Even then, I’ll still be a Mozillian; once a Mozillian, always a Mozillian. When people complain about their jobs, it’s completely foreign to me. You always have the power to change your situation, and I view disagreement to that point as self-doubt. I love the work that I do and if your company does something that you don’t agree with, then I challenge you to change your situation. Be that speaking out or moving on.


At the end

It’s about expressing your legacy; that will have repercussions that will shape our world. Chasing paychecks and enduring crappy working conditions is not a way to live. So what do you think, world traveler?

Rust: Pattern Matching and the Option Type

| Comments

The other day I was thinking about the function for performing dynamic memory allocation in the C standard library, malloc. From the manual pages, If successful, the malloc() function returns a pointer to allocated memory. If there is an error, it returns a NULL pointer and sets errno to ENOMEM. One of the most common errors when using malloc is not checking for allocation failure. The allocation is not guaranteed to succeed and trying to use a NULL reference can lead to program crashes.

So a common pattern we’ll see is:

C
1
2
3
4
5
6
7
8
9
int* nums = (int*) malloc(sizeof(int));
if (nums == NULL) {
  // handle error
} else {
  *nums = 7;
  // operate on nums
  free(nums);
  nums = NULL;
}

Here we allocated space for an integer, cast the void* returned from malloc to an int*, compared it against the NULL pointer, then freed the allocated memory and removed the reference.

One of the big problems with the null pointer has to do with safely dereferencing it. In C, dereferencing the null pointer is undefined and usually leads to a segfault and program crash.

It can be so unsafe to work with null pointers that C. A. R. Hoare refers to them as his billion-dollar mistake:

I call it my billion-dollar mistake. It was the invention of the null reference in 1965. At that time, I was designing the first comprehensive type system for references in an object oriented language (ALGOL W). My goal was to ensure that all use of references should be absolutely safe, with checking performed automatically by the compiler. But I couldn’t resist the temptation to put in a null reference, simply because it was so easy to implement. This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years.

So how do we represent the value of nothing? With an integer, for example, your first instinct might be to use 0 to refer to no value. But 0 is a value, so how do we represent the range of integer values but also whether there is a value or not?

Rust, a systems programming language with a focus on safety and concurrency, does not have the concept of a null pointer. Instead, it has a different construct to represent the absence of value, a whole other structure called an Option. It is an enumerated type that can either be None (no value) or Some(T) (a specialization of type T).

So what the heck is an option type and how do we use it? The option type is a polymorphic type (generic type that can be specialized) that encapsulates either an empty constructor or the constructor of the original data type. Let’s take a trip down the rabbit hole to see how we use one. First, let’s look at some C++ code, and then translate it to Rust.

option.cpp
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#include <iostream> // cout, endl
#include <stdlib.h> // rand
#include <time.h> // time
using namespace std;

// If there is a 'random error' returns NULL
const int* const may_return_null () {
  srand(time(NULL));
  return rand() % 2 == 1 ? new int(666) : NULL;
}

int main () {
  // if else
  const int* const x = may_return_null();
  if (x) {
    // switch
    switch (*x) {
      case 777: cout << "Lucky Sevens"        << endl; break;
      case 666: cout << "Number of the Beast" << endl; break;
      case 42: cout  << "Meaning of Life"     << endl; break;
      default: cout  << "Nothing special"     << endl; break;
    }
  } else {
    cout << "No value" << endl;
  }

  // single if
  if (*x == 666) {
    cout << "Did I mention that Iron Maiden is my favorite band?" << endl;
  }
}

Let’s step through this program line by line, starting in main.

Line 14
1
const int* const x = may_return_null();

Here we’re calling a function that may return null, just like malloc!

Lines 8-9
1
2
srand(time(NULL));
return rand() % 2 == 1 ? new int(666) : NULL;

In the body of may_return_null we seed the random number generator, generate a random number, mod it by 2 (so it can either be 0 or 1, 50-50 chance, hopefully), then either return a pointer pointing to memory allocated on the heap or the null pointer. We also use the succinct ternary operator, which gives us the power of a conditional statement in the form of a concise expression.

Line 15
1
if (x) {

We check if the pointer is valid, that is that it is safe to use, as NULL is falsy in a C++ conditional. If it is, then we can safely dereference it. Let’s switch on the (dereferenced) value. Notice how we need to break to explicitly prevent fall through, though 97% of the time that’s what you intend.

Lines 17-22
1
2
3
4
5
6
switch (*x) {
  case 777: cout << "Lucky Sevens"        << endl; break;
  case 666: cout << "Number of the Beast" << endl; break;
  case 42: cout  << "Meaning of Life"     << endl; break;
  default: cout  << "Nothing special"     << endl; break;
}

If the pointer was null, the else branch of the conditional would execute printing a different result.

Line 24
1
cout << "No value" << endl;

Finally in we check the value pointed to. Did I forget something here? Save that thought, we’ll come back to it.

Lines 28-30
1
2
3
if (*x == 666) {
  cout << "Did I mention that Iron Maiden is my favorite band?" << endl;
}

Let’s see my rough translation of this program into Rust.

option.rs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
use core::rand;

fn may_return_none () -> Option<int> {
  if rand::Rng().next() % 2 == 1 { Some(666) } else { None }
}

fn main () {
  let x: Option<int> = may_return_none();

  io::println(match x {
    Some(777) => "Lucky Sevens",
    Some(666) => "Number of the Beast",
    Some(422) => "Meaning of Life",
    Some(_) => "Nothing special",
    None => "No value"
  });

  match x {
    Some(666) => {
      io::println("Did I mention that Iron Maiden is my favorite Band?")
    },
    _ => {}
  };
}

Both programs, when compiled should randomly print either:

option.rs
1
2
3
4
No Value
// or
Number of the Beast
Did I mention that Iron Maiden is my favorite Band?

Now let’s walk through the Rust code, starting at main and compare it to the equivalent C++ code.

option.rs Line 8
1
let x: Option<int> = may_return_none();
option.cpp Line 14
1
const int* const x = may_return_null();

I read this Rust code as ‘let x be type Option, specialized to type int initialized to the return value of may_return_none.’ I read the C++ as ‘x is a const pointer to a const integer initialized to the return value of may_return_none.’ It’s important to note that values and pointers in Rust default to being immutable, where as in c++ they default to being mutable, which is why we need to be explicit about their const’ness. In Rust, we could declare x as being explicitly mutable: let mut x: ... = ...;. In both, we can also leave the explicit types to be inferred by the compiler.

option.rs Line 4
1
if rand::Rng().next() % 2 == 1 { Some(666) } else { None }
option.cpp Lines 8-9
1
2
srand(time(NULL));
return rand() % 2 == 1 ? new int(666) : NULL;

Now for the body of may_return_none. Rust does not have a ternary operator, so a single line if {} else {} block will have to do. We also don’t need parentheses around the predicate in the conditional statement. If we add them, no harm is done, as we’d just be being redundant about order of operations. There are no semicolons in this expression, nor a return statement in this function. Rust will return the last expression in a function or method if the semicolon is left off. Also there is no such thing as a conditional statement, only conditional expressions; the if is itself something that can be evaluated and assigned to a variable! We see this later in the code where we pass the evaluation of a match expression directly into a call of io::println. Rust returns the evaluation of the if expression, which is the evaluation of the branch’s block specified by the predicate, which will randomly be one of the two enumerated types of Option<int>, either an encapsulation of a int whose literal value is 666, Some<666>, or the representation of no value, None. I believe the RNG code has changed in 0.7, this code was written in 0.6.

In Rust, to get the value out of an option type, we pattern match on it. Pattern matching is something I became familiar with through the Haskell programming language. Pattern matching is a powerful language construct that can entirely replace conditionals. In fact, the one line if expression could have been written using a match expression:

1
2
3
if rand::Rng().next() % 2 == 1 { Some(666) } else { None }
// or
match rand::Rng().next() % 2 { 1 => Some(666), _ => None }

The basic design pattern for accessing the value of an option type in Rust looks like:

1
2
3
4
match x { // x: Option<T>
  Some(y) => { *y },
  None => {}
}

The curly braces are optional for one liners (no block needed). Pattern matches have to be exhaustive. That means I have to exhaust all possibilities for what the deconstructed value could be. You can use a branch that looks like:

1
_ => 'everything else'

to catch everything else. The underscore here means “every other possible case.” So by having to use a match expression (also not a statement, as opposed to C++’s switch statement), which itself must be exhaustive, Rust forces us to handle the case where the optional type is None! This will help us again in the future. We also don’t need parentheses around the predicate for the match expression, which in this case is just a single variable, x.

In the value Some(_), the underscore has an additional meaning here that we would not use a variable in the corresponding arm’s block. If we declared it as Some(y) we would get the warning:

1
2
3
option.rs:14:9: 14:11 warning: unused variable: `y`
option.rs:14     Some(y) => "Nothing special",
                      ^~

I hope back in my C++ code you spotted the fatal flaw. On line 28, I just dereferenced a raw pointer without checking its validity. This is a violation of memory safety.

option.cpp Line 28
1
if (*x == 666) {

When running the C++ code, instead of seeing No value printed to stdout in the case of no value, a segfault occurs.

option.cpp Line 28
1
2
No value
[1]    80265 segmentation fault  ./option

What I should have done is something more like:

Line 28 corrected
1
if (x && *x == 666) {

But, the C++ compiler let me get away with not handling the case where the pointer was invalid (even if doing nothing in the case of “handling” it). By leaving out the check for a valid pointer, I have instructed the machine to behave the same or follow the same code path with and without a reference to a valid memory location. Let’s see what happens when I don’t handle the None case by deleting line 15 or option.rs, None => "No value":

Line 28 corrected
1
2
3
4
5
6
option.rs:10:14: 16:3 error: non-exhaustive patterns: None not covered
option.rs:10   io::println(match x {
option.rs:11     Some(777) => "Lucky Sevens",
option.rs:12     Some(666) => "Number of the Beast",
option.rs:13     Some(42) => "Meaning of Life",
option.rs:14     Some(_) => "Nothing special"

Not only did the compiler prevent me from generating an executable, it told me that a pattern was not exhaustive, explicitly which one, and what case that was not covered.

Coming back to encoding a lack of value for an int, we had left off that 0 is a valid value. For instance, how should we represent any other integer divided by 0, as an integer? Signed integers use a single bit to store whether their value is positive or negative, the tradeoff being the signed integers represent up to one less power of two than unsigned integers. Maybe we could use an additional bit to represent valid or invalid, but this would again cost us in terms of representable values. The IEEE 754 floating point representation has encodings for plus and minus Infinity, plus and minus 0, and two kinds of NaN. To solve the validity problem, we can use enumerated types, which in Rust occur as specializations of the Option type. It’s up to the compiler to implement either additional information for the type, or use raw null pointers. And to get the value back out of the Option type, we must handle the case where there is no valid value.

The key takeaways that I wish to convey are:

  1. In C and C++, I can use NULL to refer to a pointer that has no value.
  2. A C/C++ compiler, such as clang, will allow me to compile code that violates memory safety, such as dereferencing NULL pointers.
  3. Rust instead uses Option types, an enumeration of a specialized type or None.
  4. Rust forces me to use a match statement to access the possible value of an Option type.
  5. Pattern matching in Rust must be exhaustive.
  6. The Rust compiler, rustc, forces me to handle the case where the pointer has no value, whereas the C++ compiler, clang++, did not.
  7. All conditional and switch statements can be replaced with pattern matching.

The Persistence of Memory

| Comments

I would like to die on Mars…

Well, isn’t that forward thinking? Granted, the full quote I would like to die on Mars, just not on impact is meant to sound hopeful of his company, SpaceX. I agree that some day, humans will be buried on Mars. But is it forward thinking, enough? Does it push the needle to where it needs to be pushed? Does it strive for innovation and push our imagination like Star Trek did? Or is it just a realistic goal for SpaceX?

The person with big dreams is more powerful than one with all the facts.

Albert Einstein

While Elon would like to die on Mars, I think it would be neat to transfer my “soul” into a computer. We refer to data persistence within a computer as memory, using transistors and capacitors to store charges, effectively “remembering” a particular state. While neurons, dendrites, and synapses within the human brain don’t quite work the same as RAM, I believe we will find a way to interface them some day. If one could transfer to a computer, maybe they could transfer back, or swap even. One could theoretically trade entire bodies. The jury is still out for me as far my opinion of Google’s Glass, but I see wearable computing as a step in the right direction towards interfacing with the human body. I guess it reminds me of the Heads in Jars from Futurama. I recognize that the old guard must give way for the new guard, lest the new be held in place, bound socially by the old.

In database systems, there is this notion of Atomicity, where a set of operations is guaranteed to either complete as a group, or all fail should at least one fail. One of the most popular examples is a bank transfer. Say you’re transferring $100 from Bank A to Bank B. Bank A’s database should not dock your account $100 until Bank B has added $100, and vice versa. Imagine if the power went out after Bank A docked your account, but before Bank B added to your account; that money could just disappear. Bank B may have received it but not credited your account, or Bank A may have it, just not a record that it’s yours. You may have just doubled your money, or lost all of it. The notion of Atomicity provides for “transactions,” or sets of instructions that are guaranteed to all pass or all fail.

Atomicity would be important for a memory transference device. Once you were transferred into a machine, would it still be “you” if you remained in your body? A physical clone, is as much you as an identical twin, but will never have the same memories and experiences that shape their mind. If you merely copied your mind to a machine rather than transferred it, then you would still remained trapped in your rotting piece of meat.

This is what I mean by transfer my “soul.” An actual atomic transaction moving all of your memories and experiences from body to machine, all at once, or not at all. I probably wouldn’t want to be the first volunteer for such a machine, because if the machine failed at the atomic transaction, your soul would either be as lost as the $100 from the earlier example, or you’d just be copied, not moved.

In a world where diseases are too profitable to cure, I can imagine a world where the richest 1% live forever while the poor are forced to die. Frozen Walt Disney just sat up. An abuse of the technology, not too different a world from the one in Surrogates.

Have you not done tormenting me with your accursed time! It’s Abominable! When! When! One day, is that not enough for you, one day he went dumb, one day I went blind, one day we’ll go deaf, one day we were born, one day we shall die, the same day, the same second, is that not enough for you? They give birth astride a grave, the light gleams an instant, then it’s night once more.

Pozzo Waiting for Godot by Samuel Becket

They say that it is fear of death and what comes after death that makes men turn to religion.

Mustafa Mond Brave New World by Alduos Huxley

Ever since my father passed away, I’ve occasionally suffered from existential anxiety attacks. It usually starts with me thinking about those Discovery Channel shows about how the old the universe is, and how old it will become. Then my mind starts to think of all of the really cool technological advancements that have occurred in my lifetime, and how I wont be around to experience all future advancements, like a game that others get to play while I have to sit on the sidelines and watch. Of course, if I’m dead then I guess I won’t care. Currently, I calm myself down my telling myself that such an event is a long time away and I will live a long and happy life with many descendants, though I have no guarantees. My father only lived to be 48.

‘That’s such a weak excuse,’ said Valentine. ‘Everyone dies. Everyone leaves. What matters is the things you build together before they go. What matters is the part of them that continues in you when they’re gone.’

Valentine Children of the Mind by Orson Scott Card

Where before they would bloom and wither in the space of a single day, now they hold their blooms for three and four days at a time. Moistened by the dew of night, bathed in the light of the sun, the white flowers strive to live their lives to the fullest, beautifying the town as if striving to live out the portion of life denied to those whose “tomorrows” were snatched away from them forever.

White Flowers Lost Odyssey

Human beings, who cannot live forever, daring to take a journey without end. This might be the ultimate tragedy, but it could just as well be the ultimate comedy. Kaim knows one thing, however: one cannot simply dismiss it as an exercise in futility.

The Upstreamers Lost Odyssey

Randy Pausch’s The Last Lecture taught me that legacy is the greatest thing you can leave behind. While I haven’t yet started the business I’ve always wanted, my first legacy I will leave my kiddos is my collection of Iron Maiden vinyl records. They and their children will probably think I was nuts, but there is the slim chance they might really appreciate my taste in music. I also want to write a book someday, that way I can marvel at how my ideas will outlast me.

I like to think that something survives after you die, it’s strange to think that you accumulate all this experience, and maybe a little wisdom, and it just goes away. So I really want to believe that something survives, that maybe your consciousness endures. But on the other hand, perhaps it’s like an on-off switch. “Click!” And you’re gone.

Steve Jobs Steve Jobs by Walter Isaacson

It terrifies me to no end to think what could possibly drive someone to suicide. What doom a person would have to be surrounded with to honestly believe that no matter what they did, that they could not change their current situation. I believe the journey is ultimately worth it. It terrifies me to think of all of the amazing minds this world has seen, among so many who choose not to develop their own, that have come and gone. Those who could change the world yet are snatched away.

This body holding me reminds me of my own mortality. Embrace this moment. Remember. We are eternal. All this pain is an illusion.

Parabola Tool

I’m sorry this post turned a little dark. By talking about my fears, I hope to not let them control me. I believe publishing this will be a relief. I think I’ll join Elon on Mars, but unlike Elon, I believe I will transcend my physical being. It’s important therefor, to treasure your body while it lasts.

‘But it [life] won’t have amounted to anything,’ said Grego. ‘If your children die, then it was all a waste.’ ‘No’ said Olhado quietly. Grego shook his head. ‘Yes it does, Olhado. Death undoes everything.’ Olhado shrugged. ‘Then why do you bother doing everything, Grego? Because someday you will die. Why should anyone ever have children? Someday they will die, their children will die, all children will die. Someday stars will wind down or blow up. Someday death will cover us all like the water of a lake and perhaps nothing will ever come to the surface to show that we were there. But we were there, and during the time we lived, we were alive. That’s the truth-what is, what was, what will be-not what could be, what should have been, what never can be. If we die, then our death has meaning to the rest of the universe. Even if our lives are unknown, the fact that someone lived here, and died, that will have repercussions, that will shape the universe.’

Children of the Mind Orson Scott Card

There will allways be hope, wherever you are, until you yourself abondon it.

Old Revolutionary Lost Odyssey

Basic JIT

| Comments

Ever since I learned about Just In Time Compilation from the various Ruby VMs and JavaScript VMs, I’ve been inspired. I could tell you all about how just in time (JIT) compilation worked, and how it could give your interpreted language a speed boost. It was so cool. Well, it still is! There’s a ton of research going on around JIT compilation. But the problem for me is that I could never figure out, let alone guess, how it was being done. How could you compile code at runtime, and then execute it? I asked Benjamin Peterson about how he had learned so much about JITs, and he referred me to pypy (a Python JIT)’s source. But digging through source was too much; I just wanted a simple example I could grok quickly. So I’ve always been left wondering.

Luckily, I have an awesome job where I can meet face to face with the people who are doing the work on making awesome production JIT compilers. I spent part of last week at GDC demoing Unreal Engine 3 running in the browser. The demo was actually the hard work of many, and I’ll dive into it more in another post following up the events, but Luke Wagner, a Mozillian working on the JavaScript engine, added OdinMonkey to SpiderMonkey to allow optimizations of asm.js.

Luke is super friendly, and just listening to him talk with Dave Herman and Alon Zakai is a treat. I asked Luke the basics of JIT’ing at tonight’s Mozilla Research Party and he explained clearly. Compile a simple object file, use objdump to get the resulting platform specific assembly, use the mmap system call to allocate some memory that you can write to AND execute, copy the instructions into that buffer, typecast it to a function pointer and finally call that.

So my goal was to at runtime, create a function that for simplicity’s sake multiplied two integers. The first thing I did was write up a simple .c file, then compile that to an object file with the -c flag.

1
2
3
4
// Compile me with: clang -c mul.c -o mul.o
int mul (int a, int b) {
  return a * b;
}

As a heads up, I’m on 64bit OSX. So the generated assembly may differ on your platform. Obviously, the production JIT maintainers have abstracted away the platform dependency, taking into account what platform you’re on. My example doesn’t, but that why I’m walking you through the steps I took to find out the x86_64 instructions. The next step is to grab binutils, which is not installed by default in OSX. I used homebrew to install it: brew install binutils. Homebrew installs gobjdump but it works with the same flags.

Once you have binutils and [g]objdump, the next step is to read out the machine code from your object file, which is represented in hexadecimal. By running gobjdump -j .text -d mul.o -M intel you should get something similar (remember, architecture dependent).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ gobjdump -j .text -d mul.o -M intel

Disassembly of section .text:

0000000000000000 <_mul>:
0: 55 push rbp
1: 48 89 e5 mov rbp,rsp
4: 89 7d fc mov DWORD PTR [rbp-0x4],edi
7: 89 75 f8 mov DWORD PTR [rbp-0x8],esi
a: 8b 75 fc mov esi,DWORD PTR [rbp-0x4]
d: 0f af 75 f8 imul esi,DWORD PTR [rbp-0x8]
11: 89 f0 mov eax,esi
13: 5d pop rbp
14: c3 ret

Ok, so those instructions vary in size. I don’t know any x86 so I can’t comment too much on that particular Instruction Set Architecture but they’re obviously in pairs of hexadecimal digits. 162 == 28 meaning that each pair of hex digits can be represented by a single byte (or char of memory). So these can all be thrown in an unsigned char []. The man page for mmap explains all of the fun flags, but the important point is that this way of allocating memory makes it executable, so it can do bad things that memory allocated from malloc can’t. I’m sure the JavaScript engine guys have fun with that. Once memory is copied in, you can typecast the memory to a function pointer. Make sure to check out the syntax that reminds me of a function pointer in an argument list, but being used as an L-value. Of course, you could just put the cast right in front of the memory before you use it, but I find this as neat, not so common C syntax. I kind of have a thing for stuff like that. Then we can call it! The resulting code looks like this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
#include <stdio.h> // printf
#include <string.h> // memcpy
#include <sys/mman.h> // mmap, munmap

int main () {
// Hexadecimal x86_64 machine code for: int mul (int a, int b) { return a * b; }
unsigned char code [] = {
  0x55, // push rbp
  0x48, 0x89, 0xe5, // mov rbp, rsp
  0x89, 0x7d, 0xfc, // mov DWORD PTR [rbp-0x4],edi
  0x89, 0x75, 0xf8, // mov DWORD PTR [rbp-0x8],esi
  0x8b, 0x75, 0xfc, // mov esi,DWORD PTR [rbp-04x]
  0x0f, 0xaf, 0x75, 0xf8, // imul esi,DWORD PTR [rbp-0x8]
  0x89, 0xf0, // mov eax,esi
  0x5d, // pop rbp
  0xc3 // ret
};

  // allocate executable memory via sys call
  void* mem = mmap(NULL, sizeof(code), PROT_WRITE | PROT_EXEC,
                   MAP_ANON | MAP_PRIVATE, -1, 0);

  // copy runtime code into allocated memory
  memcpy(mem, code, sizeof(code));

  // typecast allocated memory to a function pointer
  int (*func) () = mem;

  // call function pointer
  printf("%d * %d = %d\n", 5, 11, func(5, 11));

  // Free up allocated memory
  munmap(mem, sizeof(code));
}

Voila! How neat is that! It works! Luke’s directions, an hour of working on this, and this article in particular, and we have a simple JIT working. Again, this simple example is super non-portable and dealing with memory in this fashion is generally unsafe. But now I know the basics of JIT’ing code, and now so do you!

Thanks Luke!