Brain dump in text format

Technology / Software / Hacks by Jason Mobarak

Prius Total Cost of Ownership

| Comments

Disclaimer: I like our Prius Plug-in, but I really don’t like our Prius C. The plug-in version came with a little green sticker that lets you whiz by other people that didn’t think to plop down the extra dough for a “range extended” electric (which the Prius Plug-in really is not that, but I digress).

First, a question for Prius owners: do you think the $2500-$4000 cost of a replacement batter pack at 9 years would nullify your fuel saving over that time period?

For us, the answer is mostly no, but it depends on what you drove before (or what you would drive instead). Versus the 2000 Passport (driving 350 miles a week for commuting), which has a 21 gallon tank (that we usually filled at about 16 gallons). It would cost about $26k in fuel over 9 years (16g/w * $3.50/g * 52w/y * 9y) versus the Prius’ $14k (8.5g/w * $3.50/g * 52w/y * 9y).

It does put a dent in the savings though, and when compared to other high mpg cares the “win” isn’t as clear. If you consider something like Ford’s EcoBoost which get’s good milage and performance from tiny engines, it probably would not win, this is assuming the total cost of ownership for a Ford is similar to the Toyota, but maybe that’s a laughable thing to assume…

If you compare it to a Camry, the Prius doesn’t seem to win. Camry is a Toyota, so you’re safer in assuming the platform’s reliability and TCO are similar. For me the Camry would be about $35 a week in gas, or $16k over a 9 year period, which clearly does not win over the Prius plus replacing it’s battery pack.

Least Bits to Transmit a Permutation

| Comments

Consider the 52 cards of a deck. You generated a random sequence for these cards and want to send that sequence to a receiver. You want to minimize the communication between you and the receiver, i.e., minimize the number of bits required to send the sequence.

What is the minimum number of bits required to send the sequence?

To arrive at the solution, I’m going to consider a smaller example: What are all the permutations of [0, 1, 2, 3]?

[0,1,2,3] => perm 00 => binary: 0 0 0 0 0   || [0,3,1,2] => perm 12 => binary: 0 1 1 0 0
[0,2,1,3] => perm 01 => binary: 0 0 0 0 1   || [0,3,2,1] => perm 13 => binary: 0 1 1 0 1
[1,0,2,3] => perm 02 => binary: 0 0 0 1 0   || [1,3,0,2] => perm 14 => binary: 0 1 1 1 0
[1,2,0,3] => perm 03 => binary: 0 0 0 1 1   || [1,3,2,0] => perm 15 => binary: 0 1 1 1 1
[2,0,1,3] => perm 04 => binary: 0 0 1 0 0   || [2,3,0,1] => perm 16 => binary: 1 0 0 0 0
[2,1,0,3] => perm 05 => binary: 0 0 1 0 1   || [2,3,1,0] => perm 17 => binary: 1 0 0 0 1
[0,1,3,2] => perm 06 => binary: 0 0 1 1 0   || [3,0,1,2] => perm 18 => binary: 1 0 0 1 0
[0,2,3,1] => perm 07 => binary: 0 0 1 1 1   || [3,0,2,1] => perm 19 => binary: 1 0 0 1 1
[1,0,3,2] => perm 08 => binary: 0 1 0 0 0   || [3,1,0,2] => perm 20 => binary: 1 0 1 0 0
[1,2,3,0] => perm 09 => binary: 0 1 0 0 1   || [3,1,2,0] => perm 21 => binary: 1 0 1 0 1
[2,0,3,1] => perm 10 => binary: 0 1 0 1 0   || [3,2,0,1] => perm 22 => binary: 1 0 1 1 0
[2,1,3,0] => perm 11 => binary: 0 1 0 1 1   || [3,2,1,0] => perm 23 => binary: 1 0 1 1 1

How many bits do you need to represent a number in general? Another way of phrasing: how many digits do I need to represent this number? Take a base 10 number, say 123, written as a function combination of it’s bases: (10^2 * 1) + (10^1 * 2) + (10^0 * 3) = 100 + 20 + 3. So, the largest component is 100, and we need at least 3 digits (or 1 + the power of largest component).

So, 10N = X, we want to know what N is, thus, log_10(X) = N. This holds for base 2 numbers, thus:

log_2(X) = N

So, if we have 24 permutations (4 factorial), we need this many bits:

log_2(24) = 4.58; ceil(4.58) = 5

Since we can’t have partial bits we round up… and arrive at 5.

If we need only 5-bits to number each sequence, we could just store a lookup table… but for larger inputs, you cannot just transmit a sequence number and have the remote end look-up which permutation to use in a lookup table, this would use too much memory…

However, what if we choose to map these as choices, noting that choices cannot be repeated and the that pool of choices shrinks each time. Since the pool is [0, 1, 2, 3], the choices would be for example:

                       Sequence     Choices
                       --------     -------
Perm 1 -                            [0,1,2,3]
       - choose(00) => [0]          [1,2,3]
       - choose(00) => [0,1]        [2,3]
       - choose( 0) => [0,1,2]      [3]
       - choose( _) => [0,1,2,3]    []

    binary: 000 00 => decimal: 0

---------------------------------------------

                       Sequence     Choices
                       --------     -------
Perm 2 -                            [0,1,2,3]
       - choose(00) => [0]          [1,2,3]
       - choose(01) => [0,2]        [1,3]
       - choose( 0) => [0,2,1]      [3]
       - choose( _) => [0,1,2,3]    []

    binary: 00 01 0 => decimal: 2

Then, we can rewrite the list of permutation like this:

Permutation   Choices    |  Permutation   Choices
-----------   ---------  |  -----------   ---------
[3,2,1,|]     [D,C,A,B]  |  [1,2,1,|]     [B,D,A,C]
[3,2,0,|]     [D,C,B,A]  |  [1,2,0,|]     [B,D,C,A]
[3,1,1,|]     [D,B,A,C]  |  [1,1,1,|]     [B,C,A,D]
[3,1,0,|]     [D,B,C,A]  |  [1,1,0,|]     [B,C,D,A]
[3,0,1,|]     [D,A,B,C]  |  [1,0,1,|]     [B,A,C,D]
[3,0,0,|]     [D,A,C,B]  |  [1,0,0,|]     [B,A,D,C]
[2,2,1,|]     [C,D,A,B]  |  [0,2,1,|]     [A,D,B,C]
[2,2,0,|]     [C,D,B,A]  |  [0,2,0,|]     [A,D,C,B]
[2,1,1,|]     [C,B,A,D]  |  [0,1,1,|]     [A,C,B,D]
[2,1,0,|]     [C,B,D,A]  |  [0,1,0,|]     [A,C,D,B]
[2,0,1,|]     [C,A,B,D]  |  [0,0,1,|]     [A,B,C,D]
[2,0,0,|]     [C,A,D,B]  |  [0,0,0,|]     [A,B,D,C]

Writing it this way, a pattern emerges… the first column is never greater than 3, the second is never great than 2, the third is only every 1 or 0.

We can map these choices into a number that uses only 5-bits by noticing:

Always 0-3 for most significant
  [00, 01, 10, 11] => [0, 1, 2, 3]

Then always 0-2 for next bits
  [00, 01, 10] => [0, 1, 2]

Then only 1 or 0 left:
  [0, 1] => [0, 1]

So, the bit pattern is roughly:
  ZZ YY X (Z = first choice, Y = second choice, X = final choice)

The “bit pattern” idea isn’t really a good model of the general idea, since we’ll be spreading ZZ portion into the coded sequence number. Let max = 3, to prepare the first choice to be mapped we multiply the choice by max, then we can use integer division to recover the number:

b000 * 3 => b0000 (0)
b001 * 3 => b0011 (3)
b010 * 3 => b0110 (6)
b011 * 3 => b1001 (9)

Now, we can add the second choice to this number, if we mod by 3, we’ll get back the second choice, if we use integer division by 3, we’ll get back the first choice:

(b0000 + b00) => b0000 ( 0) | mod 3 => 0 | div 3 => 0
(b0000 + b01) => b0001 ( 1) | mod 3 => 1 | div 3 => 0
(b0000 + b10) => b0010 ( 2) | mod 3 => 2 | div 3 => 0

(b0011 + b00) => b0011 ( 3) | mod 3 => 0 | div 3 => 1
(b0011 + b01) => b0100 ( 4) | mod 3 => 1 | div 3 => 1
(b0011 + b10) => b0101 ( 5) | mod 3 => 2 | div 3 => 1

(b0110 + b00) => b0110 ( 6) | mod 3 => 0 | div 3 => 2
(b0110 + b01) => b0111 ( 7) | mod 3 => 1 | div 3 => 2
(b0110 + b10) => b1000 ( 8) | mod 3 => 2 | div 3 => 2

(b1001 + b00) => b1001 ( 9) | mod 3 => 0 | div 3 => 3
(b1001 + b01) => b1010 (10) | mod 3 => 1 | div 3 => 3
(b1001 + b10) => b1011 (11) | mod 3 => 2 | div 3 => 3

To map the final choice into this number we multiply each of these numbers by max-1 = 2, and add in the final choice:

(b0000 * 2) + 0 => b0000 00 ( 0) | (b0110 * 2) + 0 => b0001 10 (12)
(b0000 * 2) + 1 => b0000 01 ( 1) | (b0110 * 2) + 1 => b0001 11 (13)

(b0001 * 2) + 0 => b0000 10 ( 2) | (b0111 * 2) + 0 => b0011 10 (14)
(b0001 * 2) + 1 => b0000 11 ( 3) | (b0111 * 2) + 1 => b0011 11 (15)

(b0010 * 2) + 0 => b0001 00 ( 4) | (b1000 * 2) + 0 => b1000 00 (16)
(b0010 * 2) + 1 => b0001 01 ( 5) | (b1000 * 2) + 1 => b1000 01 (17)

(b0011 * 2) + 0 => b0001 10 ( 6) | (b1001 * 2) + 0 => b0100 10 (18)
(b0011 * 2) + 1 => b0001 11 ( 7) | (b1001 * 2) + 1 => b0100 11 (19)

(b0100 * 2) + 0 => b0010 00 ( 8) | (b1010 * 2) + 0 => b0101 00 (20)
(b0100 * 2) + 1 => b0010 01 ( 9) | (b1010 * 2) + 1 => b0101 01 (21)

(b0101 * 2) + 0 => b0010 10 (10) | (b1011 * 2) + 0 => b0101 10 (22)
(b0101 * 2) + 1 => b0010 11 (11) | (b1011 * 2) + 1 => b0101 11 (23)

Each number from the above table can be divided by 2 (integer division) to recovery the 2nd choice, or modulo by 2 to recover the last choice.

CyanogenMod (AOSP) and Vim Build Integration

| Comments

The build system AOSP and CyanogenMod have some interesting shell scripts that make it easier to build various parts of the system:

  • breakfast [device] – selects a particular device for building
  • brunch [device] – building the entire image for the device
  • mka bacon – almost the same as brunch, but, mmm…. bacon….
  • mm [make_args] – builds whatever module you happen to be in, just that module
  • mmp [make_args] – builds whatever module you happen to be in, just that module, and pushes it to the device

So… the mm and mmp directives seem like great shortcuts to shorten your develop / compile / test cycle, right? But what about editor completion? You can’t invoke mm and mmp in a vim session in order to munch on the compile errors?! Fret not! Now you can!

The first step is a couple of nifty shell scripts. They probably represent a horrible compromise of your security, so use with care. But here they are…

The first is a server of sorts, it sits in a open shell session, one that’s all setup to build Android:

do_over_there_listen.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#!/bin/bash

FIFO_STDINP="/tmp/do_over_there.stdinp.fifo"
FIFO_STDOUT="/tmp/do_over_there.stdout.fifo"
FIFO_STDERR="/tmp/do_over_there.stderr.fifo"

do_stuff_here() {
    rm -vf $FIFO_STDINP
    rm -vf $FIFO_STDOUT
    rm -vf $FIFO_STDERR

    mkfifo $FIFO_STDINP
    mkfifo $FIFO_STDOUT
    mkfifo $FIFO_STDERR

    while read line <$FIFO_STDINP ; do
        echo "Running: $line"
        eval $line >$FIFO_STDOUT 2>$FIFO_STDERR
    done
}

You run this script like so:

run do_stuff_here
1
2
3
4
$ . ~/Scripts/do_over_there_listen.sh ; do_stuff_here
removed `/tmp/do_over_there.stdinp.fifo'
removed `/tmp/do_over_there.stdout.fifo'
removed `/tmp/do_over_there.stderr.fifo'

The next step is a script that takes the place of make in your vim session:

do_over_there
1
2
3
4
5
6
7
8
9
10
11
#!/bin/bash

D=$(dirname $(readlink -f ${BASH_SOURCE}))
source "$D/do_over_there_listen.sh"

echo "mm $@" >$FIFO_STDINP

cat $FIFO_STDOUT &
cat $FIFO_STDERR >&2 &

wait

Then, the final piece of the puzzle is to tell vim to use the new “make” command with (after chmod +x do_over_there of course):

vimrc mods
1
set makeprg=~/path/to/do_over_there

Finally, you’ll be able to compile with the AOSP and CyanogenMod build system niceties but then suck those error logs into vim. Like so:

Android Build Tools SNAFU

| Comments

So, I started developing for the Android platform in earnest around April of 2013. I found that the Android project had release not just a mere plugin for Eclipse but a rebranded, pre-customized version of Eclipse called ADT or Android Development Toolkit.

This seemed pretty cool, I guess, though it seems better to just release a plugin so you can slice and dice your dev environment however you want.

Cool, but can it do Vim?

At least for the February release of ADT it did not work so well (the 20130219 version)– installing vrapper breaks all kinds of things:

This:

Turned in to this:

Afterwards, all of my Android integration pre-packaged in ADT seemed to disappear. I had to manually re-install the Android integration that was previously there. Not sure why I did… maybe it was user error… but it was not a good user experience.

Alternate Reality: Android Studio

So far, my experience with trying the Android Studio preview has been great, but mostly because the of the dark UI, super easy vim installation that didn’t nuke any Android integration … and because it does stuff like this:

Finally, something that can at least make Java not look so stupid. To quote the Sarah Palin jibe, “it’s just lipstick on a pig”, but I’ll take it! Yes sir, uh-huh.


So… this brings me to new version of ADT (verstion 20130522) – I’m starting a new project I figured I’d try to upgrade to the May release of ADT. Lo and behold it doesn’t come pre-packaged with something called Android Build Tools and it has a really shitty way to install it, basically it pops up a nag box and says:

1
2
3
4
5
Hey, we couldn't take the time to implement auto download or to auto package
this new tool we're requiring, so go grab it yourself, good luck!

Yours truly,
-Asshats

This wasted a good hour as I groped around trying to figure out what was wrong. Finally the Internet saved me as I discovered that many other developers can’t read either:

The thing is, I did read the nag dialog, and I tried to download whatever it wanted, but for whatever reason (poor up bringing – loose morals – just plain dumb) I didn’t succeed. So, this proves at least one thing: Android developers can’t read. Or does it? Maybe it proves that Google wasted a lot of people’s time with this decision.

Google Apps Script: Insert Header Numbers

| Comments

Google Apps Script presents an interesting solution to extend the features of Google applications. The usual way to do this is with client side browser extensions, but they are not portable (that is, specific to a browser, and specific to the web)– and cannot run unattended.

Google’s apps (and similarly, 99% of other web apps) have long lacked scriptability that native applications enjoy. Apps script is a play to resolve that deficiency.

TL;DR

Jump to the solution for the script that adds header numbers.

The problem

Google docs has undergone some interesting simplifications in recent times, and it’d be interesting to learn why certain features got chopped.

In particular, at one point it was possible to insert numeric headers into a Google doc, though it looks like this was a CSS hack. Since Google docs isn’t exposing the underlying CSS of the document anymore, this method no longer works. You can also turn your headers into numbered lists … but this is really cumbersome.

Why do you need numeric headers? Well, it’s a pretty standard word processor feature– if you’re packaging information into a fairly static format (say a PDF), it’s useful for communication purposes to reference a specific part of the document; for example, if you were writing a technical review of someone’s software.

Background

Google offers several different types of scripts:

The biggest distinction appears to be if a script is bound to a container or not – that is, whether the script lives inside a doc, form, spreadsheet, or if it sits outside one of these (sitting in the script editor). Scripts can also become a standalone WebApp.

As first, standalone scripts seemed useful for testing things out, but the inability to interact with the user quickly made a standalone scripts useless… I found myself thinking that as I developed useful things with standalone scripts (macros!) that it’d be nice to share these, and simply insert some silly UI prompts to make the script generic– it’s unclear how to do this without making the script specific to a container, then sharing it in the document specific script gallery – right now, there’s no script gallery for text documents.

Power users, macro sharing, and the apps script gallery

The apps script gallery seems sadly lacking… why is the script gallery buried inside the document? This hardly seems like the “happening” place to showcase cool macros:

I did manage to find apps script “gallery” at one point, but it was full of commercial apps geared at implementing solution on top of Google Apps for businesses. This is great, but if Google wants to crowd source all the crazy features that users want by providing power-users with scripting capabilities, there should probably be something more like the chrome extensions gallery to showcase macros. That is, something easily accessible from a Google search.

The solution

The solution is to simply use the Google apps script API to create your own header numbers. It’s somewhat cumbersome, and will probably break in many cases, but it works.

There’s probably a better way to do this… for example, one improvement would be to figure out how to share this to a common repository so it could be imported into other documents.

Start by creating a new apps script in the document:

Then insert the following script (also maintained in a gist):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
var state = {
  headerLevel1: 0,
  headerLevel2: 0,
  headerLevel3: 0,
}

var private = {
  getSectionNumber: function (paragraph)
  {
    if ( paragraph.getHeading() == DocumentApp.ParagraphHeading.HEADING1 )
    {
      state.headerLevel2 = 0;
      state.headerLevel3 = 0;

      return (++state.headerLevel1) + " ";
    }
    else if ( paragraph.getHeading() == DocumentApp.ParagraphHeading.HEADING2 )
    {
      if (state.headerLevel1 == 0)
        state.headerLevel1 = 1;

      state.headerLevel3 = 0;

      return state.headerLevel1 + "." + (++state.headerLevel2) + " ";
    }
    else if ( paragraph.getHeading() == DocumentApp.ParagraphHeading.HEADING3 )
    {
      if (state.headerLevel2 == 0)
        state.headerLevel2 = 1;

      return state.headerLevel1 + "." + state.headerLevel2 + "." + (++state.headerLevel3) + " ";
    }

    return "";
  },

  isNumber: function (text)
  {
    return !isNaN(text);
  },

  removeExistingHeaderNumber: function (text)
  {
    for (var i = 0; i < text.length; i++)
    {
      if ( private.isNumber(text.charAt(i)) || text.charAt(i) == "." )
          continue;

      break;
    }

    return text.substr(i);
  }
};

function addHeaderNumbers()
{
  var doc = DocumentApp.getActiveDocument();
  var body = doc.getBody()

  for (var i = 0; i < body.getNumChildren(); i++)
  {
    var element = body.getChild(i);

    if ( element.getType() != DocumentApp.ElementType.PARAGRAPH )
      continue;

    var paragraph = element.asParagraph();

    if ( paragraph.getHeading() != DocumentApp.ParagraphHeading.NORMAL )
    {
      var sectionNumber = private.getSectionNumber(paragraph);
      paragraph.setText(sectionNumber + private.removeExistingHeaderNumber(paragraph.getText()))
    }
  }

  doc.saveAndClose();
}

Save the project and return to the document. Then use the script manager to open the list of available functions to run:

Then select the ‘addHeaderNumbers’ script:

And magically your document will now have header numbering:

EOT

Infinite Terminal Logs

| Comments

The Problem: Unsatisfying Terminal Emulators

Sometimes when you’re hacking on things it’s a great resource to be able to recall / search everything that you’ve done in the past so that you can reconstruct what’s happened in a coherent matter.

This is, a crutch, or a substitute for a structured approach to a problem, but I find it useful during “hack and slash” sessions when you don’t necessarily want or need a structured approach.

Some commercial terminals offer logging (SecureCRT for example). Linux, terminals… not so much. No such option exists in the default Ubuntu terminal. There are two terminals named Terminator one is Java based and does not have a package in the standard Ubuntu repositories, and the other is present in Ubuntu’s standard repositories.

You can image this led to some confusion on my part… because both offer some functionality that I wanted, but what I really wanted was a combination of the two terminals. It’s ironic that SecureCRT offers the best solution for what I wanted.

Java Terminator:

  • Terminal logging
  • Good search, ala OSX Term.app (does not really work though)

Tenshu Terminator:

  • Infinite scrollback (also, does not work very well)
  • Tiling

So, what is a hacker to do? Why… hack up something that works… a good citizen of the FLOSS world would try to add these features to the terminal emulator that they liked, but why bother when you can create a solution that solves most of the problem in a few minutes… Really, I ended up not like the java based Terminator at all because it was ugly and did not ingrate well with Ubuntu, so created my own logging solution seemed like it would solve the logging, search and infinite scrollback use cases.

The Solution: Old Fashioned ‘Script’ Utility

The script tool is a unix utility to record everything that passing from the shell to the terminal emulator to control things like cursor movements, colors, and the contents of the cells on the terminal device.

This is pretty heavy handed, and allows you to replay your entire terminal session (complete with timings) if you so desire. But this isn’t the goal, the goal is to get a terminal log, that easily searchable.

So, to start, data is needed first, so as soon as the shell is started, everything is recorded:

shell profile additions
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
if [ -z "$UNDER_SCRIPT" ]; then

    logdir=$HOME/Logs
    logname=$(date +%F_%T).$$.script

    logfile=$logdir/.$logname
    final_logfile=$logdir/$logname

    trap '( mv $logfile $final_logfile )' EXIT

    if [ ! -d $logdir ]; then
        mkdir $logdir
    fi

    gzip -q $logdir/*.script &>/dev/null

    export UNDER_SCRIPT=$logfile

    script -e -f -q $logfile
    exit $?
fi

Then a terminal emulation library is added:

1
2
$ cd ~/Logs
$ wget https://raw.github.com/liftoff/GateOne/master/terminal/terminal.py

Then a script is implemented to emulate a terminal and generate a usable log file:

~/Logs/stripesc.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import sys
import terminal
import locale

encoding = locale.getpreferredencoding()
term = terminal.Terminal(200,200) # bigger than we could possibly need 

fp = sys.stdin

def process_scrollback(term):
    if not term.scrollback_buf:
        return
    return term.scrollback_buf[-1].tounicode().rstrip().encode(encoding)

def make_scroll_up_cb(term):
    def _():
        print process_scrollback(term)
    return _

term.add_callback(terminal.CALLBACK_SCROLL_UP, make_scroll_up_cb(term))

Then, as need you’d invoke a shell script to decode a log, and grep for something interesting:

~/Logs/strip-escape-codes.sh
1
cat - | python stripesc.py

Also periodically, you could throw away the terminal emulation data that’s no longer needed:

~/Logs/strip-all-escape-codes.sh
1
2
3
4
5
6
7
#!/bin/bash
for log_file in `ls -1 *.script.gz`; do

    zcat $log_file | python stripesc.py
        >`echo $log_file | sed -e 's/.script.gz/.log/'`

done

Thus, we’ve solved the problem with minimal effort, and maximum utility (albeit a little cumbersome).