Build R package from GNU Emacs on Windows 10

I use Windows 10 machine, at work, with GNU Emacs installed from choco repository. Script from my article Run shiny R application from Emacs was not working like in my old Windows 7. First thing is that it was not founding my R binary and I’ve got error:

apply: Spawning child process: Invalid argument

First thing was adding path to R binary second issue was path to Root of the package

Here is first function based on the one from previous article:

(defun call-r-command (command-list)
  "run R command in shell buffer window

  if there is displayed buffer that have shell it will use that window"
  (let* ((name "*R*")
         (new-buffer (get-buffer-create name))
         (old-buff (get-buffer name))
         ;; run R command
         (script-proc-buffer
          (apply 'make-comint-in-buffer
                 "script"
                 new-buffer
                 "c:/Program Files/R/R-3.5.2/bin/R.exe"
                 nil
                 command-list))
         ;; if there is *R* buffer try to find its window
         (old-window (if old-buff (get-buffer-window old-buff)))
         ;; use old window or find any that have shell mode
         (window (or old-window
                     (get-window-with-mode '(comint-mode eshell-mode))))
       (script-proc (get-buffer-process script-proc-buffer)))
    ;; reuse window or create new one
    (if window
        (set-window-buffer window new-buffer)
      (switch-to-buffer-other-window new-buffer))))

This function execute R command, so my new shiny function is shorter

(defun shiny ()
  "run shiny R application in shell buffer"
  (interactive)
  (let ((R (concat "shiny::runApp('" default-directory "')")))
    (call-r-command `("-e" ,R "--no-save"))))

Next is function that return root of git repo (since my R package is also git repo):

(defun shell-line (command)
  "Function execute shell command and return the result as single line"
  (replace-regexp-in-string "\n" "" (shell-command-to-string command)))

The problem with git on Windows 10 in GNU Emacs (or because of shell-command-to-string function) is that path is in Unix format not in Windows format, so I need to fix the path to use it in my R calling script.

(defun git-root-repo ()
  "Function return root of the git repo"
  (interactive)
  (replace-regexp-in-string "/mnt/c/" "c:/" (shell-line "git rev-parse --show-toplevel")))

Now my final function that build the R package:

(defun build-r-package ()
  "Build R package using git root directory as path to R package"
  (interactive)
  (call-r-command `("CMD" "INSTALL" ,(git-root-repo))))

Missing function for finding buffer with same major mode can be found in my previous article Run shiny R application from Emacs.

Better event type in Angular event handlers

I was learning Angular and Udemy video course as well as the official docs show this code:

export class KeyUpComponent_v1 {
  values = '';


  onKey(event: KeyboardEvent) { // with type info
    this.values += (<HTMLInputElement>event.target).value + ' | ';
  }
}

this looks much better in ReactJS with typescript, where event target have proper type:

onClick(event: MouseEvent<HTMLButtonEvent>) {

}

But this don’t work in Angular, you got Error: “Type ‘MouseEvent’ is not generic.”

So I’ve thought that I should try to create this type for Angular. And it look like this:

type Omit<T, K extends keyof T> = Pick<T, Exclude<keyof T, K>>;

interface GenericEvent<T> extends Omit<Event, 'target'> {
    target: T;
}

You can use this type like this:

@Component({
    selector: 'app-input',
    template: '<input (input)="onInput($event)"/>'
})
export class InputComponent {
    value = '';
    onInput(event: GenericEvent<HTMLInputElement>) {
        this.value = event.target.value;
    }
}

NOTE: in this example I’ve used Event interface as base for GenericEvent, but for input event there is more specific type which is InputEvent.

And you can easily create event for each Event type in TypeScript (here are the list of Events interfaces, I’m not sure, but probably they all are types (interfaces) in TypeScript).

If you would like single type with two values, check this Stack Overflow post: Generic interface extending parameterized interface in TypeScript.

How to install Gnash from source on Fedora

Gnash was removed from Fedora so the only way to have it, is to install it from source, here is instruction how to do this:

Installing Depenecies

Depenecies that can be installed from Fedora repositories

Some of them are from RPM fustion so you need to install that repository.
Here are instruction how to do this.

sudo dnf install swftools haxe dejagnu csound boost-devel SDL-devel \
gstreamer-plugins-base-devel gstreamer-devel speexdsp-devel \
speex-devel jemalloc-devel

Then to install app-devel adn swfmill packages you need to add Sphere RPM repo

sudo dnf install "http://ftp.gwdg.de/pub/opensuse/repositories/home:/zhonghuaren/Fedora_27/noarch/rpmsphere-release-27-1.1.noarch.rpm"

You can change Fedora 27 to your version of Fedora (for Fedora 29, repo file with version 28 also works)

sudo dnf install agg-devel swfmill

Then you need to install one library from source. libming is also missing in Fedora:

git clone https://github.com/libming/libming.git --depth 1
cd libming
./autogen.sh
./configure
make
sudo make install

if you get error because of missing yacc/bison and lex/flex use:

sudo dnf install bison flex

Installing Gnash

Then main part which is installing gnash, First you need to install dependencies:

sudo dnf install libjpeg-devel giflib-devel gtk2-devel curl-devel

Unfortunately you will not be able to use RTMP streaming becasue curl is build without support for it. If you need this you can try to build curl from source.

then clone the git repo:

git clone git://git.sv.gnu.org/gnash.git --depth 1
cd gnash

invoke

./autogen.sh

Then you need to execute this command from gnash directory:

sed -i -e 's/${JEMALLOC_CONFIG} --cxxflags/${JEMALLOC_CONFIG} --cppflags/' configure

In configure script there was an error from jemalloc-config command, it was executed with
--cxxflags option. Probably option got removed, it’s now --cppflags or maybe that’s only on Fedora.
This was causing that Makefiles was broken.

then you need to run configure:

./configure --disable-npapi --prefix=/usr

by default make will install gnash in /usr/local and if you’re using lightspark it will
not find gnash, so we use prefix option to install it in /usr directory.

I’ve used this config option --disable-npapi since AFAIK FireFox and Chrome don’t
support NPAPI anymore and there are no xulrunner package in Fedora.

Then you need to execute this command:

sed -i 's/<jemalloc.h>/<jemalloc\/jemalloc.h>/' libbase/jemalloc_gnash.c

that will fix error in one if the c files.

now you can run build and install gnash:

make && sudo make install

You should be able to run swf files using gnash now.

Tested on Fedora 27 and Fedora 29.

Reload buffer when file changes in Emacs

I’ve started learning typescript and one of the cool feature of compiler is –watch options that recompile the file when it change it’s content. In Emacs if file is changed the buffer still contain old file and if you try to edit that file you’re ask if you really want to ed it that file because it change content.

It found that it would be nice to have reload when file changes, so I wrote this little bit of Emacs Lisp:

;; http://emacs.stackexchange.com/questions/22647/reload-single-file-in-every-window
(defun reload ()
  "Revert buffer, then restore previous position."
  (interactive)
  (let ((pt  (point)))
    (revert-buffer t t)
    (goto-char pt)))

;; two helpers to work with alists

(defun add-to-alist (symbol name value)
  "Add new pair to ALIST"
  (let* ((list (symbol-value symbol))
         (pair (assoc name list)))
    (if pair
        (setcdr pair value)
      (let ((pair (cons name value)))
        (if list
            (setf (symbol-value symbol) (cons pair list))
          (setf (symbol-value symbol) (cons pair nil)))))
    (symbol-value symbol)))

(defun remove-alist (symbol name)
  "Remove pair from ALIST"
  (let* ((list (symbol-value symbol))
         (pair (assoc name list)))
    (if pair
        (let ((ptr list))
          (if (eq (caar ptr) name)
              (setf (symbol-value symbol) '())
            (while ptr
              (if (eq (caadr ptr) name)
                  (progn
                    (setcdr ptr (cddr ptr))
                    (setq ptr nil))
                (setq ptr (cdr ptr))))
            list)))))

(require 'filenotify)

(setq file-watchers '())


(defun watch ()
  "Function watch file changes and reload the file from buffer"
  (interactive)
  (let* ((fname (buffer-file-name))
         (pair (assoc fname file-watchers)))
    (if pair
        (progn
          (file-notify-rm-watch (cdr pair))
          (remove-alist 'file-watchers descritor)))

    (let ((descriptor (file-notify-add-watch fname
                                             '(change)
                                             (lexical-let ((filename fname))
                                               (lambda (event)
                                                 (with-current-buffer (get-file-buffer filename)
                                                   (if (not (buffer-modified-p))
                                                       (progn
                                                         (message "update %s" filename)
                                                         (reload)))))))))
      (add-to-alist 'file-watchers fname descriptor))))

Then to watch the file You just need to invoke the function using M-x watch. The function will not reload the buffer if you have modifications. In that case you need to invoke reload function using M-x reload.

This will probably don’t work on windows but I’m not sure, here is documentation for file-notify-* functions.

Run shiny R application from Emacs

Right now I use Windows box for development at work and I use Emacs, but I needed to run RStudio to run the shiny application that I was working on. I though it would be nice to run shiny app from Emacs. On windows I was not able to run ansi-term but eshell was working.

Here is a function that run shiny app if the buffer you’re in is directory that contain server.R:

(defun shiny ()
  "run shiny R application in new shell buffer

  if there is displayed buffer that have shell it will use that window"
  (interactive)
  (let* ((R (concat "shiny::runApp('" default-directory "')"))
         (name "*shiny*")
         (new-buffer (get-buffer-create name))
         (script-proc-buffer
          (apply 'make-comint-in-buffer "script" new-buffer "R" nil `("-e" ,R)))
         (window (get-window-with-mode '(comint-mode eshell-mode)))
       (script-proc (get-buffer-process script-proc-buffer)))
    (if window
        (set-window-buffer window new-buffer)
      (switch-to-buffer-other-window new-buffer))))


(defun search-window-buffer (fn)
  "return first window for witch given function return non nil value"
  (let ((buffers (buffer-list))
        (value))
    (dolist (buffer buffers value)
      (let ((window (get-buffer-window buffer)))
        (if (and window (not value) (funcall fn buffer window))
            (setq value window)))))))

(defun get-window-with-mode (modes)
  "return window with given major modes"
  (search-window-buffer (lambda (buff window)
                        ((let ((mode (with-current-buffer buffer major-mode)))
                          (member mode modes)))))

I’ve needed to use sentinels because If I run simple eshell command it was freezing Emacs and I need to press C-g to cancel the command, it din’t kill R but it was not very good solution.

I’ve also need to update my switch to other major mode buffer, because I was using Eshell that had eshell-mode and my new shiny buffer had comint-mode, I wanted those buffers it to be treated as the same. Above I’ve defined two functions for searching windows because I was using eshell and I wanted to have R from shiny run in the same window. If you’re using ansi-term (on windows it don’t work) you can check it’s major mode and add it to the list of modes in call to get-window-with-mode.

Below is the code for switching to other buffer with same mode (it’s should be common maybe), using C-M-tab and C-tab.

(setq same-mode-alist '((eshell-mode comint-mode) (comint-mode eshell-mode)))

;; SAME BUFFER SWITCH
(defun buffer-same-mode (change-buffer-fun)
  "execute function for next buffer with the same major mode
  or mode on the list in same-mode-alist variable"
  (let ((current-mode major-mode)
        (next-mode nil))
    (while (not (or (eq next-mode current-mode)
                    (member next-mode (cdr (assoc current-mode same-mode-alist)))))
      (funcall change-buffer-fun)
      (setq next-mode major-mode))))

(defun previous-buffer-same-mode ()
  "switch to previous buffer with same mode"
  (interactive)
  (buffer-same-mode #'previous-buffer))

(defun next-buffer-same-mode ()
  "switch to next buffer with same mode"
  (interactive)
  (buffer-same-mode #'next-buffer))

(global-set-key [C-M-tab] 'previous-buffer-same-mode)
(global-set-key [C-tab] 'next-buffer-same-mode)

Show code coverage from jest framework in emacs

I recently switched from jasmine/istanbul to jest framework for my project jQuery Terminal and I’ve got coverage drop from 80-81% to about 69% with some update of the code and commented out code that I was working on.

So I thought it would be cool to highlight the lines that where covered and not covered by tests and since I use emacs as my editor I thought that I write a function that will do that.

Here it is:

(defun root-git-repo ()
  (interactive)
  (replace-regexp-in-string "\n"
                            ""
                            (shell-command-to-string "git rev-parse --show-toplevel")))

(defun line-pos-at-line (line)
  (interactive)
  (save-excursion
    (goto-line line)
    (line-beginning-position)))

(defun coverage-mark-buffer ()
  (interactive)
  (let* ((dir (root-git-repo))
         (json-object-type 'hash-table)
         (json-array-type 'list)
         (json-key-type 'string)
         (json (json-read-file (concat dir "/coverage/coverage-final.json")))
         (filename (buffer-file-name (current-buffer)))
         (coverage (gethash filename json))
         (statments (gethash "statementMap" coverage)))
    (save-excursion
      (maphash (lambda (key value)
                 (let* ((statment (gethash key statments))
                        (start (gethash "start" statment))
                        (end (gethash "end" statment))
                        (start-line-pos (line-pos-at-line (gethash "line" start)))
                        (start-pos (+ start-line-pos (gethash "column" start)))
                        (end-line-pos (line-pos-at-line (gethash "line" start)))
                        (end-pos (+ end-line-pos (gethash "column" end)))
                        (color (if (= value 0) "dark red" "dark green"))
                        (face `((t (:background ,color)))))
                    (hlt-highlight-region start-pos end-pos face)))
               (gethash "s" coverage)))))

The function is using hlt-highlight-region from highlight.el by Drew Adams.

The function don’t check if coverage file exists. It assume that you’re opening from from git repo and that coverage file is in coverage directory (default for jest) in git root.

The function is quite slow, and it need to process a lot, if the file is big. You can see how it look like in this page, generated by Emacs library htmlize.

If you want to clear the buffer you can use this function:

(defun coverage-clear-buffer ()
  (interactive)
  (save-excursion
    (end-of-buffer)
    (hlt-unhighlight-region 0 (point))))

Clear is much faster.

If you’re iterested in code coverage in Emacs you can take a look at jest-coverage minor mode that I’ve created based on this solution.

How to create Web Server in Browser

In my GIT Web Terminal I use BrowserFS to write files (it’s required by isomorphic-git that I use to create git interface in browsers).
BrowserFS is implementation of node fs module but for browsers, it use few types of storage, I use indexedDB same as example for isomorphic-git.

I thought that i would be cool to edit the project itself and be able to view the files when I edit them in the browser (before I commit), like they where serve from web server. And it turn out it’s possible with Service Worker since you can access IndexedDB from that worker and you can create HTTP response from string or from arrayBuffer (BrowserFS is returning arrayBuffer from its readFile function).

Here is the code for service worker that show pages and directory listing:

self.addEventListener('install', function(evt) {
    self.skipWaiting();
    self.importScripts('https://cdn.jsdelivr.net/npm/browserfs');
    BrowserFS.configure({ fs: 'IndexedDB', options: {} }, function (err) {
        if (err) {
            console.log(err);
        } else {
            self.fs = BrowserFS.BFSRequire('fs');
            self.path = BrowserFS.BFSRequire('path');
        }
    });
});

self.addEventListener('fetch', function (event) {
    event.respondWith(new Promise(function(resolve, reject) {
        function sendFile(path) {
            fs.readFile(path, function(err, buffer) {
                if (err) {
                    err.fn = 'readFile(' + path + ')';
                    return reject(err);
                }
                resolve(new Response(buffer));
            });
        }
        var url = event.request.url;
        var m = url.match(/__browserfs__(.*)/);
        function redirect_dir() {
            return resolve(Response.redirect(url + '/', 301));
        }
        if (m && self.fs) {
            var path = m[1];
            if (path === '') {
                return redirect_dir();
            }
            console.log('serving ' + path + ' from browserfs');
            fs.stat(path, function(err, stat) {
                if (err) {
                    return resolve(textResponse(error404(path)));
                }
                if (stat.isFile()) {
                    sendFile(path);
                } else if (stat.isDirectory()) {
                    if (path.substr(-1, 1) !== '/') {
                        return redirect_dir();
                    }
                    fs.readdir(path, function(err, list) {
                        if (err) {
                            err.fn = 'readdir(' + path + ')';
                            return reject(err);
                        }
                        var len = list.length;
                        if (list.includes('index.html')) {
                            sendFile(path + '/index.html');
                        } else {
                            var output = [
                                '',
                                '',
                                '',
                                '<h1>BrowserFS</h1>',
                                '<ul>'
                            ];
                            if (path.match(/^\/(.*\/)/)) {
                                output.push('<li><a href="..">..</a></li>');
                            }
                            (function loop() {
                                var file = list.shift();
                                if (!file) {
                                    output = output.concat(['</ul>', '', '']);
                                    return resolve(textResponse(output.join('\n')));
                                }
                                fs.stat(path + '/' + file, function(err, stat) {
                                    if (err) {
                                        err.fn = 'stat(' + path + '/' + file + ')';
                                        return reject(err);
                                    }
                                    var name = file + (stat.isDirectory() ? '/' : '');
                                    output.push('<li><a href="' + name + '">' + name + '</a></li>');
                                    loop();
                                });
                            })();
                        }
                    });
                }
            });
        } else {
            fetch(event.request).then(resolve).catch(reject);
        }
    }));
});
function textResponse(string) {
    var blob = new Blob([string], {
        type: 'text/html'
    });
    return new Response(blob);
}

function error404(path) {
    var output = [
        '',
        '',
        '',
        '<h1>404 File Not Found</h1>',
        `File ${path} not found in browserfs`,
        '',
        ''
    ];
    return output.join('\n');
}

Service worker is using __browserfs__ marker to distinguish normal url from urls that are from BrowserFS/IndexedDB. Everything after it it’s serve from BrowserFS. So if you write file foo you can access it using __browserfs__/foo. Here is the code sample that will create a file using browserFS:

fs.writeFile('/foo', 'hello world', function(err) {
   if (err) {
     console.log('Error write file');
   }
});

You can see how this work in my GIT Web Terminal just clone any repo (best is the one that have directories) and view files with url https://jcubic.github.io/git/__browserfs__/repo/path/to/file.

How to create D3 Plugin

I was learning D3 and wanted to write plugin attrs, that will allow to use object to set attributes on svg DOM nodes. It seems that this functionality was in D3 but was removed.

I was searching and was not able to fine how to create the plugins, but then I look at source code for d3 transition (it’s easier to search bundle file than original files on github).

And at the end, there was code that I’ve needed to use. That’s what I thought it require to add to prototype.

So here is my attrs plugin:

d3.selection.prototype.attrs = function(attrs) {
  this.each(function(d, i) {
    var element = d3.select(this);
    Object.keys(attrs).forEach((key) => {
      element.attr(key, attrs[key]);
    });
  });
  return this;
};

You can use it like normal attr:

var g = d3.select('body')
  .append('svg')
  .attrs({width: 200, height: 200})
  .append('rect').attrs({
    x: 50,
    y: 50,
    width: 100,
    height: 100,
    stroke: 'rgb(255, 100, 100)',
    'stroke-width': 10,
    fill: 'black'
  });

It works the same as jQuery plugins.

How to add GPS coordinate to your photo using data from google

I have Nikon DSLR and I wanted to add GPS location so when I upload them to flickr I have it on a map and don’t need to add location by hand.

If you have android phone, you don’t need any extra GPS hardware, to have GPS location on your photos taken with your camera.

The solution, if you have GPS enabled in your phone is it take location history from google map using this url: https://www.google.com/maps/timeline?pb (export option is in dropdown in gear icon at the bottom)

You will have JSON file with GSP coordinates and timestamps for all your locations from your phone.

Now to extract the GPS out of JSON file I use this small script written in Python, it use exiftool to write EXIF back to the file because PIL can only read exif not write it.

#!/usr/bin/env python

from __future__ import division
import simplejson as json
from PIL import Image
from dateutil import parser
from optparse import OptionParser
from subprocess import call
from datetime import datetime, timedelta

def get_date_taken(path):
    return Image.open(path)._getexif()[36867]

def nearest(items, pivot):
    return min(items, key=lambda x: abs(x - pivot))

def comparator(date, hours_shift = None):
    def compare(x):
        current = datetime.fromtimestamp(int(x['timestampMs']) / 1000.0)
        if hours_shift is not None:
            current = current + timedelta(seconds = hours_shift * 60 * 60)
        return abs(current - date)
    return compare

def get_gps(gps, date, hours_shift = None):
    return min(gps['locations'], key=comparator(date, hours_shift))

def parse_date(str):
    return datetime.strptime(str, "%Y:%m:%d %H:%M:%S")

def timestamp(dt, epoch=datetime(1970,1,1)):
    td = dt - epoch
    return (td.microseconds + (td.seconds + td.days * 86400) * 10**6) / 10**6

def timezone(date, hours):
    return date - timedelta(seconds = hours * 60 * 60)

if __name__ == '__main__':
    from sys import argv
    opt = OptionParser()
    opt.add_option('-l', '--location')
    opt.add_option('-t', '--timezone')
    (options, args) = opt.parse_args()
    if options.location is None or len(args) != 1:
        print "usage %s [--timezone <hours shift>] --location [History JSON File] <IMAGE FILE>" % argv[0]
    else:
        gps_list = json.loads(open(options.location).read())
        date = parse_date(get_date_taken(args[0]))
        if options.timezone is not None:
            loc = get_gps(gps_list, date, -float(options.timezone))
        else:
            loc = get_gps(gps_list, date)
        found = datetime.fromtimestamp(
            int(loc['timestampMs']) / 1000.0
        )
        print "%s == %s" % (date, found)
        call([
            'exiftool',
            '-m',
            '-GPSLatitude=%s' % str(int(loc['latitudeE7']) / 1e7),
            '-GPSLongitude=%s' % str(int(loc['longitudeE7']) / 1e7),
            args[0]
        ])

To use it you need to use terminal and execute gps.py --tomezone <hours shift> --location <Path to JSON> <Image File>

The only one issue I’ve found is that PIL can’t extract exif from RAW/NEF files, so you can only use JPEG but you can write exif to RAW/NEF but read create time out of JPG file, if you shot in both JPG and RAW like I do.

EDIT: I was on a trip to Cracow in March of 2019, I was walking from my hotel to train station taking pictures and when I was, at home GPS was connected without timezone so maybe the shift only came from Day Light saving. (I was in Tuscany in April 2018 the shift was 1 which would be about right because in April there is summer time in Poland).

How to simulate error for http requests in R application

I was working on shiny R application and the app was using xmlParse(path) to parse url from web API I was using. I needed to add error handling to respond to 404 and 500 errors, so I’ve changed my code to use httr library. That way I could use two steps process of getting xml from API.

My code look like this:

      res <- GET(url)
      if (res$status_code == 500) {
        message <- "Error From API" 
        simpleFatalError(message)
        stop(message)
      } else if (res$status_code == 404) {
        message <- paste("Study", .session$sid, "not found")
        simpleFatalError(message)
        stop(message)
      } else if (res$status_code != 200) {
        message <- paste(
          "Unknown response code",
          res$status_code,
          "for Study",
          .session$sid
        )
        simpleFatalError(message)
        stop(message)
      }
      studyXML <- tryCatch(
        xmlRoot(xmlTreeParse(
          content(res, "text", encoding = "UTF-8"),
          useInternalNodes=TRUE
        )),
        error = function(e) {
           simpleFatalError("parse error")
        }
      )

and now I had the problem how to simulate 500 error because that error in the API that was needed to be handled was fixed by the API team. So what I’ve did was installing Fiddler I work on windows if you work on Linux better would be to use Burp Suite.

And I used Fiddler script which seems simpler solution because it was just one line of code:

    static function OnBeforeResponse(oSession: Session) {
        if (m_Hide304s && oSession.responseCode == 304) {
            oSession["ui-hide"] = "true";
        }
        if (oSession.url.Contains("study.xml")) { // this was part of the url that returned xml from API
            oSession.oRequest.FailSession(500, "Blocked", "Fiddler blocked this file");
        }
    }

But I had the problem that Fiddler was not handling my backed requests. And it turn out that I needed to enable proxy for httr which is as simple as:

res <- GET(path, use_proxy("127.0.0.1", 8888))

Then I was able to test my connecting to the API, and get proper error in shiny app.