Posts Tagged linux

How backup partition on Linux with progress bar from command line

On GNU/Linux you can do everything from command line. The same goes to copy whole partition to a file. You can use powerful dd command to save your partition to a file. You can also use pv to show progress bar. Here is useful function that will backup your partition (device) to a file with progress bar (you need sudo to access device):

function backup() {
    sudo dd if=$1 | pv -s $(sudo blockdev --getsize64 $1) | bzip2 -9f > $2
}

and you can use this function like this:

backup /dev/sda1 /home/me/part.img.bz2

You can create similar function for restoring the partition:

function restore() {
    bunzip2 -dc $1 | pv -s $(ls -l $1 | cut -d' ' -f5) | dd of=$2
}

and you can use this function similar to backup (but the file is first and device second):

restore /home/me/part.img.bz2 /dev/sda1

NOTE: using dd can damage your partition, use it with caution.

Advertisements

,

Leave a comment

How to replace tabs by spaces in whole codebase?

I’ve stared working on a project on github, written in php, that have mixed spaces and tabs for indent, I’m using GNU/Linux so I thought that I just replace all tabs by spaces using bash. Here is the command I’ve used that just did that:

find . -type f -name '*.php' | while read file; do
     grep $'\t' $file > /dev/null && sed -ie 's/\t/    /g' $file
done

Explanation:

  • find . -type f -name '*.php' – this will search for all files with php extension, in current directory. I’ve used -type f to return only files since there where directories that end with php.
  • while read file; do ... done – while loop over files found by find, the filename will be in file variable
  • grep $'\t' $file > /dev/null – this will return true if file contain tabs
  • & & execute next command if previous is true
  • sed -ie 's/\t/ /g' $file – replace tabs by spaces in file inline (-i option)

, , , ,

Leave a comment

How to backup 8GB file or larger on fat32 from linux

I have external USB hard drive that use Fat32 (so I can use it on windows machine as well on GNU/Linux). I needed to reinstall my OS (Xubuntu) and repartition the disk, so I need to copy all my files to external drive. But I’ve got 15GB virtualbox file.

On GNU/Linux or MacOS X you can use standard tools to split the file and then merge it back.

Here is a code that will compress and create 4GB chunks.


tar czvf file.tar.gz directory/
split -b $(echo "4*(2^30)-1" | bc) --verbose file.tar.gz file.tar.gz-

it will create files like file.tar.gz-aa file.tar.gz-ab that can be copy to Fat32 partition because they are smaller then 4GB (2^30 is 1GB in bytes).

to merge those files together you can just cat them


cat file.tar.gz-aa file.tar.gz-ab > file.tar.gz

if you want to see progress bar you can use pv command. This is the smallest way I came up with to do this:


cat file.tar.gz-aa file.tar.gz-ab | pv -s $(ls -l file.tar.gz-aa file.tar.gz-ab | cut -d' ' -f5 | tr '\n' '+' | sed -e 's/+$//' -e 's/.*/echo -n $(( & ))/' | bash) > file.tar.gz

Here is a function that can be put into .bashrc file


function cat-pv () {
    cat $@ | pv -s $(ls -l $@ | cut -d' ' -f5 | tr '\n' '+' | sed -e 's/+$//' -e 's/.*/echo -n $(( & ))/' | bash)
}

Here I found shorter way to calculate size of list of files it use awk:


ls -lrt | awk '{ total += $5 }; END { print total }'

So the function can be strink to


function cat-pv () {
     cat $@ | pv -s $(ls -lrt $@ | awk '{ total += $5 }; END { print total }')
}

, , ,

1 Comment

Shell without ssh or root on your shared hosting

My site jcubic.pl is on shared hosting that don’t have ssh, and I wanted to have a shell access, so using my jQuery Terminal, I’ve stared a project leash. It’s a shell written in javascript with help from php and python cgi. It will give you a shell access, no need to install any new software, and you don’t need to be root (like with other shells like shell in a box, AjaxTerm or Buterfly).

, , , ,

Leave a comment

How to have access to a shell without ssh or telnet using jQuery terminal

If you have, like I do, a virtual server on hosting service, that support cgi but don’t have access to a shell and php have passthru disabled you still can have access to a shell, even if you can’t install anything on that machine – like anyterm.

There is this project, which I did, called jQuery terminal, that emulate a Unix terminal. All you need is interpreter that will execute shell commands.

Lots of hosting services disable php passthru but leave cgi script, so if you have them you can create simple shell like this one:

UPDATE: If you don’t want to code it yourself (or look at working solution), you can check my project LEASH – Browser Shell.

#!/bin/bash
echo -en "Content-Type: text/plain\r\n\r\n"
query=$(/usr/bin/python -c "import urllib; print urllib.unquote_plus('$QUERY_STRING')")
eval $query 2>&1

I use python subprocess to decode QUERY_STRING. Probably it can be done using some Unix tools.

On my server I use Python CGI script like this one:.

#!/usr/bin/python
import cgi
import subprocess as sub

try:
    from json import dumps as json_serialize
except ImportError:
    def json_serialize(obj):
        result = ''
        t = type(obj)
        if t == types.StringType:
            result += '"%s"' % obj.replace('\\', '\\\\').replace('"', '\\"').replace('\n', '\\n')
        elif t == types.NoneType:
            result += 'null'
        elif t == types.IntType or t == types.FloatType:
            result += str(obj)
        elif t == types.LongType:
            result += str(int(obj))
        elif t == types.TupleType:
            result += '[' + ','.join(map(json_serialize, list(obj))) + ']'
        elif t == types.ListType:
            result += '[' + ','.join(map(json_serialize, obj)) + ']'
        elif t == types.DictType:
            array = ['"%s":%s' % (k,json_serialize(v)) for k,v in obj.iteritems()]
            result += '{' + ','.join(array) + '}'
        else:
            result += '"unknown type - ' + type(obj).__name__ + '"'
        return result

class Singleton(object):
    __single = None
    def __init__(self):
        Singleton.__single = self

    def __new__(cls):
        if Singleton.__single:
            return Singleton.__single
        else:
            return super(Singleton, cls).__new__(cls)


class Form(Singleton):
    def __init__(self):
        super(Form)
        self.form = cgi.FieldStorage()

    def __getitem__(self, name):
        return self.form.getvalue(name)

def execv(command, path):
    command = 'cd %s && %s && pwd 1>&2' % (path, command)
    proc = sub.Popen(['/bin/bash', '-c', command],
                     stdout=sub.PIPE, stderr=sub.PIPE)
    stderr = proc.stderr.read()[:-1]
    stdout = proc.stdout.read()[:-1]
    if not os.path.exists(stderr):
        raise Exception(stdout, stderr)
    return {
        "cwd": stderr,
        "stdout": re.sub('.\x08', '', stdout)
    }

if __name__ == '__main__':
    print "Content-Type: text/plain"
    print
    form = Form()
    if token() == form['token']:
        response = {}
        try:
            response['result'] = execv(form['command'], form['path'])
        except Exception, e:
            response['error'] = e.args[1]
            response['result'] = {'stdout': e.args[0], 'cwd': form['path']}
        else:
            response['error'] = None
    else:
        response = {'error': 'You are not authorized', 'result': None}
    stdout.write(json_serialize(response))

You have enabled passthru in php you can write your shell in php.

Now when you have access to a shell you need to setup interface using jQuery terminal.

<!DOCTYPE HTML>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <meta charset="utf-8" />
    <title>Shell</title>
    <script src="http://code.jquery.com/jquery-1.7.1.min.js"></script>
    <script src="terminal/js/jquery.mousewheel-min.js"></script>
    <script src="terminal/js/jquery.terminal.js"></script>
    <link href="terminal/css/jquery.terminal.css" rel="stylesheet"/>
    <style>
    .terminal a.ui-slider-handle:focus { outline: none; }
    body { margin: 0; padding: 0; }
    html { background-color: #000; }
    .clear { clear: both; }
    /* This works only in Safari and Google Chrome */
    @media screen and (-webkit-min-device-pixel-ratio:0) {
        .terminal, .terminal .terminal-output, .terminal .terminal-output div,
        .terminal .terminal-output div div, .cmd, .terminal .cmd span, .terminal .cmd div {
            font-weight: bold;
        }
    }
    </style>
<script>
$(function() {
    var pwd, last_dir, home_dir;
    var terminal = $('#shell').terminal(function(command, term) {
        if (command.replace(/^ *(.*) *$/, '$1') == '-') {
            pwd = last_dir;
            return;
        }
        term.pause();
        $.post("/cgi-bin/cmd.py", {
            token: term.token(),
            command: command.replace(/(\\)?~/, function($0,$1) {
                return $1 ? $0 : home_dir;
            }),
            path: pwd
        }, function(response) {
            term.echo(response.result.stdout);
            pwd = response.result.cwd;
            term.resume();
        });
    }, {
        login: function(user, password, authenticate) {
            $.post("/cgi-bin/cmd.py", {
                user: user,
                password: password
            }, function(token) {
                if (token) {
                    home_dir = pwd = last_dir = '/home/ + user;
                }
                authenticate(token);
            });
        },
        prompt: function(callback) {
            var username = terminal.login_name();
            var re = new RegExp("^" + '/home/' + username);
            var username = '[[;#44D544;]' + username + ']';
            var path = '[[;#5555FF;]' + pwd.replace(re, '~') + ']';
            callback(username + '[[;#989898;]:]' + path + '[[;#989898;]$] ');
        },
        name: 'shell'
    }).css({
        overflow: 'auto'
    });

    $(window).resize(function() {
       terminal.css('height', $(window).height()-20);
    }).resize();

});
</script>
</head>
<body>
  <div id="shell"></div>
</body>
</html>

, ,

1 Comment

Concatenate files with progress bar on GNU/Linux

When I download files form rapidshare or other file hosting sites I occasionally download movies that are splited into parts, and to watch the movie they need to be concatenated.

I came up with this bash function which I put in my .bashrc file:

function concat() {
  files=${@:1:$(($#-1))}
  cat $files | pv -s $(du -cb $files | tail -n1 | cut -d $'\t' -f1) > ${@: -1}
}

If you have files like Movie.avi.001…Movie.avi.020, to concatenate these files just type:

concat Movie.avi.* Movie.avi

And you’ll see progress bar when output file is created.

, ,

1 Comment

[Ruby] Wyświetlanie definicji ze “Słownika Wyrazów Obcych Kopalinskiego”

To jest kod skryptu w Ruby do wyświetlania definicji ze “Słownika Wyrazów Obcych Kopalińskiego”. Pobierz plik kopalinski.rb.

Żeby użyć tego skryptu musisz mieć zainstalowanego Ruby-ego.

W terminalu wpisz:

kopalinski.rb Słownik

Skrypt wyświetli definicje słowa “Słownik”. Jeśli w słowniku Kopalińskiego nie ma jakiegoś słowa skrypt nie wyświetla nic.

,

3 Comments

%d bloggers like this: