Posts Tagged linux

How to backup 8GB file or larger on fat32 from linux

I have external USB hard drive that use Fat32 (so I can use it on windows machine as well on GNU/Linux). I needed to reinstall my OS (Xubuntu) and repartition the disk, so I need to copy all my files to external drive. But I’ve got 15GB virtualbox file.

On GNU/Linux or MacOS X you can use standard tools to split the file and then merge it back.

Here is a code that will compress and create 4GB chunks.


tar czvf file.tar.gz directory/
split -b $(echo "4*(2^30)-1" | bc) --verbose file.tar.gz file.tar.gz-

it will create files like file.tar.gz-aa file.tar.gz-ab that can be copy to Fat32 partition because they are smaller then 4GB (2^30 is 1GB in bytes).

to merge those files together you can just cat them


cat file.tar.gz-aa file.tar.gz-ab > file.tar.gz

if you want to see progress bar you can use pv command. This is the smallest way I came up with to do this:


cat file.tar.gz-aa file.tar.gz-ab | pv -s $(ls -l file.tar.gz-aa file.tar.gz-ab | cut -d' ' -f5 | tr '\n' '+' | sed -e 's/+$//' -e 's/.*/echo -n $(( & ))/' | bash) > file.tar.gz

Here is a function that can be put into .bashrc file


function cat-pv () {
    cat $@ | pv -s $(ls -l $@ | cut -d' ' -f5 | tr '\n' '+' | sed -e 's/+$//' -e 's/.*/echo -n $(( & ))/' | bash)
}

Here I found shorter way to calculate size of list of files it use awk:


ls -lrt | awk '{ total += $5 }; END { print total }'

So the function can be strink to


function cat-pv () {
     cat $@ | pv -s $(ls -lrt $@ | awk '{ total += $5 }; END { print total }')
}

, , ,

Leave a comment

Shell without ssh or root on your shared hosting

My site jcubic.pl is on shared hosting that don’t have ssh, and I wanted to have a shell access, so using my jQuery Terminal, I’ve stared a project leash. It’s a shell written in javascript with help from php and python cgi. It will give you a shell access, no need to install any new software, and you don’t need to be root (like with other shells like shell in a box, AjaxTerm or Buterfly).

, , , ,

Leave a comment

How to have access to a shell without ssh or telnet using jQuery terminal

If you have, like I do, a virtual server on hosting service, that support cgi but don’t have access to a shell and php have passthru disabled you still can have access to a shell, even if you can’t install anything on that machine – like anyterm.

There is this project, which I did, called jQuery terminal, that emulate a Unix terminal. All you need is interpreter that will execute shell commands.

Lots of hosting services disable php passthru but leave cgi script, so if you have them you can create simple shell like this one:

UPDATE: If you don’t want to code it yourself (or look at working solution), you can check my project LEASH – Browser Shell.

#!/bin/bash
echo -en "Content-Type: text/plain\r\n\r\n"
query=$(/usr/bin/python -c "import urllib; print urllib.unquote_plus('$QUERY_STRING')")
eval $query 2>&1

I use python subprocess to decode QUERY_STRING. Probably it can be done using some Unix tools.

On my server I use Python CGI script like this one:.

#!/usr/bin/python
import cgi
import subprocess as sub

try:
    from json import dumps as json_serialize
except ImportError:
    def json_serialize(obj):
        result = ''
        t = type(obj)
        if t == types.StringType:
            result += '"%s"' % obj.replace('\\', '\\\\').replace('"', '\\"').replace('\n', '\\n')
        elif t == types.NoneType:
            result += 'null'
        elif t == types.IntType or t == types.FloatType:
            result += str(obj)
        elif t == types.LongType:
            result += str(int(obj))
        elif t == types.TupleType:
            result += '[' + ','.join(map(json_serialize, list(obj))) + ']'
        elif t == types.ListType:
            result += '[' + ','.join(map(json_serialize, obj)) + ']'
        elif t == types.DictType:
            array = ['"%s":%s' % (k,json_serialize(v)) for k,v in obj.iteritems()]
            result += '{' + ','.join(array) + '}'
        else:
            result += '"unknown type - ' + type(obj).__name__ + '"'
        return result

class Singleton(object):
    __single = None
    def __init__(self):
        Singleton.__single = self

    def __new__(cls):
        if Singleton.__single:
            return Singleton.__single
        else:
            return super(Singleton, cls).__new__(cls)


class Form(Singleton):
    def __init__(self):
        super(Form)
        self.form = cgi.FieldStorage()

    def __getitem__(self, name):
        return self.form.getvalue(name)

def execv(command, path):
    command = 'cd %s && %s && pwd 1>&2' % (path, command)
    proc = sub.Popen(['/bin/bash', '-c', command],
                     stdout=sub.PIPE, stderr=sub.PIPE)
    stderr = proc.stderr.read()[:-1]
    stdout = proc.stdout.read()[:-1]
    if not os.path.exists(stderr):
        raise Exception(stdout, stderr)
    return {
        "cwd": stderr,
        "stdout": re.sub('.\x08', '', stdout)
    }

if __name__ == '__main__':
    print "Content-Type: text/plain"
    print
    form = Form()
    if token() == form['token']:
        response = {}
        try:
            response['result'] = execv(form['command'], form['path'])
        except Exception, e:
            response['error'] = e.args[1]
            response['result'] = {'stdout': e.args[0], 'cwd': form['path']}
        else:
            response['error'] = None
    else:
        response = {'error': 'You are not authorized', 'result': None}
    stdout.write(json_serialize(response))

You have enabled passthru in php you can write your shell in php.

Now when you have access to a shell you need to setup interface using jQuery terminal.

<!DOCTYPE HTML>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <meta charset="utf-8" />
    <title>Shell</title>
    <script src="http://code.jquery.com/jquery-1.7.1.min.js"></script>
    <script src="terminal/js/jquery.mousewheel-min.js"></script>
    <script src="terminal/js/jquery.terminal.js"></script>
    <link href="terminal/css/jquery.terminal.css" rel="stylesheet"/>
    <style>
    .terminal a.ui-slider-handle:focus { outline: none; }
    body { margin: 0; padding: 0; }
    html { background-color: #000; }
    .clear { clear: both; }
    /* This works only in Safari and Google Chrome */
    @media screen and (-webkit-min-device-pixel-ratio:0) {
        .terminal, .terminal .terminal-output, .terminal .terminal-output div,
        .terminal .terminal-output div div, .cmd, .terminal .cmd span, .terminal .cmd div {
            font-weight: bold;
        }
    }
    </style>
<script>
$(function() {
    var pwd, last_dir, home_dir;
    var terminal = $('#shell').terminal(function(command, term) {
        if (command.replace(/^ *(.*) *$/, '$1') == '-') {
            pwd = last_dir;
            return;
        }
        term.pause();
        $.post("/cgi-bin/cmd.py", {
            token: term.token(),
            command: command.replace(/(\\)?~/, function($0,$1) {
                return $1 ? $0 : home_dir;
            }),
            path: pwd
        }, function(response) {
            term.echo(response.result.stdout);
            pwd = response.result.cwd;
            term.resume();
        });
    }, {
        login: function(user, password, authenticate) {
            $.post("/cgi-bin/cmd.py", {
                user: user,
                password: password
            }, function(token) {
                if (token) {
                    home_dir = pwd = last_dir = '/home/ + user;
                }
                authenticate(token);
            });
        },
        prompt: function(callback) {
            var username = terminal.login_name();
            var re = new RegExp("^" + '/home/' + username);
            var username = '[[;#44D544;]' + username + ']';
            var path = '[[;#5555FF;]' + pwd.replace(re, '~') + ']';
            callback(username + '[[;#989898;]:]' + path + '[[;#989898;]$] ');
        },
        name: 'shell'
    }).css({
        overflow: 'auto'
    });

    $(window).resize(function() {
       terminal.css('height', $(window).height()-20);
    }).resize();

});
</script>
</head>
<body>
  <div id="shell"></div>
</body>
</html>

, ,

1 Comment

Concatenate files with progress bar on GNU/Linux

When I download files form rapidshare or other file hosting sites I occasionally download movies that are splited into parts, and to watch the movie they need to be concatenated.

I came up with this bash function which I put in my .bashrc file:

function concat() {
  files=${@:1:$(($#-1))}
  cat $files | pv -s $(du -cb $files | tail -n1 | cut -d $'\t' -f1) > ${@: -1}
}

If you have files like Movie.avi.001…Movie.avi.020, to concatenate these files just type:

concat Movie.avi.* Movie.avi

And you’ll see progress bar when output file is created.

, ,

Leave a comment

[Ruby] Wyświetlanie definicji ze “Słownika Wyrazów Obcych Kopalinskiego”

To jest kod skryptu w Ruby do wyświetlania definicji ze “Słownika Wyrazów Obcych Kopalińskiego”. Pobierz plik kopalinski.rb.

Żeby użyć tego skryptu musisz mieć zainstalowanego Ruby-ego.

W terminalu wpisz:

kopalinski.rb Słownik

Skrypt wyświetli definicje słowa “Słownik”. Jeśli w słowniku Kopalińskiego nie ma jakiegoś słowa skrypt nie wyświetla nic.

,

3 Comments

Really strange thing with Fedora

Recently I’ve create icons for GTK and I’ve notice strange thing. If you put this comment:

<!--
    icon is vector   for gtk

    Copyright  (c)

    You should have received a copy  the GNU General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/<.

-->

In svg file between xml prolog and svg root node, the file will not display in Nautilus. The same things is with icons.

here is link to example file.

This was tested on Fedora 14 64bit liveCD on VirtualBox

, , ,

Leave a comment

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: