rsync - the better scp

3 minute read Regard this Post, i make this post. First: scp is great. But i recomand to use rsync instead. It has replace for me also the cp and scp command. Why? Simple rsync is more powerful. I allows me dry runs, then it's very efficient. File moves should be easy and save. That is what rsync does! It's some tool that need a little bit use. rsync is simply for backups, but that is almost the same coping around like the cp/scp.

Some examples

Start with a dry run:
rsync -nv  somefile user@server:/path/to/copy
This will connect as user to server. Then show what he will copying, That is very useful to check when your running recusive. A / makes a lots of different in the path showing.
 rsync -rnv  somefolder user@server:/path/to/copy/
somefolder/a
rsync -rnv  somefolder/ user@server:/path/to/copy/
./a
 

Problems with scp

You have a big file you want to push to your vServer, you don't want fill up all of your bandwidth so you set a set a limit of for bw.  Your using scp -l, so we setting a limit of 150 kB/s
scp -l 150 somefile user@server:/path/to/save
I got some wired problem with the scp limiter. Sometimes it works, sometimes it didn't. Why? That is the version with rsync
rsync --bwlimit 150 somefile user@server:/path/to/save
It's basically the same, but it worked reliable on rsync. But when the copy process stops before your done (lost connection for example) you have to start over again. .  The solution is using inplace option in rsync:   inplace copy: rsync is able to copy a file on inplace. This very important you are able to suspend a copy process.
rsync --inplace somefile user@server:/path/to/save
(Lost Connection)
rsync --inplace somefile user@server:/path/to/save
  Keeping the rights, scp will copy and place with the default umask of the system user. rsync allow you to keep the right permission with -a option
ls -l somefile
-rw-r--r-- 1 akendo akendo 347  4. Dez 16:21 somefile
scp -a somefile user@server:/path/to/save
user@server: ls -l somefile
-rw-r----- 1 user user 347  Mar  7 00:19 somefile
rsync -a somefile user@server:/path/to/save
user@server: ls -l somefile
-rw-r--r-- 1 akendo akendo 347  4. Dez 16:21 somefile
As you can see, lose the User/Groups , Time Stamp will be lost with scp. In rsync this will remain the same.   Using Humanreadable
rsync -vh somefile user@server:/path/to/copy
somefile
sent 2.02K bytes received 112 bytes 1.42K bytes/sec
total size is 964.25M speedup is 452912.60 (DRY RUN)
  Some nice looking example, to clone a folder:
rsync -rauvh /file user@server:/path/to/save
This will copy the folder "file" to "/path/to/save" on the host server as user. In some other post i show the best way of using it as a backup method. Feel free to add comments or improve this post.  

Howto fix libgeos_c.so not found

1 minute read I'm working with Python Django and PostGIS. When i try to add the Geo support to the Project, i hit this error message:
OSError: /usr/local/lib/libgeos_c.so: cannot open shared object file: No such file or directory
As I notfied about that he is try to work with the /usr/local/ path what is wrong. I found the lib installed corretly to the /usr/lib/libgeos_c.so . So i simply link it to the /usr/lib/ folder:
sudo ln -s /usr/lib/libgeos_c.so /usr/local/lib/
That does the job.
Make sure that you have the sci-libs/geos installed. 

svn2git

2 minute read

I'm currently moving our old svn repositories to git. There a lots of reason to do this. Performance, Disk Space, SVN sucks, decentralization and some more.

Here is a Video from Linux Tolvads talking about Git. Tech Talk: Linus Torvalds on git

Tools Used:

I recommand this svn2git tool! It has a good documentation, easy to understand and work almost out of the box!  

Some Words:

I start looking in my local repository from Gentoo.  eix showed  git2svn from http://gitorious.org/svn2git/svn2git . There was a problem running it from our SVN Repo. He wan't able to found some URL part. So it was complete useless. I started  to google a bit. I found some nice links that explain and have some examples. Our svn contains lots of different repositories, i used this link here as a sample. The Script fits  as basic, i adjusted it, so that it fit fine for me. I change it so it can uses SSH to create the Git repositories on a remote Host.  Addtional i created a authors file. Example: stnick = Santa Claus <nicholas@lapland.com>
Code:
#!/usr/bin/env bash
REPOS="project1 project2 project3 project4"
URL="https://10.0.4.7:/Musicpictures"
REMOTE_IP="10.0.4.5"
for repo in $REPOS
do
 echo "Create local repor"
 mkdir $repo
 cd $repo

 echo "Start of svn2git progress"
 cmd="svn2git $URL/$repo -m --authors ../authors.txt"
 echo $cmd
 `$cmd`

 echo "Prepare remote Host with SSH"
 ssh root@10.0.4.5 "mkdir -p /var/git/svn2git/$repo.git && cd /var/git/svn2git/$repo.git && git init --bar"
 cmd="git remote add origin ssh://root@10.0.4.5/var/git/svn2git/$repo.git"
 echo $cmd
 `$cmd`
 cmd="git push --all"
 echo $cmd
 `$cmd`
 cd ..
 echo "DONE EXPORTING $repo"
done
exit

Addtional i change the the verbose and also i add -m for the orgirnal SVN Link, this coaint then also the classic comit ID.

The Main Problem is that not all of the repo are in the default SVN way store, some has no trunk/tags/branches structures. So i have to check that all has been converted in a right way.

Sources:
http://chrisjean.com/2009/02/21/git-project-description-file-hasnt-been-set/

How to monitor MonogoDB with Nagios

1 minute read     I  need some Monitoring for Nagios3 of my MongoDB. I found quick some refer in the  MongoDB wiki to this github link. All what you have to do is downloading the Code from github. You need to install pymongo and  python-dev, then you can using it. Here my steps:
wget -c  https://github.com/mzupan/nagios-plugin-mongodb/zipball/master
unzip master -d nagios_plugin_mongodb
rm master
apt-get install python-dev
pip install pymongo
cd nagios_plugin_mongodb/mzupan-nagios-plugin-mongodb-59a9247
cp check_mongodb.py /usr/lib/nagios/plugins/check_mongodb
/usr/lib/nagios/plugins/check_mongodb -h localhost -d monogodb_test_db
/usr/local/lib/python2.6/dist-packages/pymongo/connection.py:378: DeprecationWarning: slave_okay is deprecated. Please use read_preference instead.
 "use read_preference instead.", DeprecationWarning)
OK - Connection took 0 seconds
You can ignore the DeprecationWarning from Python.
Sources:
http://www.mongodb.org/display/DOCS/Monitoring+and+Diagnostics
https://github.com/mzupan/nagios-plugin-mongodb

Gentoo migration to Gnome 3.2

2 minute read Currently is there no developer for compiz in Gentoo. That is a problem, it has bug and no one is fix them. So the Gentoo Stuff deciecd to remove them.
!!! The following installed packages are masked:
- x11-libs/libcompizconfig-0.8.4-r2::gentoo (masked by: package.mask)
/usr/portage/profiles/package.mask:
# Jorge Manuel B. S. Vicetto <jmbsvicetto@gentoo.org> (22 Jan 2012)
# Mask compiz for last-rites unless someone steps up
# to maintain it. Removal in 30 days.
- x11-wm/compiz-0.8.6-r3::gentoo (masked by: package.mask)
- dev-python/compizconfig-python-0.8.4-r3::gentoo (masked by: package.mask)
- x11-apps/ccsm-0.8.4-r1::gentoo (masked by: package.mask)
- x11-plugins/compiz-plugins-unsupported-0.8.4-r1::gentoo (masked by: package.mask)
- x11-wm/compiz-fusion-0.8.6::gentoo (masked by: package.mask)
- x11-plugins/compiz-plugins-extra-0.8.6-r1::gentoo (masked by: package.mask)
- x11-themes/emerald-themes-0.5.2::gentoo (masked by: package.mask)
- x11-libs/compizconfig-backend-gconf-0.8.4-r2::gentoo (masked by: package.mask)
- x11-libs/compiz-bcop-0.8.4::gentoo (masked by: package.mask)
- x11-wm/emerald-0.8.4-r2::gentoo (masked by: package.mask)
- x11-plugins/compiz-plugins-main-0.8.6-r1::gentoo (masked by: package.mask)
For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook
  That why i started to move to gnome 3.2, i have try it on my Workstation. Addtional i found some nice extionas that allow me to bring some useable to Gnome 3. 2. But more late, that how i did it. Start with a look on the gnome-base/gnome package, then unmask it.
[I] gnome-base/gnome
 Available versions: (2.0) 2.32.1-r1 ~3.2.1
 {accessibility (+)cdr cups dvdr +extras +fallback ldap mono policykit}
 Installed versions: 2.32.1-r1(2.0)(15:44:58 12/20/11)(cdr cups dvdr ldap policykit -accessibility -mono)
 Homepage: http://www.gnome.org/
 Description: Meta package for GNOME 3, merge this package to install
 
 echo "gnome-base/gnome ~amd64" >> /etc/portage/package.keywords
emerge -DNuvaq world
This will enter some error message, i recommand to use the --autounmask-write to save the time by unmasking. This made end in some resolve error. Sometime it's no unmask everything in the right way, make sure you got x11-libs/cairo-1.10.2-r2 unmask, i used x11-libs/cairo-1.10.2-r1 and this caused some trouble. When you done, wait till it's compied. Restart the the Xserver or better the host.   After it, i went to https://extensions.gnome.org, here a list of some extensions i recommand: Note you need to use Firefox, Chrome has some bug and will not work yet.
https://extensions.gnome.org/extension/130/advanced-settings-in-usermenu/
https://extensions.gnome.org/extension/38/windows-alt-tab/
https://extensions.gnome.org/extension/6/applications-menu/
 
 
My feeling about Gnome 3.2: I'm very impress,  even my Suspend to Ram work almost Perfect.  It's quite workable, there some problem out like some long time for single application to start, links to file browser a broken or sometime you just have to relog. But i like it, it's smooth working and lots of fun!    

Nginx with SSL

2 minute read For a demo Application that has to be show in the Internet, we need a Scala Lift online. This contains sensitive data, so i need SSL. But the Jetty Server were all the Scala Lift Application is no able to handle Name or SSL so easliy (Or it can, but i have no idea how to do it). So i want that my Website allowing access via http, but then redicting to the HTTPS Page. Just made sure, i add a password so no one unknown can see it. First create a new SSL Certificate :
cd /usr/nginx/
mkdir SSL
cd SSL
openssl genrsa -des3 -out server.key 1024
openssl req -new -key server.key -out server.csr
cp server.key server.key.org
openssl rsa -in server.key.org -out server.key
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
add in your configuration /etc/nginx/site-avaiable/YOUR_DOMAINNAME_HERE
server {
 listen 0.0.0.0:80;
 server_name YOUR_DOMAINNAME_HERE,;
 log_format main '$remote_addr - $remote_user [$time_local] $status '
 '"$request" $body_bytes_sent "$http_referer" '
 '"$http_user_agent" "$http_x_forwarded_for"';
 access_log /var/log/nginx/YOUR_DOMAINNAME_HERE.log;
 rewrite ^(.*) https://YOUR_DOMAINNAME_HERE permanent;
 }

 server {
 listen 0.0.0.0:443;
 ssl on;
 ssl_certificate /etc/nginx/SSL/server.crt;
 ssl_certificate_key /etc/nginx/SSL/server.key;
 server_name YOUR_DOMAINNAME_HERE,;
 access_log /var/log/nginx/ssl-YOUR_DOMAINNAME_HERE.log;
location / {
 auth_basic "Restricted";
 auth_basic_user_file /etc/nginx/auth/dev-htpasswd;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-NginX-Proxy fale;
 proxy_set_header Host $host;
 proxy_pass http://localhost:8080;
}
}
Now enable the configuration and restart the nginx.  
Source:
Nginx Wiki 
josephscott.org 

Basic xserver on a vServer

1 minute read For fetching some date, my Application need a xserver. The Problem is that it's running on VM. So i install a basic xdm. I'll accessing this via VNC.
apt-get install tightvncserver
apt-get install xdm
Now starting the xdm and  tightvncserver
/etc/init.d/xdm start
tightvncserver :1
tightvncserver will ask for vnc password. This will be ask during a VNC client is connecting. Now connect to the server $IP:5901. There is a lonley bash wating  ;-)

[Gentoo]Jetty with a .WAR file

2 minute read
Jetty with a .WAR file
In my company i currenly have a lot to do with Scala Lift. We're developing some stuff with it, but because it takes a lots of time to setup a Scala environment and we need quick show solution, we'll go with a .war file. The problem on my gentoo is, there is no jetty packages, so i searched for some overlays. I still got the java-overlay. With eix jetty, i'm searching for jetty:
* www-servers/jetty [1]
Available versions: (5) ~5.1.3-r1
 {doc source test}
 Homepage: http://www.mortbay.org/
 Description: A Lightweight Servlet Engine
The jetty version is outdate, so i'm downloading the jetty from the offical webpage:
wget -c http://dist.codehaus.org/jetty/jetty-6.1.26/jetty-6.1.26.zip
#Now unzip it
mkdir jetty && unzip jetty-6.1.26.zip -d jetty/
during the download i found a useful blog entry about jetty for debian.  How to deploy a .WAR file there. It's little bit like tomcat but just easier. Move you .WAR File to the $JETTY_HOME/webapps/ folder and start the Jetty Server.
java -jar start.jar
I found also on the overlays.gentoo an updated version of jetty, this can be found on the dustin overlay
layman -a dustin
eix-sync && eix jetty
* www-servers/jetty [1]
 Available versions: (6) ~6.1.24
 {elibc_FreeBSD}
 Homepage: http://jetty.codehaus.org/
 Description: Jetty Web Server; Java Servlet container
[1] "dustin" /var/lib/layman/dustin
I add the =www-servers/jetty-6.1.24 ~amd64 to /etc/portage/package.keywords, now install the jetty:
emerge -avq jetty
so far Akendo

Web Sources:

http://lifeofaprogrammergeek.blogspot.com/2010/06/deploying-war-in-jetty.html
http://overlays.gentoo.org/proj/java
http://gpo.zugaina.org/www-servers/jetty
http://www.gentoo.org/proj/en/java/
 

No iwlagn in Linux 3.2+

1 minute read I  installed today the new Kernel 3.2 and didn't have any Wifi, that is related to some changes in the iwlagn divers. It have being rename to iwlwifi. I keep using a static Kernel for my Laptop and so i didn't enable the new driver automatically. My first idea was that the sys-kernel/linux-firmware has to be updated. But after short check on the Wifi devices everything was clear. No Devices was found with wlan0, so  i check the driver and a try of loading the related module iwlagn. After wounder some minutes i check on the old and the new kernel. The dirver seems to be the same, just the name has change.  i reanabled the new driver, re-compile the new kernel! Everything works normaly.   I check on the h-online and found this git diffs of the kernel.
Rename the iwlagn module as iwlwifi in preparation for future
changes.  Add an alias to iwlagn for backward compatibility.
Normaly there have to a an alias, but for static kernel the driver has to renable.   so far Akendo.  
Sources:
Ubuntu Forum:"Confused between iwlwifi vs iwlagn" H-online:"Kernel Log Part 1 Network" H-online:"Kernel Log Minor gems"    

Update my Blog

1 minute read I just updated my Blog. It's now move to my a host of my own. My old blog on blogger is still online, you can reach it with original URL 4k3nd0.blogger.com or b2.blog.akendo.eu   Have fun!