Bulding a virtualenv

1 minute read

virutalenv

Virutalenv is a python program that allows you to create system separate container with an own version of python/pip. All installed packages(with pip) are store there. This allow a developer to install all necessarily packages into the userspace. It's smoother and allow to have separate version of the same project, for example django. Starting command:
virtualenv --no-site-packages .env
To apply the virtualenv to your local environment run
source .env/bin/activate
(.env)akendo @ akendo :: .../Django
That will change you bash environment to use the python/pip version of the virtualenv. Now you can install you python packages via pip without interfering with other version in the system. An example for our django project, this the list of need packages, all stored in a "requirements.txt" file. To create a requirements.txt, push all the requiert package into the file. You can use the pip freeze command to create file for all installed packages.
 django==1.2.7
 django-celery
 psycopg2
 PIL
 BeautifulSoup
 Markdown
 django-tastypie
 django-oauth
 oauth2
 simplejson
When you have all, just use the -r option to the pip call:
pip install -r path/to/requirements.txt
 
Sources:
Setting up Django virtual environment via python  
 

Enable remote access for PostgreSQL

2 minute read

Remote access to a Postgtres Database

In the last days i have worked a lot with PostgreSQL. We have some Django Application which needs some extra SQL love. I have a installation script what insert all the script to the extra SQL into the database. The Problem: " How to do this on a remote database host, without coping everything to it more often?" A try with psql command show also support for remote host. The great thing about psql is that will automatic using a SSL connection. The command is:
psql -h $DATABASE_HOST -d DATABASE -U $ROLE
To make this work, i have first enable that the socket, to  listing on a different Interface then localhost. Change listen address in postgresql.conf
#------------------------------------------------------------------------------
 # CONNECTIONS AND AUTHENTICATION
 #------------------------------------------------------------------------------
 # - Connection Settings -
 listen_addresses = '0.0.0.0' # what IP address(es) to listen on;
 # comma-separated list of addresses;
 # defaults to 'localhost', '*' = all
 # (change requires restart)
 port = 5432 # (change requires restart)
 max_connections = 100 # (change requires restart)
 # Note: Increasing max_connections costs ~400 bytes of shared memory per
 # connection slot, plus lock space (see max_locks_per_transaction).
Add your accessing host to the pg_hpa.conf
# Database administrative login by UNIX sockets
local all postgres trust
# TYPE DATABASE USER CIDR-ADDRESS METHOD
# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 trust
host myDB all 172.17.7.2/27 trust
Now restarting the postgreSQL and i can access via psql from my host.
psql -h POSTGRESQL_HOST -U $USER -d myDB
psql (8.4.11)
Type "help" for help.
myDB=#
 

Security note:

The "trust" is taking every request reguardless who is doing. This IS very DANGEROUS! I did it becourse i'm a lazy admin and don't want to insert everytime a password during deployments. Be aware of Spoofing Attacks!

rsync - the better scp

3 minute read Regard this Post, i make this post. First: scp is great. But i recomand to use rsync instead. It has replace for me also the cp and scp command. Why? Simple rsync is more powerful. I allows me dry runs, then it's very efficient. File moves should be easy and save. That is what rsync does! It's some tool that need a little bit use. rsync is simply for backups, but that is almost the same coping around like the cp/scp.

Some examples

Start with a dry run:
rsync -nv  somefile user@server:/path/to/copy
This will connect as user to server. Then show what he will copying, That is very useful to check when your running recusive. A / makes a lots of different in the path showing.
 rsync -rnv  somefolder user@server:/path/to/copy/
somefolder/a
rsync -rnv  somefolder/ user@server:/path/to/copy/
./a
 

Problems with scp

You have a big file you want to push to your vServer, you don't want fill up all of your bandwidth so you set a set a limit of for bw.  Your using scp -l, so we setting a limit of 150 kB/s
scp -l 150 somefile user@server:/path/to/save
I got some wired problem with the scp limiter. Sometimes it works, sometimes it didn't. Why? That is the version with rsync
rsync --bwlimit 150 somefile user@server:/path/to/save
It's basically the same, but it worked reliable on rsync. But when the copy process stops before your done (lost connection for example) you have to start over again. .  The solution is using inplace option in rsync:   inplace copy: rsync is able to copy a file on inplace. This very important you are able to suspend a copy process.
rsync --inplace somefile user@server:/path/to/save
(Lost Connection)
rsync --inplace somefile user@server:/path/to/save
  Keeping the rights, scp will copy and place with the default umask of the system user. rsync allow you to keep the right permission with -a option
ls -l somefile
-rw-r--r-- 1 akendo akendo 347  4. Dez 16:21 somefile
scp -a somefile user@server:/path/to/save
user@server: ls -l somefile
-rw-r----- 1 user user 347  Mar  7 00:19 somefile
rsync -a somefile user@server:/path/to/save
user@server: ls -l somefile
-rw-r--r-- 1 akendo akendo 347  4. Dez 16:21 somefile
As you can see, lose the User/Groups , Time Stamp will be lost with scp. In rsync this will remain the same.   Using Humanreadable
rsync -vh somefile user@server:/path/to/copy
somefile
sent 2.02K bytes received 112 bytes 1.42K bytes/sec
total size is 964.25M speedup is 452912.60 (DRY RUN)
  Some nice looking example, to clone a folder:
rsync -rauvh /file user@server:/path/to/save
This will copy the folder "file" to "/path/to/save" on the host server as user. In some other post i show the best way of using it as a backup method. Feel free to add comments or improve this post.  

Howto fix libgeos_c.so not found

1 minute read I'm working with Python Django and PostGIS. When i try to add the Geo support to the Project, i hit this error message:
OSError: /usr/local/lib/libgeos_c.so: cannot open shared object file: No such file or directory
As I notfied about that he is try to work with the /usr/local/ path what is wrong. I found the lib installed corretly to the /usr/lib/libgeos_c.so . So i simply link it to the /usr/lib/ folder:
sudo ln -s /usr/lib/libgeos_c.so /usr/local/lib/
That does the job.
Make sure that you have the sci-libs/geos installed. 

svn2git

2 minute read

I'm currently moving our old svn repositories to git. There a lots of reason to do this. Performance, Disk Space, SVN sucks, decentralization and some more.

Here is a Video from Linux Tolvads talking about Git. Tech Talk: Linus Torvalds on git

Tools Used:

I recommand this svn2git tool! It has a good documentation, easy to understand and work almost out of the box!  

Some Words:

I start looking in my local repository from Gentoo.  eix showed  git2svn from http://gitorious.org/svn2git/svn2git . There was a problem running it from our SVN Repo. He wan't able to found some URL part. So it was complete useless. I started  to google a bit. I found some nice links that explain and have some examples. Our svn contains lots of different repositories, i used this link here as a sample. The Script fits  as basic, i adjusted it, so that it fit fine for me. I change it so it can uses SSH to create the Git repositories on a remote Host.  Addtional i created a authors file. Example: stnick = Santa Claus <nicholas@lapland.com>
Code:
#!/usr/bin/env bash
REPOS="project1 project2 project3 project4"
URL="https://10.0.4.7:/Musicpictures"
REMOTE_IP="10.0.4.5"
for repo in $REPOS
do
 echo "Create local repor"
 mkdir $repo
 cd $repo

 echo "Start of svn2git progress"
 cmd="svn2git $URL/$repo -m --authors ../authors.txt"
 echo $cmd
 `$cmd`

 echo "Prepare remote Host with SSH"
 ssh root@10.0.4.5 "mkdir -p /var/git/svn2git/$repo.git && cd /var/git/svn2git/$repo.git && git init --bar"
 cmd="git remote add origin ssh://root@10.0.4.5/var/git/svn2git/$repo.git"
 echo $cmd
 `$cmd`
 cmd="git push --all"
 echo $cmd
 `$cmd`
 cd ..
 echo "DONE EXPORTING $repo"
done
exit

Addtional i change the the verbose and also i add -m for the orgirnal SVN Link, this coaint then also the classic comit ID.

The Main Problem is that not all of the repo are in the default SVN way store, some has no trunk/tags/branches structures. So i have to check that all has been converted in a right way.

Sources:
http://chrisjean.com/2009/02/21/git-project-description-file-hasnt-been-set/

How to monitor MonogoDB with Nagios

1 minute read     I  need some Monitoring for Nagios3 of my MongoDB. I found quick some refer in the  MongoDB wiki to this github link. All what you have to do is downloading the Code from github. You need to install pymongo and  python-dev, then you can using it. Here my steps:
wget -c  https://github.com/mzupan/nagios-plugin-mongodb/zipball/master
unzip master -d nagios_plugin_mongodb
rm master
apt-get install python-dev
pip install pymongo
cd nagios_plugin_mongodb/mzupan-nagios-plugin-mongodb-59a9247
cp check_mongodb.py /usr/lib/nagios/plugins/check_mongodb
/usr/lib/nagios/plugins/check_mongodb -h localhost -d monogodb_test_db
/usr/local/lib/python2.6/dist-packages/pymongo/connection.py:378: DeprecationWarning: slave_okay is deprecated. Please use read_preference instead.
 "use read_preference instead.", DeprecationWarning)
OK - Connection took 0 seconds
You can ignore the DeprecationWarning from Python.
Sources:
http://www.mongodb.org/display/DOCS/Monitoring+and+Diagnostics
https://github.com/mzupan/nagios-plugin-mongodb

Gentoo migration to Gnome 3.2

2 minute read Currently is there no developer for compiz in Gentoo. That is a problem, it has bug and no one is fix them. So the Gentoo Stuff deciecd to remove them.
!!! The following installed packages are masked:
- x11-libs/libcompizconfig-0.8.4-r2::gentoo (masked by: package.mask)
/usr/portage/profiles/package.mask:
# Jorge Manuel B. S. Vicetto <jmbsvicetto@gentoo.org> (22 Jan 2012)
# Mask compiz for last-rites unless someone steps up
# to maintain it. Removal in 30 days.
- x11-wm/compiz-0.8.6-r3::gentoo (masked by: package.mask)
- dev-python/compizconfig-python-0.8.4-r3::gentoo (masked by: package.mask)
- x11-apps/ccsm-0.8.4-r1::gentoo (masked by: package.mask)
- x11-plugins/compiz-plugins-unsupported-0.8.4-r1::gentoo (masked by: package.mask)
- x11-wm/compiz-fusion-0.8.6::gentoo (masked by: package.mask)
- x11-plugins/compiz-plugins-extra-0.8.6-r1::gentoo (masked by: package.mask)
- x11-themes/emerald-themes-0.5.2::gentoo (masked by: package.mask)
- x11-libs/compizconfig-backend-gconf-0.8.4-r2::gentoo (masked by: package.mask)
- x11-libs/compiz-bcop-0.8.4::gentoo (masked by: package.mask)
- x11-wm/emerald-0.8.4-r2::gentoo (masked by: package.mask)
- x11-plugins/compiz-plugins-main-0.8.6-r1::gentoo (masked by: package.mask)
For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook
  That why i started to move to gnome 3.2, i have try it on my Workstation. Addtional i found some nice extionas that allow me to bring some useable to Gnome 3. 2. But more late, that how i did it. Start with a look on the gnome-base/gnome package, then unmask it.
[I] gnome-base/gnome
 Available versions: (2.0) 2.32.1-r1 ~3.2.1
 {accessibility (+)cdr cups dvdr +extras +fallback ldap mono policykit}
 Installed versions: 2.32.1-r1(2.0)(15:44:58 12/20/11)(cdr cups dvdr ldap policykit -accessibility -mono)
 Homepage: http://www.gnome.org/
 Description: Meta package for GNOME 3, merge this package to install
 
 echo "gnome-base/gnome ~amd64" >> /etc/portage/package.keywords
emerge -DNuvaq world
This will enter some error message, i recommand to use the --autounmask-write to save the time by unmasking. This made end in some resolve error. Sometime it's no unmask everything in the right way, make sure you got x11-libs/cairo-1.10.2-r2 unmask, i used x11-libs/cairo-1.10.2-r1 and this caused some trouble. When you done, wait till it's compied. Restart the the Xserver or better the host.   After it, i went to https://extensions.gnome.org, here a list of some extensions i recommand: Note you need to use Firefox, Chrome has some bug and will not work yet.
https://extensions.gnome.org/extension/130/advanced-settings-in-usermenu/
https://extensions.gnome.org/extension/38/windows-alt-tab/
https://extensions.gnome.org/extension/6/applications-menu/
 
 
My feeling about Gnome 3.2: I'm very impress,  even my Suspend to Ram work almost Perfect.  It's quite workable, there some problem out like some long time for single application to start, links to file browser a broken or sometime you just have to relog. But i like it, it's smooth working and lots of fun!    

Nginx with SSL

2 minute read For a demo Application that has to be show in the Internet, we need a Scala Lift online. This contains sensitive data, so i need SSL. But the Jetty Server were all the Scala Lift Application is no able to handle Name or SSL so easliy (Or it can, but i have no idea how to do it). So i want that my Website allowing access via http, but then redicting to the HTTPS Page. Just made sure, i add a password so no one unknown can see it. First create a new SSL Certificate :
cd /usr/nginx/
mkdir SSL
cd SSL
openssl genrsa -des3 -out server.key 1024
openssl req -new -key server.key -out server.csr
cp server.key server.key.org
openssl rsa -in server.key.org -out server.key
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
add in your configuration /etc/nginx/site-avaiable/YOUR_DOMAINNAME_HERE
server {
 listen 0.0.0.0:80;
 server_name YOUR_DOMAINNAME_HERE,;
 log_format main '$remote_addr - $remote_user [$time_local] $status '
 '"$request" $body_bytes_sent "$http_referer" '
 '"$http_user_agent" "$http_x_forwarded_for"';
 access_log /var/log/nginx/YOUR_DOMAINNAME_HERE.log;
 rewrite ^(.*) https://YOUR_DOMAINNAME_HERE permanent;
 }

 server {
 listen 0.0.0.0:443;
 ssl on;
 ssl_certificate /etc/nginx/SSL/server.crt;
 ssl_certificate_key /etc/nginx/SSL/server.key;
 server_name YOUR_DOMAINNAME_HERE,;
 access_log /var/log/nginx/ssl-YOUR_DOMAINNAME_HERE.log;
location / {
 auth_basic "Restricted";
 auth_basic_user_file /etc/nginx/auth/dev-htpasswd;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-NginX-Proxy fale;
 proxy_set_header Host $host;
 proxy_pass http://localhost:8080;
}
}
Now enable the configuration and restart the nginx.  
Source:
Nginx Wiki 
josephscott.org 

Basic xserver on a vServer

1 minute read For fetching some date, my Application need a xserver. The Problem is that it's running on VM. So i install a basic xdm. I'll accessing this via VNC.
apt-get install tightvncserver
apt-get install xdm
Now starting the xdm and  tightvncserver
/etc/init.d/xdm start
tightvncserver :1
tightvncserver will ask for vnc password. This will be ask during a VNC client is connecting. Now connect to the server $IP:5901. There is a lonley bash wating  ;-)

[Gentoo]Jetty with a .WAR file

2 minute read
Jetty with a .WAR file
In my company i currenly have a lot to do with Scala Lift. We're developing some stuff with it, but because it takes a lots of time to setup a Scala environment and we need quick show solution, we'll go with a .war file. The problem on my gentoo is, there is no jetty packages, so i searched for some overlays. I still got the java-overlay. With eix jetty, i'm searching for jetty:
* www-servers/jetty [1]
Available versions: (5) ~5.1.3-r1
 {doc source test}
 Homepage: http://www.mortbay.org/
 Description: A Lightweight Servlet Engine
The jetty version is outdate, so i'm downloading the jetty from the offical webpage:
wget -c http://dist.codehaus.org/jetty/jetty-6.1.26/jetty-6.1.26.zip
#Now unzip it
mkdir jetty && unzip jetty-6.1.26.zip -d jetty/
during the download i found a useful blog entry about jetty for debian.  How to deploy a .WAR file there. It's little bit like tomcat but just easier. Move you .WAR File to the $JETTY_HOME/webapps/ folder and start the Jetty Server.
java -jar start.jar
I found also on the overlays.gentoo an updated version of jetty, this can be found on the dustin overlay
layman -a dustin
eix-sync && eix jetty
* www-servers/jetty [1]
 Available versions: (6) ~6.1.24
 {elibc_FreeBSD}
 Homepage: http://jetty.codehaus.org/
 Description: Jetty Web Server; Java Servlet container
[1] "dustin" /var/lib/layman/dustin
I add the =www-servers/jetty-6.1.24 ~amd64 to /etc/portage/package.keywords, now install the jetty:
emerge -avq jetty
so far Akendo

Web Sources:

http://lifeofaprogrammergeek.blogspot.com/2010/06/deploying-war-in-jetty.html
http://overlays.gentoo.org/proj/java
http://gpo.zugaina.org/www-servers/jetty
http://www.gentoo.org/proj/en/java/