Archive for the ‘ Linux ’ Category

Locked SVN Repo

Have you ever been working on an SVN server and had to ask yourself, “Why in tar-nation is this file not commiting???!! And who is Joey to be so important to lock a file.” For which you gather all your rage and ask Joey why he had the audacity to lock the files you needed to commit? He tells you he didn’t lock them, nor is he working in the same area of the code. Then you discover that by some way he accidentally locked 1/4 of all the files scattered throughout the repository. Since you are very smart you went to the server command line and did:
svnadmin lslocks [path to repo]
And it told you all the locks in the system.

Now you should be thinking, how can I take that list and then unlock the whole server? I too faced this same problem. This is how I dealt with it. First don’t mess with svnadmin rmlock… I couldn’t make it work to save my life. The setup is like this, we are going to grep out some keywords from the svnadmin lslocks to get only paths, then we are going to use awk to help us build perfect paths, then we are going to use the svn client to finish the job. And it all fits on one line.
*Important: Make sure your local copy is updated to the latest revision.
svnadmin lslocks /usr/local/svn/repos/[repo-name]/ |grep Path | awk '{print "file:///usr/local/svn/repos/[repo-name]" $2}' | xargs svn unlock --force

Thanks to some nifty piping, you have just unlocked your whole SVN. Let me explain some key points here, “Path” next to grep will draw out something that looks like this:

Path: [Path to locked file]
Here is a really quick run down of awk, basically it takes any white space and then seperates the information into variables starting with $0 being the whole line, $1 being the first, $2, the second, etc. $1 would be the word “Path:” and $2 is our path. But that is not enough because we have to make it a qualified repo path, and since I am doing it on my local server I can use the “file:///” prefix. After that we send it to xargs and then use svn unlock and xargs will apend the argument to the end of the line. –force is also important because that will make sure you steal the lock in the unlock process.

FINE PRINT aka Caution: SVN locks were designed to protect files while a person is working on those files so no one else could over write them while they were developing on those files. The idea is that someone can get exclusive access to the files, change them, commit them, and then release the lock when the work is completed. If you are using this type of idea for your system, please send the list of locks to all of your developers and then have them manually unlock the locks they set. Otherwise you can ruin and destroy work in progress if you are not careful. Now if you do not care about locks and know what is being developed, by all means use the function above.

If you have a better way to unlock your svn repository I’d love to hear it, I created this method from my own knowledge of the shell. If you need some help or have questions, post a comment and I’ll get back with you.

Cheers,

-m

Ubuntu 10.10 and Grub 2 Fun

I am a few distros behind the current Ubuntu. I have been using 9.04 and 8.10 since I have found them very stable and familiar. For kicks, I installed 10.10 Server on a new project and thought I would find the same things I found in previous distros. For the most part I did. However, I stumbled across a stupid change that crippled my server. First off, the server is a headless and keyboardless setup. I know, I should run an IP KVM for complete control, but my other servers haven’t warranted it yet.

The issue is that when the server looses power, and then during boot it looses power again, it throws a “Recordfail” flag that can be used to change the way Grub2 boots. In the default configuration of Ubuntu 10.10, they have chosen to display the boot menu without a timer when there is a record fail. It is much reminiscent of Windows when it fails to boot properly and gives you the boot menu for different modes.

In a desktop environment, this would be fine because I could choose to run recovery, run normally or whatever. However, in my configuration, I want this thing to boot even it if is on fire. If it doesn’t boot, it should be cause some hardware needs to be replaced. After some digging, I found the solution. Edit /etc/grub.d/10_linux and commment the following lines like so:

# recordfail=1
# save_env recordfail

Save the file and then run sudo update-grub to generate a new grub.cfg file. Viola, no more stalled boots.

Cheers,

-Mike

Boot Script (Startup Script) with Ubiquiti AirOS

I have been hitting my head against the wall trying to set the bridge priority of a few radios involved in spanning tree. It is easy to log in to the radio and change the priority with the brctl setbridgeprio br0 . But what if you want to change it on boot and automatically? I dug through the Ubiquiti forums only to find bits and pieces that lead me to the final solution.

There are a number of key files that don’t exist by default, but can be setup to provide the scripting functionality you may need.


/etc/persistent/rc.presysinit
/etc/persistent/rc.postsysinit

/etc/persistent/rc.prestart
/etc/persistent/rc.poststart

/etc/persistent/rc.prestop
/etc/persistent/rc.poststop

If you have read any of my other posts you will see that I am a huge supporter of VIM, and will assume you are using it. You can simply do vi /etc/persistent/ and then write your script. For my example, I will give you a copy of my bridge priority script:


#!/bin/sh
brctl setbridgeprio br0 7000
brctl setpathcost ath0 10

In a previous post I talked about the command that makes your /etc/ directory and changes to it persistent.


cfgmtd -w -p /etc/

This command will write the changes to /etc/ to flash and then you will be able to reboot the system and the new script will take effect. Hopefully you found this helpful.

Cheers,

-Mike

Another Round With Qmail

In today’s match we have a Qmail Server that has been demoted from sending and receiving email to just sending. The server had been properly configured for the new changes and email was successfully being sent off the server. Today, however mail stop sending to only one domain, the old domain that was hosted on the server.

The first place to look is the logs. Read the logs, know the logs, they will help you. Here is a command that will take the non-human readable date code and make it readable for better understanding:
cat /var/log/qmail/qmail-send/current |tai64nlocal |less

There are 2 major logs to check when dealing with qmail:
/var/log/qmail/qmail-smtp/current
/var/log/qmail/qmail-send/current

Here is a sample of a bounce message log entry:

2010-10-08 11:04:05.068209500 starting delivery 280: msg 1328309 to local dump@maul.com
2010-10-08 11:04:05.075688500 status: local 1/10 remote 0/120
2010-10-08 11:04:05.141754500 delivery 280: failure: Sorry,_no_mailbox_here_by_tha
t_name._(#5.1.1)/
2010-10-08 11:04:05.146678500 status: local 0/10 remote 0/120
2010-10-08 11:04:05.155633500 triple bounce: discarding bounce/1328309
2010-10-08 11:04:05.158451500 end msg 1328309
2010-10-08 11:08:06.159425500 new msg 1327108
2010-10-08 11:08:06.159435500 info msg 1327108: bytes 1025 from qp 5008 uid 0

The important thing to follow here is the “delivery 280.” That is the delivery number that will allow you to follow the message. In this case we can see it was a local delivery that failed and now is bouncing.

It can be daunting to dig through the log files, but they will have the answer if you look.

Cheers,

Mike

USB to Serial, Linux comes with built-in drivers!

I was trying to communicate with a serial device today and found that Linux is much more resourceful than M$oft products (Obviously). I plugged in my USB to serial adapter, and typed in dmesg and saw it there in the list, ready to use. But now, how to use it?

First determine where you USB device is located. You can ls /dev/ and look for it… Since there are 700+ items there, I’ll give you a hint: ttyUSB is what it should start with. Mine was /dev/ttyUSB0.

I found a tutorial of a little program called minicom. Minicom is a lightweight terminal client that is quiet cool in my opinion. Start minicom with the following:

minicom -s

This will put you in interactive configuration mode. Change your port information to what you discovered above and make sure to set the baud rate and other parameters. Now choose to save the config as dfl (default) and away you go.

Thanks again Linux for being awesome!

-Mike

Qmail gets stuck, Qmail gets slow, and finally QMAIL GETS FIXED!!!

Recently, I have been tasked to deal with a Qmail mail server that was very foreign to me. I have never administered a Qmail server before. Since I have experience with Postfix, Exchange Server, Xim and others, I figured Qmail couldn’t be too difficult right? Wow, I was wrong. Turns out Qmail is a culmination of many tools that do different things that enable it to be a complete software. There are a large number of plugins and modifications one can make. One forum I found suggested that a person install Qmail from source and get all the addons and plugins running they want and after it works, they will know how Qmail works. One person I talked with said it is nearly impossible to troubleshoot complex issues with Qmail because each installation can be different.

The Problem
If the title wasn’t clear enough of what the problem was, I’ll completely explain here. I was seeing issues with a queue that would never drop lower than 2000. Also we were having problems with mail clients that would never complete the send process and messages would get stuck in peoples outboxes causing a large number of duplicate emails being sent to customers. This issue also caused messages that were trying to be sent to our domain to be delayed or stuck… or worse duplicated.

The Solution
For this problem there was no simple fix. It took a lot of testing and configuring to determine where and what was causing the issues. First off, the guys at irc.freenode.net #qmail had a bunch of helpful suggestions that ultimately lead me to success. Remember to check out IRC communities, and make sure you give back. The next thing that really helped was Google. Like most problems in life, someone else has probably done the exact same thing. Google the issues and find the answers.

First part of this solution is to determine what exactly is wrong. How do you know what is wrong? Do predefined tests to see what works and what doesn’t. In my case, I used a Qmail-inject tool, telnet, Thunderbird and Outlook. I thought of each tool as a different level of mail sending so that I could eliminate where the problem was at. Qmail-inject is very easy, just type the following at your root command line:

echo to: email@host.co | /var/qmail/bin/qmail-inject
This will send the email directly to the Qmail queue and will tell you if you are able to send messages directly from the local machine/server.

The next tool is to connect to your server with telnet:
telnet host.co 25
Here is a web site that explains how to do an Auth session and send mail beautifully!
SMTP AUTH TELNET

If you have any problems with this process, send me a comment and I can help out. Should be fairly easy to see this.

Ok, next you need to get qmHandle. Go To Sourceforge Now

This tool is awesome. It has a large number of features that can only be explained by the help: ./qmHandle

Next we have qfixq. A brilliant program I found written by someone who was more frustrated than me. Go to qfixq.
While you are at the qfixq site, be sure to poke around for other cool scripts and tips. Lots of good information there. I think I have tried out everyone of the scripts on his site now.

qfixq is a script that will let you rebuild the Qmail queue if it gets jammed up. Mine was very jammed up. It also has a cool mode where it will empty everything as well. It is a life saver.

Next up is the logs. I now know every meaningful log for Qmail. Let me go through the important logs with you:

/var/log/qmail/qmail-smtpd/current
This has information about SMTP connections into the server. It will display current activity and can give you a status report. For me, our server was maxed at 350/350 and remote concurrent connections were being blocked causing some of our issues.

/var/log/qmail/qmail-send/current
This has information about things being sent in the queue. This is very helpful once items are dropped off from SMTP. I was able to follow from the SMTP log to this log to track down what is happening to messages. Check for local and remote delivery queues. You can also see the state of messages that are trying to be delivered.

/var/log/maillog
This has information about Spamd and ClamAV. You can see spam scores and scanning results here.

Here are some important config files that you must keep in mind so that you can make sure things are setup correctly:

/var/qmail/supervise/qmail-smtpd/run
This script has the run elements for the SMTPD daemon that is running. When you edit this with a text editor, you see a lot of things happening here. This was actually the fix for me. Our issue originated from an overwhelming spam attack. I was able to add some real time black lists (RBL’s) to thwart the evil spammers. All I added was thet following:
/usr/local/bin/rblsmtpd -t 5 \
-b -r b.barracudacentral.org \
-b -r zen.spamhaus.org \

I added this after the “smtp” tag and the rest was history. I reboot the server and there were very low connections to it. Everything was getting blocked, like it should.

Summary
The moral of the story is to use strong black lists and watch your logs. Things seem to be running quite well now. I also ran qfixq to make sure there were not any issues with that. These steps to test, research and repair should be very helpful to anyone who was stumped like me. Good luck!

Cheers,

Mike

Bluehost with SVN

Recently we have been tasked to provide SVN support with Bluehost.  Initially I was reading and found it was not possible to complete.  After a bit more searching I found that it was possible.  The steps here are not as simple as “apt-get install svn”, we actually have to compile it from source.  Since I have been extremely busy lately, I tasked my friend Henry for the job and then told him I could provide support when needed.

With the tutorial in hand, he went to work.  Here is the steps outlined in the tutorial:

cd ~
mkdir src
cd ~/src
wget http://www.gtlib.gatech.edu/pub/apache/apr/apr-util-1.2.12.tar.gz
wget http://www.gtlib.gatech.edu/pub/apache/apr/apr-1.2.12.tar.gz
wget http://subversion.tigris.org/downloads/subversion-1.5.4.tar.gz
wget http://www.webdav.org/neon/neon-0.28.0.tar.gz
tar -xzf apr-util-1.2.12.tar.gz
tar -xzf apr-1.2.12.tar.gz
tar -xzf subversion-1.5.4.tar.gz
tar -xzf neon-0.28.0.tar.gz
cd ~/src/apr-1.2.12
./configure --prefix=$HOME LDFLAGS="-L/lib64"
make
make install
cd ~/src/apr-util-1.2.12
./configure --prefix=$HOME --with-apr=$HOME LDFLAGS="-L/lib64"
make
make install
cd ~/src/neon-0.28.0
./configure --enable-shared --prefix=$HOME LDFLAGS="-L/lib64"
make
make install
cd ~/src/subversion-1.5.4
./configure --prefix=$HOME --without-berkeley-db  --with-ssl LDFLAGS="-L/lib64"
make
make install

After making it through all of the above configuring and compiling, the binaries are sitting in ~/system/bin ready for use.

Now to add a new repo, type:

svnadmin create /[dir]/[repo name]

Now import files to the repo:

svn import /[dir]/[repo name] [svn string, starting with svn://]

Now the repo is ready to add files or to check them out. But, you will soon learn that you need to add somethings to the path in order to properly execute. Once I added the things to the path, it still didn’t work. I found out that bluehost doesn’t allow execution of the .bashrc if it is a single execution item. I found a work around:

First you will have to add a public key to the .ssh/authorized_keys from your system so that you can login without using a password.  If you need help with that, here is an awesome tutorial that works very well! SSH Key Tutorial

Add the following before your rsa key in ~/.ssh/authorized_keys

command="/home/USERNAME/system/bin/svnserve -t" ssh-rsa [KEY REMOVED]

Now you should be able to login using the CLI for the system that you have added the key for and use SVN.

Now that you have everything else setup, it is time to get TurtoiseSVN client up and running.

First install Putty, if you need help with that, click here.

For those that didn’t click the link, click it for fun :D .

Next you are going to need to get WinSCP or Filezilla or some other program that can do file transfer over SSH.  I prefer Filezilla because it is Filezilla.  If you don’t have either of those, simply Google them and you’ll find them.

Ok, now you will need to log in to your Bluehost server with Putty.  Once you have logged in, type the following:

ssh-keygen -t rsa

This will create 2 files id_rsa and id_rsa.pub in the .ssh/ directory. First you will want to copy the public key to make it so you can use this key on this server, type the following:

cp .ssh/id_rsa.pub .ssh/authorized_keys

Now I will assume you are using Filezilla. Open Filezilla and then for the hostname, use “sftp://[domainname.com]” and your username and password. Navigate to the ~/.ssh directory and download id_rsa to a directory on your computer of your choosing.

When you downloaded putty, you should have also downloaded PuttyGen. Open PuttyGen and then select the Conversions menu and select Import Key. Find the id_rsa file you just downloaded and import it. After it is imported select the “Save Private Key” button and export the .PPK file to your choosing.

Now we are going to open Putty and save a new session with our newly created key. Once Putty is opened, go to SSH and then to AUTH and then select the private key you exported above. After that go back to the sessions option and type the whole domainname (including .com/.net/.whatever) as the “Saved Session” and push the Save button. It is important to save the session because Tortoise will use this in just a bit.

You can test your SSH login to make sure it doesn’t ask for a password and you will know that you are on the right path.

Next up, we have to make some changes to your Tortoise config file located at C:\Documents and Settings\YOUR_LOCAL_USERNAME\Application Data\Subversion\config
If you have Vista or Win7, it will be located here:
C:\Users\YOUR_LOCAL_USERNAME\AppData\Roaming\Subversion\config
Now find the section called [tunnels] and add the following:

ssh = c:\\Program\ Files\\TortoiseSVN\\bin\\TortoisePlink.exe

Link in Linux, you have to escape the backslashes with a backslash.

Ok, you should be ready to rock at this point. Time to install Tortoise if you haven’t already. Download it here. Follow the installer, should be easy.

Time to check something out. Try to checkout a repo now by right clicking in Windows Explorer and choosing “SVN Checkout.” Use the following string but add your own info:

    svn+ssh://youraccount@[domain]/home/[your account]/[repo path]/[Project]

Hopefully everything worked, if it didn’t, hit me up and I’ll help out.

Mike

Trixbox, VNCServer Skype, and SipToSis

Today I was on a mission to get Skype and SipToSis to automatically start on boot. Since Trixbox only loads a web based GUI and everything else is command line, you have to include other packages to get Skype working. Another helpful fact is that Trixbox uses CentOS and the yum package manager. Yum has a nice feature of the group install which will include all required packages and dependencies to get a group running. Here is the command to get Gnome installed:

  • yum groupinstall “GNOME Desktop Environment” –exclude=gimp-print –exclude=gimp-print-utils

Once you get Gnome installed (you can choose KDE or another DM if you would like) you need to set the run level of your system so that it boots to the GUI, unless you don’t want that.  Crack open your favorite text editor (VIM for me) and edit your inittab:

  • vi /etc/inittab

You are looking for a line that says id:3:initdefault:, if you find it, make sure you change the 3 to a 5.  This could break your machine, so make sure you know what you are doing, or are at the machine when you make the change.   The init level is the run level for the machine.  Run level 5 is typically the GUI level.  This command just makes it so that items set to run on level 5 start on boot.

Next we have to start VNC on boot.  To do that use chkconfig.

  • /sbin/chkconfig vncserver on

To finish up vncserver, you have to edit the startup config in sysconfig.

  • vi /etc/sysconfig/vncserver

Make sure you uncomment

  • VNCSERVERARGS[2]=”-geometry 800×600 -nolisten tcp -nohttpd -localhost”

and add:

  • VNCSERVERS=”2:root”

This will make the port 5902 since it will start vncserver on display 2.

To get Skype to startup on boot I just added skype to the path and then added skype to the session for Gnome.  Then to get SipToSis to startup on boot, I wrote the following bash script:

#!/bin/sh
cd /root/siptosis/
./SipToSis_linux >> /var/log/siptosis

This will give us the output in a log so that we can review it in case there are errors.

Alright, time to head to bed… Made some good progress with this.

Mike

Trixbox, USB Stick, Grub, TDM card, what more could you want?

I stumbled across Trixbox some time ago and thought it was an excellent replacement for Shortel and Switchvox. Trixbox is a PBX system that is built on top of Asterisks. It is much like Switchvox with its web interface and ease of use. The nice thing is that the Community Edition (CE) is completely free, unsupported, but free. For this adventure, I have been working with an Openvox TDM card, Trixbox on a USB Stick, and SATA RAID. The reason I am using a USB stick is that when the RAID is active, you can not hook up a CDRom to the SATA controller. Ok, this seems fine, but when you choose IDE for SATA in BIOS, it breaks the RAID and according to the controller, it must be “Rebuilt.” I was able to get Trixbox to load onto a USB stick thanks to the people at Pendrivelinux.com and their MultibootISO application. This application uses Grub4Dos to emulate a CD-Rom drive from an ISO file. The only issue I had was when Trixbox installed the bootloader (Grub), it installed it to the MBR of the Flash drive. I was able to get Trixbox to boot off of the Hard drive by using the kernel (hd0,0)/[dir to kernel] and initrd (hd0,0)/[dir to initrd]. Once I had it booted up, I was able to modify /etc/grub.conf to remove the Flash drive. After that, I had to edit /boot/grub/menu.1st with the changed HD info. I was able to fix the MBR by using the grub-install command on the SATA Raid device, which was in a strange place. I think it was /dev/mapper/[name of RAID]. Once I referenced that, I was able to get Trixbox booting all by its self.

Now it was time to configure the TDM card and the extensions. I looked in the system config for the TDM card, but I didn’t see it. For some reason, it was not recognized as Zaptel. I am not entirely sure I understand the difference between the 2 device drivers/emulators, but all I knew is that the guide I was working from said I needed zaptel. Openvox has Zaptel drivers on their website, but I didn’t feel like compiling from scratch.

When I was on Openvox’s website, I noticed that the extra power connector on the TDM card was only needed for FXS ports. The reason is that FXS provides voltage across and FXO receives voltage. This helped explain why the all FXO card was being recognized as FXS. Once I rebooted the system, the TDM card was not being recognized by Asterisk. I tried to run the first_run scripts from Trixbox (/var/Trixbox/first_run…), but they couldn’t initialize the card. Modprobe was failing with a FATAL message meaning it can’t find the driver.

I tried to compile the driver from source, but not all of the Linux-Kernel Header files were included when I got the development files. I then tried to uninstall Trixbox with ‘yum remove [trixbox package]‘ but that wasn’t very successful. I tried to re-install Trixbox with Yum using the repository. I had a bunch of issues and dependencies issues, but eventually got Trixbox installed and running again. Now the only issue was that the system still didn’t recognize the TDM card. Now I am going to do a fresh install from USB and then start from there.

Wish me luck.

Mike

Awesome Commands for Linux

I used to have a Wiki that I would keep track of all of the “Cool Commands” that I found or created.  But now that Wiki is down and I haven’t had the time to put it back up.  So, I need to keep track of commands so I don’t forget them.   Alright, starting off, have you ever wanted to change a few lines of text in a bunch of files?  I have.  I found a one line Perl script that will take care of it for you, and leave you a backup file for your trouble.

  • perl -pi -i.bak -e ‘s/searchval/replaceval/’ *.html

This command can use regular expressions for the search value and the replace value.

This next command will sync your information and can be used to only keep updated copies of files.   There are lots of options too, this only scratches the surface.

  • rsync -uav --progress /srcdir/ /dstdir/

This command will will look at the srcdir and then determine if the dstdir file is newer, if it is, it won’t copy.  Very useful if you need to merge multiple copies of files, but don’t want duplicates.

This next command will find files for you based on all sorts of criteria:

  • find . -mtime -10

This command will look for files that are -10 days old.

You can combine rsync and find to search for files and then sync those files.

  • find . -mtime -10 -print0 | rsync -av --progress --files-from=- -from0

This command will first compile a list of files that meet the find criteria and then will port them to rsync.  Very helpful.

Ok, that is all the cool commands for today, I will probably find some more tomorrow.

Found more… This command will make a file of any size for you:

  • dd if=/dev/zero of=file.out bs=1MB count=100

This command will create a 100 meg dummy file. Works great.

Mike