Automating ssh, ftp, scp

A really good way to automate ssh, ftp or scp is to use "Expect" (http://expect.nist.gov/). To kick it off, just use "autoexpect" which records your key stroke into an expect script and you just need to run the expect script and it can be cron. Easy as that.

Then again, if everything is so easy, we would lose our jobs. More often than not, there is some customization required. Expect is based on tcl, so if you are familiar with tcl, you are good. If not just search for what you want to do in tcl and usually there should be someone who has already came across the problem and someone else has helped.

In my case, I need to scp all the files in one directory into a remote server. In autoexpect, I used the following command.

autoexpect scp /somedir/* user@remoteserver:/somewhere/

The problem is that the /somedir/* is expanded to list out all the files in the directory. In the generated expect file it looks like this

spawn scp /somedir/f1 /somedir/f1 /somedir/f3 user@remoteserver:/somewhere/

Now we know that the files will change so this won't work. In this case, a bit of scripting is needed:

set files [glob -directory /home/koopt/user_backups/ *]
set source ""


foreach f $files {
  set source "$source $f"
}
spawn scp $source user@remoteserver:/somewhere/

The function "glob" will list out all the files in full path and we just loop through the files to get a long list of files to scp to the remote server.

Less is More

My professor used to say "Less is more" especially when answer questions in exams. We interpret that he does not want to read through long answers. i.e. He's just lazy. :)

But there are some quotes that talks about this as well.

It seems that perfection is reached not when there is nothing left to add, but when there is nothing left to take away. - Antoine de Saint-Exupéry, Wind, sand and stars, 1939

Everything should be made as simple as possible, but not simpler. - Albert Einstein

So...I'm reading "Getting Real" written by the guys are 37signals. And the same concept of "less is more" is highlighted again. Of course, being the people who brought us Ruby On Rails. This concept should be the center of what they do everyday.

Interestingly, I received a phishing email asking me to login to 37signals to change my account password. I immediately realized that it was a phishing email and went to the real website to warn them about it. I guess the success of your web application depends on whether hackers are trying to phish your users data.

Singapore NRIC check

There are various nric check information pages, but most of them are giving you the algo. Here is one implemented in PHP.

<?php
function check_nric($nric) {
  if ( preg_match('/^[ST][0-9]{7}[JZIHGFEDCBA]$/', $nric) ) 
  { // NRIC
    $check = "JZIHGFEDCBA";
  } else if ( preg_match('/^[FG][0-9]{7}[XWUTRQPNMLK]$/', $nric) ) 
  { // FIN
    $check = "XWUTRQPNMLK";
  } else 
  {
    return false;
  }

  $total = $nric[1]*2
    + $nric[2]*7
    + $nric[3]*6
    + $nric[4]*5
    + $nric[5]*4
    + $nric[6]*3
    + $nric[7]*2;

  if ( $nric[0] == "T" OR $nric[0] == "G" ) 
  {
    // shift 4 places for after year 2000
    $total = $total + 4; 
  }

  if ( $nric[8] == $check[$total % 11] ) {
    return TRUE;
  } else {
    return FALSE;
  }
}
?>

phpBB3 with youtube

To allow users to embed youtube videos in your pbpBB3 forum, do the following:

1. Go to the Administration Control Panel (ACP)

2. Click on the "Posting" tab, click "Add new BBCode".

3. In BBCode usage, enter the following:

[youtube]http://www.youtube.com/watch?v={SIMPLETEXT}{TEXT}[/youtube]

4. In HTML replace, enter the following:

<object width="425" height="350"><param name="movie" value="http://www.youtube.com/v/{SIMPLETEXT}"><param name="wmode" value="transparent"><embed src="http://www.youtube.com/v/{SIMPLETEXT}" type="application/x-shockwave-flash" wmode="transparent" width="425" height="350"></embed></object>

5. In Helpline, put anything you want, I use : Embed youtube links

6. Check "Display on posting page".

7. Submit.

With this, you can embed youtube links like this:

[youtube]http://www.youtube.com/watch?v=ZGx2Mu2Yaig&feature=channel[/youtube]

On some sites, the solution requires you to enter only the video id e.g. [youtube]ZGx2Mu2Yaig[/youtube], which is crazy. Which user will carefully select the id and paste it? Users will just copy everything in the address bar and paste it into the forum.

You may have tried other similar solution, and wondered why those do not work. The trick is in the bbcode, we have {SIMPLETEXT}{TEXT}. Often, a user will copy a youtube link with additional parameters (e.g. feature=channel ), which these parameters are handled by the {TEXT} and we throw these parameters away when we display the youtube object on screen. We only want the video id, which is supplied by the "v" parameter.

PHP - killing long running exec process

Was writing a php script to run in commandline to process some files. Part of it is to run a 'whois' command from within PHP, but it hang (ip is 74.220.96.0). I don't know why.

Anyway, I searched and found this:

<?

exec("command & sleep 3 ; kill $!");

?>

S.M.A.R.T.

I'm attending the PMP (Project Management Professional) course in the mornings and the NCAP (National Coaching something) in the evenings.

One thing that came up in two very different courses is the concept of S.M.A.R.T. for setting goals.

S - Specific
M - Measurable
A - Achievable
R - Relevant
T - Time bound

Threading?

So we have to process a huge amount of data and send them to a different machine. It seems that the sending part (the actual HTTP request in the SOAP call) is slow, and it was suggested to speed things up with multiple threads.

Firstly, my knowledge of PHP is limited on threading, and secondly it seems PHP don't have sophisicated threading APIs like Java or dotNET.

So I came up with a solution that makes multiple SOAP calls to the remote server, but not using threads.

In processing the data, the table has a row id. Simply decide how many concurrent processes you want, and pick out the data to process by the modulus. e.g. if you want to have 2 concurrent processes to send out SOAP messages, pick out the data as such

where id%2=0

and

where id%2=1

Two cron jobs are created, one to call each script with the different parameter (of course, we can do this via a bash script so only one cron job is created).

So there, no race-conditions, no hung threads (ok, maybe hung process), and no headaches.