Return Boolean True

Backing up and Archiving Google Mail Accounts

Thu, 27 Oct 2011 14:52:00 -0400

The easiest way to backup Google Mail accounts is to enable IMAP on the account and download the messages using getmail.

Once getmail is installed you need to create a file called getmailrc. If you plan to download multiple gmail accounts then you might want to create a directory for each account and point the getmail script to that directory. Here is an example of a getmailrc file for Google Mail:

[retriever]
type = SimpleIMAPSSLRetriever
server = imap.gmail.com
username = username@example.com
password = examplepassword
mailboxes = (“[Gmail]/All Mail”,)
port = 993

[destination]
type = Maildir
path = ~/username@example.com/

[options]
received = false
delivered_to = false
read_all = false
verbose = 1

After this file is saved you can procede to run getmail. I needed getmail to run all night and in the background. I outputted all the stdout to a logfile so I used the following command:

getmail —getmaildir . > output.txt 2>&1 &

Don’t forget to create the directories cur, new, tmp as these are the directories that are needed for IMAP.

Now that you have your mail in a Maildir format what do you do with it? In my case I wanted to delete the account off Google Apps but still be able to search the mail if I needed it at a later date.

The strategy I came up with to bring up a copy of courier and serve the Maildir using a webmail script (in this case Roundcube).

I installed PHP through Nginx first. The easiest way to get a PHP environment up and running on Nginx is to use the Ubuntu packages:

php5-cgi
php5-common

For additional functionality such as PostgreSQL support you can install the package:
php5-pgsql

I then configured nginx with the following script:

server {
listen 80;
server_name webmail.example.com;
access_log /var/log/nginx/access.log;
log_subrequest off;

location / {
root /www/webmail.example.com;
index index.php;

location ~ .php$ {
include fastcgi_params;
fastcgi_pass localhost:9000;
fastcgi_param SCRIPT_FILENAME /www/webmail.example.com/$fastcgi_script_name;
}
}
}

I then created the script: /etc/init.d/php-fcgi
BIND=127.0.0.1:9000
USER=www-data
PHP_FCGI_CHILDREN=15
PHP_FCGI_MAX_REQUESTS=1000

PHP_CGI=/usr/bin/php-cgi
PHP_CGI_NAME=`basename $PHP_CGI`
PHP_CGI_ARGS=”- USER=$USER PATH=/usr/bin PHP_FCGI_CHILDREN=$PHP_FCGI_CHILDREN PHP_FCGI_MAX_REQUESTS=$PHP_FCGI_MAX_REQUESTS $PHP_CGI -b $BIND”
RETVAL=0

start() {
echo -n “Starting PHP FastCGI: “
start-stop-daemon —quiet —start —background —chuid “$USER” —exec /usr/bin/env — $PHP_CGI_ARGS
RETVAL=$?
echo “$PHP_CGI_NAME.”
}
stop() {
echo -n “Stopping PHP FastCGI: “
killall -q -w -u $USER $PHP_CGI
RETVAL=$?
echo “$PHP_CGI_NAME.”
}

case “$1” in
start)
start
;;
stop)
stop
;;
restart)
stop
start
;;
*)
echo “Usage: php-fastcgi {start|stop|restart}”
exit 1
;;
esac
exit $RETVAL

After php was up and running I installed courier:
apt-get install courier-imap courier-imap-ssl

I then downloaded roundcube and configured roundcube as necessary.

Google Apps Split Delivery for Email - Have your cake and eat it too

Wed, 19 Oct 2011 13:57:00 -0400

Split delivery for e-mail is when you have a single piece of mail but want a copy of it sent to multiple destinations. There are a couple of reasons you would want to do this:

- You are getting ready to migrate to Google Apps but don’t want to go all in yet with your current e-mail server. This is understandable since you want to test out Google Apps first to see if it will work.

- You like Google Apps but don’t want to pay the yearly fee. You rather just stay under the limit of the free accounts but still want to have e-mail accounts on your domain (i.e. @example.com).

- You have other special circumstances where you want a copy of all the mail that comes in and have it delivered to some other server.

This post is actually more toward the 2nd point. 90% of my e-mails are on Google Apps but there is a remaining 10% that I rather not have a Google Apps account. However I still want them to have e-mail via some webmail client.

To get this working I am using Ubuntu with Postfix running the primary mail server for example.com. I set the MX records for example.com to this:
10 mail.example.com
20 ALT1.ASPMX.L.GOOGLE.COM

Pretty straight forward so far. The trick is that you need to get Postfix to forward a copy of all Google Apps mail to their mail servers. To do this I use Postfix’s Before-Queue Content Filter: http://www.postfix.org/SMTPD_PROXY_README.html.

This allows me to create a SMTP server that Postfix will delivery a copy of the mail to Google Apps. In the file /etc/postfix/master.cf I put the following at the end:

# =============================================================
# service type private unpriv chroot wakeup maxproc command
# (yes) (yes) (yes) (never) (100)
# =============================================================
#
# Before-filter SMTP server. Receive mail from the network and
# pass it to the content filter on localhost port 10025.
#
smtp inet n - n - 20 smtpd
-o smtpd_proxy_filter=127.0.0.1:10025
-o smtpd_client_connection_count_limit=10

This makes Postfix delivery a copy of the incoming mail to the SMTP server at 127.0.0.1:10025.

Okay but now you are asking where do I get an SMTP server to do the processing? It so happens Python comes with a smtp server library. I wrote a script that basicallyhttp://www.blogger.com/img/blank.gif inherits the SMTPServer (called CustomSMTPServer). It also implements the one command that is expected by Postfix (EHLO) because Postfix actually speaks ESMTP. I did this by subclassing the smtpd.SMTPChannel class. One caveat is that I had to use the _SMTPChannel__variablename syntax because some variables like fqdn and greeting were made private by the SMTPChannel class. So with Python you use the special syntax of prepending the class name to access it. This is generally bad practice but in this case it was all I had.

You can download the script here:
http://dl.dropbox.com/u/2177278/pymailforwarder.py

Simply run the script to start a basic SMTP server listening on port 10025 and localhost. The script simply accepts a piece of mail and forwards it to Google App’s mail server.

So what does this let you do? In my case this lets me run Roundcube, Horde, or SquirrelMail for users that don’t need a Google Apps e-mail account. For those that do I simply create that user on Google Apps.

Things to watch out for with this type of deployment:

- You have to edit /etc/postfix/main.cf and add the value
local_recipient_maps =
(Yes that is a blank or equals nothing). This makes Postfix accept the mail even though the recipient is not in the list. This can be bad. Postfix says this in the documentation:

With this setting, the Postfix SMTP server will not reject mail with “User unknown in local recipient table”. Don’t do this on systems that receive mail directly from the Internet. With today’s worms and viruses, Postfix will become a backscatter source: it accepts mail for non-existent recipients and then tries to return that mail as “undeliverable” to the often forged sender address.

To get around this you should really have an alias map file. For temporary testing though this setting will work wonders.

- The Python SMTP relay should not be exposed to the internet. It listens on localhost but ideally it would be nice to modify the script to accept authentication of some sort.

Locales in Ubuntu

Mon, 17 Oct 2011 00:09:00 -0400

The list of locales that you have installed on a system are in the directory:

/usr/lib/locale

To generate a locale you can run:

locale-gen en_US.UTF-8

This is very useful when you need to generate a UTF-8 locale.

Microsoft Office Communicator - Problem verifying certificate from the server.

Thu, 13 Oct 2011 17:17:00 -0400

When using Microsoft Office Communicator with a server that has TLS enabled you might get an error message “Problem verifying certificate from the server.”.

This message means that the computer you are on does not trust the certificate that is being presented to it.

The first way to troubleshoot this is to figure out what certificate it is receiving. The easiest way I’ve found to do this is to use openssl’s s_client:

openssl s_client -connect lcs.example.com:5061

By doing this you will see the entire certificate chain. You now need to go into the windows certificate management tools and make sure that chain is valid.

Generally this will involve running mmc.exe, then adding the snap in “Certificate Management” for the computer itself.

Another option is to cut and paste the BEGIN and END certificate lines into a text file. Name the text file with a .der extension and install the certificate. Then browse to the certificate in MMC and see if anything is wrong. Things that might go wrong include the validity date or being unable to trust the certificate chain (most likely from missing certificates).

If you are missing certificates you need to track them down and install them. After this is done you should be able to connect to communicator server.

Getting Around the YouTube Duplicate Content Filter

Wed, 12 Oct 2011 21:52:00 -0400

I was recently trying to upload a video to YouTube but kept getting the Rejected (duplicate upload) message.

I found the easiest way to get around this is to change the metadata on the video you are trying to upload. In my case it was an mp4 video.

To change the metadata I used a program called AtomicParley:

http://atomicparsley.sourceforge.net/

After downloading it simply run

AtomicParsley.exe “example.mp4” —artist “Me”

Or whatever artist you want. The new file will be written out with new metadata. This should then pass any YouTube duplicate content check.

Wordpress htaccess

Wed, 05 Oct 2011 12:40:00 -0400

One of my friends recently had a desire to rewrite some URLs with a Wordpress installation. The theme they were using was called Solid-WP and it supported a concept called “Projects”. When you create a new project it actually gives it the URL

http://www.example.com/project/

However there was a requirement that the URL should be renamed to http://www.example.com/apps/.

To do this I broke out mod_rewrite. Wordpress has some default rewrites stored inside .htaccess. Here is what it looks like:

# BEGIN WordPress

RewriteEngine On

RewriteBase /
RewriteRule ^index.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

# END WordPress

Rewrite can be hard to understand so I wanted to breakdown exactly what Wordpress was doing. First line turns on the Rewrite engine and 2nd line is used to define a base URL for rewrites.

RewriteRule ^index.php$ - [L]

If there is a request for /index.php then don’t do any rewriting and just end processing right here (the L flag).

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

These rewrite conditions are to test for real files and directories. Basically we don’t want to rewrite a URL if the URL points to a directory or file. We just want to serve the file up. !-f tests whether or not the file exists and !-d tests whether or not the directory exists.

RewriteRule . /index.php [L]

This last line as I understand it rewrites all requests to index.php.

What I ended up adding was this:

RewriteEngine On

RewriteRule project/(.+) /apps/$1 [L,R]
RewriteRule apps/(.+) /example-wp/index.php/project/$1/ [L]

RewriteBase /
RewriteRule ^index.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

The first time a user might hit /project/example. They would hit the first rule and get redirected. Processing would stop.

The second time a user hit the URL it would say /apps/example which would trigger the 2nd rule. It would get rewritten to the long form of WordPress’s controller action.

Designing Good Web APIs

Thu, 20 Jan 2011 20:42:00 -0500

I am not going to claim eons of experience in designing good APIs. I am going to approach it from the background of a developer who has to use and integrate with them.

What makes a good API? Here are some of my tenets:

1. Make your API calls as RESTful-like as possible

Because this term has evolved quite a bit I am going to stick with the primarily principles. Make API calls nouns describing “what” and not “do something”.

2. API calls should have defaults.

Don’t give me 100 required parameters just to see some action. Try as hard as you can to give me a basic call so I know I am doing something right. Along with this is to keep a minimum of API calls. The more calls you have documented the more I have to figure out which one it is I am suppose to be calling to get the right information. This is an area where I actually do want to be spoon-fed.

3. Use HTTP basic over SSL for authentication.

If you have the time and energy also support OAuth. The reason I hesitate with going straight to OAuth is because if you are just building an API your resources are probably limited. You are going to make mistakes. Getting it up and running with the lowest (and secure) common denominator is key.

Also use dedicated API keys that consist of an ID and a shared secret. Make sure these can be rolled.

4. Make your return format JSON.
For the love of god please don’t use XML.

5. Version your API call in the end point:

/api/1.0/user/3
/api/2.0/user/3

6. Make the calls easily testable.

To me this means I can plug it into a web browser, type in some credentials, and get a response back. As a developer this makes me feel good early on.

7. Document your API calls with examples.

Especially the ones where I can click on a link and it does an API call for me. This is also great for testing your call.

A number of these guidelines are geared heavily to web development. If you are designing an API for a message passing system with latency and size requirements a good deal of this would change.

How to get a 2yr single root SSL certificate for $10

Mon, 15 Nov 2010 00:22:00 -0500

I recently had to do some shopping for a SSL certificate for my startup Stratismo (I linked to it incase anyone wants to see the actual certificate). Obviously I wanted one that was widely supported and didn’t really want to pay a ridiculous amount for it.

So shopping around I landed on RapidSSL’s site. RapidSSL has a program where you can switch to them if you have a competitor’s certificate. On top of that they also give you a free year. All of this for free. The program details are here:

http://www.rapidssl.com/switch-ssl/index.html

I just happened to have a Comodo certificate so I gave this a test run. The Comodo certificate had some issues because of the number of intermediate certificates I had to include. While this shouldn’t cause problems in theory in reality it is just one more thing to deal with.

So what I did was apply for RapidSSL. They ended up giving me one of their SSL certificates for free after verifying I had a Comodo SSL certificate. They also extended the validation by a year so basically I got two years for $10. RapidSSL certificates are also signed by Equifax which is installed on almost every browser.

Overall a great deal.

PySAML - A SAML 2.0 Toolkit for Python

Mon, 13 Jul 2009 18:27:00 -0400

Security Assertion Markup Language (SAML) can be a bit confusing to understand. At its core SAML is just a protocol with defined messages written in XML. The main purpose of SAML is to enable you to log in at one place such as a website and then jump over to another website without having to log in again. This setup is commonly called “federation”.

It is similiar to OAuth. I see SAML continuing to make headroom in the enterprise space while OAuth stays strong in the consumer space. However in the future I hope these two technologies will end up playing well together.

In either case I am releasing a small Python library for generating SAML assertions. The main purpose of this is to learn SAML by doing (actually having to create an assertion gives me a good idea of the complexity of the protocol).

The library depends on M2Crypto so download and install it:
http://chandlerproject.org/bin/view/Projects/MeTooCrypto

M2Crypto depends on SWIG so you might need that as well.

Once that is installed you can download my distribution of PySAML here:

Python 2.6 - Win32:
http://darkeneddesire.com/PySAML/PySAML-1.0.win32.exe

Unix (source):
http://darkeneddesire.com/PySAML/PySAML.tar.gz

For windows you just run the executable. For unix you should run the following commands:

# cd PySAML
# python setup.py build
# python setup.by install

There are examples in the “examples” folder if you download the full source.

The whole project is available on github as well:
http://github.com/tachang/PySAML/tree/master

Trying to get Dual-SIM to work in T-Mobile G1

Thu, 09 Jul 2009 21:18:00 -0400

There is a product being sold called “Magic SIM”.

http://www.magicsim.com/en/dual_sim_show.asp?id=49&cp_id=&sort2name=14

Basically it is a SIM card that has two slots and a small memory chip. After physically cutting up your two SIM cards and putting them into the slots you insert the whole thing into your phone.

If your phone supports the GSM STK (SIM Tool Kit) standard then you can switch back and forth between the two SIM cards.

However for your phone to actually support the standard the manufacturer had to have built it into the operating system as well as provide a tool to “talk” using the STK.

I currently see that the Android development tree has some Stk development going on. I downloaded a precompiled Stk.apk and installed it on my phone with the card. However it does not seem to work.

While I do see the SIM Toolkit menu icon and it does seem to read from the custom card I am unable to obtain a carrier signal from either of my SIM cards. When I scroll to any of the menu options such as “SIM 1” or “SIM 2” and press the trackball the phone simply gives me a rotating circle on the upper right saying it is busy. And at that point it hangs.

Here is a quick and dirty image of my phone as well as what the SIM menu looks like:

Using Github Through Draconian Proxies (Windows And Unix)

Thu, 18 Jun 2009 20:38:00 -0400

Here is a pretty standard scenario at most corporations:

- All access to the internet is restricted to a proxy
- The proxy only allows connections out on port 80 and 443
- CONNECT method is only enabled for 443
- Proxy Authentication is required (NTLM or Basic)

I like to use both Windows and Unix environments. On Unix tunneling to Github is a bit easier because lots of tools are included.

Unix

1. Download Git. At the time I was writing this I was using Ubuntu so I simply did apt-get install git-core

2. Download and install corkscrew (http://www.agroman.net/corkscrew/). This is a tool for tunneling SSH through HTTP proxies.

3. Edit or create the file ~/.ssh/config and put the following:

ProxyCommand /usr/bin/corkscrew proxy.example.com 443 %h %p ~/.ssh/myauth

Host github.com
User git
Port 22
Hostname github.com
IdentityFile “/media/truecrypt1/Keys/GitHubKey.private”
TCPKeepAlive yes
IdentitiesOnly yes

Host ssh.github.com
User git
Port 443
Hostname ssh.github.com
IdentityFile “/media/truecrypt1/Keys/GitHubKey.private”
TCPKeepAlive yes
IdentitiesOnly yes

The ProxyCommand is invoked when ssh needs to make a connection. We are telling ssh to use /usr/bin/corkscrew. This is a 3rd party program that sets up a socket through the HTTP proxy.
The program /usr/bin/corkscrew takes as its 5th argument a file containing credentials for your HTTP proxy. Not all proxies need authentication but if you do just put in the file a single line formatted username:password.
The Host github.com indicates to ssh that if we are connecting to github.com to use these specific settings. There is nothing special here except we specify the location of the private key that corresponds to the public key we had over in http://www.github.com/
Notice we have another entry titled “Host ssh.github.com” . This is to get around proxies that only allow the CONNECT command over 443 (the truly locked down ones). To get around this github setup a whole separate host that listens on port 443. We add both entries here since they are both valid.

4. If everything is setup correctly you should be able to run:
# ssh github.com

Hi tachang! You’ve successfully authenticated, but GitHub does not provide shell access.
Connection to github.com closed.

If this doesn’t work you can run
# ssh ssh.github.com

And get the exact same thing. If the first command didn’t work it means you are using a proxy that blocks CONNECT on port 22. Almost no proxies block CONNECT on port 443 because you need that for SSL.

We get a no shell access message from github because we are trying to obtain a shell and github has it disabled. However this indicates everything is working. This concludes the setup for Unix.

————————————

Windows

1. Download msysgit http://code.google.com/p/msysgit/

Some settings:

- “Run Git from the Windows Command Prompt”
- “Use OpenSSH” (this one is very important)
- Pick your line endings

2. Download connect.c
http://bent.latency.net/bent/darcs/goto-san-connect-1.85/src/connect.html

This tool deserves its own post mostly because of its utter simplicity. It mirrors the open source tool corkscrew and is used for tunneling through proxies. Yes the tool’s name is really called “connect.c”.

For Window’s users, a pre-compiled binary is available:
connect.exe
I put my connect.exe in C:\Windows\connect.exe

3. Decide whether you like to use the Windows cmd.exe to do stuff or the Cygwin style shell. Or both.

Cygwin Git Bash Shell
For the Cygwin style shell start up the Git icon and edit the file ~/.ssh/config
*Make sure the file has no extension.

Put the following in that file:

ProxyCommand /c/windows/connect.exe -H username@proxy.example.com:443 %h %p

Host github.com
User git
Port 22
Hostname github.com
IdentityFile “/c/Keys/GitHubKey.private”
TCPKeepAlive yes
IdentitiesOnly yes

Host ssh.github.com
User git
Port 443
Hostname ssh.github.com
IdentityFile “/c/Keys/GitHubKey.private”
TCPKeepAlive yes
IdentitiesOnly yes

Notice the slash style in order to access the file system.
The proxy username is specified as part of the proxy setting. The password for the proxy is prompted for. Read more about connect.c to figure out how to get rid of this prompt.

At this point, using the Git Bash shell should yield:

$ ssh github.com

Hi tachang! You’ve successfully authenticated, but GitHub does not provide shell access.
Connection to github.com closed

Windows cmd.exe shell

Suppose you don’t like the Git Bash shell. You prefer the cmd.exe interpreter.

- Go to your config file at C:\Documents and Settings\.ssh\config
- Make a copy of it or make a new one. I called mine config-windows

Put the following in the file:

ProxyCommand C:/Windows/connect.exe -H username@proxy.example.com:443 %h %p

Host github.com
User git
Port 22
Hostname github.com
IdentityFile “C:\Keys\GitHubKey.private”
TCPKeepAlive yes
IdentitiesOnly yes

Host ssh.github.com
User git
Port 443
Hostname ssh.github.com
IdentityFile “C:\Keys\GitHubKey.private”
TCPKeepAlive yes
IdentitiesOnly yes

Notice the mixture of slash styles. I find this rather odd but it is what works. We have a forward slash style for the ProxyCommand but for the IdentityFile a forward slash or backward slash both work.

Running the command (making sure we run Git\bin’s ssh.exe and not some other one in the PATH):

C:\Program Files\Git\bin>ssh.exe -F “C:\Documents and Settings\.ssh\config-windows” github.com

Hi tachang! You’ve successfully authenticated, but GitHub does not provide shell access.
Connection to github.com closed

General Git Cloning

- Make sure you are using the right Git URL:

Suppose your Public Clone URL is: git://github.com/tachang /EyeFiServer.git

You should use the following URL that utilizes the SSH transport:

git clone ssh://git@github.com:22/tachang/EyeFiServer.git
git clone ssh://git@ssh.github.com:443/tachang/EyeFiServer.git

Django TypeError Exception

Mon, 15 Jun 2009 20:26:00 -0400

One of the central pieces of Django is urls.py:

http://docs.djangoproject.com/en/dev/topics/http/urls/

This file is well documented and serves as a central place to match what pages get served by what URLs. Seems simple enough since they accept regular expressions.

One thing I did run into though was getting a TypeError exception. It looked something like this:

TypeError at /applicationname/

index() takes exactly 1 argument (2 given)

To me I was puzzled why this was happening. The reason is that when Django parses the regular expression it not only does a match but saves each of the tuples in the regular expression. In the documentation it is this line:

“The view gets passed an HttpRequest as its first argument and any values captured in the regex as remaining arguments.”

So what happens is that when you start to get fancy with your regular expressions the function signature in your views.py needs to change as well to accommodate the extra parameters.

SiteMinder R12 Admin GUI

Thu, 11 Jun 2009 13:47:00 -0400

If you are upgrading or installing SiteMinder R12 you may have noticed that the traditional admin GUI has changed. The Java applet now asks you for 4 fields.

Username: This is the username found in the “Administrators” list on the old version 6 GUI. The applet is not case sensitive.
Password: The password in the “Administrators” list.
Host Name: A 4.x compatible Agent’s name
Passphrase: The Shared Secret of the 4.x compatible agent

The Host Name and Passphrase fields are where things get interesting. SiteMinder R12 is trying heavily to move the functions of this applet to a web based system. The web based system communicates with the SiteMinder policy server using the API. However, the API still uses 4.x agents as the method of authentication to the policy server. 4.x agents have an associated shared secret.

Thus to get into the traditional 6.0 GUI you need to have a 4.x compatible agent. But what happens when you upgrade to SiteMinder R12 and don’t have one? You have two options:

1. Install the SiteMinder Web Access Manager GUI - Unfortunately the installer for Windows is over 2GB
2. Manually create a 4.x Web Agent

Option 2 is easiest. We are going to create a file with the proper parameters and then use smobjimport to import it directly to the policy store. To do this create a file called “Generic4xAgent” and put the following in the file:

objectclass: Agent
Oid: 01-39c83ef9-5c51-4fb4-ba13-193543b8a9d4
Name: siteminder
Desc:
AgentType: 10-8d78bb96-ae15-11d1-9cdd-006008aac24b
RealmHintAttrId: 0

objectclass: TrustedHost
Oid: 24-5ea55269-d8a9-47a0-864c-c97026c00b99
Name: siteminder
Desc:
IpAddr: 127.0.0.1
Secret: password
Is4xHost: true
RolloverEnabled: false
SecretGenTime: 00000000-00000000-000000000000000000000000000000000000000000000000
SecretUsedTime: 00000000-00000000-000000000000000000000000000000000000000000000000
PrevSecret:

Save the file as “Generic4xAgent”. Now run the command:
smobjimport -iGeneric4xAgent -dsiteminder -wpassword -c

-i is the filename
-d is the SiteMinder admin username
-w is the SiteMinder admin password
-c indicates that the file has cleartext passwords

This creates a 4.x agent named “siteminder” with a shared secret of “password”.

At this point you should be able to login to the traditional admin UI.

Fixing pkg_add -r on FreeBSD (things no one tells you)

Fri, 29 May 2009 15:18:00 -0400

On FreeBSD when you want to install a program you have to make a choice. Your two choices are to install a package or compile from ports.

Installing a package means you download a binary package and copy all the files to the right places. Everything is already compiled. This is how Windows works. When you install a piece of software it has already been compiled.

Your second option is to compile from ports. What this means is that the source files are downloaded and then compiled on your system. You need to have all the prerequisites for the compile to be successful.

Why choose one over the other? Well with packages sometimes the binary doesn’t work. Either you happen to have a newer or older file than what the creator compiled it with or you changed something with your system and it doesn’t match anymore with the author’s compiled environment. However packages are fast to install. No need to break out gcc and compile source code.

Ports on the other hand will almost always work. They are targetted for your system because you are compiling them. It takes a bit longer but you also get to customize any of the build options.

To install a program from package you use this command:

pkg_add -r

Examples:

pkg_add -r xorg
pkg_add -r nano
pkg_add -r curl

I like pkg_add -r because it fetches the package for me and installs it. It does that by looking at your system’s uname (I think) and forms a URL to FreeBSD’s software repository. The problem is that after a while repositories are taken offline. So if you are using some older version of FreeBSD and trying to do a pkg_add -r it will return that it can’t find the package.

The thing is, the package is probably still around but in a newer form. The newer package is compatible with your version if you can find it. FreeBSD makes a statement that says packages in the same stable branch will always be compatible.

To force pkg_add -r to bend to your will you edit the PACKAGESITE environment variable. If you want to force pkg_add -to download FreeBSD 7-STABLE packages, set PACKAGESITE to ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7-stable/Latest/.

Be very careful though that you point to the right URL. I actually pointed to the wrong URL and downloaded 32-bit packages instead of the amd64 packages which I needed:

http://ftp2.at.freebsd.org/pub/FreeBSD/ports/amd64/packages-7-stable/Latest/

Ubuntu: Getting files off your phone using bluetooth

Tue, 12 May 2009 15:14:00 -0400

Sometimes you just want to get a file off your phone. One of the easiest ways to do this is bluetooth. The problem is that some carriers (such as Verizon) lock their phones down so you can’t transfer files from your phone to your PC over bluetooth.

The real technicality is that while the phone can’t initiate the sending of the file, you can still browse the phone for files and pull them.

So far the most useful tool I found to get files off my phone (which is an LG Chocolate) is obexftp.

To get started install the package using Synaptic then open up a console and type:

$ obexftp -b

When you find a device you want you connect to it like so:

$ obexftp -b 00:1E:75:ED:62:2C -l
Browsing 00:1E:75:ED:62:2C …
Channel: 7
Connecting…done
Receiving “(null)”…
done
Disconnecting…done

The way obexftp works is that you basically have to run the command over and over with different options to get what you want. So after running it with this “list” option I say I want to browse the folder “MyPictures”.

$ obexftp -b 00:1E:75:ED:62:2C -c MyPictures -l
Browsing 00:1E:75:ED:62:2C …
Channel: 7
Connecting…done
Sending “MyPictures”… done
Receiving “(null)”…
done
Disconnecting…done

Now that I know the location of the file I can procede to actually get the file.

$ obexftp -b 00:1E:75:ED:62:2C -c MyPictures -g 0512091207.jpg
Browsing 00:1E:75:ED:62:2C …
Channel: 7
Connecting…done
Sending “MyPictures”… done
Receiving “0512091207.jpg”…|done
Disconnecting…done

Easy enough. Probably not the most intuitive way to do things but it gets the job done.

Obtaining a Free Fully Trusted SSL Certificate

Mon, 27 Apr 2009 18:57:00 -0400

Ever get that popup when you visited a supposedly “secure” website that says the certificate is not trusted? Reason you get that popup is because the person who signed/issued the certificate is not trusted by default on your computer.

The business of selling certificates is a lucrative one. Large certificate authorities such as Thawte and VeriSign make a killing issuing digital certifcates. They can arbitrarily adjust the parameters on the certificate and charge for it on a case by case basis. The reason they are able to do so is because almost every computer by default trusts these companies. You can check this in Windows by going to start, run, mmc.exe and adding the Certificate snap in. Look under Trusted Root Certification Authorities.

In general you have to pay quite a bit of money to get a certificate signed by one of these authorities. However there is one way to get a virtually “free” one.

Thawte offers a program called their Web of Trust:

https://www.thawte.com/cgi/personal/wot/contents.exe

Basically after going through the steps Thawte will give you a free certificate signed by an authority called “Thawte Personal Freemail CA”. Because this CA is installed by default on all Windows installations it is trusted everywhere.

Getting this certificate involves a combination of sending in documents to Thawte or finding people who have already gone through the process to vouch for you. Once you have it though you keep it forever (as far as I can see) and can generate certificates with your e-mail address in it.

PyXMLSec Windows Binary

Wed, 15 Apr 2009 17:35:00 -0400

Xmlsec Python Bindings for Windows

Getting the python xmlsec library to work on windows is a bit of work though it really shouldn’t be. Provided here are windows installers for xmlsec. The installer itself is produced by python distutils.

In order to get it to work you will also need libxml2. I am providing a copy of it here as well.

pyxmlsec-0.3.0.win32-py2.6.exe
libxml2-python-2.7.3.win32-py2.6.exe

I used MinGW for compilation. The entire environment I used to compile pyxmlsec is available here. Look inside the pyxmlsec directory for build.bat.
PythonWindowsCompileEnvironment.zip

Eye Fi Standalone Server Version 2.0

Thu, 02 Apr 2009 22:56:00 -0400

I’d like to release an updated version of the Eye Fi standalone server in python that I have been working on. This version should work on Linux, Mac OS X, Windows, Solaris, or wherever else you can load a Python interpreter. As always I love comments so if you are using this feel free to e-mail me or drop me a note!

Source on GitHub: http://github.com/tachang/EyeFiServer

Download (zip): http://www.darkeneddesire.com/EyeFiServer/2.0/EyeFiServer-v2.0.zip

I know some people just like to browse around the source without having to download stuff (I’m one of those people):

http://www.darkeneddesire.com/EyeFiServer/2.0/Release/

This new version has the following features:

The server can now execute an arbitrary command on each uploaded photo. This is a very dangerous feature and should be used with caution. On the other hand it is also very cool. You can have the server FTP files, display them using an image viewer, or even run sorting programs on the images.
Improved security: the server now generates its own nonces instead of using one that was hard coded. The nonce is based on the random library provided by python. The INTEGRITYDIGEST field is also checked.
Ability to read settings from a configuration file (there is a included DefaultSettings.ini for reference). The file allows you to configure the listen port, console output, logging, download location, and execute on upload, and upload key.

Some other notable improvements but not really features are the addition of regression tests and support for Python 2.5. The regression tests are interesting since I run them against the official Eye-Fi Manager to make sure my behavior is a close match.

Getting usage information as to how to specify a configuration file:

C:\EyeFiServer\Release>EyeFiServer.py -h
Usage: EyeFiServer.py [options]

Options:
-h, —help show this help message and exit
-c CONFIGFILE, —config=CONFIGFILE
Path to configuration file (example in
DefaultSettings.ini)

Actually specifying a configuration file:
C:\EyeFiServer\Release>EyeFiServer.py -c DebugSettings.ini

Converting LDIF/LDAP data into a CSV file

Tue, 27 Jan 2009 18:33:00 -0500

LDIF to CSV

I work with LDAP on a regularly basis. I frequently have to pull data using ldapsearch. While the data that ldapsearch spits out is a decent representation sometimes I want something a bit easier to work with.

The format I use most tends to be CSV. While the acronym stands for comma separated value the format may be used to describe data that is printed out line by line and separated by anything you can imagine.

The problem for me is finding a decent LDIF to CSV converter. The ones that I have tried tend to choke on any number of issues. Some of these include binary data or failing to normalize the multivalued attributes.

I finally got sick enough of these issues that I decided to write my own. You can download it here:

Python/Linux/Source

Windows

Hopefully the Windows binary works for you. If it doesn’t then just download Python 2.6 and run the source manually. I am providing the binary just for convenience.

Using LDIFtoCSV

Running “python LDIFtoCSV.py” should give you the usage text.

usage: LDIFtoCSV.py [options]

-o <filename> : File to write output. By default this is set to sys.stdout
-l <filename> : File to write logging output. By default there is no logging.
-F <char> : Character to separate the fields by. By default this is a comma. i.e. -F”,”
-D <char> : Character to delimit the text value by. By default this is a double quote. i.e. -D”“”
-M <num> : The maximum number of columns a multivalued attribute should take up (default: 5). This is common with the objectClass attribute where it can have over 20 values. Do you want to have 20 columns each with the same heading objectClass or do you want to limit it.

Here are some common command lines that I use (assuming you have a test.ldif):

Outputs the CSV straight to standard output:
python LDIFtoCSV.py test.ldif

Outputs CSV to standard output with semicolons as the delimiter:
python LDIFtoCSV.py -F”;” test.ldif

Outputs CSV to standard output with pipes as the delimiter and text surrounded by carrots:
python LDIFtoCSV.py -F”|” -D”^” test.ldif

Downloading the Global Address List from Outlook/Exchange

Wed, 21 Jan 2009 16:24:00 -0500

Getting the Data

Recently I had a need to dump the entire Global Address List from within Outlook. Usually the address list will contain the contact information for your entire organization.

The most effective way I have found to do this is a straight LDAP search on the Active Directory and pull everything into an LDIF file. Conceptually it is straightforward but I have found the most difficult part is usually trying to get all the pieces together. So subscribing to the French idea of “mise en place” here is what you’ll need:

Username
Your username is your fully qualified distinguished name. It is not your Windows login name. It commonly looks something like “CN=Tchang, Jeff,OU=Users,OU=USA,DC=example,DC=com”.

The easiest way I have found to obtain your username is to login to a Windows machine and then put the following two lines in a text file with the extension .vbs:

Set objADSysInfo = WScript.CreateObject(“ADSystemInfo”)
result = InputBox(“Active Directory Username (copy and paste as necessary)”, “Active Directory Username”, objADSysInfo.UserName, 100, 100)

Password
The password will be your domain password.

Domain controller hostname
Couple of ways to get this. This will be the Active Directory/LDAP server that you will extract the entries. An echo %logonserver% at the command prompt usually works.

Ldapsearch tool
Easiest way to get the tools is from Sun’s iPlanet Directory SDK. Go to http://www.sun.com/download/products.xml?id=3ec28dbd and select the iPlanet Directory SDK downloads. After download the zipfile find the executable named ldapsearch.exe.

Performing the search
Here is an example syntax to extract out all the users:

Windows:
ldapsearch.exe -b OU=Users,OU=USA,DC=example,DC=com -h host.example.com -p 389 -D “CN=Tchang, Jeff,OU=Users,OU=USA,DC=example,DC=com” -w - (objectClass=*)

Unix:
ldapsearch -b OU=Users,OU=USA,DC=example,DC=com -h host.example.com -p 389 -D “CN=Tchang, Jeff,OU=Users,OU=USA,DC=example,DC=com” -W -x “(objectClass=*)”

The -b is for the base. In this example I knew the users were all stored in that branch. -D is for username. -w - (that is a dash w followed by a dash) means to read the password from the console. On Unix the big -W is to prompt and the -x indicates you want simple authentication (cleartext username/password).

You might want to add a ” > output.txt” to send the output to a file. If you do that you will have to supply the password on the command line.