Archive for the 'Code' Category

HTTrack: new go-to program for web mirroring / archiving

Thursday, April 2nd, 2009

Faced with a big site full of URLs like http://mysite.com/Internal1.asp?id=357 to mirror & archive, I recently tried out a new (to me) tool, HTTrack. I’ve fiddled with wget for this sort of job in the past, but it always takes me ages of man-page reading to get my options right, and even then not everything seems to work out.

This time around, for example, I’d convinced myself that wget -r -N -l inf --no-remove-listing -E -k -p http://mysite.com would do the trick. It mostly did, except for seemingly random pages that didn’t get all of their links converted.

HTTrack, on the other hand, did The Right Thing without any switches or arguments whatsoever. It was a bit more of a pain to get running; even though it’s in macports, right now the port is lagging behind the available versions, so I had to actually type ./configure and ./make myself. Well worth it for a usable mirror.

Migrating from mongrel to passenger

Wednesday, December 24th, 2008

In short, I’m happy to report that passenger a.k.a. mod_rails is awesome. I see no need to fiddle with mongrel, mongrel_cluster & mod_proxy for future rails apps.

Miscellaneous notes from the project:

  • My environment is now ubuntu server 8.04, on slicehost. This project finally prompted me to upgrade from 7.10, which went very smootly — the long-term support releases of Ubuntu seem like the way to go for most server situations.
  • How to handle the apache config? We have some php stuff running on this server, too, so the 3rd party ubuntu passenger package, which wants to use the worker mpm, isn’t the hot ticket. No problem, passenger’s default gem-based install is smooth as butter.
    This configuration also seems to require RailsAutoDetect off, which had the side benefit of letting me leave an existing mongrel-using configuration alone until I had a few minutes hours to upgrade it to rails 2.2.2, then switch it over to git & passenger.
  • Slicehost has decent docs for this, in particular ubuntu-hardy-mod_rails-installation and ubuntu-hardy-using-mod_rails-to-serve-your-application & its comments.
  • Capistrano mods for passenger
  • This was also my first git-powered rails app, which proved a little challenging to get going with cap, but has turned out really nicely. Very fast deployments.
    Along the way, having a plugin as a git submodule proved to be more trouble than it was with, though I’m not sure if that’d apply to my final configuration, which involves checking the project out from the same server as it’s deployed on,  using the :local_repository argument to capistrano.
  • This project helped me realize that having a deployed app as a public github project is more trouble than it’s worth, what with the various configuration informations that I’d rather have under SCM but don’t want to share.

slightly updated us_state_select plugin

Thursday, December 4th, 2008

I’ve used technoweenie’s handy us_state_select rails plugin in past projects. Just tried it out on a new rails 2.2 project & found out that the call to InstanceTag needed to be updated, and since technoweenie’s SVN version of the plugin seems moribund, I thought I’d take 30 extra seconds to put it up on github. Fork away!

P.S. github is way cool.

FileMaker error 100

Friday, November 21st, 2008

or, Why To Use Dedicated Layouts When Connecting To FileMaker Via PHP

I’d read that it’s a good practice to always use a dedicated layout for any PHP scripts you have that are talking to a FileMaker database. While I’d seen reasons of efficiency and reliability, today I learned another reason that’s true: it can eliminate otherwise hard-to-debug problems.

At first when working on my current FileMaker <-> PHP project, I was attempting to reuse an existing layout that had all the info I needed. While my permissions seemed to be fine for the data file and layout I was attempting to access, actually running the script kept resulting in “Error 100: File is missing” coming back at me as soon as I added any criteria to my search. FileMaker doesn’t bother putting anything useful in its server logs, either, so it wouldn’t have been much fun picking through the layout & figuring what linkage(s) were to blame.

However, by simply creating a dedicated layout, everything started working as planned. A practice I’ll be following in the future.

FCKeditor + Drupal 6 FTW

Wednesday, November 5th, 2008

I’ve had a number of experiences setting up rich text editors online over the last few years, usually in the context of a Drupal site, and it’s always been a pain.

My most recent experience has mostly been the same, except it ends a little better than usual:

  1. Start with TinyMCE, which has more-or-less worked in the past. This time around with Drupal 6, unfortunately, my experience was mostly less: it just wouldn’t respond to the customization settings I was setting in the Drupal backend.
  2. Do some more research, find excitement about the YUI editor. It seemed promising, but ultimately failed to allow image uploads despite hours of fiddling.
  3. Turn to the other oft-mentioned option, FCKeditor. After a few minutes of fiddling supported by the included readme.txt, it’s actually working and uploading images easily. Amazing!

Not everything in FCK is customizable through the web, but that’s fine. It’s probably easier to comment out a line in fckeditor.config.js, anyhow. I started with the DrupalFull toolbar & took off a few things we don’t want.

Since versions are so often key with these things, I’m using Drupal 6.6, fckeditor-6.x-1.3-rc3 (the drupal module), FCKeditor_2.6.3 (the javascript bit).

Including a reasonably debugged WYSIWYG in the Acquia distribution would get me to take it for a test drive next time around, because this cycle is such an amazing waste of time whenever I go through it.

Sending email from FileMaker via PHP, revisited

Wednesday, October 22nd, 2008

We last looked at this a few months ago, but have been revisiting it to come up with something a little more robust. Notes:

  • there’s a good overview of using PHP with FM at the sixfriedrice blog.
  • the API for FileMaker’s PHP interface is available at http://YOURSERVER.URL:16000/docs/PHP%20API%20Documentation/index.html
  • I’d missed this last time around, but accounts & permissions are a little funky. fmphp needs to be added to the Extended Privileges of the database you’re trying to get to, and must have the same privilege set as the account you’re connecting as.
  • The solution we settled on is a CLI PHP script running hourly, checking for mail to send. Launchd would be the logical way to do the scheduling, but always drives me nuts. Fortunately the server in question has cron set up (so much simpler!)

The code we’re more or less using:


#!/usr/bin/php
<?php
set_include_path(get_include_path() . PATH_SEPARATOR .
  '/Library/FileMaker Server/Web Publishing/publishing-engine/php/lib/php/');
require_once('FileMaker.php');

echo "PHP email-sending-script, running at " .
  date('m/d/Y H:i') . "\n";
$layout = 'Outgoing_Email';
$fm = new FileMaker('Layout Name');
$fm->setProperty('username', 'your filemaker username');
$fm->setProperty('password', 'your filemaker password');

$findCmd =& $fm->newFindCommand($layout);
$findCmd->addFindCriterion('Sent_Flag', '< 1');
$result = $findCmd->execute();
if (FileMaker::isError($result)) {
  if ($result->code == 401) {
    exit("No emails to send.\n");
  } else {
    exit("trouble: " . $result->message . "(" . $result->code . ")");
  }
}

$records = $result->getRecords();
foreach($records as $record) {
  echo "To: " . $record->getField('Recipient') . "\n";
  echo "Subject: " . $record->getField('Subject') . "\n";
  $headers = array(
    "From: filemaker@example.com",
    "MIME-Version: 1.0",
    "Content-type: text/html"
    );

  /*
   FM helpfully encodes < and >...
   */
  $body = preg_replace('/&lt;/', '< ', $record->getField('Body'));
  $body = preg_replace('/&gt;/', '>', $body);
  $rc = mail($record->getField('Recipient'),
       $record->getField('Subject'),
       $body,
       implode("\r\n", $headers)
      );
  if ($rc) {
    $update = $fm->newEditCommand($layout, $record->getRecordId());
    $update->setField('Sent_Date', date('m/d/Y'));
    $update->setField('Sent_Time', date('H:i'));
    $update->setField('Sent_Flag', '1');
    $result = $update->execute();
    if (FileMaker::isError($result)) {
      exit("trouble updating the database after sending email: " .
        $result->message . "(" . $result->code . ")");
    }
    echo "Mailed!\n";
  } else {
    exit("Mail didn't work.\n");
  }
}

Whoops — my rails app was open to a common vulnerability

Tuesday, September 23rd, 2008

Thanks to Hacker News for bringing this common problem with Rails apps to my attention. Nobody seems to have taken advantage of it on my app, but still, it’s a drag having insecure applications, and a little disappointing that there aren’t more heads-up about this, or a more secure default as Merb apparently has.

Sending server-side emails from FileMaker via PHP

Thursday, June 12th, 2008

Some context — my new gig features a big ‘ol FileMaker installation, which has a number of automated maintenance routines. Some of those routines send emails, through a convoluted process involving FileMaker calling a GUI MUA (e.g. Mail.app). There are a number of practical problems with this, such as the requirement of another computer running, always logged into an account capable of sending the emails.

We’d done a bit of research about strictly server-side alternatives, and found surprisingly little. The best resource was Graham Sprague’s page about sending emails via FileMaker’s XSLT Web Publishing tool. We gave that approach a try, but didn’t get any results, or anything useful from FileMaker’s logs to explain why things weren’t working. I’m not sure what FileMaker version Graham’s example was written for, perhaps something’s changed with version 9?

Rather than dive into FileMaker’s proprietary XSLT system to debug things, it occurred to me that this might be a job for FileMaker’s PHP API. Sure enough, after about 15 minutes of consulting the API Doc, we were sending emails based on the contents of a FileMaker record.

We’re still working on ironing out the details, but here’s the rough proof of concept PHP file. It works with the example email database from Graham’s XSLT sample, with the php permission added to the database. Plenty of missing features such as cc & bcc fields, actually checking for the ‘send’ flag, checking for errors, any kind of authorization or authentication, etc. In other words, you probably don’t want to be running this on a publicly accessible webserver, but at least it presents the basic idea in a simple form.


<?php
require_once('FileMaker.php');
$fm = new FileMaker();
$fm->setProperty('username', 'send_email');
$fm->setProperty('password', 'whatever_the_password_is');
$fm->setProperty('database', 'Email');

$findCmd =& $fm->newFindAllCommand('Utility_Email');
$result = $findCmd->execute();
$records = $result->getRecords();
foreach($records as $record) {
  $headers = array("From: " . $record->getField('from'));

  mail($record->getField('to'),
       $record->getField('subject'),
       $record->getField('message'),
       implode("\r\n", $headers)
      );
}
echo "Emails sent";

latest rsync looking good for OS X metadata

Tuesday, June 10th, 2008

As regular readers saw earlier, I’ve been casting about trying to find an open source backup solution that handles OS X metadata reliably. Having been disappointed by rdiff-backup, I’ve turned by eyes to the similar rsnapshot project, which uses the venerable rsync.

Since macports includes the latest version of rsync, 3.0.2, I gave it a try with the familiar rsync -avz /from /to syntax, but it performed disappointingly on n8’s handy Backup Bouncer test suite. Thanks to Mike Bombich, I learned about some extra flags to add (though my copy of rsync doesn’t seem to know about the -N or — fileflags he has):


$ sudo rsync -aHAXx  /Volumes/Src/ /Volumes/rsynctest/
$ ./bbouncer verify -d /Volumes/Src/ /Volumes/rsynctest/
Verifying:    basic-permissions ... ok
Verifying:           timestamps ...
   Sub-test:    modification time ... ok
ok
Verifying:             symlinks ... ok
Verifying:    symlink-ownership ... ok
Verifying:            hardlinks ... ok
Verifying:       resource-forks ... ok
Verifying:         finder-flags ... ok
Verifying:         finder-locks ... FAIL
Verifying:        creation-date ... FAIL
Verifying:            bsd-flags ... FAIL
Verifying:       extended-attrs ...
   Sub-test:             on files ... ok
   Sub-test:       on directories ... ok
   Sub-test:          on symlinks ... ok
ok
Verifying: access-control-lists ...
   Sub-test:             on files ... ok
   Sub-test:              on dirs ... ok
ok
Verifying:                 fifo ... FAIL
Verifying:              devices ... FAIL
Verifying:          combo-tests ...
   Sub-test:  xattrs + rsrc forks ... ok
   Sub-test:     lots of metadata ... ok
ok

Sure, there are a few FAILS in there, but they’re not important:


$ ./bbouncer verify -T important -d /Volumes/Src/ /Volumes/rsynctest/
Verifying:    basic-permissions ... ok
Verifying:           timestamps ...
   Sub-test:    modification time ... ok
ok
Verifying:             symlinks ... ok
Verifying:            hardlinks ... ok
Verifying:       resource-forks ... ok
Verifying:         finder-flags ... ok
Verifying:       extended-attrs ...
   Sub-test:             on files ... ok
   Sub-test:       on directories ... ok
   Sub-test:          on symlinks ... ok
ok
Verifying: access-control-lists ...
   Sub-test:             on files ... ok
   Sub-test:              on dirs ... ok
ok

(note the -T important flag telling Backup Bouncer to remove the extra-finicky tests). Good enough! On to get familiar with rsnapshot.

rdiff-backup 1.1.15 better with OS X metadata, but still room for improvement

Monday, June 9th, 2008

I’ve finally had the chance to repeat my test of the rdiff-backup-devel package from MacPorts, using the same steps as I used for the stable package. The results are better but not fantastic:

$ ./bbouncer  verify -d /Volumes/Src/ /Volumes/rdifftest
Verifying:    basic-permissions ... ok
Verifying:           timestamps ...
   Sub-test:    modification time ... ok
ok
Verifying:             symlinks ... ok
Verifying:    symlink-ownership ... ok
Verifying:            hardlinks ... ok
Verifying:       resource-forks ... ok
Verifying:         finder-flags ... FAIL
Verifying:         finder-locks ... FAIL
Verifying:        creation-date ... ok
Verifying:            bsd-flags ... FAIL
Verifying:       extended-attrs ...
   Sub-test:             on files ... ok
   Sub-test:       on directories ... ok
   Sub-test:          on symlinks ... FAIL
FAIL
Verifying: access-control-lists ...
   Sub-test:             on files ... FAIL
   Sub-test:              on dirs ... FAIL
FAIL
Verifying:                 fifo ... ok
Verifying:              devices ... ok
Verifying:          combo-tests ...
   Sub-test:  xattrs + rsrc forks ... ok
   Sub-test:     lots of metadata ... FAIL
FAIL

With promising reports out on rsync 3, looks like it’s time to take another look at rsnapshot