Archive for the 'Code' Category

python script for importing maildirs to gmail

Thursday, November 5th, 2009

In fact the script in question should work also for mboxes and for other SMTP servers, but maildir-to-gmail was the problem I was trying to solve.

The most promising starting point was an old script by Mark Lyon. After a little rejiggering so I could see what error was coming back from Google, I made a couple of more tweaks to use TLS & to take the user’s password.

If anyone’s interested in what seemed to me a strange hoop to hop through before connecting, check the src.

Considering how to reliably jam stuff into FileMaker from the web

Friday, October 23rd, 2009

I’m sure I’m not the only person with this situation:

  1. FileMaker database sitting behind a firewall (though similar issues would pertain for other internal databases / services)
  2. Website hosted elsewhere (i.e. other side of firewall)
  3. Need to get data from #2 to #1 reliably and securely

Up until today, I’ve only had one instance of #2 in this situation. I dealt with it by storing data collected on the website (which happened to be written in Rails) in a database on the web server, and then running a periodic PHP script on the FileMaker server that connects to the Rails app via phpactiveresource, pulls in pending data, and inserts it into FileMaker via its PHP api.

That instance was such a roaring success that the requests have been pouring in for more of the same. Some of the new requests will be handled by a site running PHP, so I’ve got a bit of rewiring to do — I can’t see any sense in the getting the data from the PHP app into something the Active Resource client can talk to.

Stepping back and looking at the bigger picture, issues here include:

  • the connection from the website to the FileMaker server could be down, so data collected by the website needs to be stored until it can be confirmed to have made it to FileMaker.
  • it would be nice for this to happen in a timely fashion
  • multiple technologies on the web side (PHP & ruby) are going to be collecting data to be submitted to FileMaker, so it’d be nice if the transfer machinery can be agnostic and just accept JSON or XML or something.

Sounds like a problem for a queue system, huh? So my current plan is to run a beanstalkd instance on the webserver, deposit JSON-endocded data into it from the web sites, and run workers that write to FileMaker using the Ruby FM API. I have no experience with beanstalkd, but a bit of googling suggests that it’s at a nice point in simplicity to configure & run, maturity, light weight, and easy access from PHP & Ruby.

A further benefit of working in beanstalkd is that, based on a quick perusal of the recommended Rails integration, it should be really easy to break Observers out to async code, thus making my rails apps snappier.

Any advice to the contrary is of course welcome. I’ll try to remember to update y’all on how this turns out.

Custom flickr sidebar via wget, cron & PHP

Wednesday, July 29th, 2009

The new site launched with a sidebar that shows two random photos from our flickr account, using their javascript widget. This was a great way to get things going, but now we’ve developed slightly more involved needs and I’ve had to come up with a custom solution.

Getting the list of photos

You need a flickr API key, which is quick & easy to get. Then wget & cron to get ‘em: wget --quiet 'http://api.flickr.com/services/rest/?method=flickr.photos.search&api_key=YOUR_API_KEY&user_id=YOUR_USER_ID&tags=website&per_page=500' -O photos.xml

Note that this includes a tags argument. The thing that pushed me to switch the workflow was the desire to be able to upload photos to our flickr account that don’t necessarily fit into the sidebar format, such as panoramics. To handle this, everything that belongs on the website gets the tag website, and we only fetch those ones. We’ve also talked about just getting landscape oriented photos, but haven’t implemented that.

I’m running this daily, which is plenty often to update the available photo list. I believe this gets the newest 500, which seems more than adequate, particularly since we don’t have close to 500 photos yet.

Parsing the list & generating the HTML

PHP5’s SimpleXML is pretty nice — here’s what we’re doing:

try {
  $xml = new SimpleXMLElement(file_get_contents('photos.xml'));
    $number_of_photos = count($xml->photos->photo);
    $displayed_photos = array();
    array_push(
      $displayed_photos,
      $xml->photos->photo[rand(0, $number_of_photos - 1)]);
    array_push(
      $displayed_photos,
      $xml->photos->photo[rand(0, $number_of_photos - 1)]);
    foreach ($displayed_photos as $photo) { ?>
  <div>
<?php
      print "<a href=\"http://www.flickr.com/photos/8562013@N07/" .
        $photo['id'] . "\"><img src=\"http://farm" . $photo['farm'] .
        ".static.flickr.com/" . $photo['server'] . "/" . $photo['id'] . "_" .
        $photo['secret'] .  "_m.jpg\" alt=\"" . $photo['title'] . "\" /></a>";
?>
  </div>
<?php
 }
} catch (Exception $e) {
  error_log("flickr badge had some troubles: " .
    $e->getMessage());
}

This snippet takes my laptop less than 1/20th of a second to run from the command line, which suits me fine. The actual code sits in page.tpl.php.

Flickr’s API docs, in particular the API Explorer, were awful handy in figuring this all out.

Joe’s hacky approach to getting arbitrary behavior out of Drupal

Tuesday, July 28th, 2009

Wherein I disqualify myself for work as a “drupal developer”

Self indulgent history part

Way back in 1999 when I started out writing web apps, Embperl seemed like the logical tool to use — so I’m used to having direct access to every step in the request-to-response process. In years since, I’ve progressed through a bunch of different tools and approaches, which have included writing a few rudimentary content management systems and using a bunch of prexisting ones.

Over the last few years I’ve become a big fan of ruby on rails, so if a site does much of anything (simple, recent example: The Food Project’s online summer program application), that’d be my starting point.

so, why use Drupal?

Even so, when a site is basically about some content that’s written and edited on an ongoing basis by other folks, Drupal starts looking like a reasonable tool. It’s pretty easy to get it to generate semantic output with reasonable URLs, and the editor-facing UI does a fair job of offerring a large degree of control and customizability without being overwhelmingly complex.

Despite having done some custom work on a few live Drupal sites, I still get confused about the best way to do some of the arbitrary web interaction tasks that were so straightforward back in the Embperl days. There are a billion modules in the Drupal ecosystem, but I often find that it takes more time to figure out if there is an appropriate one for my task & if it works with the current version of Drupal than it would to code up a solution on my own. Also, while I’ve made several attempts to solve programming tasks in what I understand to be the Drupal way, wrapping my head around all the relevant APIs (which can totally change every 6-12 months, with each Drupal release) similarly tends to take more time than bypassing Drupal’s functionality & handling things in Plain Old PHP.

Here’s a common approach I’ll take, then. I have no interest in extensive coding through a web browser (i.e. putting a bunch of PHP in a node or block), so I’ll set up a Drupal module to hold my application’s arbitrary php functions. Then I’ll create a page at the desired URL and enter a line of PHP to route the GET and/or POST to my code, as appropriate.

Example

I’m working on an email list signup that needs to communicate with a FileMaker database. An HTML form POSTs to http://thefoodproject.org/mailing_list_signup, which is just a Drupal page using the PHP input filter (which you now need to turn on via admin settings) containing <?php tfp_process_list_signup($_POST); The code for that function lives in sites/all/modules/tfp/tfp.module, and does the normal PHP things to pull information out of the POST & hand that info off to the FileMaker database. Here’s a bit of it:

function tfp_process_list_signup($_POST) {
  $reply = '';
  $sent_plausible_address = FALSE;
  if ($_POST['email']) {
    $supplied_email = $_POST['email'];
    if (preg_match(PLAUSIBLE_EMAIL, $supplied_email)) {
      $sent_plausible_address = TRUE;
      $reply = "Thanks! We'll add your to our mailing list.";
    } else {
      $reply = "'$supplied_email' doesn't look like an email address to me. ";
    }
    //. . .
  }
  return $reply;
}

On the upside, this is quick & easy. The downside with this approach vs. writing a module the Drupal way is that I’m losing out on Drupal’s form API and its associated validation, antispoofing, etc. services. On the upside, last time I checked the linked documentation for the form API, it said

Warning - this page has only been partially updated for the Drupal 6.x API
Until it has been fully updated, reference this page as well: Drupal 5.x to 6.x FormAPI changes

so I’m quite happy to not have to wade through all of that! (more…)

HTTrack: new go-to program for web mirroring / archiving

Thursday, April 2nd, 2009

Faced with a big site full of URLs like http://mysite.com/Internal1.asp?id=357 to mirror & archive, I recently tried out a new (to me) tool, HTTrack. I’ve fiddled with wget for this sort of job in the past, but it always takes me ages of man-page reading to get my options right, and even then not everything seems to work out.

This time around, for example, I’d convinced myself that wget -r -N -l inf --no-remove-listing -E -k -p http://mysite.com would do the trick. It mostly did, except for seemingly random pages that didn’t get all of their links converted.

HTTrack, on the other hand, did The Right Thing without any switches or arguments whatsoever. It was a bit more of a pain to get running; even though it’s in macports, right now the port is lagging behind the available versions, so I had to actually type ./configure and ./make myself. Well worth it for a usable mirror.

Migrating from mongrel to passenger

Wednesday, December 24th, 2008

In short, I’m happy to report that passenger a.k.a. mod_rails is awesome. I see no need to fiddle with mongrel, mongrel_cluster & mod_proxy for future rails apps.

Miscellaneous notes from the project:

  • My environment is now ubuntu server 8.04, on slicehost. This project finally prompted me to upgrade from 7.10, which went very smootly — the long-term support releases of Ubuntu seem like the way to go for most server situations.
  • How to handle the apache config? We have some php stuff running on this server, too, so the 3rd party ubuntu passenger package, which wants to use the worker mpm, isn’t the hot ticket. No problem, passenger’s default gem-based install is smooth as butter.
    This configuration also seems to require RailsAutoDetect off, which had the side benefit of letting me leave an existing mongrel-using configuration alone until I had a few minutes hours to upgrade it to rails 2.2.2, then switch it over to git & passenger.
  • Slicehost has decent docs for this, in particular ubuntu-hardy-mod_rails-installation and ubuntu-hardy-using-mod_rails-to-serve-your-application & its comments.
  • Capistrano mods for passenger
  • This was also my first git-powered rails app, which proved a little challenging to get going with cap, but has turned out really nicely. Very fast deployments.
    Along the way, having a plugin as a git submodule proved to be more trouble than it was with, though I’m not sure if that’d apply to my final configuration, which involves checking the project out from the same server as it’s deployed on,  using the :local_repository argument to capistrano.
  • This project helped me realize that having a deployed app as a public github project is more trouble than it’s worth, what with the various configuration informations that I’d rather have under SCM but don’t want to share.

slightly updated us_state_select plugin

Thursday, December 4th, 2008

I’ve used technoweenie’s handy us_state_select rails plugin in past projects. Just tried it out on a new rails 2.2 project & found out that the call to InstanceTag needed to be updated, and since technoweenie’s SVN version of the plugin seems moribund, I thought I’d take 30 extra seconds to put it up on github. Fork away!

P.S. github is way cool.

FileMaker error 100

Friday, November 21st, 2008

or, Why To Use Dedicated Layouts When Connecting To FileMaker Via PHP

I’d read that it’s a good practice to always use a dedicated layout for any PHP scripts you have that are talking to a FileMaker database. While I’d seen reasons of efficiency and reliability, today I learned another reason that’s true: it can eliminate otherwise hard-to-debug problems.

At first when working on my current FileMaker <-> PHP project, I was attempting to reuse an existing layout that had all the info I needed. While my permissions seemed to be fine for the data file and layout I was attempting to access, actually running the script kept resulting in “Error 100: File is missing” coming back at me as soon as I added any criteria to my search. FileMaker doesn’t bother putting anything useful in its server logs, either, so it wouldn’t have been much fun picking through the layout & figuring what linkage(s) were to blame.

However, by simply creating a dedicated layout, everything started working as planned. A practice I’ll be following in the future.

FCKeditor + Drupal 6 FTW

Wednesday, November 5th, 2008

I’ve had a number of experiences setting up rich text editors online over the last few years, usually in the context of a Drupal site, and it’s always been a pain.

My most recent experience has mostly been the same, except it ends a little better than usual:

  1. Start with TinyMCE, which has more-or-less worked in the past. This time around with Drupal 6, unfortunately, my experience was mostly less: it just wouldn’t respond to the customization settings I was setting in the Drupal backend.
  2. Do some more research, find excitement about the YUI editor. It seemed promising, but ultimately failed to allow image uploads despite hours of fiddling.
  3. Turn to the other oft-mentioned option, FCKeditor. After a few minutes of fiddling supported by the included readme.txt, it’s actually working and uploading images easily. Amazing!

Not everything in FCK is customizable through the web, but that’s fine. It’s probably easier to comment out a line in fckeditor.config.js, anyhow. I started with the DrupalFull toolbar & took off a few things we don’t want.

Since versions are so often key with these things, I’m using Drupal 6.6, fckeditor-6.x-1.3-rc3 (the drupal module), FCKeditor_2.6.3 (the javascript bit).

Including a reasonably debugged WYSIWYG in the Acquia distribution would get me to take it for a test drive next time around, because this cycle is such an amazing waste of time whenever I go through it.

Sending email from FileMaker via PHP, revisited

Wednesday, October 22nd, 2008

We last looked at this a few months ago, but have been revisiting it to come up with something a little more robust. Notes:

  • there’s a good overview of using PHP with FM at the sixfriedrice blog.
  • the API for FileMaker’s PHP interface is available at http://YOURSERVER.URL:16000/docs/PHP%20API%20Documentation/index.html
  • I’d missed this last time around, but accounts & permissions are a little funky. fmphp needs to be added to the Extended Privileges of the database you’re trying to get to, and must have the same privilege set as the account you’re connecting as.
  • The solution we settled on is a CLI PHP script running hourly, checking for mail to send. Launchd would be the logical way to do the scheduling, but always drives me nuts. Fortunately the server in question has cron set up (so much simpler!)

The code we’re more or less using:


#!/usr/bin/php
<?php
set_include_path(get_include_path() . PATH_SEPARATOR .
  '/Library/FileMaker Server/Web Publishing/publishing-engine/php/lib/php/');
require_once('FileMaker.php');

echo "PHP email-sending-script, running at " .
  date('m/d/Y H:i') . "\n";
$layout = 'Outgoing_Email';
$fm = new FileMaker('Layout Name');
$fm->setProperty('username', 'your filemaker username');
$fm->setProperty('password', 'your filemaker password');

$findCmd =& $fm->newFindCommand($layout);
$findCmd->addFindCriterion('Sent_Flag', '< 1');
$result = $findCmd->execute();
if (FileMaker::isError($result)) {
  if ($result->code == 401) {
    exit("No emails to send.\n");
  } else {
    exit("trouble: " . $result->message . "(" . $result->code . ")");
  }
}

$records = $result->getRecords();
foreach($records as $record) {
  echo "To: " . $record->getField('Recipient') . "\n";
  echo "Subject: " . $record->getField('Subject') . "\n";
  $headers = array(
    "From: filemaker@example.com",
    "MIME-Version: 1.0",
    "Content-type: text/html"
    );

  /*
   FM helpfully encodes < and >...
   */
  $body = preg_replace('/&lt;/', '< ', $record->getField('Body'));
  $body = preg_replace('/&gt;/', '>', $body);
  $rc = mail($record->getField('Recipient'),
       $record->getField('Subject'),
       $body,
       implode("\r\n", $headers)
      );
  if ($rc) {
    $update = $fm->newEditCommand($layout, $record->getRecordId());
    $update->setField('Sent_Date', date('m/d/Y'));
    $update->setField('Sent_Time', date('H:i'));
    $update->setField('Sent_Flag', '1');
    $result = $update->execute();
    if (FileMaker::isError($result)) {
      exit("trouble updating the database after sending email: " .
        $result->message . "(" . $result->code . ")");
    }
    echo "Mailed!\n";
  } else {
    exit("Mail didn't work.\n");
  }
}