Joe’s Amazing Technicolor Weblog

or, Why To Use Dedicated Layouts When Connecting To FileMaker Via PHP

I’d read that it’s a good practice to always use a dedicated layout for any PHP scripts you have that are talking to a FileMaker database. While I’d seen reasons of efficiency and reliability, today I learned another reason that’s true: it can eliminate otherwise hard-to-debug problems.

At first when working on my current FileMaker <-> PHP project, I was attempting to reuse an existing layout that had all the info I needed. While my permissions seemed to be fine for the data file and layout I was attempting to access, actually running the script kept resulting in “Error 100: File is missing” coming back at me as soon as I added any criteria to my search. FileMaker doesn’t bother putting anything useful in its server logs, either, so it wouldn’t have been much fun picking through the layout & figuring what linkage(s) were to blame.

However, by simply creating a dedicated layout, everything started working as planned. A practice I’ll be following in the future.

Because rsync 3 is one of if not the only OS X backup solutions that actually gets all the possible forms of metadata, I’m in the market for a backup solution that uses it. The current candidate is rsnapshot, a 6000 line perl program (!), conveniently located in macports, that wraps rsync to do smart backup things like keep snapshots via hard links. Thanks to the O’Reilly Backup book for pointing me to it.

• It’s pretty easy to set up to run locally. The main trick is that the configuration file requires tabs. First time in ~10 years I’ve had to turn tabs back on in vim (:set noexpandtab).
• For OS X, we want the magic -aNHAXx --fileflags --force-change args to make rsync behave properly with all the metadata.
• OS X has a weird directory structure, so if you try to backup /etc you just get the symlink that is to /private/etc, or if you try to exclude something under /var, you miss it because it’s really /private/var/bigdirectory. Buzzkill.
• Lchown.pm is necessary for symlinks to have the right ownership in snapshots. CPAN’d.
• Running things automatically on OS X as the privileged user is a bit odd, lacking as it does a traditional root account. Sudo does nicely, with NOPASSWD: /opt/local/bin/rsnapshot
on the backup machine. For the clients we need appropriate ssh settings, with some tricks to run sudo on the remote machine. Getting this running took a while, since I missed the fact that running rsnapshot via sudo on the backup machine meant that rsync would try to use root’s ssh key, not the backup user’s — fixed this with the -i arg to ssh.
• Restoring backups is just a matter of copying them from the appropriate snapshot dir, probably using rsync & the same arguments rsnapshot uses (easily extracted from the rsnapshot log).
• Multiple servers are done serially. If you wanted to run backups in parallel, you’d need one configuration file for every server you’re backing up, and they each need their own snapshot_root, logfile, and lockfile. For my installation, this’d be more trouble than it’s worth.
  

Another issue that temporarily gave me pause, now that we’ve got everything backed up with presumably correct permissions, is that our off-site backup procedures involve creating tar files & encrypting them. Fortunately, backup bouncer shows that OS X tar gets all the important stuff right, though it’s no rsync v3.

Thanks to MSCPA, I finally tracked down the Governor’s press release which, at first glance, has a reasonably clear description of the regulation’s intent. Also came across an analysis by Beth Israel’s CIO, a positive blurb from a Maine consultancy, and a brief mention by a MA payroll company.

As for me, I still need to do my official audit of our procedures vs. those specified by the regulation.

We last looked at this a few months ago, but have been revisiting it to come up with something a little more robust. Notes:

• there’s a good overview of using PHP with FM at the sixfriedrice blog.
• the API for FileMaker’s PHP interface is available at http://YOURSERVER.URL:16000/docs/PHP%20API%20Documentation/index.html
• I’d missed this last time around, but accounts & permissions are a little funky. fmphp needs to be added to the Extended Privileges of the database you’re trying to get to, and must have the same privilege set as the account you’re connecting as.
• The solution we settled on is a CLI PHP script running hourly, checking for mail to send. Launchd would be the logical way to do the scheduling, but always drives me nuts. Fortunately the server in question has cron set up (so much simpler!)

The code we’re more or less using:

#!/usr/bin/php
<?php
set_include_path(get_include_path() . PATH_SEPARATOR .
  '/Library/FileMaker Server/Web Publishing/publishing-engine/php/lib/php/');
require_once('FileMaker.php');

echo "PHP email-sending-script, running at " .
  date('m/d/Y H:i') . "\n";
$layout = 'Outgoing_Email';
$fm = new FileMaker('Layout Name');
$fm->setProperty('username', 'your filemaker username');
$fm->setProperty('password', 'your filemaker password');

$findCmd =& $fm->newFindCommand($layout);
$findCmd->addFindCriterion('Sent_Flag', '< 1');
$result = $findCmd->execute();
if (FileMaker::isError($result)) {
  if ($result->code == 401) {
    exit("No emails to send.\n");
  } else {
    exit("trouble: " . $result->message . "(" . $result->code . ")");
  }
}

$records = $result->getRecords();
foreach($records as $record) {
  echo "To: " . $record->getField('Recipient') . "\n";
  echo "Subject: " . $record->getField('Subject') . "\n";
  $headers = array(
    "From: filemaker@example.com",
    "MIME-Version: 1.0",
    "Content-type: text/html"
    );

  /*
   FM helpfully encodes < and >...
   */
  $body = preg_replace('/&lt;/', '< ', $record->getField('Body'));
  $body = preg_replace('/&gt;/', '>', $body);
  $rc = mail($record->getField('Recipient'),
       $record->getField('Subject'),
       $body,
       implode("\r\n", $headers)
      );
  if ($rc) {
    $update = $fm->newEditCommand($layout, $record->getRecordId());
    $update->setField('Sent_Date', date('m/d/Y'));
    $update->setField('Sent_Time', date('H:i'));
    $update->setField('Sent_Flag', '1');
    $result = $update->execute();
    if (FileMaker::isError($result)) {
      exit("trouble updating the database after sending email: " .
        $result->message . "(" . $result->code . ")");
    }
    echo "Mailed!\n";
  } else {
    exit("Mail didn't work.\n");
  }
}

Update: I’d forgotten to check the flags that had been missing from the macport version last time around, but they’re all there. Now, running rsync -aNHAXx --fileflags --force-change /Volumes/Src/ /Volumes/rsync3test passes every bbouncer test! Cool.

Original post: On Leopard (10.5.5) using rsync 3.0.4 from macports, a few more tests are passing backup bouncer 0.1.3 (compare to my June post):

$ sudo ./bbouncer verify -d /Volumes/Src/ /Volumes/rsync3test/
Verifying:    basic-permissions ... ok (Critical)
Verifying:           timestamps ... ok (Critical)
Verifying:             symlinks ... ok (Critical)
Verifying:    symlink-ownership ... ok
Verifying:            hardlinks ... ok (Important)
Verifying:       resource-forks ...
   Sub-test:             on files ... ok (Critical)
   Sub-test:  on hardlinked files ... ok (Important)
Verifying:         finder-flags ... ok (Critical)
Verifying:         finder-locks ... FAIL
Verifying:        creation-date ... FAIL
Verifying:            bsd-flags ... ok
Verifying:       extended-attrs ...
   Sub-test:             on files ... ok (Important)
   Sub-test:       on directories ... ok (Important)
   Sub-test:          on symlinks ... ok
Verifying: access-control-lists ...
   Sub-test:             on files ... ok (Important)
   Sub-test:              on dirs ... ok (Important)
Verifying:                 fifo ... ok
Verifying:              devices ... ok
Verifying:          combo-tests ...
   Sub-test:  xattrs + rsrc forks ... ok
   Sub-test:     lots of metadata ... ok

Would you believe that a branch office of a certain unnamed organization hasn’t had a firewall (or even a router doing NAT) for close to a year? You could configure the printer from across the world and everything. That is, until the other day, when I got this little guy installed. Questions I had to answer in this project:

1. Which soekris? Current needs are reasonably simple firewalling for a smaller office, so the 4501 answers nicely. Would consider 5501s for an office with more traffic / servers.
2. What media? 2 GB CF cards are cheap as dirt now, I remember running OpenBSD on systems with smaller hard drives… the lower heat / power / footprint and higher reliability of CF is very attractive for this application, compared to a hard drive.
3. What OS / distro? Initially, I’m going with what’s basically a default i386 OpenBSD installation. Commonly used for firewalls, and given the cheapness and reliability of CF these days, a generic setup should work for now. Once I’m comfortable with how things are running I’ll look into more options for minimizing writes, and possibly mounting some or all of the FSes ro. The most promising reference I’ve found so far is Michiel van Baak’s guide.
4. What style of installation? Having serial access seems like a much better idea than just blindly booting CF cards, for troubleshooting purposes if nothing else. This also flows from the previous question, if I’d settled on a flashdist type approach, I’d be writing via a CF card. As it was, I got a keyspan USB-to-serial connector (drivers for OS X and linux!) & a null modem, and talked to the 4501 via screen /dev/cu.KeySerial1 19200 from my laptop. I then booted the Soekris from a convenient Debian server, with some helpful tips from here.

Stumbling blocks I ran into:

• The whole TFTP booting situation was tricky due to a missing ‘next-server’ directive in my dhcpd conf file, which I fixed thanks to this.
• After that, I got hung at

  
  boot> boot bsd.rd
  booting tftp:bsd.rd: 4780308+874136 [52+178240+163973]=0x5b821c
  entry point at 0x200120

  which was fixed by doing

  
  boot> set tty com0
  boot> stty com0 19200

  before booting bsd.rd (thanks to google’s cache of an mailing list discussion for that).

As any of you who’ve tried it know, it’s not that hard, though it did take a little trial and error to teach myself based on web howto’s. I ended up referring to this, this and this.

Why bother? It really is much nicer when your cables are the right length, so you don’t have unruly spools of unneeded cable everywhere. After a recent rewire, my server closet was a real cabling disaster. Also, it’s way more cost effective if you’re talking lots of cables.

Lessons:

• you don’t really need a cable tester. At least the simple kind I got wasn’t even that helpful — it gave a passing score to a cable that wouldn’t reliably work. Just plug it into your network & see if it transfers a big file properly. (If I’m missing something here, speak up!)
• a good wire cutter is absolutely essential. My crimper has a decent little cutter built in that’s just the right size, which is handy.
• T568B is what you want unless you have some weird situation
• Never realized how I’ve taken patch cables for granted, even in my days of installing servers. I guess Willy must have made sure we had a common supply, but I can’t really recall.

Networks Unlimited just sent out a note (thanks!) about the Mass Office of Consumer Affairs’ new
Standards for The Protection of Personal Information of Residents of the Commonwealth, aka 201 CMR 17.00: M.G.L. c. 93H. It outlines the responsibilities of anyone who gathers personal information on Mass residents. At a glance, they look pretty reasonable. From the intro:

Every person that owns, licenses, stores or maintains personal information about a resident of the Commonwealth shall develop, implement, maintain and monitor a comprehensive, written information security program applicable to any records containing such personal information.

It’ll be interesting to sit down with this & see how our policies & procedures match up.

We already had a serviceable in-house email server, so why change? In no particular order,

• Cost savings. We’d been spending $1k+ annually on postini’s spam protection service alone, which Google Apps conveniently bundles in for free. Also a maintenance contract on the email software, depreciation on the server, electricity
• One less server to manage
• Far superior webmail and group calendaring to what we’ve been using

When first thinking about doing this conversion I talked to a bunch of people whose organizations were already using google apps. Nobody really had any complaints. Given the upside, I certainly don’t feel like I’m in a position to complain, either, but I thought it’d be worth mentioning a few issues I encountered:

• I applied July 22nd for nonprofit status, and heard back (in the affirmative) on August 5th. Fortunately, it’s entirely possible to switch over with just a standard account, but I wish I’d applied earlier and not had to worry about whether or not that was going to go through.
• While it’s nice that there’s an open source LDAP solution for syncing user accounts, it has had some issues and even when working it only handles people’s names, email addresses, and passwords. At the time I was working with it, the best bet was to use the latest tarball, into whose directory I had to copy the above-mentioned file. Even after that, I continued to have enough trouble to abandon ship — was quicker to do the work by hand than to fix the bugs.
  In case anyone’s also trying to hook this tool up with Apple’s Open Directory, I had some initial luck with the following:
  Command: set ldap_url LDAP://servername.example.org
Command: set ldap_base_dn dc=servername,dc=example,dc=org
Command: set ldap_user_filter (objectclass=apple-group)
  though it doesn’t look like anyone’s worked through the mapping from Apple’s schema.
  Beyond just working stably, it would be great if group membership could be used to configure mail lists — I see that’s listed under possible improvements (bottom of page). I’ll try to take a look at that some day & see how feasible it would be to work it in.
• Transferring old emails is a pain if you’re not on a mainstream email server (raise your hand if you’ve heard of Stalker Software’s Communigate Pro). We’re doing it by setting both the old and new servers up in the same email client & copying messages over through it, but then we lose the date emails were sent / received.
• I couldn’t find any way to turn on IMAP access across the board, so I did a lot of logging in & setting people’s account preferences. This wouldn’t be a big deal if it weren’t for the old email transferring issue mentioned above.
• While it’s possible to create an email list containing all addresses in the domain, only admins can send mail to it; to get an all-staff list, I had to actually type everybody’s address in. More motivation to work on the LDAP group-to-list mapping mentioned above.

As promised, I spent yesterday at Organizer’s Collaborative’s Grassroots Use of Technology conference, up in Lowell. I went hoping particularly to pick up tips on donor management and fundraisining tools, and came away with some good leads. It was also fun to reconnect with folks.

Keynote speakers

Nick Jehlen of Action Mill shared his approach to social change projects, and how that approach played out for Turn Your Back on Bush, Winter Soldier, and Enough Fear. His basic premise is to take Ghandi’s idea of being the change that you want to see in the world, and bring it to the commons, so that principled actions have a chance to influence others. In addition to having interesting stuff to say, Nick really put together a handsome presentation, so if you get a chance to catch him speak sometime, go for it.

After lunch, Paul Niwa talked about his Boston Chinatown site, which provides a visualization of the community members’ connectedness. He’s a professor of journalism at Emerson, so his initial goals were mostly based in journalistic concerns, but one of the interesting results of the project is that it may have provided incentive for some people to become more involved in their community, to boost their importance on the visualiztion! It was also interesting how what Paul called his “journalistic arrogance” led him to publish people’s information on the web much more freely than many of us in the nonprofit / activist space would be likely to do.

Sesssions

The first breakout session I went to was horrible. No names, to protect the guilty.

Sura Hart and Katie Winterbottom of Grassroots.org ran a helpful session on SEO. Props to them on running the presentation from Google Apps, on a KDE laptop. As for the content,

• it was helpful to see specifics about keyword research, and the tradeoffs between keyword popularity in searches and the existing presence for that term on the web
• will have to think harder about the working of intrasite hrefs
• hadn’t really thought about using the title attribute on tables, forms, etc.
• Google Grants sounds like an amazing opportunity.

Nate Aune of jazkarta had a ton of useful tool suggestions. He started with the constituent database, as that’s at the core of almost any successful organization. His recommendation is salesforce.com now that they’re giving their service away free to nonprofits. This is significant because it’s a best-of-breed solution, with a thriving ecosystem of parters enabled by its comprehensive API.

From there, Nate went on a whirwind tour of helpful tools. I’ll only note here the ones that I can see looking into in the future:

Online donations

• we already use paypal, but I’m not sure if we’re taking advantage of the fact that they give nonprofits a lower fee than for-profits. Also, all processing can be done via their API - no need to send folks to paypal.com’s ugly pages (which we do now…)
• fundable
• chipin
• for nonprofits, google checkout is totally free through next year. Interesting, I wonder what happens then?

Mass email

• campaign monitor - we’ve just been giving them a try, so it was nice to hear that jazkarta has good luck with them. However:
• VerticalResponse is also supposed to be excellent, and is integrated to salesforce.com

Misc.

• eventbrite sounds very handy for online ticketing
• phone.com’s integration of voice mail and email could be very handy