Rails3 Custom Password Validators

As I was writing validators for the User class of a Rails 3 app, I wanted to make sure that people wouldn't use their names, usernames, or email addresses as passwords.
Unfortunately I couldn't find a way to accomplish this with the built-in validators. Fortunately Rails 3 makes it easy to write your own custom validators.

Here's an extract of my User class

The "password => true" tells Rails to call my custom validator which, in this case, has to be called password_format.rb.

I keep my custom validators in /lib/validators, so I need to add the following to my config/application.rb file:

And finally the validator itself:

(Don't forget to write the specs to test this! :-)

Mining the OSX Console for Fun and Profit

Well... Fun? Yes (if you're a geek). Profit? Not very likely but if you search long enough who knows? :-)

I rarely open the OSX Console app. This morning, while waiting for a call, I did and found this cry for help from Firefox:

11.1.31 9:25:12 [0x0-0xbb0bb].org.mozilla.firefox[2734] SHOULD NEVER HAPPEN

In fact I found dozens of them. Intrigued I started looking for other messages. Chrome, it seemed, was having problems of a more existential nature.

11.1.31 9:34:37 [0x0-0x2a42a4].com.google.Chrome[12790] objc[12794]: Class CrApplication is implemented in both [...] and [...]. One of the two will be used. Which one is undefined.

As long as one of them's used that's OK right?

Pages should really know better.

11.1.31 8:58:43 Pages[12487] *** WARNING: Method setDrawsGrid: in class NSTableView is deprecated. It will be removed in a future release and should no longer be used.

Uh-oh.

11.1.30 17:56:04 AppleMobileBackup[8848] WARNING: Backing up bf6f8237f787cbf4206d1e107b24aacd55c44b5b

Hmmm... Does MDRP phone home when you rip a DVD? Kind of: by default it will "Anonymously report rip statistics" but you can turn this behavior off in its preferences. I don't think this should preference should be checked by default but I like the fact that MDRP is at least logging this activity.

The most common message I saw? Variations of this one:

11.1.29 1:12:15 <Many Different Programs Here>[229] Can't open input server /Users/[...]/Library/InputManagers/Edit in TextMate

It seemed as if everyone was complaining about TextMate (I know, I should move to Vim, it's on the list...). Fortunately, there's a solution.

After scanning a day's worth of logs I decided I had better things to do!

Tech Trends I'm Watching in 2011

The good news for us techies is that there are so many things to choose from... So, in no particular order, here are some that I'm excited about.

Continuous Deployment / Delivery

Different people give it different names but the concept is the same: Since you're already running continuous integration servers against your codebase to make sure they build correctly and pass automated tests, why not go one step further and deploy to production instead of stopping at the integration environment?

Companies that follow this approach will often release to production dozens of times a day. A key goal is reducing the "wait time" for a bug fix or a new feature to be available to users from days, weeks, or even months to a couple of hours at most.

On the technical side, continuous deployment calls for extensive automation, comprehensive unit and integration test coverage, clear procedures for making (reversible!) database updates, improved production server monitoring to detect any flaws ASAP, and the ability to efficiently stage a deployment through multiple "waves" of servers (e.g. start with 1, then 10, then 100 servers) and of course roll it back in case of issues. 

On the business side, continuous deployment means more flexibility in how features are rolled out. While bug fixes may hit all users very quickly, Product Managers can now batch features together as needed and release them to a few clients for beta testing, before "flicking the switch" of a full deployment. Of course, all this entails tight cooperation between Development, Operations, and Product Management.

While some will understandably be gun shy of pushing out code so quickly, I believe that companies that achieve continuous deployment will reap significant competitive advantage.

Kanban

Another example of how our world is moving from the discrete to the continuous is in development "methodologies". Long ago, waterfall gave us release cycles measured in months and years. Today Agile approaches like Scrum typically aim for a "production-ready" codebase at the end of each sprint (typically two weeks). But two weeks is too slow if you're embracing continuous deployment.

Consequently I'm seeing more and more activity around Kanban or Kanban-like models for software development. Based on Toyota's Lean Manufacturing principle Kanban focuses on eliminating waste and improving efficiency based on (among other things) maintaining a smooth flow of features and fixes through the "development pipeline". This "pull" system is a lot more suited to rapidly responding to external changes than waterfall or even Scrum (where early sprint termination is usually fairly costly). 

In reality, it can often make sense to combine elements of both Scrum and Kanban. In my own experience we've found Kanban well suited for operations teams, esp. when they're dealing with many emergent tasks.

Kanban's challenge when used with continuous deployment is being able to decompose stories into small enough items. A four week work item defeats the purpose of continuous flow! :-)

NOSQL

Alternatives to SQL really exploded in 2010 when NOSQL hit mainstream. Products like CouchDBMongoDBRedis, and Cassandra are powering very large sites. These databases are in some ways the descendants of the OODBs that seemed poised to break SQL's hold on our data 10-15 years ago and, more directly, related to the BigTable technology Google developed. 

While the tech community has a well-known affinity for bright, new, shiny things and we're still learning how best to use NoSQL, there's no doubt we'll be seeing a lot of action in this space in 2011. We'll also see a backlash fueled in part by highly publicized failures like this one (though there will always be counter-examples!).

Smart Phones

Despite Microsoft's re-entry in the mobile space with Windows 7 Phone and Palm's impending release of webOS based tablets (oh, and RIM too), I doubt Apple's iOS and Google's Android will lose much if any market share.

Last year Adobe implemented Flash on iOS but was blocked by Apple's strict licensing agreement, which Apple then quietly relaxed a few months later. Given that you need your apps on both Android & iOS to reach the majority of consumers I expect cross platform development environments to get more airtime in 2011. Companies include RhomobileAirplay, and appMobi (which just announced a $6M round of funding yesterday).

In the meantime, since Android and iOS have sophisticated (and mostly compliant) HTML5 browsers,  some developers are building their mobile apps as web apps and releasing a "shell app" to both Apple and Google app stores. Netflix is a prime example of this approach.

Security

It's not clear to me whether we'll see new classes of security products in 2011 but what is clear is that with attack surfaces multiplying, demand will be strong. Companies are expanding their apps into the mobile space, rolling out APIs, dealing with ever increasing browser complexity, maybe even experimenting with NFC, and they often do so at the expense of security. Just as importantly, many web apps now outsource key functions to third parties (e.g. customer supportpaymentsemails...). Criminals targeting these firms could potentially affect hundreds or thousands more.

Financial services may be where most of the money is but cyber criminals are nothing if not opportunistic. They'll keep looking for other industries to target, esp. ones that might not be so security minded. Speaking of branching out, I'm not sure if it will happen this year but with Apple's growing PC penetration it's only a matter of time before virus and trojan writers focus on OSX.

Concerns of "mini wikileaks" within large organizations will likely increase demand for employee monitoring tools, egress filtering, comprehensive audit trails. None of this will prevent a determined data thief, but it will satisfy auditors and shareholders.

There's more to talk about: Rails becoming the defacto startup dev platform, the problems of complex systems, and the progress towards AI, but these will have to wait for another time.

Fusing RSS feeds together from Posterous and other blogs

Posterous is a great web app: they've managed to the keep it simple and elegant to use while making it ever more sophisticated (thanks in large part to Garry Tan's leadership).

They're also very quick to respond to questions from users:

Why did I need such a script? My oldest domain, cyberclip.com, was registered in 1995 (back when "cyber" was cool! ;-) and went through early years of excitement and later years of neglect. A few months ago I streamlined it to a very simple design:

Recently I thought it would be nice to add just a little more content and decided to include a list of the newest posts from all my blogs. Since Posterous doesn't supply a consolidated feed, here's the ruby script I wrote to generate one. I've only tested it with FeedBurner and Posterous but it should work with all properly formatted RSS feeds.

I've purposely kept error handling out of this script for simplicity's sake because that's handled by the program that invokes it. That prog runs as a cron job and regens the homepage on a regular basis. It won't replace the homepage if RSSFusion errors out for whatever reason. So worst case is a stale homepage, not a broken one.

Combining blog posts for inclusion in another page is one use, but this script could also be used to generate a fused RSS feed of multiple blogs, which is something I should probably add to my homepage as well...

Software Architecture should be forged in Fire, not Carved in Ice

(Picture by Jason Bolonski)

I've seen a number of corporate environments that carve software architectures out of ice.Why ice? Because an ice architecture sure looks great: sparkling, pristine, perfect even. This approach often feels right, esp. in cost conscious, slower moving organizations. You know how it goes: spend the bulk of your time in design, figure things out properly, measure twice (or thrice), cut once, and then you're set. Sadly once carved, often the only thing left to do with such an architecture is freeze it, lest it melt. And a frozen architecture is rarely useful.

The problem is that you're never set. Needs keep changing and if your architecture can't evolve with them, you've (best case) got a working but unmaintainable and unevolvable app, or (worst case) something that becomes unusable, even by the people who need the application the most and who are willing to put up with its flaws.

The architecture you want is not one carved in ice. Rather, you want something you've not only heated and beaten into shape to serve your current purpose, but also a design that you can reforge in to something new as needs dictate. So how do you achieve this?

Understand your problem space & key challenges
I'm not advocating no design, I'm advocating just enough design. Knowing how far to go is both art and science. Two things that will help is a good understanding of the problem space and the main obstacles you'll face. If you're building a social app you need to have at least broad designs for your sharing / trust model for users, how you will distribute data and scale, what security choices to make, etc. If you don't have strong expertise in house, hire someone who does, even as a part time consultant. It will be money well spent.

Rapid Iteration
Focus on speed. Not at the expense of quality but at the expense of features. Build your Minimum Viable Product, get it live, get it used, and iterate. The only way to really learn what works and what doesn't is to let your users at your app. The faster your iteration the more you can adapt (reforge) to changing needs. This is one of the reasons that agile development has become the de facto development approach in the past decade. Continuous deployment and delivery are other, welcome, instances of this trend.

Less is more
Build what you need and improve when needed. This goes hand in hand with rapid iteration and goes against the "what if" architecture. "What if I want to go live in other countries? Oh better internationalize", "What if we need to support suppliers as well as customers? Good point, better code them in". "What if I need to scale to hundreds of millions of users?" The list goes on and the longer you let it go the slower you'll be, not just in development but in maintenance and new feature additions.

This doesn't mean making uneducated decisions. If you think there's a good chance you'll need to go international in future, leverage a framework that supports it. But don't build it internationalization (i18n) until you're ready to use it. I'd argue that the cost to support i18n - testing multiple languages across the app, different formats & interfaces, new & altered business logic, etc. - is not worth saddling your dev team with against the day when you finally need the feature.

Ultimately, the fewer lines of code the better. To paraphrase Dijkstra: it's not lines of code produced, it's lines of code spent!. So spend wisely.

Test Driven / Refactoring
If there was one software engineering practice I'd enforce, this is it. A comprehensive and robust set of tests gives you the confidence to make radical structural changes to your application and still have a working product at the end of process. I've seen business leaders question the value of putting in effort here. Understandably they're concerned that all the time spent coding tests could have been better spent coding new features.

A counter-argument to this is to remind the business folks that, once tests are written, all the downstream QA is not only free but extremely rapid. That's when you can reassign developers to new areas confident in the knowledge that you'll detect breaks long before they ever reach end users. As your codebase grows these tests will be a godsend to help avoid spending all your time on maintenance.

"Less is more" is your friend here too: focus on DRY (Don't Repeat Yourself). The DRYer your code, the less of it you have, so the fewer tests you'll need and the easier it will be to reforge.

Culture
The most fundamental and so the most important. The culture of an organization is represented by people's shared values, goals, and behavior. Whether implicit or explicit, culture is the bedrock on which all else rests. The more individuals align with the culture, the more effective the team. This buy-in means that a successful culture cannot be dictated (typically by management), it must be nurtured.

Culture will obviously vary by company but these common values will support all the principles listed above:
  • Continuous improvement: Continually striving to make things better, to achieve ever higher quality, and redefine goals as necessary 
  • Trust: Despite the best laid plans, failures happen. When they do, an organization needs to display enough trust in individuals and team to allow them to fix the problem, learn from the experience, and come out stronger
  • Collaboration: In my experience, tech and business are often in push-pull. Tech rarely gets the necessary time or resources, and business rarely gets all its desired features. The principles above drive long term value. If the culture prizes collaboration, openess, and sustained value, then neither group will want to sacrifice the short term for the long term

Update (2011.2.3): Great article on the importance of culture from Wealthfront.

 

    What about...?
    What about separation of concerns, SOA, AOP, and more? All of these have their place and can certainly improve your application and architecture. Design patterns, if properly used, can make your software design more flexible and reduce the amount of refactoring needed to add new features. Still, these practices are should haves, not must haves. They can make good architecture great, but on their own won't keep it great in the long run.

    Ultimately the key to building great software is great people. Finding them, building the right culture together, and continuously evolving the organization and its processes. Over the years, new software engineering practices will come to light but these principles will evolve much more slowly.

    Optimizing Facebook's "Hoppity" Puzzle

    I found Facebook's puzzle page the other day. While it has some very meaty challenges, it also has a couple of trivial ones. These easy puzzles are there to allow you to make sure you can submit solutions (I'm not consistently getting mine to run but I'm hopeful that turning off "Always send Windows-friendly attachments" in OSX Mail will do the trick).

    One easy puzzle is called Hoppity. Essential you count up from 1 to a specified number and follow these rules
    • For integers that are evenly divisible by three, output the exact string Hoppity, followed by a newline.
    • For integers that are evenly divisible by five, output the exact string Hophop, followed by a newline.
    • For integers that are evenly divisble by both three and five, do not do any of the above, but instead output the exact string Hop, followed by a newline.
    This very simple program is typically written with a few if then else's, though you could also simulate a bitmask and use a case statement:

    As soon as I'd written it I started wondering: can I optimize this? I mean, this is Facebook we're talking about. Endless scale. So if they offered a Hoppity function on the main page, you bet they'd have to make it run fast! :-)

    Looking at this program it's clear that the output should repeat every 15 counts. Here's a Mathematica plot to illustrate this where I've replaced Hoppity, Hophop, and Hop with 1, 2, and 3 respectively.

    So if you're ever interviewing at Facebook and you're given this problem (which would be surprising, I agree), you can offer this optimization to make sure the Hoppity feature doesn't take down their servers when they release it to all their members :-)

    Pre-computing is always useful!

    External network card: Essential travel equipment

    We've traveled extensively in Europe over the past few months, and throughout these trips one of the few constants is the search for Internet connectivity. As soon as we arrive somewhere, the quest for wifi starts. Sometimes we're lucky enough to be staying with friends or at a "wired" hotel. Often though we're renting an apartment, or just in a place that hasn't seen the light yet :-)
    One solution is to visit the local Internet cafe, which is seldom practical. For one thing they're rarely close by. For another, spending time in one isn't practical: evenings are when we like to catch up with mail, tweets, blogs, and plan our upcoming activities. Carting the family off to the cafe after a long day of sightseeing is no fun.
    Our solution is to look for a generous neighbor with an open wireless access point. But to stand any chance of finding one, your laptop needs help. Its wifi capabilities just don't have the range you need. Prior to our last trip to Europe, I purchased an Alfa external wifi card with extra antenna.
    At 1,000 milliwatts and with the larger +9 dB antenna, only once was I unable to find a friendly neighborhood access point (that was in the suburbs of Paris, seems like the Parisians don't like to share, or are just more secure :-) At less than $40, this kit is now one of my "must bring" items.
    Tips:
    • Installing and using one of these cards is a piece of cake on Windows (just follow instructions and use the drivers on the included CD, or download the latest from the web) but getting it to work on OSX takes a little more work, see below
    • Walk around the premises as you look for that open access point, you'll detect different networks as you go from room to room
    • Set your card on a window sill (i.e. with the window open) for even better reception, it will make a noticeable difference
    • Be courteous: don't start downloading huge amounts of data, watching YouTube, etc. You're getting free internet access, don't be a pest
    • Be careful: you never know who else is listening to traffic on this network. Use HTTPS wherever possible. I heartily recommend Firefox with the HTTPS Everywhere extension (which isn't really everywhere, but it's a lot better than not using it)
    • I've passed through airport security with the +9dB antenna and no one's made an issue of it (rightly so but you never know what's going to tweak airport security these days...)
    • Added bonus: this card works great with BackTrack
    As I mentioned above getting the RTL8187L (that's the chipset in the card) drivers working on OSX 10.6 is a little more involved than Windows but I've successfully installed them on two MacBook Pros. These instructions come from http://www.insanelymac.com/forum/index.php?showtopic=208763
    1. Download RTL8187L driver for Mac OS X 10.5 fromRealTek
    2. Install it, including restarting. Ignore the error about the kext not being installed poperly
    3. Open /Applications/Utilities/Terminal, and type the following commands in order:
      1. cd /System/Library/Extensions/
      2. sudo chown -R root:wheel RTL8187l.kext/
      3. sudo chmod -R go-rwx RTL8187l.kext/
      4. sudo chmod -R go+rX RTL8187l.kext/
      5. sudo kextutil -t -v /System/Library/Extensions/RTL8187l.kext
    4. Agree when it pops up and tells you that there's a new network interface that's been added.
    5. You should then be able to open the /Applications/Realtek USB WLAN Client Utility and configure it to connect to your network.

    Rails3 Mind Map

    I find mind maps useful for many purposes. The process of clearing your screen, fullscreening your mind mapping tool, and immersing yourself in a topic of interested is a great brainstorming exercise.

    Today, my goal wasn't creativity, it was to build a map of main components of Rails 3. You'll likely find the PDF more useful as its nodes are clickable and refer back to the Rails API (and github in a couple cases where I found documentation to be more useful).

    This isn't a comprehensive map. Let me know what I've missed.

    (This map was created with MindNode Pro, an easy to use, cheap, and high quality OS X app, proving that sometimes you can have your cake and eat it :-)

    No More Excuses! Using RVM to Play with Rails 3

    Now that Rails 3.0 is out, it's high time to start using it. But what if you want to keep Rails 2.x around for your current projects? Fortunately, on OS X, there's a simple solution: RVM.

    Once you've installed RVM, you'll need to install a version of ruby compatible with Rails 3. There are two choices: 1.8.7 and 1.9.2. Given its new features and speed improvements, 1.9.2 is the one to choose, unless you have particular dependencies on 1.8.7.

    Installing 1.9.2 is simple: rvm install 1.9.2. This will download, compile, and install 1.9.2 to a .rvm folder in your home directory.

    Once that's done, type rvm 1.9.2 to switch over and rvm info to confirm that you're now running 1.9.2. Note: this will only apply to the current terminal window, here's how to make it the default.

    Type gem list and you should see just two gems: rake and bundler.

    Now go ahead and install Rails 3: gem install rails. Confirm by way of rails --version and gem list.

    That's it, you're done... Now have fun!

    What to go back to your previous version of ruby? Just type: rvm system and you'll revert back to your standard ruby installation and the gems that went with it.

    Steps to coax The Gimp into compiling under Macports

    Macports is a great tool to easily install a ton of open source software on your Mac. The packages are generally very well maintained but that can be a little tough for very complex ones like The Gimp, the best (?) open source image processing software out there. Though you can download a prebuilt version of The Gimp, I wanted to build my own in order to leverage some third party plugins, like Tilt Shift photography.

    If you haven't yet, install Macports.

    Then kick things off with "sudo port install gimp". This will likely generate a very impressive list of dependencies and start chugging through them. Expect this to take hours, not minutes. Hopefully, it will build cleanly all the way through. If not, the following notes may be of use... 

    My first failure (bug #26084) was quickly resolved thanks to the lightning support of the Macports maintainers (kudos to ryandesign [at] macports.org). Turns out I was a building a slightly out of date package. Following the advice to "sudo port selfupdate" and "sudo port clean libgnomeui", the build proceeded smoothly.

    The next build failure was already captured by bug #25962. There are three errors listed, though I only saw the first two. These were easily addressed with a couple of "ln -s" commands as described in the bug. I also installed select_python ("port install python_select") and pointed it to python2.6 as described in the comments.

    The final bug occurred when building gimp itself. You can check out #26095 and its simple fix.