Technoclippings

Technoclippings

Paul Clip  //  Living life as a glorious adventure!

Archive for

January 2011

Jan 31 / 10:30am

Mining the OSX Console for Fun and Profit

Well... Fun? Yes (if you're a geek). Profit? Not very likely but if you search long enough who knows? :-)

I rarely open the OSX Console app. This morning, while waiting for a call, I did and found this cry for help from Firefox:

11.1.31 9:25:12 [0x0-0xbb0bb].org.mozilla.firefox[2734] SHOULD NEVER HAPPEN

In fact I found dozens of them. Intrigued I started looking for other messages. Chrome, it seemed, was having problems of a more existential nature.

11.1.31 9:34:37 [0x0-0x2a42a4].com.google.Chrome[12790] objc[12794]: Class CrApplication is implemented in both [...] and [...]. One of the two will be used. Which one is undefined.

As long as one of them's used that's OK right?

Pages should really know better.

11.1.31 8:58:43 Pages[12487] *** WARNING: Method setDrawsGrid: in class NSTableView is deprecated. It will be removed in a future release and should no longer be used.

Uh-oh.

11.1.30 17:56:04 AppleMobileBackup[8848] WARNING: Backing up bf6f8237f787cbf4206d1e107b24aacd55c44b5b

Hmmm... Does MDRP phone home when you rip a DVD? Kind of: by default it will "Anonymously report rip statistics" but you can turn this behavior off in its preferences. I don't think this should preference should be checked by default but I like the fact that MDRP is at least logging this activity.

The most common message I saw? Variations of this one:

11.1.29 1:12:15 <Many Different Programs Here>[229] Can't open input server /Users/[...]/Library/InputManagers/Edit in TextMate

It seemed as if everyone was complaining about TextMate (I know, I should move to Vim, it's on the list...). Fortunately, there's a solution.

After scanning a day's worth of logs I decided I had better things to do!
Filed under  //  fun  
Jan 25 / 8:13pm

Tech Trends I'm Watching in 2011

The good news for us techies is that there are so many things to choose from... So, in no particular order, here are some that I'm excited about.

Continuous Deployment / Delivery

Different people give it different names but the concept is the same: Since you're already running continuous integration servers against your codebase to make sure they build correctly and pass automated tests, why not go one step further and deploy to production instead of stopping at the integration environment?

Companies that follow this approach will often release to production dozens of times a day. A key goal is reducing the "wait time" for a bug fix or a new feature to be available to users from days, weeks, or even months to a couple of hours at most.

On the technical side, continuous deployment calls for extensive automation, comprehensive unit and integration test coverage, clear procedures for making (reversible!) database updates, improved production server monitoring to detect any flaws ASAP, and the ability to efficiently stage a deployment through multiple "waves" of servers (e.g. start with 1, then 10, then 100 servers) and of course roll it back in case of issues. 

On the business side, continuous deployment means more flexibility in how features are rolled out. While bug fixes may hit all users very quickly, Product Managers can now batch features together as needed and release them to a few clients for beta testing, before "flicking the switch" of a full deployment. Of course, all this entails tight cooperation between Development, Operations, and Product Management.

While some will understandably be gun shy of pushing out code so quickly, I believe that companies that achieve continuous deployment will reap significant competitive advantage.

Pastedgraphic-7

Kanban

Another example of how our world is moving from the discrete to the continuous is in development "methodologies". Long ago, waterfall gave us release cycles measured in months and years. Today Agile approaches like Scrum typically aim for a "production-ready" codebase at the end of each sprint (typically two weeks). But two weeks is too slow if you're embracing continuous deployment.

Consequently I'm seeing more and more activity around Kanban or Kanban-like models for software development. Based on Toyota's Lean Manufacturing principle Kanban focuses on eliminating waste and improving efficiency based on (among other things) maintaining a smooth flow of features and fixes through the "development pipeline". This "pull" system is a lot more suited to rapidly responding to external changes than waterfall or even Scrum (where early sprint termination is usually fairly costly). 

In reality, it can often make sense to combine elements of both Scrum and Kanban. In my own experience we've found Kanban well suited for operations teams, esp. when they're dealing with many emergent tasks.

Kanban's challenge when used with continuous deployment is being able to decompose stories into small enough items. A four week work item defeats the purpose of continuous flow! :-)

Pastedgraphic-8

NOSQL

Alternatives to SQL really exploded in 2010 when NOSQL hit mainstream. Products like CouchDBMongoDBRedis, and Cassandra are powering very large sites. These databases are in some ways the descendants of the OODBs that seemed poised to break SQL's hold on our data 10-15 years ago and, more directly, related to the BigTable technology Google developed. 

While the tech community has a well-known affinity for bright, new, shiny things and we're still learning how best to use NoSQL, there's no doubt we'll be seeing a lot of action in this space in 2011. We'll also see a backlash fueled in part by highly publicized failures like this one (though there will always be counter-examples!).

Pastedgraphic-9

Smart Phones

Despite Microsoft's re-entry in the mobile space with Windows 7 Phone and Palm's impending release of webOS based tablets (oh, and RIM too), I doubt Apple's iOS and Google's Android will lose much if any market share.

Last year Adobe implemented Flash on iOS but was blocked by Apple's strict licensing agreement, which Apple then quietly relaxed a few months later. Given that you need your apps on both Android & iOS to reach the majority of consumers I expect cross platform development environments to get more airtime in 2011. Companies include RhomobileAirplay, and appMobi (which just announced a $6M round of funding yesterday).

In the meantime, since Android and iOS have sophisticated (and mostly compliant) HTML5 browsers,  some developers are building their mobile apps as web apps and releasing a "shell app" to both Apple and Google app stores. Netflix is a prime example of this approach.

Pastedgraphic-10

Security

It's not clear to me whether we'll see new classes of security products in 2011 but what is clear is that with attack surfaces multiplying, demand will be strong. Companies are expanding their apps into the mobile space, rolling out APIs, dealing with ever increasing browser complexity, maybe even experimenting with NFC, and they often do so at the expense of security. Just as importantly, many web apps now outsource key functions to third parties (e.g. customer supportpaymentsemails...). Criminals targeting these firms could potentially affect hundreds or thousands more.

Financial services may be where most of the money is but cyber criminals are nothing if not opportunistic. They'll keep looking for other industries to target, esp. ones that might not be so security minded. Speaking of branching out, I'm not sure if it will happen this year but with Apple's growing PC penetration it's only a matter of time before virus and trojan writers focus on OSX.

Concerns of "mini wikileaks" within large organizations will likely increase demand for employee monitoring tools, egress filtering, comprehensive audit trails. None of this will prevent a determined data thief, but it will satisfy auditors and shareholders.

There's more to talk about: Rails becoming the defacto startup dev platform, the problems of complex systems, and the progress towards AI, but these will have to wait for another time.
Filed under  //  future  
Jan 18 / 11:51pm

Fusing RSS feeds together from Posterous and other blogs

Posterous is a great web app: they've managed to the keep it simple and elegant to use while making it ever more sophisticated (thanks in large part to Garry Tan's leadership).

They're also very quick to respond to questions from users:

Pastedgraphic-1

Why did I need such a script? My oldest domain, cyberclip.com, was registered in 1995 (back when "cyber" was cool! ;-) and went through early years of excitement and later years of neglect. A few months ago I streamlined it to a very simple design:

Pastedgraphic-2

Recently I thought it would be nice to add just a little more content and decided to include a list of the newest posts from all my blogs. Since Posterous doesn't supply a consolidated feed, here's the ruby script I wrote to generate one. I've only tested it with FeedBurner and Posterous but it should work with all properly formatted RSS feeds.

I've purposely kept error handling out of this script for simplicity's sake because that's handled by the program that invokes it. That prog runs as a cron job and regens the homepage on a regular basis. It won't replace the homepage if RSSFusion errors out for whatever reason. So worst case is a stale homepage, not a broken one.

Combining blog posts for inclusion in another page is one use, but this script could also be used to generate a fused RSS feed of multiple blogs, which is something I should probably add to my homepage as well...
Filed under  //  ruby  
Jan 16 / 4:15pm

Software Architecture should be forged in Fire, not Carved in Ice

Pastedgraphic-8

(Picture by Jason Bolonski)

I've seen a number of corporate environments that carve software architectures out of ice.Why ice? Because an ice architecture sure looks great: sparkling, pristine, perfect even. This approach often feels right, esp. in cost conscious, slower moving organizations. You know how it goes: spend the bulk of your time in design, figure things out properly, measure twice (or thrice), cut once, and then you're set. Sadly once carved, often the only thing left to do with such an architecture is freeze it, lest it melt. And a frozen architecture is rarely useful.

The problem is that you're never set. Needs keep changing and if your architecture can't evolve with them, you've (best case) got a working but unmaintainable and unevolvable app, or (worst case) something that becomes unusable, even by the people who need the application the most and who are willing to put up with its flaws.

The architecture you want is not one carved in ice. Rather, you want something you've not only heated and beaten into shape to serve your current purpose, but also a design that you can reforge in to something new as needs dictate. So how do you achieve this?

Understand your problem space & key challenges
I'm not advocating no design, I'm advocating just enough design. Knowing how far to go is both art and science. Two things that will help is a good understanding of the problem space and the main obstacles you'll face. If you're building a social app you need to have at least broad designs for your sharing / trust model for users, how you will distribute data and scale, what security choices to make, etc. If you don't have strong expertise in house, hire someone who does, even as a part time consultant. It will be money well spent.

Rapid Iteration
Focus on speed. Not at the expense of quality but at the expense of features. Build your Minimum Viable Product, get it live, get it used, and iterate. The only way to really learn what works and what doesn't is to let your users at your app. The faster your iteration the more you can adapt (reforge) to changing needs. This is one of the reasons that agile development has become the de facto development approach in the past decade. Continuous deployment and delivery are other, welcome, instances of this trend.

Less is more
Build what you need and improve when needed. This goes hand in hand with rapid iteration and goes against the "what if" architecture. "What if I want to go live in other countries? Oh better internationalize", "What if we need to support suppliers as well as customers? Good point, better code them in". "What if I need to scale to hundreds of millions of users?" The list goes on and the longer you let it go the slower you'll be, not just in development but in maintenance and new feature additions.

This doesn't mean making uneducated decisions. If you think there's a good chance you'll need to go international in future, leverage a framework that supports it. But don't build it internationalization (i18n) until you're ready to use it. I'd argue that the cost to support i18n - testing multiple languages across the app, different formats & interfaces, new & altered business logic, etc. - is not worth saddling your dev team with against the day when you finally need the feature.

Ultimately, the fewer lines of code the better. To paraphrase Dijkstra: it's not lines of code produced, it's lines of code spent!. So spend wisely.

Test Driven / Refactoring
If there was one software engineering practice I'd enforce, this is it. A comprehensive and robust set of tests gives you the confidence to make radical structural changes to your application and still have a working product at the end of process. I've seen business leaders question the value of putting in effort here. Understandably they're concerned that all the time spent coding tests could have been better spent coding new features.

A counter-argument to this is to remind the business folks that, once tests are written, all the downstream QA is not only free but extremely rapid. That's when you can reassign developers to new areas confident in the knowledge that you'll detect breaks long before they ever reach end users. As your codebase grows these tests will be a godsend to help avoid spending all your time on maintenance.

"Less is more" is your friend here too: focus on DRY (Don't Repeat Yourself). The DRYer your code, the less of it you have, so the fewer tests you'll need and the easier it will be to reforge.

Culture
The most fundamental and so the most important. The culture of an organization is represented by people's shared values, goals, and behavior. Whether implicit or explicit, culture is the bedrock on which all else rests. The more individuals align with the culture, the more effective the team. This buy-in means that a successful culture cannot be dictated (typically by management), it must be nurtured.

Culture will obviously vary by company but these common values will support all the principles listed above:
  • Continuous improvement: Continually striving to make things better, to achieve ever higher quality, and redefine goals as necessary 
  • Trust: Despite the best laid plans, failures happen. When they do, an organization needs to display enough trust in individuals and team to allow them to fix the problem, learn from the experience, and come out stronger
  • Collaboration: In my experience, tech and business are often in push-pull. Tech rarely gets the necessary time or resources, and business rarely gets all its desired features. The principles above drive long term value. If the culture prizes collaboration, openess, and sustained value, then neither group will want to sacrifice the short term for the long term

Update (2011.2.3): Great article on the importance of culture from Wealthfront.

 

What about...?
What about separation of concerns, SOA, AOP, and more? All of these have their place and can certainly improve your application and architecture. Design patterns, if properly used, can make your software design more flexible and reduce the amount of refactoring needed to add new features. Still, these practices are should haves, not must haves. They can make good architecture great, but on their own won't keep it great in the long run.

Ultimately the key to building great software is great people. Finding them, building the right culture together, and continuously evolving the organization and its processes. Over the years, new software engineering practices will come to light but these principles will evolve much more slowly.
Filed under  //  architecture