On the Path

What is Crowdsourcing?

We’ve all see the online CAPTCHAs that make you type in a pair of garbled words to prove you’re not a spam-bot. What you might not know is that by doing this, you are participating in crowd sourcing, a way to gather small amounts of data from large numbers of people. In this case, many of those words are used to digitize old books from the Library of Congress. When a computer can't read the scanned text, it "crowd sources" the unknown word. Once two or more people enter the same word for the garbled image, the computer now knows what that scanned word is. It’s pretty amazing to think a few seconds of time from millions of people can create public works of real value. This is the value of “crowd sourcing”.

Can your business use crowd sourcing? It works well for things that are “hard” for a computer to figure out, but easy for humans. For example, “which of these pictures contain pasta dishes?”. Amazon has a service called 'Mechanical Turk' which farms out work like this to interested people in exchange for micro-payments. Say you're willing to pay a penny a picture to have your images categorized or tagged. Amazon’s service will match up your job with interested parties. Amazon gives you back the results just as if a computer had processed the images, but instead the work was done with human intellegence. The results can be integrated into your data, allowing your systems to act on the new information. In fact, we’ve used services like this to verify that profile photos on a social network were all 'G-rated', and to generate transcripts for video files.

Can your business make use of crowd sourcing?

Protect Your Intellectual Assets with a Reconstitution Plan

I once knew a small company that was effectively shut down because their internet service provider (ISP) had gone out of business.  In order to recover the 'intellectual assets' of their business, they had to go to the liquidation auction of their ISP and bid on (and win!) the physical machines that had been running their websites. Would you fare any better?

As a business owner, you want to make sure you have a complete reconstitution plan. Backups are a good start, but aren't enough by themselves. Make sure your reconstitution plan includes:

• a tested backup and restore of the database, source code, deploy procedures, uploaded images/documents, and all of the configuration files specific to your servers. Make sure the backup contains all the files needed by your web app, in a format you can access and use for a restoration.

• a map of all of the servers in use, including their names, IP addresses, and services they are providing. This includes things like your name server and mail server, which are probably handled by your service provider (and ignored by most plans that think just having a backup is enough).

• The login credentials of your domain name registrar, in case you have to change your domain name servers.

•  The login credentials to your domain name servers, in case you have to change the ip address of a name (so your "www.your-domain.com" can move to another server when necessary).

 

Spend a little time letting your imagination run through disaster recovery scenarios...  What would happen if there were a fire in your data center?  Would your backups be destroyed too?  What would happen if your machines were hacked, data was corrupted, and you didn't discover it for a week?  Would your only backups contain that corrupted data?

I hope you never need a reconstitution plan, but if you do, you'll be grateful for the time you spent preparing. As we all know, hope isn't a good business strategy. So start your plan now!

 

The Power of Accountability

Many years ago when I was getting my red cross first aid certification, I learned that when starting any kind of emergency first aid where additional help is needed, you pick someone out of the crowd, point at them, and say, "YOU!  GO CALL 911!".  The theory goes that by singling someone out, it is more likely to happen.  If instead, you just yelled, "SOMEONE GO CALL 911!", the crowd would continue to gather and watch.

I am reminded of this constantly when working with teams who coordinate most of their interaction in email.

Today we had a question about some important batch processing that was supposed to complete last night.  Someone wondered if the processing completed as expected and sent an email to three people.  The most senior person was out of the office but saw the email and thought, "One of those other two guys will get it". the mid-level person was off on another project for the week, saw the email and thought "I'm out of the loop - one of those other guys will get it".  And the third-level person saw the email and thought "With those two other guys on the list, It's not really my place to respond", and the person needing an answer just got radio silence.

A simple change would have made this go a lot differently - hold one person responsible and ask them.  At the very least, send the message 'to:' one and cc: the others.

This is not an uncommon occurrence - I think of this analogy all of the time; today I decided to write about it.

Assessment vs. Certification

In my last blog post on assessing agility, I asked three questions. This is my answer to "how an assessment is different from a certification".

First, I'd like to give a historical perspective for my answer. For several years, I worked for a company that was infatuated with the CMM/CMMi model for process improvement (I was *not* one of the infatuated, which I think gave me a healthy perspective for its strengths and weaknesses). I originally heard this 'assessment vs. certification' explanation in the context of the CMMi and ISO 9001.

An 'assessment' is something that occurs at a specific point in time. The organization goes through some kind of audit, and after that audit, can forever say they were 'assessed', and what the results of that assessment were. That assessment might be done internally, might be done by an external entity, but it really just represents a 'point in time'. It is reasonable to say things like "I know we were assessed 3 months ago, but I want to re-assess now".

A 'certfication' might start with the same kind of audit, but then continues on to some kind of 'official blessing' with an expiration date, implying that the certification has to be earned again at some point in time (and the corollary, doesn't need to be checked in the interim). This certification also carries the weight of some external entity that can bestow this certification. In many cases, only a 'renewal fee' is needed to maintain the certification.

For my tastes, an 'assessment' seems lighter-weight and less bureaucratic. While past results might be an indicator of future success, a positive assessment 6 months ago carries less weight than a 'certification' that doesn't expire for a year. An assessment also doesn't imply the external organization to define, grant, and police the certifications. Even if benevolent, a good process framework doesn't need a 'dictator' to approve what you do, as long as it is measurably working.

Also in our industry, 'certification' is a word that implies a significant amount of vendor influence. "Microsoft Certified Software Engineer", "Sun Certified Java Developer", "Certified Scrum Master"... in each of those cases, the certification process is as much about representing vendor interests as it is proving anything to anybody. In the case of a yearly fee, these certifications become more of a checklist to make sure you are 'in the club', as opposed to really proving you know anything. Yes, thats a cynical view, but I think it represents many a vendor certification in the techn industry.

I believe that if we are going to try to define something useful for agile project guidance, this should be an 'assessment' process, not a 'certification' process. While vendors may offer tools and wisdom within the scope of the assessment, the assessment should be protected from vendor interests via some open and transparent process.

Falling in Love with a Problem

As human beings, we tend to be problem-solvers - and the geeky kind of people I hang out with tend to do this more than most. Clients come to us with business problems we write software to solve. Family members come to us with computer problems to solve. As managers, we solve schedule issues, coordination problems, and other, more 'human' kinds of problems.

We do this not just for work. In our spare time, we contribute to open source projects. We solve crossword puzzles, sudoku problems, and the like. We play games that have increasingly complex rulesets. We build inkjet printers out of lego parts just to say we can. Magazines like Make and events like Maker Faire cater to this.

Ideally, we solve a problem in a way that removes it from our path. Unit tests? Check. Admin tools for the client so they can tweak things themselves without having to get us involved? Check. Self-opening garage door that senses our car approaching? Check. Every now and then, this process goes south. We don't solve the problem in a way that leaves it solved - we solve the problem in a way that puts it on a pedestal. We 'fall in love' with the problem we are solving, so we never completely solve it.

I coined the phrase "falling in love with the problem you are trying to solve" several years ago for a presentation on Maven. Maven is a build process automation tool for Java, although fans of Maven will take issue with that description (ironically, that helps to prove my point). A build process is something that should 'just work', get out of my way, and be easy to change when need be. Ant, despite its problems, fits this description. Rake, the build tool popular in the Ruby community, fits this description too. Maven, however, doesn't just build the project... It builds a project website... it tries to be a release tool... and it is the repository for project-specific information all the way down to contact information for team members.

Looking at Maven in the wild, you see an interesting pattern. People who first use it find it amazing. "It does everything I need and more", they will say, because at first, it does. After a little bit of usage, they run into something just outside of the well-worn path of Maven. "I need to pre-process this spreadsheet I got from the translation company into a bunch of property files for localization". Good luck with that. Asking for help on the Maven mailing list will get you a response like, "Oh that's easy - just write a Maven Plugin". And suddenly, the tool that was supposed to do everything and get out of your way becomes something you need to understand at a deep, deep level. Some people will continue to love Maven, and go down the rabbit hole of complexity. Others will look up from their keyboard after 8 hours of trying to figure out the bizarre value you have to pass to Maven via the '-D' parameter in order to generate a Maven Plugin Template Project and say something about shaving a yak.

I also see this "problem loving" pattern in the Scrum community. Scrum is a process framework that software development teams use, and the community has been in an upheaval for the past year or so while the top-feeders of the Scrum food chain have been arguing about the tenets of scrum, and the future of scrum certification. A bunch of blogs have been written, a bunch of opinions expressed, and you know what? For the average developer who just wants to get their job done, this controversy is distracting. Sadly, this argument is over dogma in a community that is supposed to be guided by pragmatic principles. The dogmatists have fallen in love with the problem they are trying to solve. The debate may be beneficial for the long-term health of the Scrum methodology, but here and now, they seem disconnected from the real problems Scrum was meant to solve.

I point this out not to pick on Maven or Scrum, although I'm sure if the dogmatists in those communities see this post, they will likely comment defending their 'precious' (I use both Maven and Scrum regularly, and in another context, might be such a poster). I point this out to illustrate the problem of 'falling in love with a problem'. I'm working on a project now where I continually question whether we are building a rube-goldberg machine that workships complexity, or whether this really is the best approach with the constraints in place.

In this trap, you become distracted from the ultimate problem you need to solve, and instead build cathedrals that worship the problem. We can all dysfunctionally fall into this trap; Ask yourself what problems you might be loving too much.

Feedback

For any online venture, listening to feedback from your users is important if you want to be successful.  In particular for one of our clients, determining how users arrive at the site and why they leave provides invaluable insight.

One method of attracting users to the site relies on ads placed with the major search engines.  When a user searches for certain keywords, these ads are displayed alongside the search results.  When a user clicks on one of these ads, a cookie is placed on their computer and they are forwarded to our registration page.  Should the user sign up, another bit of code (supplied by the search engines) looks for the cookie and records a "conversion" which lets us know how the ads are performing.  Obviously, we would also like to know what keywords the user searched for that brought up the ad so that we can apply resources towards the most popular keywords and maybe even discover keywords that we should be paying attention to.  All that's needed here is some parsing of the URL that's generated when a user clicks on one of the ads.  Google, Yahoo, and MSN all have the ability to append the keywords that the user searched to the URL that is specified in the ad.  When a user lands on our registration page, we simply check for the appropriate URL parameter and save off any keyword data that may be there.

So now that we know how the users got to our site, what about why some choose to leave?  Well, to capture this data, we implemented an "exit survey" using jQuery that simply asks the users to choose a reason they are removing their account and optionally enter some comments about why they are leaving.  The form is straightforward and simple and takes just a second of the users time before removing their account.  The data is saved before removing the account and allows our client to monitor what may be frustrating users so that the site can grow and improve.

Well, there you have it.  Adding simple URL parameters and exit forms to your site is an excellent way to keep in touch with your users and to monitor trends in your membership.  Here's to more URL parsing and less exit form analysis in your app!

Massachusetts Data Security Law and Your Website

Let me preface this blog entry by saying that I'm not a lawyer, but I think if you read the references I'm pointing you to, you won't have to be to reach the same, logical conclusion. I am offering my own informed opinion regarding how a new Massachusetts law might affect the data security standards on websites you might manage. If you happen to be a lawyer with expertise in this area, I welcome your informed legal opinion as a blog comment.

There has been some FUD recently about a new law in Massachusetts: 201 CMR 17.00: STANDARDS FOR THE PROTECTION OF PERSONAL INFORMATION OF RESIDENTS OF THE COMMONWEALTH. I think this law is a great example of something that should exist at a Federal level, but I'll save that editorial for another time. I want to correct some misperceptions I see appearing in the technical media.

The first piece of mis-information comes from this 'article' at SQL Server Magazine. That article specifically says "if you store personal information like first and last names of Masachusetts residents", runs through a sample calculation of losing a database of 1,000 names, calculates a nightmare fine of five million dollars, and then shows you a link to where you can find out tons of information about 'upgrading' to Microsoft SQL server. The article's very first point about 'first and last names' is completely false, and I'll prove it to you.

Second, the venerable nerd news site Slashdot has a story with a link to that very same article, along with a few hundred comments from industry leaders through pimply-faced 14-year-olds, and everyone in between, with conclusions ranging from 'don't worry about it' through 'possession of a phone book is now illegal'. Slashdot might be a great place to read about linux hacking, but is not a great place for legal advice.

I wanted to understand for real what this law might mean to my clients. After all, virtually every site we maintain with user accounts has data that could be construed as "personally identifiable information"... especially by the slashdot crowd. I wanted to make sure we comply. So I tried to read the text of the law directly from the Massachusetts Government Website.

I found something interesting: the text of this law doesn't read like some obscure tax code - it actually reads more like something I'm familiar with - requirements for implementing software. This law, at least the technical burden it would place on us, was very understandable.

The majority of my questions were answered right on page 2, under definitions:

Personal information, a Massachusetts resident's first name and last name or first initial and last name in combination with any one or more of the following data elements that relate to such resident: (a) Social Security number; (b) driver's license number or state-issued identification card number; or (c) financial account number, or credit or debit card number, with or without any required security code, access code, personal identification number or password, that would permit access to a resident’s financial account; provided, however, that “Personal information” shall not include information that is lawfully obtained from publicly available information, or from federal, state or local government records lawfully made available to the general public.

Read that carefully - its not just the first and last name - it is the first and last name in combination with one or more of (a) Social Security Number, (b) one of several state-issued IDs, (c) one of several pieces of financial information. The combination is what leads to a violation, and that makes perfect sense.

Notice also there is an exemption for "information that is lawfully obtained from publicly available information, or from federal, state or local government records lawfully made available to the general public".

Clearly, this law isn't talking about "first and last names of Massachusetts residents" as the SQLServer article would have you believe. Surprise, surprise, an article in a magazine named after a commercial software product is more interested in selling you said commercial product than offering you legal advice. Clearly, phone books are not illegal in the state of Massachusetts. Again, I'm not a lawyer, but I don't think I need to be to reach that conclusion.

But how might it affect the design of a website for a client?

If you are a tech firm designing websites that financial information of the type covered by this law, than you should already be aware of the Payment Card Industry Data Security Standard. If you are, then the contents of this law shouldn't surprise you. If you aren't, then I hope a million dollar fine puts you out of business, because that would make our whole industry more secure.

Both this law and the PCI-DSS specify some pretty sane policies on everything from the encryption of data in the database through the physical security of the machines that store that data, and includes sane definitions on what constitutes a breach, and how this sensitive data should be stored, used, transmitted, and transferred. This new Massachusetts law specifies who has this duty under the law, and extends these sane policies for sites that store social security numbers and other state-issued IDs. That is a great thing, and is pretty easy for a good tech firm to do for you. An even better tech firm will ask you "are you sure you want to be collecting that stuff in the first place?"...

Other Salient Sections of the Law

There are two sections of this law that pertain to technologists rather than lawyers: 17.03: Duty to Protect and Standards for Protecting Personal Information outlines the policies and procedures you and your company should have in place to safeguard data, and 17.04: Computer System Security Requirements outlines the security mechanisms and acess restrictions that should be on your servers, such as the usage of user IDs and passwords, restricted access for administration, firewalls, virus scanning, and so on. These sections are written well enough to be an auditing template for your technical staff to follow. I'll take this opportunity to give our company a plug - if you'd like an independent opinion on how your compliance fares, give us a call.

Should this be a Model for the Nation?

Why do I think this could be a good starting point for a Federal law? First, because if I'm doing this properly for Massachusetts, it would actually be more work to not do it for every other state out there. Second, the law doesn't say anything about things like your passport ID number, a federal Tax ID number, or the like, probably because the State doesn't have jurisdiction over the Federal space. Third, I think I have proven above the exemptions for publically available data and the 'in combination with' clause make this a pretty rational law. And finally, with a Federal law, there won't be a million variants to support... and I'm sure several companies would come out with compliance and auditing services (like Sarbanes-Oxley), and I'll be able to prove our worth to a client by having the systems we design and implement sail through those with flying colors.

Keep It Clean

We manage a high profile public website for a client that has thousands of visitors a day.  And while we attract a lot of legitimate users, we also attract a lot of ‘undesirable’ traffic - hackers, people trying to collect email addresses for spam, kids looking for places to dump music files, and people otherwise just out to be a nuisance.  Initially, we manually filtered out these ‘bad guys’ by doing a periodic scrub of new accounts.  We had even developed some profiles of ‘likely nuisance’ users so we could do this job quickly.  As the website grew in traffic though, this became a larger and larger activity.  Clearly, this wouldn’t work long-term - we’d be crushed under our own success.  So what did we do?

Enter Iovation.  Iovation is a company with a product that is always checking your user base of these problem users. At every account creation, and even at every login, Iovation can hook in and flag problem users.

Integrating our Ruby on Rails site with Iovation is accomplished by including a piece of javascript (supplied by Iovation) to your account creation and/or login pages that grabs what they call a "device id" for each user that visits your site.  This id is uniquely identifies the users computer and is then sent by means of a web service to Iovation to be checked against a database of known malicious users.  Should the check prove positive, you can choose to remove the user from your site, place their account in some sort of pending state, or whatever you wish.  The point is that those users don't make it to your site and that the process is automated. 

"So, what is this database of "known malicious users" and where did that come from?", you're asking.  Well, there are two ways in which Iovation can help you gather the evidence to keep your site clean.  The first is through your own admin utility where you manually remove accounts.  Iovation again supplies the code to add an additional step to your delete process that allows you to select from a list of offenses such as "SPAM", "Inappropriate Content", "Identity Theft", and many others.  When you delete this person from your site, another web service call is made to store this user in Iovations database.  Should this user try to create another account on your site, they will be rejected and there are rules that also allow you to determine how many accounts can be created within a given time range from a particular device.  The second way Iovation helps you keep your site clean is by leveraging other sites that offer similar services as your site and who are also clients of Iovation.  This is the real power of this technology.  Our application is in the social networking arena, so immediately we were able to use thousands of records that have been flagged by other social networking sites to cut down the number of unwanteds in our site.  Iovation allows you to choose which evidence types you'd like to check against when using data that isn't generated by you. 

And finally, I should mention that Iovation provides a web tool that you can use to generate reports, find a specific device or account, view business rules for the types of evidence that you trust from other companies, and even retract evidence should there be some kind of misunderstanding.  Our member complaints of harassment and solicitation dramatically declined after implementing this technology and would recommend giving them a look if your site's success is based on having a community free of online scam artists.

Rewarding Failure

At last weekend's RubyNation conference, I had the pleasure of attending a talk by Danny Blitz. Danny's personality came close to stealing the show - if you ever have a chance to hear him speak - or better yet sit down and have a one-on-one conversation with him, do it. He is quite a presence. He's a fantastic manager too - given the opportunity, I'd love to work on a team he put together.

One of the key points of his presentation on successful management was "rewarding failure". When he said this, he got a combination of enthusiastic applause and quizzical looks (as well as a woop woop from Aaron Bedra). His point can be summed up with the old saying, "If you want to bake a cake, you need to break a few eggs". The crux of his point: If you don't fail occasionally, you aren't taking the risks you need to have a big success.

I thought of a story I tell at NFJS conferences - "The Naive Manager and the Case of the Broken Builds".

I once worked with a manager who knew very little about software - he managed the paperwork of the processes, and at that he was very thorough. Once I was putting a really good employee through their annual review process. When I turned in the review paperwork, the manager said, "I don't know about this... I see from the team's emails he occasionally breaks the build..."

You see, we had set up a continuous integration server. Every 15 minutes, the server would wake up, build the code, and send out a success or failure email. Occasionally, in the course of doing our jobs, we would occasionally forget to check in a new file, have to update the build system to use a new library, etc. The manager had noticed that over the course of the last year, this employee had broken the builds a few times.

These are the kinds of failures that happen *all the time* on a software project... A good project will have a continuous integration server to catch them as early as possible, before there is a stack of problems on top of it to solve. This manager seemed to think that breaking the build was bad, but in reality, its not a problem at all when it is caught and fixed immediately. A *continuously* broken build is bad, which is a story for another time. Risk: good. Failure: allowed. Just fix it when you screw up.

Take the chances you need to succeed big - but do it with a net. The conclusion Danny left us with is that good management and development practices comprise that net.