wissel.net

Usability - Productivity - Business - The web - Singapore & Twins

Structuring a Proof of Concept


A common practise in IT, as run up to a sale or a project is to proof that the intention of the undertaking can be fulfilled.

The challenge

A PoC needs to strike a challenge between effort and coverage. A final proof of a project is its completion, so the temptation lures to try to proof everything. On the flip side: if they core functionalities aren't covered the proof has little value.

The second challenge is to define concise successs criteria. Quite often, especially for standard product PoC, it is left to ?how users like it' - which isn't a really qantifiable result.

Use cases

A workable approach is to define use cases, that cover a typical scenario, like ?Sale of an ice cream'. This scenario needs to be broken down into business steps until a step can be looked at: ?did work / did not work'.
The breakdown needs to be business level, business language. So ?Can click on customer info' should rather read ?Customer info is retrievable'.

Use cases and steps are hierarchical, typically 2-3 levels are sufficient for most PoC. Deeper levels are a smell that you are looking at a pilot or full fledged project, not a PoC.

So, in a nutshell: A PoC line item needs to have a binary answer. If a binary answer isn't possible break the line item into smaller units. Stick to the domain specific language (usually: the business steps)

Measurements

When a use case line item has a binary outcome (works / doesn't work), the simplest measure is to check if everything worked to declare the PoC a success. Usually doesn't help.

The next level is to define a pass percentage. Like 70% of 200 line items must pass. Again a simple solution. Challenge there: nice to have and essential features have equal weight. You could end with an outcome that has all nice-to-have features, but might miss essentials.

So the next level is to define weights for each items, including a showstopper flag for must-have features. Weighting discussions are popular battle grounds for feuding fractions, since the weight determines outcomes, especially for concurrent PoC execution.

Another weakness of this approach: works/doesn't work as binary value doesn't cover: ?Does it work well?'. Like ?Is a pair of sneakers suitable to get from Boston to New York?' The binary answer: Yes you can walk, but the real answer: use a car, train, bus or plane.

Balanced Scorecard to the rescue

Looking at the definition of Usability, one can find 3 criteria:

  • Does it work?
  • Is it efficient?
  • Is the user pleased?

I would treat the first column as a binary value and the later two as scales from 1-5. This allows to generate a balanced score card that reflects important aspects of a proof. Depending on the nature of the system, you could add additional columns like ?failure resistance, error recovery, risk'.

While it doesn't relieve you from the weight bickering, it provides a clearer picture of actual outcomes.

As usual: YMMV


Posted by on 04 October 2018 | Comments (2) | categories: Salesforce Software

Comments

  1. posted by Ben Langhinrichs on Saturday 06 October 2018 AD:

    This is a good discussion of the issues, as well as the pitfalls, of a PoC. One of the biggest challenges I've seen with such efforts, and one that I've seen doom "successful" proofs-of-concept, is the scalability issue. Your last comment about walking vs. taking another mode of transportation touches on the issue, but this can be very hard to prove or disprove if the PoC is based on too minimal a subset of functionality, or based on an incomplete system. For that reason, I think any PoC should identify areas likely to degrade with load. Once you ask the question that way, it can be fairly easy to differentiate, as it often is the difference between self-contained functions (e.g., client-side) vs. remote-dependent (e.g., server-side or database lookup or extensive validation determinant). It is still somewhat of a guess how it will operate under load, but this allows educated guesses, and perhaps identifies areas to be completed early even after a successful PoC in order to allow a cut off if there is a failure.


  2. posted by Erik Brooks on Wednesday 10 October 2018 AD:

    Stephan, I've been reading your blog for years and always loved it, especially your insight into "fuzzy" problems such as these. Keep rockin'!