The post has been updated since it was last published.

Big Data pilot

A fast growing data integration company, Informatica, observed that “the explosive growth of Big Data is no longer a news. Being within the reach of most organizations today, the challenge is to adopt the big data or hadoop technologies by starting with an executable big data pilot. A smart place to start is with a clearly defined Big Data pilot plan, led and owned by technology and governed by experienced data scientists”.

A pilot project plan describes all the details necessary to conduct a pilot plan successfully. This includes details on how to evaluate the pilot, the results of which will facilitate the decision whether or not to move the solution to production stage.

This checklist is designed for project managers, analysts in the IT services industry, data architectures and data scientists who plan to conduct a pilot project for successful execution of a Big Data solution. It enlists the core steps that will be a precursor in the implementation of the pilot project. By following this guide, Big Data scientists and project managers would not only be able to define the selection criterion for a big data pilot, they would also be able to execute the pilot plan successfully while remaining risk-aware.

This checklist is for use in the initial planning stages of a Big Data pilot. Being of a precise nature, it intends to serve as a guide for Big Data experts in the execution and implementation stages, helping them through the transition stages, evaluation and completion of the pilot.

1. Describe what the pilot selection criterion is

• Identify company goals
• Identify the compelling need for pilot
• Ensure the pilot can verify the architecture and roll-out plans in production environment
• Identify if a single pilot task is enough, or there shall be multiple (at any given time)

  Ensure pilot provided greater insight to actual solution of a problem

2. Identify pilot scope

• Define what are the solution components deployed and the span of customer’s operational environment for that deployment
• Identify what are the business functions and technologies to be exercised during the pilot
• Identify the number of sites, users and workstations
• Define the duration of evaluation period

3. Define site selection criterion

• Recognize testing goals
• Identify pilot visibility
• Identify if this pilot task as for technical, operational or business functions

4. Identify target user profile

• Define the number of target users and teams
• Define the user subset that will participate in the pilot

5. Define incremental milestone

• Define any milestones that are to be covered
• Identify number and class of additional users

6. Analyze success criterion and metrics

• Identify the target metrics and acceptable range of values for these success criteria
• The success criteria may fall in the following categories:
  • TCO – Total Cost of Ownership/ROI-Return on Investment
  • System performance
  • Operations cost
  • Stability –down time
  • Increased productivity
  • User satisfaction
  • Business goals

7. Execute/build transition plan

• Identify the transition strategy
• Define the procedures for rolling back a pilot to pre-deployment state
• Identify any freeze (or rapprochement) strategy that is to be carried out in case that a pilot is not following a predictable path

8. Architect a user preparation plan

• Build a marketing or evangelism plan that will describe the key messages that will be delivered to users
• Make a User Training Plan for training users during the pilot. This will include:

1. Training program developed for future testing/training cases
2. Documentation of all iterative steps, and expected/projected result
3. Incorporating user recommendations: generic and specific.

9. Evaluate the Pilot

• Include the method for assessing quality of pilot deployment process (e.g. user  surveys,  peer  reviews,  user interviews, etc.)
• What will be the designation/functional role of the people performing the evaluation?

10. Identify risks associated with the pilot

• Define any risks associated with the development of pilot components

The pilot checklist is based on the big data pilot template published earlier. If you want to generate the internal support for larger big data initiatives, you can download the template here.