COMMONWEALTH OF MASSACHUSETTS MASSACHUSETTS OFFICE OF INFORMATION TECHNOLOGY
Request for Quotes for
Independent Verification & Validation Services for the Benefits Realization of the Enterprise eLicensing System (EES) Document Number: MassIT RFQ 15-27
THIS RFQ AND ALL RESPONSES HERETO, INCLUDING THE WINNING BID, SHALL BECOME PUBLIC RECORD AND CAN BE OBTAINED FROM MASSIT BY SENDING AN EMAILTO
[email protected]. ANY PORTIONS OF A RESPONSE THAT ARE LABELED AS CONFIDENTIAL WILL STILL BE CONSIDERED PUBLIC RECORD.
TABLE OF CONTENTS Section
Page No.
1.0
DEFINITIONS ……………………………………………………………………..
2
2.0
GENERAL PROCUREMENT INFORMATION ………………………………..
5
3.0
PROJECT OVERVIEW…………………………………………………………..
5
4.0
ABCC AND DPL OVERVIEW…………………………………………………..
6
5.0
PROCUREMENT CALENDAR…………………………………………............
6
6.0
TERM………………………………………………………………………………
7
7.0
BIDDER REQUIREMENTS ……………………………………………………...
7
8.0
IMPORTANT INFORMATION FOR BIDDERS…………………………………………………………………………
7
9.0
IV&V SERVICES REQUIRED. . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . ..
9
10.0
CONTRACTOR AND CONTRACTOR EMPLOYEE REQUIREMENTS……
13
11.0
PRE-RESPONSE PROVISIONS………………………………………………… 14
12.0
CONTENTS OF RESPONSE…………………………………………………….
13.0
SUBMISSION OF RESPONSES………………………………………………… 18
14.0
EVALUATION OF RESPONSES………………………………………………… 19
15
A list of documents referenced in this RFQ that are separately uploaded to COMMBUYS and posted with this procurement may be found at the end of this RFQ.
1.0 DEFINITIONS Terms used in this RFQ have the meanings ascribed to them below: ABCC means the Alcoholic Beverages Control Commission, an agency within the Office of the State Treasurer and Receiver General. ABCC/DPL Implementation means the implementation of the EES at ABCC/DPL under HED’s contract with Accenture. Accenture means Accenture, LLC, the systems integrator. Accenture RFQ means the Request for Quotes for Enterprise eLicensing and ePermitting Systems Integration Services, as amended. Accenture Statement of Work or Accenture SOW means the Statement of Work between Accenture and HED for the purpose of creating an Enterprise eLicensing System, with a detailed description of a task order approach of the services to be performed by Accenture to create the Enterprise eLicensing System and of the rights and obligations of both Accenture and the Commonwealth and attached to this RFQ as Exhibit D. Accenture Task Orders means the task orders entered into by DPL and Accenture in connection with the implementation of the EES for ABCC and DPL.
Page 2 of 64
Apparent Successful Bidder means the Bidder to which the SST has voted to award the Contract in the period before the IV&V SOW has been executed. Once the Apparent Successful Bidder has executed the IV&V SOW and all other required forms and documents, it is known as the IV&V Contractor or the Contractor. Best and Final Offer (BAFO) means a price offered by a Bidder in response to the SST’s request that the Bidder lower the price offered (or provide additional services or commodities at the same price offered) in its Response. Bidder means a party who submits a response to this RFQ. COMMBUYS means the official system of record for all procurement information in the Commonwealth, which can be accessed at www.commbuys.com. Commonwealth means the Commonwealth of Massachusetts. Commonwealth Information means personal and other Commonwealth data in the Contractor’s possession, or used by the Contractor in the performance of the Contract, which includes, but is not limited to, Commonwealth public records, documents, files, software, equipment or systems. DPL means the Division of Professional Licensure, an agency within HED. Effective Date means the date of execution of the SOW by MassIT and the IV&V Contractor. eLicensing Project means the tasks and activities undertaken to create the EES. The eLicensing Project comprises the DPL/ABCC Implementation and the implementation of Accela for other agencies. Employee means, without limitation, any officer, director, member, partner, full or part-time employee, temporary employee, contract employee, individual consultant, volunteer, trainee, student intern, agent or subcontractor of the IV&V Contractor, or any of the aforementioned who are employed or retained by a subcontractor or consultant of the IV&V Contractor, who will work on the Contract. Enterprise eLicensing System or EES means the eLicensing system that the Commonwealth is seeking to implement under an agreement with Accenture. EOHED or HED means the Commonwealth’s Executive Office for Housing and Economic Development. . IPA means the Intellectual Property Agreement that is attached to the SOW template attached to this RFQ. IT means information technology. IT Capital PMO means MassIT’s Capital Portfolio Management Office. ITS53 means, collectively, ITS53 IT Project Services – Solution Providers Statewide Contract and ITS53 IT Project Services – Technical Specialist Statewide Contract. IV&V Contract means the contract for IV&V services entered into by MassIT and the IV&V Contractor as a result of this RFQ, which shall consist of the following documents in the following order of precedence: (1) the Commonwealth Terms and Conditions; (2) the Commonwealth Standard Contract Form; (3) Request for Response ITS53; (4) the Bidder’s Response thereto; (5) this RFQ (including all exhibits and attachments, amendments, clarifications and supplements thereto); (6) all other documents or terms negotiated by and agreed upon by the IV&V Contractor and MassIT, including the IV&V Statement of Work and related Task Orders; and (7) the IV&V Contractor’s Response to this RFQ (including all exhibits and attachments, amendments, technical corrections, clarifications, best and final offers, and supplements thereto).
Page 3 of 64
The order of precedence of the documents comprising the Contract set forth in the above definition is non-negotiable; Bidders who do not agree to this order of precedence should not respond to this RFQ. IV&V Contractor or Contractor means the Bidder selected by the SST in accordance with this RFQ to perform IV&V services. IV&V Project or Project means the performance of IV&V services with respect to the EES in accordance with the IV&V SOW. IV&V Project Manager means the individual, who will be identified in the IV&V SOW, who is responsible for the IV&V Contractor’s day-to-day operations under the Contract. IV&V services means independent verification and validation services. IV&V Statement of Work or IV&V SOW means the Statement of Work between MassIT and the IV&V Contractor developed in accordance with this RFQ for the provision of IV&V services on the eLicensing project, which will consist of a detailed description of a task order approach of the IV&V services to be performed by the IV&V Contractor with respect to the eLicensing Project and the rights and obligations of both the IV&V Contractor and the Commonwealth. IV&V Task Order means a Task Order entered by the winning Bidder and MassIT in connection with an EES implementation under a contract between a systems integrator and a Commonwealth executive department agency. There will be a separate Task Order for every Accela implementation for which the Contractor will provide IV&V services. IV&V Task Order 1 will address the provision of IV&V services in connection with the ABCC/DPL implementation of Accela. M.G.L. means the Massachusetts General Laws. MassIT Project Manager means the individual, who will be identified in the IV&V SOW, who is responsible for the Commonwealth’s day-to-day operations under the Contract. Massachusetts Office of Information Technology or MassIT (formerly the Massachusetts Information Technology Division) means the Commonwealth Office within the Executive Office for Administration and Finance that is responsible, among other things, for running central IT operations, overseeing IT policies, standards and architecture, managing IT capital bond funding, and promoting cross-agency collaboration and adoption of shared services. Public Records has the meaning given to it in M. G. L. c. 4, § 7, cl. 26, as materials made or received by a state agency, unless they fall within a specified exemption to the definition. Request for Quotes or RFQ means this RFQ 15-27. RFQ Response or Response means a Bidder’s response to this RFQ. Solution Vendor or Systems Integrator means any external party under contract with the Commonwealth to perform implementation services. Standard Contract Form means the Commonwealth Standard Contract Form. Statement of Work or SOW means the Statement of Work between MassIT and the Contractor for the performance of IV&V services entered into under this RFQ.
Page 4 of 64
Strategic Sourcing Team or SST means Commonwealth employees (and, at MassIT’s option, Commonwealth consultants and other individuals) who are responsible for conducting a comprehensive and impartial evaluation of the Bidders’ Responses to this RFQ and selecting the IV&V Contractor. Terms and Conditions or Ts and Cs means the Commonwealth Terms and Conditions. Additional definitions may be found in the Commonwealth’s Procurement Regulations, 801 CMR 21.00.
2.0 GENERAL PROCUREMENT INFORMATION Table 1 – General Procurement Information Procuring Agency Mailing Address Procurement Contact Person E-Mail RFQ File Number RFQ File Name/Title
MassIT One Ashburton Place, Eighth Floor, Boston, MA 02108
Candice Shoemaker
[email protected] MassIT RFQ 15-27 Independent Verification & Validation Services for the Benefits Realization of the Enterprise eLicensing System (EES)
3.0 PROJECT OVERVIEW Pursuant to this RFQ, the Commonwealth is soliciting a single qualified Contractor to enter into an SOW to provide IV&V services for the benefits realization of at least one, and potentially multiple, implementations of the EES. Each specific implementation will be memorialized in a task order entered into under the SOW. The task order for the ABCC/DPL Implementation must coincide with the task order approach specified in the Accenture SOW. The development and implementation of the EES, including IV&V services, will be funded from the proceeds of bonds issued by the Commonwealth pursuant to an IT bond authorization. Consequently, the eLicensing Project is sponsored by the IT Capital PMO. This sponsorship is based upon the premise that the Accela eLicensing product has the potential to be an Enterprise-wide system, capable of fitting the needs of any licensing agency or municipality within the Commonwealth. Because MassIT is providing the funds to pay for the services of the IV&V Contractor, this RFQ is posted by MassIT, and the parties to the IV&V SOW will be the IV&V Contractor and MassIT. Pursuant to the Accenture SOW, Accenture will implement Accela software that has previously been procured by MassIT for the purpose of creating the baseline of the EES. The objective of the ABCC/DPL Implementation is to deploy a new eLicensing system for ABCC and DPL, which are in the process of replacing their current applications and tools that manage license applications and renewals, consumer complaints, disciplinary actions, audits, and inspections. The current applications and tools utilize legacy technologies and do not seamlessly handle the business needs of the agencies. If the ABCC/DPL implementation is successful, the Commonwealth may extend the use of the Accela-based system to other Commonwealth agencies. The Commonwealth envisions that EES will be a secure, flexible, user-friendly system that will ultimately support many Commonwealth licensing functions and the various needs of Massachusetts licensing agencies, licensees, applicants for licenses, municipalities and the Commonwealth as a whole. For the ABCC/DPL Implementation the Contractor may bill MassIT an agreed-upon fixed price every fourteen (14) days. If MassIT subsequently engages the Contractor selected under this RFQ to provide IV&V services for additional Commonwealth agencies, each such engagement will be performed on a time Page 5 of 64
and materials basis, and a task order in the form of Attachment B hereto must be completed and executed by the Contractor for each engagement. Throughout the course of the ABCC/DPL Project, Accenture, as the ABCC/DPL System Integrator, is required to deliver three (3) separate releases - “A,” “B,” and “C,” which are all public-facing. Based on the current Project plan, Release A went live in March 2015. (For a description of the Releases, see Exhibit D.) Building on a series of successful releases, the Commonwealth envisions that the end-state of the Project will be the baseline for an enterprise level comprehensive eLicensing and ePermitting solution, using Accela COTS software.
4.0 ABCC and DPL OVERVIEW ABCC provides uniform control over the sale, purchase, transportation, manufacture, consumption and possession of alcoholic beverages in the Commonwealth. Each year ABCC issues approximately 13,000 retail and 10,000 state alcoholic beverage licenses. Retail establishments include all general on premises, restaurants, taverns, clubs, hotels, and stores. Retail license applications require approval from the licensing authorities of the community in which the retail establishment is located as well as from ABCC. On the other hand, ABCC is the sole agency responsible for approving and issuing state licenses. All manufacturers, wholesalers and importers, out-of-state suppliers, brokers, salespeople, warehouses, planes, trains, ships, ship chandlers and motor vehicles transporting alcoholic beverages for a commercial purpose into Massachusetts require an ABCC-issued state license or permit. DPL currently oversees and provides administration and support for thirty-one (31) licensing boards and the Office of Private Occupational School Education (the “Licensing Entities”). The Licensing Entities regulate more than 365,000 individuals and businesses as well as more than 200 private occupational schools, encompassing over fifty (50) trades and professions. DPL’s mission is to protect the public health, safety and welfare by licensing qualified individuals, businesses and private occupational schools and to provide services to consumers by fair and consistent enforcement of the licensing laws and regulations. DPL seeks to promote consumer protection and a fair and competitive marketplace through education and outreach.
5.0 PROCUREMENT CALENDAR The following Table indicates the dates of important events in the procurement process:
Table 2 –Procurement Calendar Date/Time 4/3/2015 4/13/2015 5PM
Event IV&V RFQ Release on COMMBUYS Deadline for the Submission of Questions under Bid Q&A
4/17/2015 Estimated
Answers to Bidder Questions Posted on COMMBUYS
4/30/2015 3:00 p.m.
Response Submission Deadline/Bid Opening Date
TBD 5/8/2015 Estimated 5/13/2015 Estimated
Oral Presentations by Bidders (if required) Notification of Apparent Successful Bidder Estimated Commencement Date of IV&V Services Estimated
All times indicated are prevailing Eastern time.
Page 6 of 64
The dates in the above Table may change and are not binding on MassIT. MassIT will post any change to the above Table on COMMBUYS. MassIT will reject any Responses not received by the Response Submission Deadline indicated in the above Table. Information regarding the Bid Q&A process is found in Section 8.2 of this RFQ, and the procedure for submitting a Response is found in Section 13 of this RFQ.
6.0 TERM The latest date on which a renewal or extension may be executed under this RFQ is June 30, 2019. The SOW will have an initial term of three (3) years from its Effective Date. However, MassIT reserves the right to extend the term of the IV&V Contract through multiple one- to three-year extensions through June 30, 2022, as the needs of MassIT require for the reasons indicated below. Any renewal or extension may have a term of no more than three (3) years and must expire no later than June 30, 2022. HED has executed an SOW with Accenture as well as Accenture Task Orders 1, 2, and 3. Accenture Task Order 4 is currently being re-evaluated by MassIT and Accenture. The Commonwealth expects that the IV&V Task Order 1 services will be needed throughout the remaining term of the Accenture SOW, including all changes to existing Accenture task orders and any remaining Accenture task orders. Therefore, MassIT anticipates that IV&V Task Order 1, which will address the ABCC/DPL Project, will terminate thirty (30) calendar days following the completion of the final Accenture Task Order outlined in the Acccenture SOW, which date may be less than three (3) years, or more than three (3) years, from the Effective Date. Therefore, MassIT will not be able to definitively determine the date of completion of IV&V Task Order 1 until after the final Accenture Task Order is completed. Consequently, MassIT may engage the winning Bidder under additional IV&V Task Orders in connection with Accela implementations at other agencies. In addition to considering extensions and renewals based upon the Commonwealth’s needs pertaining to the EES schedule, MassIT will also consider the overall performance of the IV&V Contractor and the availability of funding for the Contract.
7.0 BIDDER REQUIREMENTS Every Bidder must possess at least the following, as described in its Response: (a) the ability to evaluate various aspects of the eLicensing Program in order to achieve the purpose described in this RFQ; (b) demonstrable, successful experience in providing IV&V services for projects similar to the EES Program in size, scope and complexity; and (c) recent experience with the implementation of COTS solutions and systems integration. The proposed IV&V Project Manager should have achieved and currently hold Project Management Professional certification from the Project Management Institute or equivalent (i.e., project management degree or certification). The IV&V Project Manager is expected to utilize standard project management tools, including, but not limited to, Microsoft SharePoint, Project and Project Server. All proposed IV&V Project Team staff must have experience with the Bidder’s proposed IV&V methodology. The SST may award additional points to Bidders with sucessful experience implementing COTS solutions for the Commonwealth, implementing Accela software, or providing IV&V services relating to such an implementation. Since MassIT is currently undergoing an Agile transformation project, extra points may also be awarded to Responses that propose a Project Manager who also has Agile/Scrum experience. .
8.0 IMPORTANT INFORMATION FOR BIDDERS 8.1 General Page 7 of 64
This RFQ is issued pursuant to statewide contract ITS53,solution providers and technical specialists. Only authorized vendors with a contract under ITS53 are eligible to submit a Response. However, the following companies may not submit a Response: (1) Accenture or any of its affiliates; (2) Accela, Inc.; and (3) any vendor awarded an Accela implementation contract by MassIT under RFR 15-21, “Accela Integrators Master Contract,” COMMBUYS Bid No. BD-15-1060-ITD00-ITD00-00000002072. This RFQ and the successful Bidder’s Response will be incorporated into the IV&V SOW, in the form required by ITS53, including any modifications agreed upon by Mass IT. Unless otherwise provided, or unless clearly inconsistent with, or superseded by, this RFQ, the Successful Bidder/Contractor remains bound by the terms of ITS53 and the documents it executed in connection therewith, including, but not limited to, the Terms and Conditions and the Standard Contract Form. All goods and services offered by a Bidder under this RFQ are subject to the terms of ITS53, including, without limitation, any minimum or mandatory requirements of such contract. Bidders may not (manually or electronically) alter the RFQ language or any RFQ component files. Modifications to the body of the RFQ specifications, terms and conditions, or which change the intent of the RFQ, are prohibited and may disqualify the Bidder’s Response. Bidders must review all the Exhibits referred to in this RFQ in order to obtain background information on the ABCC/DPL Implementation. MassIT urges Bidders to thoroughly read and understand this material prior to preparing their Responses to this RFQ. Bidders must submit a Response for all of the services requested by this RFQ. The SST will disqualify any Response that indicates a Bidder (alone or with subcontractors or partnering companies) is unable or unwilling to provide all of the services requested. The IV&V Contractor must act as the prime contractor, but may partner or subcontract with other companies, with the exception of those companies listed in the first paragraph of this Section 8.1. The provision of all services to be provided by the IV&V Contractor will be set forth in the IV&V SOW, incorporating a task order approach. The task orders in the IV&V SOW must coincide with the task orders identified in the Accenture SOW, which is Exhibit D to this RFQ. The form of the IV&V SOW attached to this RFQ as Attachment A must be completed by the Bidder to the extent possible and submitted (unsigned) as part of its Business and Technical Response.
8.2 Bid Q&A Process The Bid Q&A provides the opportunity for Bidders to ask written questions and receive written answers from the SST regarding the RFQ. All Bidders’ questions must be submitted through the Bid Q&A found on COMMBUYS. To post a question on COMMBUYS, log into COMMBUYS, locate the Bid, acknowledge receipt of the Bid, and scroll down to the bottom of the Bid header page. The “Bid Q&A” button allows Bidders access to the Bid Q&A page. Questions must be asked prior to the Deadline for the Submission of Questions indicated in the Procurement Calendar. MassIT reserves the right not to answer questions submitted after this date, although questions submitted after the Deadline may be answered if the MassIT SST believes that providing an answer may improve the quality of the Responses. It is the Bidder’s responsibility to verify that the questions entered into the Bid Q&A are recorded in COMMBUYS and appear on the Bid Q&A page. Please note that any questions submitted to MassIT using any other medium (including those that are sent by mail, fax, email, or voicemail) will not be answered. To reduce the number of redundant or duplicate Page 8 of 64
questions, Bidders are asked to review all questions previously submitted to determine whether the Bidder’s question has already been posted. Bidders are responsible for entering content are suitable for public viewing, since all of the questions and answers are immediately accessible to the public. Bidders must not include any information that could be considered personal, security-sensitive, inflammatory, incorrect, collusory, or otherwise objectionable, including information about the Bidder’s company or other companies. The SST reserves the right to edit or delete any submitted questions that raise any of these issues or that are not in the best interests of the Commonwealth or this RFQ. All answers are final when posted. Any subsequent revisions to previously provided answers will be dated. It is the responsibility of the prospective Bidder and awarded Contractor to maintain an active registration in COMMBUYS and to keep current the email address of the Bidder’s contact person and prospective contract manager, if awarded a contract, and to monitor that email inbox for communications from MassIT, including requests for clarification. MassIT and the Commonwealth assume no responsibility if a prospective Bidder’s/awarded Contractor’s designated email address is not current, or if technical problems, including those with the prospective Bidder’s/awarded Contractor’s computer, network or internet service provider (ISP) cause email communications sent to/from the prospective Bidder or awarded Contractor and MassIT to be lost or rejected by any means, including email or spam filtering. All Bidders will be able to view all the Bidders’ questions and all the SST’s answers. Bidders are responsible for checking COMMBUYS for MassIT’s answers. MassIT’s SST reserves the right not to answer every question. Only written answers posted on COMMBUYS will bind MassIT.
8.3 Selection Process An SST comprised of representatives of MassIT and HED (and possibly ABCC, and/or other state agencies, consultants, and other individuals) is responsible for conducting an evaluation of the Responses to this RFQ in order to select the IV&V Contractor. Responses to the RFQ may be evaluated by any person(s) at the discretion of MassIT, including independent consultants retained by MassIT, now or in the future, for the sole purpose of evaluating Responses. See Section 14.0 for more information about the evaluation process.
8.4 IV&V Contractor Project Manager The IV&V Contractor Project Manager will report directly to Albert W. Mowatt, Director, IT Capital Portfolio Management Office, or his designee. In addition, the IV&V Contractor Project Manager will participate on the eLicensing Project Steering Committee as well as the Enterprise eLicensing Program Governance Board.
9.0 IV&V SERVICES REQUIRED 9.1 Purpose The purpose of the IV&V services is to ensure that:
the eLicensing Project requirements and specifications can achieve the Commonwealth’s vision and objectives, in accordance with the SOW entered into by the systems integrator with each agency implementing Accela; the goods and services provided by the systems integrator comply with the stated requirements and specifications;
Page 9 of 64
the resulting eLicensing implementation complies with the vision and objectives for an Enterprise eLicensing and ePermitting platform; and appropriate recommendations are made by the IV&V Contractor with respect to benefits realization and remediation of issues in the Project.
MassIT will contract with an IV&V Contractor to independently monitor and evaluate various aspects of the eLicensing Project from inception to completion, reporting whether both the Commonwealth and each systems integrator employ best practices, identify risks proactively and address them promptly, meet their contractual obligations on schedule and within budget, that Project deliverables meet quality standards,and that benefits realization planning and remediation of perceived issues have been addressed.
9.2 Critical Components As part of the Project, the IV&V Contractor will independently review the performance of the systems integrators throughout the Project lifecycle, with specific focus on the following critical components:
Project Organization and Management
Requirements Management
Cost and Schedule Management
Systems Architecture and Design
Human Resources Management
Risk Management
Issue Management and Escalation
Quality Management
Communications Management
Technology Architecture Definition
Operational Readiness
Benefits Realization
9.3 IV&V Phases and Tasks MassIT expects that the IV&V Contractor will structure its Response in accordance with the phases listed below. Furthermore, MassIT has provided recommended IV&V tasks (see Exhibit A - IV&V Tasks, Subtasks, and Artifacts) that are appropriate for this IV&V effort. However, the IV&V Contractor may refine these tasks in its Response to align with its IV&V methodology.
Initiation and Planning Phase: The primary objective is to develop a clear and mutually agreed upon understanding of the scope of work for the IV&V Project. The IV&V Contractor and Commonwealth staff will finalize and align activity timelines, resource requirements, Work Breakdown Structure and other IV&V Project management activities for the EES. o
Execution and Control Phase: The primary objective is to evaluate risks and probabilities of occurrence, develop potential mitigation strategies, and assess the overall impact on the Page 10 of 64
Project goals, scope, schedule, resources and cost. This Phase comprises the majority of IV&V tasks performed during the Project’s lifecycle. Tasks in this phase are organized into the EES Management Thread, the Change and Configuration Management Thread, the Business and Solution Alignment Thread, the Benefits Realization Thread, and the Architecture Verification Thread. o
EES Management Thread The IV&V Contractor monitors the EES management processes in this thread, to detect and address issues, increase management insight into product and process risk, and support software lifecycle process compliance with EES performance, schedule and budget requirements.
o
Change and Configuration Management Thread Tasks in this thread help monitor the EES change lifecycle, processes and supporting documentation. The IV&V Contractor will validate the adequacy of the overall support structure, will help reduce incidents of software defects via regular monitoring of product quality metrics and will help facilitate a stable software release management process.
o
Business and Solution Alignment Thread Tasks in this thread help validate application development artifacts for completeness, accuracy, traceability, consistency, readability and testability. The IV&V Contractor will verify whether the standards, practices and conventions used in solution development are satisfactory, and establish a basis for assessing completion of one development activity prior to starting others.
o
Benefits Realization Thread Tasks in this thread help determine whether the organization is prepared to realize the benefits of the IV&V effort and/or whether it has already done so. The IV&V Contractor will determine whether the organization has identified: (a) the organizational processes, systems, and resources needed to realize the benefits; (b) the changes required to these processes, systems, and resources in order to realize the benefits; and (c) how and when the benefits will be realized. Thereafter, the IV&V Contractor will determine whether such benefits have been realized. In determining whether the organization has realized benefits, the IV&V Contractor should consider the CASE tool data, the Project Charter, and the Investment Brief. See Exhibit O for a list of possible components of Benefits Realization.
o
Architecture Verification Thread Tasks in this thread help determine the architecture’s ability to deliver a system capable of fulfilling the quality requirements and to identify potential risks, This includes the fundamental organization of a system embodied in its components, their relationships to each other and to the environment, the principles governing its design and evolution as well as the structure of its components, their inter-relationships and connections to Commonwealth systems, and the principles and guidelines governing their design and evolution over time. See http://www.sei.cmu.edu/architecture/tools/evaluate/atam.cfm for information regarding a widely-used method for evaluating software architecture.
Closure Phase: This phase marks the completion of the IV&V Project. The primary objective is to estimate accurately the remaining effort required to close the EES for DPL and ABCC. The IV&V Contractor, along with the Commonwealth and Accenture, will monitor the EES schedule, budget and progress of any remaining/approved change requests required to complete the EES for DPL and ABCC. This combined team will document and estimate the effort required to complete any outstanding tasks, including the remediation of any issues identified by the IV&V Contractor.
9.4 Deliverables
Page 11 of 64
The Initiation and Planning Phase commences with the IV&V Contractor gaining a thorough understanding of the Accenture RFQ, its RFQ Response, the Accenture SOW, and the Accenture Task Orders. The IV&V Contractor will monitor the overall development of the EES by Accenture by attending meetings and reviewing documents, including all deliverables produced by the parties and suggesting opportunities for improvement to the process or relevant documents. The IV&V Contractor will provide a core team of professionals for a fixed period of time to support the EES activities described in this Section 9.0. The Commonwealth and the IV&V Contractor will determine the specific services, duration and timing during the Initiation and Planning Phase in concert with the Accenture work plan. The Commonwealth may request additional services or deliverables to support the EES (e.g., conducting detailed code walkthrough, detailed analysis of system design specifications for scope verification, etc.), and the services or deliverables would be mutually agreed upon through the change control process or the creation of a new task order. The IV&V Contractor shall deliver all written documents in machine-readable format, capable of being completely and accurately stored in an electronic format and reproduced by computer software on a laser printer. All itemized and/or annotated lists shall be delivered in computer documents or spreadsheets, capable of being imported to Microsoft Word or Microsoft Excel. All meetings between the Commonwealth and the IV&V Contractor will be held in at MassIT’s, HED’s, DPL’s or ABCC’s Boston offices, unless mutually agreed upon by both parties. Meetings will be scheduled in advance, with reasonable accommodation of attendees’ schedules. The meeting results will be described in follow-up reports generated by the IV&V Project Manager and approved by the MassIT Project Manager. At a minimum, the IV&V Contractor will develop a specific set of deliverables through the various phases of the IV&V Project as described in Table 3 below. The IV&V deliverables will include work products and outputs created from the tasks and activities described above, including the results of the reviews of Accenture’s deliverables. The work products, outputs and deliverables created each month will be consolidated and submitted as part of the IV&V Contractor’s Monthly Executive Summary Reports. The Monthly Executive Summary Reports will be utilized for reporting to: (a) Mr. Mowatt (or designee); (b) the MassIT Chief Applications Officer (or designee); (c) HED, ABCC and/or DPL. This information is required as part of the Commonwealth’s IT Bond funding compliance reporting set forth by the IT Capital PMO. Additionally, the IV&V Contractor will adhere to project management requirements set forth by MassIT in its CommonWay Project Management Methodology, which is based on the Project Management Institute’s Project Management Body of Knowledge. Additional information about CommonWay Project Management Methodology is available on MassIT’s CommonWiki at https://wiki.state.ma.us/confluence/display/Welcome/CommonWiki+Home.
Table 3 - Deliverables Phase
Initiation and Planning
Deliverable Type
Description
IV&V Project Plan
Microsoft Project Plan describing IV&V tasks, timelines, milestones and deliverables - The IV&V Project Plan will be refined as necessary to align with approved changes to the overall EES Project Plan.
Page 12 of 64
Weekly Status Reports
Execution and Control
Monthly Executive Summary Reports
Closure
Transition Plan
Weekly report including IV&V thread summaries, budget and schedule and implementation considerations - Thread components include: Achievements for Last Period Next Period Activities Milestones Issues/Risks Action Items For Major Issues, Recommended Remediations Open/Closed Issues Monthly “Executive Summary” dashboard using a “stoplight” format (red, yellow, green) indicators across all threads with additional detail for: Project Highlights Areas of Concern Milestones Issue Analysis Risk Analysis Scope Analysis Recommendations for Improvement Plan to transition processes, procedures and tools to the Commonwealth with any follow-up recommendations, including best practices, lessons learned and issues requiring remediation
10.0 CONTRACTOR AND CONTRACTOR EMPLOYEE REQUIREMENTS 10.1 Confidentiality and Security As part of its work effort, the Contractor will be required to use Commonwealth data and IT resources in order to fulfill part of its specified tasks. For purposes of this work effort, “Commonwealth Data” shall mean data provided by MassIT, HED, ABCC, or DPL (collectively, the “Agency”) to the Contractor, which may physically reside at a Commonwealth or Agency or Contractor location. In connection with such data, the Contractor will implement commercially reasonable safeguards necessary to:
prevent unauthorized access to Commonwealth Data from any public or private network; prevent unauthorized physical access to any IT resources involved in the development effort; and prevent interception and manipulation of data during transmission to and from any servers.
The Contractor will notify MassIT immediately if any breaches to the system occur. IV&V Contractor Employees who work on Commonwealth premises must comply with all written Commonwealth policies, including confidentiality and security, and acceptable use of Commonwealth telephones and computers.
10.2 Employee Requirements Before any Employee, including an Employee hired after the date of the Contract, may have access to Commonwealth Information, the IV&V Contractor must inform its Employees of the Commonwealth confidentiality and security requirements and the prohibitions relating to the disclosure of the Commonwealth Information. The link to the MassIT Workplace Policies, which should be reviewed by all Contractor Employees is: Page 13 of 64
http://www.mass.gov/itdemployee/human-resources/workplace-policies/. The HED Workplace Policies are attached to the SOW between Accenture and HED that is attached to this RFQ as Exhibit D. Additionally, the IV&V Contractor will prohibit access to the Commonwealth Information by any of its Employees, including an Employee hired after the date of the Contract, until the Employee has read and signed the IPA attached to the template Statement of Work attached to this RFQ as Attachment A. The IV&V Contractor must notify the MassIT Project Manager two (2) weeks prior to any changes in its Employees and provide a detailed transition plan to ensure there is no gap in quality service.
11.0 PRE-RESPONSE PROVISIONS 11.1 RFQ Release on COMMBUYS This RFQ has been distributed electronically on COMMBUYS. COMMBUYS is the official system of record for all procurement information and is publicly accessible at no charge at www.commbuys.com. Bidders are solely responsible for obtaining all information distributed for this RFQ via COMMBUYS. It is each Bidder’s responsibility to check COMMBUYS for any addenda or modifications to this RFQ. The Commonwealth accepts no responsibility and will provide no accommodation to Bidders who submit a Response based on an out-of-date RFQ or on information received from a source other than COMMBUYS.
11.2 Bidder Communication with the Commonwealth Bidders are prohibited from communicating directly or indirectly with any employee of MassIT, DPL, ABCC, HED, or any member of the SST, except as specified in the RFQ or in the ordinary course of business (e.g., in connection with the Bidder’s pre-existing matter involving that agency or with a pre-existing contract the Bidder may have with one of those agencies). No Commonwealth employee or representative is authorized to provide any information or respond to any question or inquiry regarding the RFQ except as part of the Bid Q&A procedure described above in this RFQ. Violation of this prohibition may be considered sufficient cause by the SST to disqualify a Response and/or an Apparent Successful Bidder. Notwithstanding the foregoing, if the Bidder is experiencing difficulty obtaining any required documents electronically through COMMBUYS, or if the Bidder has any issues with responding through COMMBUYS, it must contact the COMMBUYS Help Desk at
[email protected] or call during normal business hours (8AM – 5PM, Monday – Friday) at 1-888-627-8283 or 617-720-3197.
11.3 RFQ Amendments The SST may, at any time prior to the execution of the Contract, for any reason and without penalty, amend the RFQ or change the procurement requirements, performance specifications, budget or procurement schedule. Notice of an amendment will be posted on COMMBUYS. In such case, the SST will post a notice and an amendment to the RFQ on COMMBUYS. If the SST makes a significant change to the procurement, one that modifies the scope and requirements of the original procurement, the SST will cancel the procurement and begin a new procurement by posting a new RFQ on COMMBUYS. The SST will post a notice of the cancellation on COMMBUYS. It is the Bidder’s responsibility to check COMMBUYS for any notice or amendment, supplement or clarification to the RFQ. Only a written notice, amendment, supplement or clarification posted on COMMBUYS will bind the Commonwealth.
Page 14 of 64
11.4 Responses Public; Property of Commonwealth All Responses and related or attached information submitted in response to the RFQ are subject to the Massachusetts Public Records Law, M.G.L. c. 66, § 10 and c. 4, § 7, cl. 26. Any statements in Responses that are inconsistent with these statutes will be disregarded. Therefore, any documents submitted in response to the RFQ become public records and are subject to mandatory disclosure upon the request of third parties, after Contract execution. All Responses and other documents submitted in response to the RFQ become the property of the Commonwealth. The Commonwealth is under no obligation to return any documents submitted by a Bidder.
12.0 CONTENTS OF RESPONSE Each Response must include a Cover Letter, a Business and Technical Response and a Cost Response. Responses should present a straightforward description of the Bidder’s proposed services and should follow the outline described in this Section. Extraneous marketing or promotional materials are discouraged. Bidders must NOT include any costs in their Business and Technical Response. If any cost information is found in the Business and Technical Response, the entire Response will be disqualified. The Bidder’s Response must include three (3) separate sections: (1) the Cover Letter; (2) the Business and Technical Response; and (3) the Cost Response.
12.1 Cover Letter As part of its Business and Technical Response, the Bidder must provide a dated cover letter, with a signature by an authorized signatory of the Bidder, in which the Bidder states that: (a) it agrees to all the terms of this RFQ (mentioning this RFQ by name and number); and (b) it has no known conflicts of interest in responding to this RFQ, or, if it does have such a conflict, the nature of that conflict.
12.2 Business and Technical Response 12.2.1 Executive Summary As part of its Business and Technical Response, the Bidder must provide an executive summary, consisting of no more than five (5) pages, which briefly addresses: the corporate/business experience of the Bidder’s organization and how it relates to the Commonwealth’s expectations for the IV&V services; the Bidder’s understanding of the overall IV&V Project, goals, objectives and components; the Bidder’s understanding of the IV&V Project’s constraints and its approach to overcoming them; the Bidder’s understanding of the Project risks and the Bidder’s approach to managing them; a summary of the proposal that includes an overview of the approach the Bidder will take to complete each phase; and a statement that it meets all of the requirements listed in Section 7.0, if these requirements are not discussed elsewhere in the Response. Page 15 of 64
12.2.2 Bidder’s Approach As part of its Business and Technical Response, the Bidder must describe the approach that it will utilize when providing the services described in this RFQ.
12.2.3 Sample of IV&V Methodology/Framework As part of its Business and Technical Response, the Bidder must provide a sample framework for the IV&V Project based on similar projects completed by the Bidder. This framework should set forth the methodology and tasks by which the Bidder has delivered services substantially similar to those described in this RFQ to other clients.
12.2.4 Sample Reports As part of its Business and Technical Response, the Bidder must provide sample IV&V reports from similar projects completed by the Bidder. .
12.2.5 Other Elements As part of its Business and Technical Response, the Bidder must:
provide a sample IV&V Project Plan from another engagement of similar scope and size (redacted as needed). describe any automated tools that the IV&V Contractor will utilize in its review of deliverables and/or in coordination with the Commonwealth. Additionally, the Bidder must provide an example of an output (redacted as needed) from each proposed tool. provide a sample defect report, using an example from a prior engagement (redacted as appropriate). The Bidder must create a second sample defect report, using likely areas in the IV&V Project as examples of defects. provide a sample risk analysis report using timeliness and security of communications between systems as the topic. provide a sample Weekly Status Report. provide a sample Monthly Executive Summary Report. provide a sample Transition Plan, using an example from a prior engagement (redacted as appropriate). provide a ranked order listing of perceived concerns with the EES, based on the Accenture RFQ, Accenture‘s Response to that RFQ, and the SOW between Accenture and HED and the Bidder’s expertise.
12.2.6 IV&V Statement of Work As part of its Business and Technical Response, the Bidder must complete the draft of the ITS53 template Statement of Work that is attached to this RFQ as Attachment A, delineating the specific tasks, deliverables, and review and acceptance processes for each phase of every engagement entered under the SOW. The Apparent Successful Bidder will be given the opportunity to finalize the draft Statement of Work with MassIT, at which time the parties will more clearly define the deliverables, milestones and reporting structure for this project. The Bidder must also submit a Task Order 1 based on the task order template set forth at Attachment B. The Apparent Successful Bidder will be given the opportunity to finalize the draft Task Order for the purposes of creating IV&V Task Order 1. The same template will be used for all future Task Orders entered into by the winning Bidder under this RFQ. The SOW and Task Order the Bidder submits with its Response need not be signed by the Bidder.
Page 16 of 64
Except for descriptions of tasks and deliverables, the terms of the Statement of Work are standard and largely non-negotiable.
NO COSTS SHOULD BE INCLUDED IN THE DRAFT SOW; BIDDERS WHOSE SOW INCLUDES REFERENCES TO COST MAY BE DISQUALIFIED. 12.2.7 Digital Services Playbook Exhibit N to this RFQ is the CommonweaIth of Massachusetts Digital Services Playbook, which is derived from the U.S. Digital Services Playbook. The Playbook, which provides IT guidelines for projects, is divided into thirteen (13) “plays.” The winning Bidder must use this Playbook to inform its work under this RFQ. Every Bidder must indicate in its RFQ Response how it plans to use the Playbook in performing the IV&V services required by this RFQ.
12.2.8 Business/Commonwealth References The Bidder must provide references from at least two (2) organizations for which the Bidder has successfully completed a scope of services of similar scale to, or larger than, the scope of work contemplated by this RFQ. If the Bidder has provided these services for any Commonwealth agency, a reference from one such agency must be included as one (1) of the two (2) references. The suggested limit for each such reference is five (5) pages. MassIT reserves the right to contact some or all of the references provided by the Bidder as well as other organizations for which the Bidder has performed services similar to those required by this RFQ. References must include the following information:
reference organization’s name and address; name and title, address, telephone number, and e-mail address for a contact person; brief description of the contract requirements (including a description of the products and services offered, as appropriate); cost of the effort at contract signing and the final cost of the effort, with a brief explanation of any difference; and start and end dates for the effort, commencing with the signed contract and the actual start and end dates, with a brief explanation of any difference.
12.2.9 Organization and Staff Loading Charts The Bidder must a complete a Personnel Table in the form of Attachment E.
12.2.10 Personnel Resumes The Bidder must provide resumes for each of the IV&V Project team members proposed for this engagement. The suggested limit is four (4) pages per resume. The Bidder must describe all past working relationships between the proposed Project team members, their experience for this Project in relation to the Commonwealth’s expectations for IV&V services, and their availability. If any proposed Project team members are not employed by the Bidder, the Bidder must include a description of how these and any additional resources that may be needed will be acquired. The suggested limit is four (4) pages. The Bidder must identify any services that will be completed by a subcontractor of the Bidder, if any. Provide the name of such company, a description of its role on the IV&V Project and copies of resumes and references for any personnel that will work on the IV&V Project.
12.2.11 Commonwealth Resources Required Page 17 of 64
Note that the Commonwealth will provide the IV&V Contractor with such access to those facilities that will be required to perform its services under the Contract. HED will provide office space for the Contractor at 1000 Washington Street in Boston, Massachusetts. The Commonwealth will provide, for business use only, office furnishings and supplies, telephones, computers and peripheral devices, computer connections, copy machines, and network access and direct internet access as may be required for the IV&V Contractor to perform its work under the Contract. The Contractor’s Employees may use Commonwealth facilities, furnishings and supplies only for work to be performed under the Contract. As part of its Business and Technical Response, the Bidder must:
list any staff the IV&V Contractor will need from the Commonwealth, including the type of resource (subject matter experts, business analysts, developers, business and technical staff, etc.), the hours per week required per individual and the phase of the Project at which such staff member will be required; list any documentation required to be provided by the Commonwealth for IV&V Contractor review; list any hardware and software requirements (number of computers, Microsoft applications, system availability, etc.); and list any administrative items and equipment that the IV&V Contractor will need (cubicles, phones, building access requirements, etc.).
12.3 Cost Response Bidders must NOT include any costs in their Business and Technical Response. If any cost information is found in the Business and Technical Response, the entire Response will be disqualified. Costs that are not specifically identified in a Response, and accepted by the Commonwealth as part of the Contract, will not be compensated under the Contract. The Commonwealth will not be responsible for any costs or expenses incurred by Bidders in responding to the RFQ. The Commonwealth will not reimburse travel-related expenses such as airfare, meals, lodging or car rental. The Bidder must complete and submit as part of its Cost Response the Cost Tables found in Attachments C and D. Mass IT requires Bidders to provide all of the costs for the IV&V effort in these Attachments. The Commonwealth will not make any advance payments regardless of how they are classified (e.g., down payment, start-up costs, deposit, etc.). Unless adjusted pursuant to a change order, all prices are fixed as set forth in the Contract at the time of execution. Changes within scope should not be considered a chargeable change. The Commonwealth will not pay the IV&V Contractor’s cost for responding to a change request. Any Bidder that offers a prompt pay discount under ITS53 (as defined in Section 1 of this RFQ) is required to include the same or more favorable discounts in its Response for this IV&V Project. MassIT reserves the right to request a Best and Final Offer of any Bidder. MassIT is not required to request a BAFO from every or any Bidder. MassIT reserves the right to hire the Contractor, at the hourly rates included in its Response, for work closely related to the Project described in this RFQ, but not specifically described herein.
13.0 SUBMISSION OF RESPONSES
Page 18 of 64
Interested Bidders must submit one (1) electronic copy of their Response to the RFQ in Microsoft Word format via the appropriate Bid page on the Commonwealth’s procurement site, COMMBUYS. Documents should be editable (unlocked) to allow for any legal redlined review as necessary. Responses must be received no later than the Response Submission Deadline indicated in the Procurement Calendar or they will not be evaluated. All Responses submitted to this RFQ must be valid for a minimum of ninety (90) days after their submission through COMMBUYS.
NOTE: WHEN BIDDERS ARE UPLOADING QUOTES THEY MUST MAKE SURE THAT “NO BID” IS NOT ACTIVATED AND THEY MUST INCLUDE PRICING SO THAT “NO CHARGE” IS NOT ACTIVATED
Useful Links: Job aid on how to submit a quote: http://www.mass.gov/anf/docs/osd/commbuys/create-a-quote.pdf Webcast: How to Locate and Respond to a Bid in CommBuys, which will familiarize bidders with CommBuys terminology and basic navigation, and provide guidance for locating bid opportunities in CommBuys and submitting an online quote. If the Bidder has any issues with responding through COMMBUYS, it must contact the COMMBUYS Help Desk at
[email protected] or by calling the Help Desk during normal business hours (8AM – 5PM, Monday – Friday) at 1-888-627-8283 or 617-720-3197. All Responses must be received by the date and time specified in the Procurement Calendar. For Responses received on the due date, the SST determines the time a Response was received. All late Responses will be rejected. The Bidder must include in its Response all documents that the Commonwealth will be asked to agree to or sign. The Commonwealth shall not be bound by any document not included in the Bidder’s Response or any document that it has not executed.
14.0 EVALUATION OF RESPONSES 14.1 Evaluation Procedure The SST will conduct a comprehensive and impartial evaluation of the Responses received to this RFQ. The Commonwealth reserves the right to alter the composition of the SST or to designate other staff to assist in the evaluation process. The SST will evaluate each Bidder based on its Responses to the items in Section 12.0 and the results of any oral presentations by Bidders. The SST will select the Bidder whose Response, in aggregate, provides the “best value” for the Commonwealth. Cost will be among several factors in this consideration; however, the SST is not required to choose the Bidder that proposes the lowest cost. The SST reserves the right to request further written information or clarification from Bidders. Any such clarifications or written communications shall form part of the Bidder’s Response. The SST anticipates that it will complete its evaluation of responses and notify the Apparent Successful Bidder selected to perform IV&V services by the date specified in the Procurement Calendar.
Page 19 of 64
After an initial evaluation of Responses by the SST, Bidders may be required to conduct oral presentations. The presentations will occur at MassIT’s offices in Boston, Massachusetts and will last no more than two (2) hours. The Bidder’s Project Team members identified in their Responses are expected to attend the session. These sessions are not opportunities to submit new information or modify a Response. Rather, the purpose is to provide an opportunity for Bidders to highlight the strength and unique aspects of their Responses and to provide answers to questions from the Commonwealth regarding their Responses. The SST will not inform a Bidder of its scores or ranking at the time of the presentation. Bidders must provide copies of any presentation material (including an electronic version) at the beginning of the session. This material becomes part of the Bidder’s Response. The SST may limit the number of presentations conducted. Failure to agree to a presentation may result in the disqualification of a Bidder. The information the Commonwealth obtains from the presentations will be considered in the overall evaluation of the Responses.
14.2 Selection and Notice of Award The Apparent Successful Bidder will be the Bidder whose Response represents the best value for the Commonwealth. The SST will notify the Apparent Successful Bidder of its selection. No Bidder may make any public announcement concerning its selection for the Contract prior to the release of that information on COMMBUYS. Notice to an Apparent Successful Bidder that it may be awarded the Contract does not constitute acceptance of all the terms of a Response by the Commonwealth and does not impose any contractual obligation on the Commonwealth. Contract award is contingent upon the successful conclusion of negotiations and Contract execution with the Apparent Successful Bidder.
14.3 Evaluation Criteria The following identifies, in order of importance, the criteria by which the SST will evaluate the Bidder’s Response:
experience with IV&V engagements for software development projects of similar size, scope and complexity; experience with the Accela technology proposed for eLicensing; references; overall cost; and performance demonstrated during oral presentation (if any).
The SST may award additional points to Bidders with experience implementing COTS solutions for the Commonwealth, implementing Accela software, or providing IV&V services relating to such an implementation. Since MassIT is currently undergoing an Agile Transformation project, extra points may also be awarded to Responses that propose a Project Manager who also has Agile/Scrum experience. The SST may award additional points to Bidders with successful experience implementing COTS solution for the Commonwealth, implementing Accela software, providing IV&V services relating to such an implementation or providing a Project Manager with equivalent experience. If the Bidder has such experience, one reference must be from a Commonwealth agency for which it performed such services. Since MassIT is currently undergoing an Agile Transformation project, extra points may also be awarded to bids that include a Project Manager who also has Agile/Scrum experience. MassIT may negotiate changes with the Contractor(s) to the original performance measures, reporting requirements or payment methodologies tied to performance at any time during the Contract duration, if they Page 20 of 64
are consistent with the specifications of the RFQ. MassIT reserves the right to negotiate and execute Contract amendments with the Contractor(s) that MassIT determines as necessary to result in the intent of the RFQ, to amend the specifications for necessary requirements, or to result in a better value Contract. Amendments may include, but are not limited to, Contract dollars, Contract performance, increased or decreased obligations, scope of work, or quantity.
Page 21 of 64
List of Documents Referenced in the RFQ That Are Separately Uploaded to COMMBUYS and Posted With This Procurement ATTACHMENTS (required to be included in Response) Mandatory Submissions with Business and Technical Response: Attachment A - MassIT Template Statement of Work with Intellectual Property Agreement (IPA) attached thereto as Exhibit A Attachment B – Form of Task Order Attachment E - Personnel Table Mandatory Submissions with Cost Response ONLY: Attachment C - Cost Response - Costs for Each 14-Day Period - For Task Order 1 Only Attachment D – Cost Response - Hourly Rates
EXHIBITS Exhibit A - IV&V Tasks, Subtasks, and Artifacts Exhibit B - Accenture Project Plan Exhibit C - Accenture RFQ and RFQ Response - System Integration Service Exhibit D - Executed Statement of Work between Accenture and HED Exhibit E - Accenture Task Orders 1, 2, and 3 Exhibit F - Deliberately Left Blank Exhibit G - Current Weekly Project Status Report Exhibit H - Export of All Current Risks and Issues Exhibit I - Deliberately Left Blank Exhibit J - Recent “Go/No Go” Matrix/Decision Chart Exhibit K – Deliberately Left Blank Exhibit L - Defect List, Including the Priority and Severity of Defects Exhibit M - Current List of “Hot Fixes” Contemplated by the ESC Exhibit N – Commonwealth of Massachusetts Digital Services Playbook Exhibit O- Elements of Benefits Realization Page 22 of 64
ATTACHMENT A MassIT TEMPLATE STATEMENT OF WORK (with IPA attached as Exhibit A) STATEMENT OF WORK BETWEEN THE [NAME AGENCY] AND [NAME ITS53 IT PROJECT SERVICES VENDOR] FOR THE [NAME PROJECT]
1. INTRODUCTION The following document will serve as a statement of work (SOW) between the [Agency Name] ([Agency Abbreviation]) [of the [Name Parent Agency] and [Vendor Name] ([Vendor Abbreviation]) to apply to work on the [Name of System to be Developed] ([Abbreviated System Name]). Work effort under this SOW will be limited to service activities relating to the implementation of [Abbreviated System Name] in the Commonwealth of Massachusetts. The entire agreement between the parties (the “Agreement”) consists of the following documents in the following order of precedence: (1) the Commonwealth Terms and Conditions; (2) the Commonwealth’s Standard Contract Form; (3) RFR ITS53 – IT Project Services; (4) [Vendor Abbreviation]’s response thereto; (5) RFQ [name and date RFQ] and (6) [Vendor Abbreviation]’s response thereto and this SOW.
2. OVERVIEW, EFFECTIVE DATE AND TERM [Provide overview of project] This SOW shall become effective on the date on which it is executed by both parties and shall terminate on [termination date]. Notwithstanding the foregoing, sections [choose sections that will survive termination] shall survive the termination of the remainder of this SOW. The term of this agreement shall be ___ [months or days] commencing on ____ and ending on ____, 20__.
3. SINGLE POINT OF CONTACT [Vendor Abbreviation] and [Agency Abbreviation] will each assign a single point of contact with respect to this SOW. It is anticipated that the contact person will not change during the period the SOW is in force. In the event that a change is necessary, the party requesting the change will provide prompt written notice to the other. In the event a change occurs because of a non-emergency, two-week written notice is required. For a change resulting from an emergency, prompt notice is required. [Vendor Abbreviation]’s contact person is [Vendor Contact Name and Title], who can be reached at [Vendor Contact Address, phone number(s), eMail]. [Agency Abbreviation]’s contact is [Agency Contract Name and Title] who can be reached at [Agency Contact Address, phone number(s), email].
4. SYSTEM SECURITY As part of its work effort, [vendor abbreviation] will be required to use commonwealth data and it resources in order to fulfill part of its specified tasks. For purposes of this work effort, “commonwealth data” shall mean data provided by the [agency abbreviation] to [vendor abbreviation], which may physically reside at a commonwealth or [agency abbreviation] or [vendor abbreviation] location. In connection with such data, [vendor abbreviation] will implement commercially reasonable safeguards necessary to:
Page 23 of 64
Prevent unauthorized access to Commonwealth Data from any public or private network
Prevent unauthorized physical access to any information technology resources involved in the development effort and
Prevent interception and manipulation of data during transmission to and from any servers.
[Vendor Abbreviation] will notify the Commonwealth immediately if any breaches to the system occur.
5. ACCEPTANCE OR REJECTION PROCESS [Vendor Abbreviation] will submit the required deliverables specified in this SOW to the [Agency Abbreviation] Project Manager for approval and acceptance. [Agency Abbreviation] will review work product for each of the deliverables and evaluate whether each deliverable has clearly met in all material respects the criteria established in this agreement and the relevant Task Order specifications. Once reviewed and favorably evaluated, the deliverables will be deemed acceptable. Within ten (10) working days of receipt of each deliverable, the [Agency Abbreviation] Project Manager will notify [Vendor Abbreviation], in writing, of the acceptance or rejection of said deliverable using the acceptance criteria specified in this section. A form signed by [Agency Abbreviation] shall indicate acceptance. [Vendor Abbreviation] shall acknowledge receipt of acceptance forms in writing. Any rejection will include a written description of the defects of the deliverable. [Vendor Abbreviation] will, upon receipt of such rejection, act diligently to correct the specified defects and deliver an updated version of the deliverable to the Commonwealth. [Agency Abbreviation] will then have an additional 5 (five) business days from receipt of the updated deliverable to notify [Vendor Abbreviation], in writing, of the acceptance or rejection of the updated deliverable. Any such rejections will include a description of the way in which the updated deliverable fails to correct the previously reported deficiency. Failure of [Agency Abbreviation] to reject a deliverable within the above specified notification periods will constitute acceptance by the Commonwealth of said deliverable. Following any acceptance of a deliverable which requires additional work to be entirely compliant with the pertinent specifications, and until the next delivery, [Vendor Abbreviation] will use reasonable efforts to provide a prompt correction or workaround.
6. PROJECT MANAGEMENT 6.1 PROJECT MANAGERS 6.1.1 [Agency Abbreviation] Project Manager Project management of this engagement will be performed by [Agency Abbreviation]. The [Agency Abbreviation]’s project manager will:
Work closely with the [Vendor Abbreviation] Project Manager to ensure successful completion of the project.
Consult with the [Vendor Abbreviation] Project Manager to develop the Project Management Plan.
Review weekly status reports and schedule weekly meetings with [Vendor Abbreviation], as necessary.
Coordinate participation from [name other agencies and/or vendors] as required during the engagement.
Acquire [Agency Abbreviation] project team members as needed.
Coordinate [Agency Abbreviation]’s review of the deliverables and sign an acceptance form to signify acceptance for each accepted deliverable.
Page 24 of 64
[Agency Abbreviation]’s Project Manager reports to [name and title], who reports to [name and title][repeat this phrase until last named individual is agency head]. [Agency Project Manager Name, Agency Project Manager Title] will work directly with the [Vendor Abbreviation]’s Project Manager. [Name individual, with title] will sign this SOW and all amendments hereto on behalf of [Agency Abbreviation].
6.1.2 Vendor Project Manager The [Vendor Abbreviation] Project Manager will: Serve as an interface between the Commonwealth Project Manager and all [Vendor Abbreviation] personnel participating in this engagement.
Develop and maintain the Project Management Plan, in consultation with the [Agency Abbreviation] Project Manager.
Facilitate regular communication with the [Agency Abbreviation] Project Manager, including weekly status reports/updates, and review the project performance against the project plan.
Facilitate weekly project status meetings for the duration of the engagement.
Update the project plan on a weekly basis and distribute at weekly meetings for the duration of the engagement.
Sign acceptance forms to acknowledge their receipt from [Agency Abbreviation].
Be responsible for the management and deployment of [Vendor Abbreviation] personnel. [Vendor Abbreviation]’s Project Manager reports to _____, who reports to_____ [repeat until reaching engagement partner or equivalent]. [Name and title], being an authorized signatory named in the [Vendor Abbreviation]’s response to ITS53, will sign this SOW and all amendments thereto on behalf of [Vendor Abbreviation].
6.2 ISSUE RESOLUTION The project managers from each organization bear the primary responsibility for ensuring issue resolution. If they mutually agree that they are unable to resolve an issue, they are responsible for escalating the issue to [name and title of respective persons at agency and vendor].
6.3 CHANGES IN TASK ORDER SCOPE OF WORK The project managers from each organization bear the primary responsibility for ensuring issue resolution. If they mutually agree that they are unable to resolve an issue, they are responsible for escalating the issue to [name and title of respective persons at agency and vendor].
6.4 KEY PERSONNEL [Vendor Abbreviation] agrees to provide the following personnel for the following amounts of time for the duration of this project: TABLE 1 KEY PERSONNEL Staff Members
o Title
Time Commitment expressed as percentage of full time
Page 25 of 64
[Vendor Abbreviation] will assign all of the foregoing personnel to this engagement on the time basis set forth in Table 1. In the event that a change is necessary, [Vendor Project Manager] will provide prompt written notice to [Agency Project Manager] of the proposed change. If the personnel change is a result of a nonemergency, two-week written notice shall be provided by [Vendor Project Manager] to [Agency Project Manager]. For personnel changes that result from an emergency, prompt written notice shall be provided by [Vendor Project Manager]. [Agency Project Manager] has the right to accept or reject all personnel.
6.5 EQUIPMENT, WORK SPACE, OFFICE SUPPLIES [Agency Abbreviation] will provide [workspace, cubicles, standard office equipment, and standard network connectivity provided to state employees] for [Vendor Abbreviation] team members working on-site for activities defined in task orders, as required by this SOW. [Vendor Abbreviation] will submit a list of employees who will need access to the building and to state systems as required for execution of this SOW. Any [Vendor Abbreviation] employees who have access to IT resources must comply with the “Acceptable Use Policy” (see www.mass.gov/itd) or any alternative Acceptable Use Policy adopted by the [Agency abbreviation].
6.6 Related Project Knowledge In addition to the “Statewide Contract IT Specifications” and all other terms of ITS53, [Vendor Abbreviation] shall, prior to commencing any other work under this SOW, become familiar with the following documents: [here list any other material that the vendor must master in order to perform under the contract, such as prior studies, agreements, reports, etc.].
6.7 Intellectual Property Agreement for Vendor’s Employees, Consultants and Agents [Vendor Abbreviation] shall ensure that all [Vendor Abbreviation] personnel providing services under this SOW, regardless of whether they are [Vendor Abbreviation]’s employees, contractors, or agents, shall, prior to rendering any services under this SOW, sign the “Intellectual Property Agreement for Vendor’s Employees, Consultants and Agents,” which is included as one of the ITS53 documents, and return signed copies of the same to [Agency Abbreviation]’s Project Manager prior to the delivery of any services under this SOW.
7 ADDITIONAL TERMS 7.1 DEFINITIONS The terms used in this SOW, unless defined herein, shall have the meaning ascribed to them in the other documents that constitute the Agreement between the parties.
7.2 CODE REVIEW [Vendor Abbreviation] shall comply with any code review policy adopted by the Massachusetts Office of Information Technology (“MassIT”) (formerly the Massachusetts Office of Information Technology, which was known as “ITD”).
7.3 WARRANTY [Vendor Abbreviation] makes the following warranties with respect to any deliverables delivered under this SOW: (1) Vendor’s services shall be performed in a professional and workmanlike manner and in accordance with the specifications and description of services as set forth in the Agreement; (2) the deliverables will substantially conform with the deliverable descriptions set forth in this SOW; (3) all media on which [Vendor Abbreviation] provides any software under this Agreement shall be free from defects; and Page 26 of 64
(4) all software delivered by [Vendor Abbreviation] under this Agreement shall be free of Trojan horses, back doors, and other malicious code. 7.4 Title and Intellectual Property Rights [These terms will apply if the Contractor will be developing or modifying software. They are subject to negotiation. However, the approval of the General Counsel for MassIT is required for any changes to these terms.]
7.4.1 Definition of Property The intellectual property required by [Vendor Abbreviation] to develop, test, and install the [name product to be developed] (hereinafter the "Property") may consist of computer programs (in object and source code form), scripts, data, documentation, the audio, visual and audiovisual content related to the layout and graphic presentation of the [name product to be developed], text, photographs, video, pictures, animation, sound recordings, training materials, images, techniques, methods, algorithms, program images, text visible on the Internet, HTML code and images, illustrations, graphics, pages, storyboards, writings, drawings, sketches, models, samples, data, other technical or business information, and other works of authorship fixed in any tangible medium.
7.4.2 Source of Property The development of the [name product to be developed] will involve intellectual property derived from four different sources: (1) third party software contractors such as . . .[this provision may not apply to all contracts]; (2) that developed by [Vendor Abbreviation] for the open market (i.e. [Vendor Abbreviation]’s commercial off the shelf software); (3) that developed by [Vendor Abbreviation] for other individual clients, or for internal purposes prior to the Effective Date of this Statement of Work and not delivered to any other client of [Vendor Abbreviation]’s; and (4) developed by [Vendor Abbreviation] specifically for the purposes of fulfilling its obligations to [Agency Abbreviation] under the terms of this Statement of Work and all other documents listed in Section 1 of this Statement of Work. Ownership of the first and second categories of intellectual property is addressed in separate agreements between [Agency Abbreviation] and the contractors and resellers of such software. This section of the Statement of Work addresses exclusively ownership rights in the third and fourth categories of intellectual property.
7.4.3 Contractor Property and License [Vendor Abbreviation] will retain all right, title and interest in and to all Property developed by it, i) for clients other than the Commonwealth, and ii) for internal purposes and not yet delivered to any client, including all copyright, patent, trade secret, trademark and other intellectual property rights created by [Vendor Abbreviation] in connection with such work (hereinafter the "Contractor Property"). [Agency Abbreviation] acknowledges that its possession, installation or use of Contractor Property will not transfer to it any title to such property. [Agency Abbreviation] acknowledges that the Contractor Property contains or constitutes commercially valuable and proprietary trade secrets of the Contractor, the development of which involved the expenditure of substantial time and money and the use of skilled development experts. [Agency Abbreviation] acknowledges that the Contractor Property is being disclosed to [Agency Abbreviation] to be used only as expressly permitted under the terms of the license described in the associated Statement of Work. [Agency Abbreviation] will take no affirmative steps to disclose such information to third parties, and, if required to do so under the Commonwealth¹s Public Records Law, M.G.L. c. 66, § 10, or by legal process, will promptly notify [Vendor Abbreviation] of the imminent disclosure so that [Vendor Abbreviation] can take steps to defend itself against such disclosure. Except as expressly authorized in the associated Statement of Work, [Agency Abbreviation] will not copy, modify, distribute or transfer by any means, display, sublicense, rent, reverse engineer, decompile or disassemble the Contractor Property.
Page 27 of 64
[Vendor Abbreviation] grants to [Agency Abbreviation] a fully-paid, royalty-free, non-exclusive, nontransferable, worldwide, irrevocable, perpetual, assignable license to make, have made, use, reproduce, distribute, modify, publicly display, publicly perform, digitally perform, transmit and create derivative works based upon the Contractor Property, in any media now known or hereafter known, but only to the extent reasonably necessary for [Agency Abbreviation]’s exploitation of the deliverables to be developed. During the term of the associated Statement of Work and immediately upon any expiration or termination thereof for any reason, [Vendor Abbreviation] will provide to [Agency Abbreviation] the most current copies of any Contractor Property to which [Agency Abbreviation] has rights pursuant to the foregoing, including any related documentation. Notwithstanding anything contained herein to the contrary, and notwithstanding [Agency Abbreviation]’s use of the Contractor Property under the license created herein, [Vendor Abbreviation] shall have all the rights and incidents of ownership with respect to the Contractor Property, including the right to use such property for any purpose whatsoever and to grant licenses in the same to third parties.
7.4.4 Commonwealth Property In conformance with the Commonwealth’s Standard Terms and Conditions, on the date on which [Agency Abbreviation] reimburses [Vendor Abbreviation] for a deliverable accepted by [Agency Abbreviation] under the terms of this Statement of Work, all of [Vendor Abbreviation]’s right, title and interest in all Property developed by [Vendor Abbreviation] under the terms of this Statement of Work solely for purposes of creating the deliverables described in this Statement of Work shall pass to and vest in the Commonwealth, including all copyright, patent, trade secret, trademark and other intellectual property rights created by [Vendor Abbreviation] in connection with such work and any causes of action relating to or based upon such work (hereinafter the "Commonwealth Property"). [Vendor Abbreviation] hereby assigns to the Commonwealth, as of the date on which [Agency Abbreviation] reimburses [Vendor Abbreviation] for such deliverables, all intellectual property rights that it may now or hereafter possess in the Commonwealth Property related to such deliverable and all derivative works thereof. [Vendor Abbreviation] also agrees to execute all documents and take all actions that may be necessary to confirm such rights, including providing any code used exclusively to develop such deliverables for [Agency Abbreviation] and the documentation for such code. [Vendor Abbreviation] acknowledges that there are currently and that there may be future rights that the Commonwealth may otherwise become entitled to with respect to Commonwealth property that does not yet exist, as well as new uses, media, means and forms of exploitation, current or future technology yet to be developed, and that [Vendor Abbreviation] specifically intends the foregoing ownership or rights by the Commonwealth to include all such now known or unknown uses, media and forms of exploitation. With respect to web site development contracts, [Agency Abbreviation] will bear sole responsibility for registering the software or system domain name or URL, applying for any trademark registration relating to the software or system domain name or URL and applying for any copyright registration related to its copyright ownership with respect to any Commonwealth Property. [Vendor Abbreviation] agrees to take such actions as may be reasonably requested by [Agency Abbreviation] to evidence the transfer of ownership of or license to intellectual property rights described in this section.
7.4.5 Clearances [Vendor Abbreviation] will represent and warrant to [Agency Abbreviation] that it has obtained all rights, grants, assignments, conveyances, licenses, permissions and authorizations necessary or incidental to any materials owned by third parties supplied or specified by it for incorporation in the deliverables to be developed.
7.4.6 Third-party Intellectual Property If the deliverables contain or will contain any third-party intellectual property to which the Contractor intends to provide a sublicense, the Contractor must provide copies of all such sublicense agreements as early in the process as possible. The sublicense agreements must be included in the Contractor’s initial quotation to the Contracting Department, or, if the requirement to utilize sublicensed intellectual property is not known at the outset of the project, as soon as the requirement becomes known. Sublicenses to third-party intellectual property can ONLY be provided under ITS53 if they are provided at no charge to the Commonwealth.
Page 28 of 64
7.5 [AGENCY ABBREVIATION]’S RESPONSIBILITIES In addition to the tasks set forth in ”Equipment, Work Space, Office Supplies,” [Agency Abbreviation] shall be responsible for the following [insert any additional obligations that agency must fulfill; use this section sparingly; include responsibility for procuring hardware and commercial off the shelf software licenses or providing travel reimbursement.]
7.6 HIGH LEVEL PROJECT DESCRIPTION [Provide a high level description of the project, including design, coding, testing, and technical and user documentation responsibilities and costs.]
7.7 PROJECT MANAGEMENT [Address the vendor¹s practices and procedures for project planning, tracking, reporting and management, including the types, frequency and contents of reports that will be provided by the developer to the agency.]
7.8 SOFTWARE CONFIGURATION MANAGEMENT PROCEDURES [For projects involving software development or modification, address the physical location of software during development, and the contractor¹s procedures for controlling access and updates.]
7.9 Software Escrow [Address software escrow if applicable. If the Commonwealth will own the code, software escrow is not needed unless the code will be shared by multiple agencies.]
7.10 PERFORMANCE BOND [Agencies should weigh the use of Performance Bonds for larger projects.]
7.11 Additional CORI check requirements [Address any CORI check requirements which the agency may have over and above those specified in ITS53, including any requirement to run repeat CORI checks at specified time intervals during the engagement.]
7.12 MASSGIS WEB MAPPING SERVICES MassGIS is the Commonwealth’s Office of Geographic and Environmental Information. Its legislative mandate includes coordinating GIS activities in the Commonwealth’s public agencies and distributing GIS data. MassGIS has also developed and is the host for the Commonwealth’s e-government geospatial web mapping initiative. If [Vendor Abbreviation] will develop a capability for viewing maps and related information on an internet web site for [Agency Abbreviation], and if that web site will display map information available through MassGIS web mapping services (e.g., parcels, orthophotos, streets, wetlands), then [Vendor Abbreviation] is required to use MassGIS geospatial web mapping services. Waivers will be made if [Vendor Abbreviation] demonstrates to MassGIS’ satisfaction that using the MassGIS web mapping services for the proposed application is not reasonably practical due to one or more of the following concerns: 1. Performance of the application would be degraded due to using the MassGIS Services; 2. The proposed application requires reliability that exceeds those that MassGIS can reasonably be expected to provide; 3. The security requirements of the application preclude using the MassGIS services; 4. Cost.
Page 29 of 64
Waivers are not valid unless they are provided in writing by the Director or Assistant Director of MassGIS.
8
[VENDOR ABBREVIATION] TASKS AND DELIVERABLES
This section describes the tasks and deliverables that [Vendor Abbreviation] will provide to [Agency Abbreviation] and the tasks that [Vendor Abbreviation] will complete by the end of the engagement described in this SOW. Deliverables will be considered “complete” when all the acceptance criteria set forth in this SOW have been met or the prescribed review period for each deliverable or task has expired without written response from [Agency Abbreviation]. The task/deliverable numbers are referred to in subsequent sections throughout this SOW. All written documents shall be delivered in machine-readable format, capable of being completely and accurately reproduced by computer software on a laser printer. All itemized and/or annotated lists shall be delivered in computer spreadsheets, capable of being imported to Microsoft Excel 2000 [or name alternative desktop software used by agency]. All meetings shall be held in ______ unless agreed to otherwise by the Project Managers. Meetings must be scheduled at least three full business days in advance, with reasonable accommodation of attendees’ schedules. All meeting results will be described in a follow-up report generated by the [Vendor Abbreviation] project manager and approved by the [Agency Abbreviation] project manager
8.1 Deliverables and Tasks TABLE 2 Deliverables and Tasks Deliverable or Task 1.1
Description and Metrics of Acceptance [For each deliverable or task, describe deliverable and list metrics for acceptance]
1.2
Page 30 of 64
8.2 Milestones Table 3 below shows the dates on which tasks and deliverables set forth in Table 2 must be delivered to [Agency Abbreviation]. TABLE 3 Delivery Milestones Deliverable or Task Number
Deliverable Name
or
Task
Delivery Date
1.1 1.2
9
PAYMENT TERMS
[Vendor Abbreviation] agrees to invoice the Commonwealth for the deliverables or work completed per the requirements set forth in the appropriate Task Order. MassIT will make payments to [Vendor Abbreviation] only after receiving an accurate invoice for deliverables completed and accepted pursuant to Section VI of this SOW. Payments for specific tasks and deliverables shall be made in accordance with Table 4 below. Table 4 Deliverable Name Deliverable Number
or
or
Task
Milestone Payment
Task
1.1 1.2
Payments will be made in accordance with the Commonwealth's bill paying policy. A deliverable or task will be considered “completed” when [Agency Abbreviation] has determined that the acceptance criteria for that specific deliverable or task have been met as specified in Table 2 of this SOW.
10
KNOWLEDGE TRANSFER AND DOCUMENTATION REQUIREMENTS [Address requirements for knowledge transfer and documentation requirements here.]
11
MAINTENANCE [Address maintenance to be provided by vendor, if any, and cost thereof.]
Page 31 of 64
12
TRANSFER OF ENGAGEMENT PRODUCTS AT CONTRACT TERMINATION
[Address any special requirements for transfer of the application and/or other engagement products to the Commonwealth or to another vendor at Contract Termination.]
13
ACCESSIBILITY
This section applies only to Executive Department agencies.
13.1 Compliance with Standards [Vendor] shall ensure that all deliverables delivered under this agreement adhere to (1) the Section 508 Standards for Electronic and Information Technology Accessibility, 36 C.F.R. §1194, issued under Section 508 of the Rehabilitation Act of 1973, as amended (29 U.S.C. § 794(d)) (the “Section 508 Standards”), and (2) the Web Accessibility Standards, (the “MassIT Standards,” formerly the “ITD Standards”) issued by MassIT, available online at www.mass.gov/itd. For purposes of this Agreement, [Vendor’s] obligations pertaining to these standards shall be limited to those subsections thereof that have been certified by MassIT and the Massachusetts Office on Disability as objective and measurable. Such subsections shall be posted by MassIT at www.mass.gov/itd. The Section 508 and MassIT Standards may be modified from time to time, and Vendor is responsible for compliance with the most current version in effect on the date that [Vendor] executes this Agreement.
13.2 TRAINING [Vendor] shall coordinate with [Agency] in the identification of all prospective attendees at [Vendor] training who require accommodation, and shall cooperate with [Agency] in its provision of such accommodation. All technical and user documentation and any additional training material delivered by [Vendor] under this Agreement shall include alternative keyboard commands that may be substituted for mouse commands. Any documentation delivered under this Agreement and wholly owned by the [Agency] shall be in an agreedupon editable format.
13.3 AT/IT ENVIRONMENT LIST Attachment __ hereto sets forth a list of the specific assistive technology (AT) (including class, brand, and version) and specific desktop configuration against which [Vendor’s] deliverables will be tested under this Agreement (the “AT/IT Environment List”).
13.4 SOFTWARE DEVELOPED UNDER THE AGREEMENT Prior to commencing any design work under this Agreement, [Vendor’s] Project Manager and design professionals shall meet with [Agency]’s project manager to review the Section 508 and MassIT Standards, and the AT/IT Environment List, and to discuss their impact on the design process. [Vendor] shall test every software deliverable delivered under this Agreement, including the custom code created to customize commercial off the shelf software (COTS) (collectively, “Deliverables”), and any updates, new releases, versions, upgrades, improvements, bug fixes, patches or other modifications to the software (“Enhancements”) developed under this agreement, against Section 508 and MassIT Standards, and for interoperability with the AT and IT environment listed in the AT/IT Environment list. At the time each such Deliverable or Enhancement is delivered to [Agency], [Vendor] shall deliver to [Agency] and the MassIT Accessibility Laboratory (the “MassIT ATL”) the results of such testing. In addition, Vendor shall cooperate with the MassIT ATL, and any Accessibility Testing Vendor engaged by the MassIT ATL, or by [Agency] under the supervision of the MassIT ATL, in the performance of testing. The MassIT ATL, any Accessibility Testing Vendor engaged by the MassIT ATL, or by [Agency] under the supervision of the MassIT ATL, shall test each Deliverable or Enhancement against the Section 508 and Page 32 of 64
MassIT Standards, and for interoperability with the AT and the IT environment described in the AT/IT Environment List. The MassIT ATL shall certify such deliverables or Enhancements as compliant with the Section 508 and the MassIT Standards, and interoperable with the AT and environment described in the AT/IT Environment List. [Vendor] shall be responsible for curing each instance in which its deliverables fail to comply with the Section 508 or MassIT Standards. Vendor shall use best efforts to cooperate with [Agency], the MassIT ATL, and any pertinent AT vendor to correct any problems identified during such testing with the interoperability of the Deliverables or Enhancements with the AT and the IT environment specified in the AT/IT Environment List. [Vendor] shall provide a credit against amounts due by [Agency] under this agreement for all testing, including repeat accessibility testing required with respect to Deliverables or Enhancements that fail initial testing with respect to the Section 508 or MassIT Standards and are required by the MassIT ATL to be retested in that regard. Such credits shall not exceed 5% of either (1) the total fixed price due [Vendor] under this Agreement, or (2) the total not-to-exceed amount of this Agreement if entered under a time and materials basis. [Note: There is no functioning MassIT ATL as of the date of the posting of RFQ 15-27, and there is unlikely to be one at the time the SOW is executed.]
13.5 COTS AND ASP SOFTWARE [Vendor] shall conduct testing against the Section 508 and MassIT Standards, and for interoperability with the AT and IT environment listed in the AT/IT Environment list, on all COTS referenced in [Vendor’s] bid that must be acquired by [Agency] through another agreement (such as the Commonwealth’s statewide software reseller agreement) in order to implement the system to be delivered by [Vendor] under this Agreement, and all COTS (including for purposes of this section COTS configured by [Vendor]), or software to be provided by [Vendor] or its subcontractors in their capacity as application service providers (ASP), delivered under this agreement, and any Enhancements thereto or new versions thereof, prior to its delivery to [Agency] (collectively, COTS and ASP Software).Vendor shall deliver to both [Agency] and the MassIT ATL the results of such testing with each delivery of COTS or ASP Software. [Vendor] need not conduct such tests for COTS and ASP Software for which accessibility testing has already been conducted and test results have already been provided to the MassIT ATL. Instead, [Vendor] shall provide notice to [Agency] that such software has already been certified by the MassIT ATL. The notice shall include the name of the software or Enhancement, and the date the software was so certified. The MassIT ATL, or any Accessibility Testing Vendor engaged by the MassIT ATL, or by [Agency] under the supervision of the MassIT ATL, shall test such software for accessibility against the Section 508 Standards and the MassIT Standards, and for interoperability with the specific AT and the IT environment set forth in the AT/IT Environment List. The MassIT ATL shall certify as accessible all software so tested that complies with the Section 508 Standards and the MassIT Standards, and is interoperable with the AT and the environment specified in the AT/IT Environment List, and shall maintain a central web-based list of certified software for use by the Executive Department. [Vendor] shall be responsible for curing each instance in which its deliverables fail to comply with the Section 508 and MassIT Standards. Vendor shall use best efforts to cooperate with [Agency], the MassIT ATL, and any pertinent AT vendor to correct any problems identified during such testing with the interoperability of the Deliverables or Enhancements with the AT and the IT environment specified in the AT/IT Environment List. [Vendor] shall provide a credit against amounts due by [Agency] under this agreement for all testing, including repeat accessibility testing required with respect to Deliverables or Enhancements that fail initial testing with respect to the Section 508, MassIT Standards and are required by the MassIT ATL to be retested in that regard. Such credits shall not exceed 5% of the either the total fixed price due [Vendor] under this Agreement, or the total not-to-exceed amount of this Agreement if entered under a time and materials basis.
Page 33 of 64
[Vendor] shall not deliver COTS or ASP software under this Agreement that fails to meet such standards unless it has documented (1) that it has performed due diligence in seeking accessible alternative COTS or ASP Software, offering equivalent features and functionality to the inaccessible COTS or ASP Software, for which [Vendor] is or can readily become a licensed distributor; and (2) the cost of developing substitute accessible software under this Agreement. (Such documentation need not include reference to any specific competing COTS or ASP Software and its level of accessibility). COTS or ASP Software delivered under this Agreement or under another contract with a state agency in connection with a system delivered under this Agreement that does not meet the Section 508 Standards or the MassIT Standards shall be acceptable if either (1) the software vendor provides a roadmap for meeting such standards and interoperating with such AT or (2) the agency seeks and obtains a waiver from MassIT that it would be an undue hardship on the agency to eschew use of such COTS or ASP Software.
13.6 MAINTENANCE AGREEMENTS Any maintenance agreement entered by [Vendor] and [Agency] in connection with the system delivered under this Agreement shall require [Vendor] to cooperate with [Agency] in its efforts to resolve interoperability problems that arise during the term of the maintenance agreement related to the use of such system with specific AT in a specific IT environment. The undersigned hereby represent that they are duly authorized to execute this SOW on behalf of their respective organizations. [Agency Name]
[Agency Signatory and Title] Date
[Vendor Name]
[Vendor Signatory and Title] Date
Page 34 of 64
EXHIBIT A to SOW
Intellectual Property Agreement for Contractor’s Employees, Consultants and Agents
CONFIDENTIALITY, ASSIGNMENT OF INVENTIONS AND REPRESENTATION INFRINGEMENT AGREEMENT; OTHER REPRESENTATIONS
OF
NON-
The undersigned hereby acknowledges that he or she is an employee or consultant of the following contractor of the Commonwealth of Massachusetts:
Name of Contractor: ________________________ (“Contractor”)
and desires to be assigned by the Contractor to perform services for the Commonwealth, and that the Contractor desires to assign you to perform services on one or more projects for the Commonwealth, but only under the condition that you sign this Agreement and agree to be bound by all of its terms and conditions.
NOW THEREFORE, in consideration of your assignment to work for the Commonwealth, the access you have to the confidential information of the Commonwealth, and for other good and valuable consideration, the parties agree as follows: 1. Confidentiality of the Commonwealth’s Materials. You agree that both during your assignment at the Commonwealth and thereafter you will not use for your own benefit, divulge or disclose to anyone except to persons within the Commonwealth whose positions require them to know it, any information not already lawfully available to the public concerning the Commonwealth (“Confidential Information”), including but not limited to information regarding any web site of the Commonwealth, any ecommerce products or services, any web development strategy, any financial information or any information regarding users of or vendors to the Commonwealth’s web sites. Confidential Information also includes, without limitation, any technical data, design, pattern, formula, computer program, source code, object code, algorithm, subroutine, manual, product specification, or plan for a new, revised or existing product or web site; any business, marketing, financial or sales information; and the present or future plans of the Commonwealth with respect to the development of its web sites and web services. 2. All Developments the Property of the Commonwealth. All confidential, proprietary or other trade secret information and all other works of authorship, trademarks, trade names, discoveries, invention, processes, methods and improvements, conceived, developed, or otherwise made by you, alone or with others, and in any way relating to the Commonwealth or any of its web development projects, whether or not patentable or subject to copyright protection and whether or not reduced to tangible form or reduced to practice during the period of your assignment with the Commonwealth (“Developments”) shall be the sole property of the Contractor’s customer, the Commonwealth. To the maximum extent permitted by law, you hereby waive all moral rights in any Developments. You agree to disclose all Developments promptly, fully and in writing to the Commonwealth promptly after development of the same, and at any time upon request. You agree to, and hereby do assign to the Page 35 of 64
Commonwealth all your right, title and interest throughout the world in and to all Developments without any obligation on the part of the Commonwealth to pay royalties or any other consideration to you in respect of such Developments. You agree to assist the Contractors customer the Commonwealth, (without charge, but at no cost to you) to obtain and maintain for itself such rights. 3. Return of the Commonwealth’s Materials. At the time of the termination of your assignment with the Commonwealth, you agree to return to the Commonwealth all Commonwealth materials, documents and property, in your possession or control, including without limitation, all materials relating to work done while assigned by the Contractor to projects for Commonwealth or relating to the processes and materials of the Commonwealth. You also agree to return to the Commonwealth all materials concerning past, present and future or potential products and/or services of the Commonwealth. You also agree to return to the Commonwealth all materials provided by persons doing business with the Commonwealth and all teaching materials provided by the Commonwealth. 4. Representation of Non-Infringement. You hereby represent and warrant that, to your best knowledge, no software, no web content and no other intellectual property that you develop during your assignment to and deliver to the Commonwealth, and no Developments made by you and assigned to the Commonwealth pursuant to Section 2 above, shall infringe a patent, copyright, trade secret or other proprietary or intellectual property right of any third party. 5. No Conflicting Agreements. You represent and warrant that you are not a party to any agreement or arrangement which would constitute a conflict of interest with the obligations undertaken hereunder or would prevent you from carrying out your obligations hereunder. 6. Tax Payments. You hereby represent and warrant that you have paid all due state and federal taxes, or, if your tax status is in dispute or in the process of settlement, that you have responded as directed and within the required timeframes to all communications received from the state or federal government. 7. Not A Massachusette Employee. You acknowledge that you are not an employee of any Massachusetts state or municipal government agency, and are not entitled to any benefits, guarantees or other rights granted to state or municipal government agencies, including but not limited to group insurance, disability insurance, paid vacations, sick leave or other leave, retirements plans, health plans, or premium overtime pay. Should you be deemed to be entitled to receive any such benefits by operation of law or otherwise, you expressly waive any claim or entitlement to receiving such benefits from Massachusetts state or municipal government agencies. 8. Miscellaneous: (a) The Commonwealth is a third party beneficiary of this Agreement with full rights to enforce its terms directly. (b) This Agreement contains the entire agreement between the parties with respect to the subject matter hereof, superseding any previous oral or written agreements. (c) Your obligations under this Agreement shall survive the termination of your assignment with the Commonwealth regardless of the manner of or reasons for such termination. Your obligations under this Agreement shall be binding upon and shall inure to the benefits of the heirs, assigns, executors, administrators and representatives of the parties. (d) You agree that the terms of this Agreement are reasonable and properly required for the adequate protection of our customer the Commonwealth’s legitimate business interests. You agree that in the event that any of the provisions of this Agreement are determined by a court of competent jurisdiction to be contrary to any applicable statute, law, rule, or policy or for any reason unenforceable as written, then such court may modify any of such provisions so as to permit enforcement thereof to the maximum extent permissible as thus modified. Further, you agree that any finding by a court of competent jurisdiction that any provision of this Agreement is Page 36 of 64
contrary to any applicable stature, law, or policy or for any reason unenforceable as written shall have no effect upon any other provisions and all other provisions shall remain in full force and effect. (e) You agree that any breach of this Agreement will cause immediate and irreparable harm to the Contractor’s customer the Commonwealth not compensable by monetary damages and that the Commonwealth will be entitled to obtain injunctive relief, in addition to all other relief, in any court of competent jurisdiction, to enforce the terms of this Agreement, without having to prove or show any actual damage to the Commonwealth. (f) No failure to insist upon strict compliance with any of the terms, covenants, or conditions hereof, and no delay or omission in exercising any right under this Agreement, will operate as a waiver of such terms, covenants, conditions or rights. A waiver or consent given on any one occasion is effective only in that instance and will not be construed as a bar to or waiver of any right on any other occasion. (g) This Agreement shall be governed by and construed in accordance with the laws of the Commonwealth of Massachusetts, without regard to the doctrine of conflicts of law. This Agreement is executed under seal. The undersigned believes that this agreement imposes reasonable standards of conduct for all of the employees and the contractors on assignment at the Commonwealth, and that this agreement will serve to best protect the interests of all involved parties. If you agree with the terms set forth herein, please sign and return this Agreement. Agreed and Accepted: Name of Employee or Consultant:
_______________________________
Signature:
__________________________
Date: Name of Contractor:
__________________________ _______________________________
Signature:
__________________________
Name:
__________________________
Title:
__________________________
Date:
__________________________
Page 37 of 64
ATTACHMENT B FORM OF TASK ORDER
Task Order under the Statement of Work between MassIT (Client) and XXXXXXX (Contractor) for IV&V Services for the Enterprise eLicensing System) TABLE OF CONTENTS Page No. I.
Summary
II.
Term
III.
Description of Services
IV.
Key Documents
V.
Assumptions
VI.
Payment Schedule
VII.
Project Management
VIII. Facility Resources and Technology Resources
This Task Order is entered into as of [________], 2015__ (the “Effective Date”) under that certain Statement of Work between Client and Contractor dated as of [_______], 2015__ (the “SOW”), and describes certain Services under the SOW. Capitalized terms used herein but not defined shall have the meanings set forth in the SOW.
I.
Summary
This Task Order describes services performed by Contractor at an agreed-on price to be invoiced to Client every two weeks in relation to IV&V Services for the Enterprise eLicensing System implementation at [Name agency] based on the Accela solution.
II. Term The term of this Task Order begins on the Effective Date and ends on [__________].
III. Description of Services Page 38 of 64
The Services under this Task Order are as generally described in Exhibit [ ] to the SOW.
IV. Key Documents The following key documents related to the Accela implementation at [Name Agency] are attached hereto: Exhibit A - Updated [SI Vendor] Project Plan Exhibit B – [SI Vendor] RFQ and RFQ Response - System Integration Service Exhibit C – [SI Vendor] Executed Statement of Work between [SI Vendor] and the Commonwealth Exhibit D – [SI Vendor] Task Orders [X], [X] and [X]
V. Assumptions The following Assumptions apply to this Task Order: [
].
VI. Payment Schedule The price for this Task Order is $ ____ every fourteen (14) days. Client shall pay this amount in accordance with the payment schedule set forth in Exhibit [ __].
VII. Project Management A. Project Managers The Project Managers for Contractor and Client for this Task Order are set forth below: Contractor: [__________] Client: [__________] B. Key Personnel The Key Personnel for this Task Order are set forth below: [__________] C. Key Subcontractors The Key Subcontractors for this Task Order are set forth below: [__________] Note: If a Contractor has not previously provided information or documentation (e.g., resumes) with respect to the Project Managers, Key Personnel or Key Subcontractors identified in this Section VII., such information and documentation must accompany this Task Order. Page 39 of 64
VIII.
Facility Resources and Technology Resources
A. Client Provided Facility Resources
Client will provide the following Facility Resources to Contractor for the performance of the Services under this Task Order: [ ]. B.
Client Provided Technology Resources
Client will provide the following Technology Resources to Contractor for the performance of the Services under this Task Order: [ ]
Page 40 of 64
ATTACHMENT C COST RESPONSE - FOR TASK ORDER 1 ONLY TO BE INCLUDED IN THE COST RESPONSE ONLY ANY COST INFORMATION FOUND IN THE BUSINESS AND TECHNICAL RESPONSE WILL CAUSE THE ENTIRE BID TO BE DISQUALIFIED.
Cost Table Period Year 1
Fully-Loaded Cost for Each 14-Day Period $
Year 2
$
Year 3
$
Any Renewal or Extension Period
$
The first 14-day period will begin on the date on which the Contractor first performs billable services for MassIT. The last period may cover less than fourteen (14) days. For IV&V Task Orders following Task Order 1 (i.e., for any engagements under this RFQ following the ABCC/DPL Accela implementation), the winning Bidder will estimate its resource commitment based on the size, duration and complexity of the Project and the hourly rates submitted with its Cost Response and create a new Attachment D for that engagement.
Page 41 of 64
ATTACHMENT D COST RESPONSE – HOURLY RATES TO BE INCLUDED IN THE COST RESPONSE ONLY ANY COST INFORMATION FOUND IN THE BUSINESS AND TECHNICAL RESPONSE WILL CAUSE THE ENTIRE BID TO BE DISQUALIFIED.
Cost Table Project Team Member*
Functional Title*
Total Hours
Rate per Hour*
Total Rate
$ $ $ $ $ $ $
* Indicates Key Personnel. For Task Order 1 (the ABCC/DPL Implementation), the Bidder should complete the first, second, and fourth columns only. These hourly rates will determine the cost of services performed for MassIT that are within the scope of the RFQ, but are not specifically called for by the RFQ in connection with the ABCC/DPL implementation of Accela. In addition, such rates will be used as a basis for the winning Bidder’s cost of services for any task orders executed under this RFQ following IV&V Task Order 1 (for the ABCC/DPL Accela implementation). Insert additional lines for each proposed staff member.
Page 42 of 64
ATTACHMENT E – PERSONNEL TABLE Bidders must complete the personnel table below as part of their Business and Technical Response. Bidders must NOT include any costs in their Business Response. If any cost information is found in the Business Response, the entire Response will be disqualified. For each individual proposed to conduct IV&V activities on the Project, provide the individual’s name (if a specific person is proposed), functional title, job description, time commitment (as a percentage of full time) and duration on the Project. Identify “Key Personnel” by placing an asterisk (*) next to the individual’s name. Insert additional lines for each proposed staff member. Project Team Member
Functional Title
Job Description
Page 43 of 64
Time Commitment
Duration
EXHIBIT A – IV&V Tasks, Subtasks, and Artifacts Initiation and Planning Initiation and Planning Tasks
Subtasks
IV&V Project Contractual Requirements
•
Review Requirements
Establish the proposed IV&V team for the Project. Confirm Project objectives with Commonwealth. Work with Project sponsors to identify champions for the IV&V process. Finalize the IV&V Project plan as appropriate.
•
Validate Project objectives
•
Scope IV&V Effort
•
Finalize IV&V Contract
•
Institute change control board
Desired outcomes: •
IV&V team setup
•
Executive buy-in to the IV&V effort
•
Finalized IV&V scope and schedule
IV&V Project Concept Confirm the level of rigor to be applied to the IV&V effort for each of the requirements: analysis, design, development, test, deployment, operation and maintenance tasks. Identify constraints and potential risks. Identify IV&V deliverables and frequency of IV&V status reports.
•
Identify/Finalize roles and responsibilities
•
Finalize IV&V Traceability
•
Finalize IV&V Risks Matrix
•
Finalize IV&V Process and Procedures
•
Desired outcomes:
Document Project Vision/Strategic Direction
Artifacts Inputs •
IV&V RFQ and Bidder’s Response
•
Accenture RFQ and Accenture Response and SOW
•
Accenture project work plan
•
Commonwealth Meetings
Outputs •
Finalized Contract and/or Statement of Work
•
IV&V Project Work Plan
Inputs •
Concept Documentation
•
IV&V Project Work Plan
•
Meetings with Commonwealth
Outputs •
Updated IV&V Work plan reflecting milestones
•
IV&V Deliverable templates
•
Agreement on IV&V deliverables and frequency of status reports
•
Finalized IV&V processes and procedures
•
Specific mitigation strategies for potential risks
•
IV&V Issues/Risks Summary
•
Updated Communication Plan
IV&V Project Staff Allocation
•
Finalize Staffing plan
Socialize roles and responsibilities of the IV&V team. Review the IV&V effort and work plan against allocated resources.
•
Allocate staff per Work Plan
Desired outcomes: •
Adequately staffed IV&V Project
Inputs •
IV&V Project work plan
•
Accenture project plan and staffing matrix
•
Commonwealth and Accenture meetings
Outputs Page 44 of 64
Initiation and Planning Tasks team
Subtasks
IV&V Project environment
•
Establish the IV&V Project site, administration, and tools
Organize physical space
•
Create shared administration with MassIT
•
Configure Project software tools
Desired outcomes: •
Setup and access to required resources for the IV&V Project team
Artifacts •
Updated Project Organization Chart
•
Updated IV&V Work plan and Schedule
•
Updated IV&V staffing matrix
•
Periodic Status Report
Inputs •
IV&V Project Work Plan
•
Commonwealth and Accenture Meetings
Outputs •
Access to shared resources
•
Outlook distribution lists, MS Project calendars, etc.
Execution and Control Execution and Control – Verify and Validate Project Management Tasks Tasks Project Coordination Verify that stakeholders are appropriately addressed in the Project communication plan.
Subtasks •
Review Project Communication Plan
Inputs •
Commonwealth and Accenture Meetings
Outputs
Monitor communication between the IV&V Project, Accenture, and Commonwealth teams. Desired Outcomes: •
Artifacts
Effective ongoing communication between Commonwealth, Accenture, and IV&V Project Teams
Page 45 of 64
•
Updated IV&V Issues/Risks Summary
•
Periodic Status Report
Execution and Control – Verify and Validate Project Management Tasks Tasks Schedule Management Review planned IV&V Project schedule. Monitor actual schedules to plan through all phases relative to plan and regularly report deviations. Verify that requested scope changes are justified, measured and approved prior to implementation. For approved scope changes, verify the impact to delivery schedules. Regularly track IV&V Project progress and artifacts against schedule. Validate approved deviations from the IV&V Project plan with the Accenture and Commonwealth teams.
Subtasks •
Review Project schedules
•
Validate Project plan updates
•
Trace updates to changes in scope
•
Assess impact of scope changes to delivery schedule
•
Remediate potential slippage
Artifacts Inputs •
Accenture project plan and staffing matrix
•
Change requests
•
Commonwealth and Accenture Meetings
Outputs •
Updated IV&V Issues/Risks Summary
•
Periodic Status Report
Desired Outcomes: •
Timely completion of all IV&V Project activities
Cost Management Review planned IV&V Project budgets and costs. Review approved Project scope change requests. Analyze and report impacts to overall Project costs. Monitor and report any unplanned cost overruns.
•
Review IV&V Project costs
•
Validate IV&V Project budgets and overruns
Desired Outcomes: •
Validate the Project staffing matrix against Accenture’s project work plan. Verify that adequate resources are available to execute and support all phases of the Project.
•
Accenture budgeting, costing procedures and documentation
•
Change requests
•
Commonwealth and Accenture Meetings
Outputs
Project completion on or within budget
Resource Management
Inputs
•
Review resource allocation
•
Trace resource allocation to Accenture schedule and plan
•
Updated IV&V Issues/Risks summary
•
Periodic IV&V Project status report
Inputs •
Accenture project plan and staffing matrix
•
Change requests
•
Commonwealth and Accenture Meetings
Desired Outcomes: •
Outputs
Project completion on or within budget
Page 46 of 64
•
Preliminary Resource Assessment
•
Updated IV&V Issues/Risks summary
•
Periodic IV&V Project status report
Execution and Control – Verify and Validate Project Management Tasks Tasks Risk Management Identify and prioritize areas of potential risk through all phases of the IV&V Project lifecycle. Monitor issues, log for unresolved issues that might become risks. Develop and validate mitigation strategies with Project stakeholders.
Subtasks •
Identify and prioritize identified risks
•
Develop/validate mitigation strategies
Desired Outcomes: •
Identify and prioritize IV&V Project issues and develop and/or validate responses and strategies to resolve these issues. Maintain a project issues log with information including (but not limited to): •
Issue Definition
•
Issue Attributes (Status, Open Date, Close Date)
•
Issue Ownership (Opened By, Assigned To, Approved By, Closed By)
•
Issue Impact and Estimates
•
Issue Resolution
•
Assess issue collection and logging
•
Assess and prioritize identified issues
•
Track issue resolutions to closure
Desired Outcomes: •
Inputs •
Risk assessment Matrix (Identification, probability and mitigation strategies)
•
Issue tracking document
•
Commonwealth and Accenture Meetings
Outputs
Effective ongoing identification, analysis, quantification, contingency planning, and response to risks
Issue Management
Artifacts
Effective ongoing identification and verification of issue resolutions
Page 47 of 64
•
Updated IV&V Issues/Risks Summary
•
Periodic Project Status Report
Inputs •
Project Issue tracking documentation, process, and tools
Outputs •
Updated IV&V Issues/Risks Summary
•
Periodic Project status report
Execution and Control – Verify and Validate Project Management Tasks Tasks Deliverable Quality Review Verify that IV&V Project deliverables meet the intended purpose, have appropriate detail and conform to established standards. Verify that issues and recommendations regarding missing or inadequate deliverables are addressed in a timely manner so as not to impact Project delivery.
Subtasks •
•
Assess Accenture’s approach to manage deliverable quality Validate Accenture deliverables
Artifacts Inputs •
Deliverables produced for each development milestone
•
Existing Quality Standards documentation
Outputs •
Deliverable Quality Assessment
•
Updated IV&V Issues/Risks Summary
•
Periodic Project status report
Desired Outcomes: •
Ongoing verification that deliverables meet appropriate standards and acceptance criteria
Execution and Control – Verify and Validate Change and Configuration Management Tasks Tasks Change Control Validate IV&V Project change control standards, policies and procedures. Monitor change impacts on Project schedule. Validate coordination of Project activities associated with a change request
Subtasks •
Review Change Impact Analysis
•
Review Change Management Plan
•
Review Communications Plan
Desired Outcomes: Ongoing impact assessment of Project changes to delivery schedules
•
Prevention of unauthorized changes and scope creep
Inputs •
Accenture’s Change Management Plan
•
Accenture’s Communication Plan
Outputs •
Assess change control process and verify that established change control procedures help reduce or eliminate scope creep. •
Artifacts
Page 48 of 64
Periodic Project Status report
Execution and Control – Verify and Validate Change and Configuration Management Tasks Tasks Governance Validate that adequate staff are in place to identify, collect and address IV&V Project changes.
Subtasks •
•
Verify the change request and tracking process for completeness and consistency.
Validate enhancements collection process Validate version/release control impact process
Review established software quality assurance (QA) metrics. Validate that software quality is maintained through the IV&V Project lifecycle and reported consistently with established metrics.
Change requests (Help desk, regulatory drivers, business analysis, etc.)
•
Change prioritization process
•
•
Validate Accenture’s QA Metrics criteria
•
Audit solution configuration
Periodic Project Status Report
Inputs •
Accenture’s QA metrics documentation
•
Established Baseline metrics
Outputs •
Desired Outcomes:
Periodic Project Status Report
Software quality effectiveness based on established metrics
Software Build Management Review the established source code management processes and tools. Review source code and software build versioning and management procedures. Validate that source code and build management procedures are applied to all development environments across all phases of the development lifecycle.
•
•
•
Validate Accenture’s source code management process Validate Accenture’s software build management processes/ procedures Analyze issues and risks
Review the construction iteration plan to verify that the build label for each development environment (e.g., development, production) in the build management tool matches the build label deployed to that environment. Verify that the correct source code label is used to build executable code. Desired outcomes: •
•
Ongoing impact assessment of Project changes to delivery schedules
Project Metrics Reporting
•
Inputs
Outputs
Desired Outcomes: •
Artifacts
Reduced versioning errors in the software release and build management process
Page 49 of 64
Inputs •
Accenture version control and release documentation and tool
•
Construction Iteration Plan
Outputs •
Updated IV&V Issues/Risks Summary
•
Periodic Project Status report
Execution and Control – Verify and Validate Change and Configuration Management Tasks Tasks Training and Support Management Review support process. Verify that a support issue (e.g. helpdesk ticket, incident record) is tracked to closure and escalated (as applicable) consistent with the support process. Validate that adequate user documentation and manuals are in place to enable users to understand available functionality. Verify that user documentation is updated when new functionality is delivered to users. Review training program and plans. Monitor execution of training plans and validate that training materials are developed and delivered consistent with the plan.
Subtasks •
Validate Training plan and artifacts
•
Validate Knowledge Transfer plan
•
Validate Helpdesk Artifacts
•
Validate User Documentation/ Manuals
•
Validate Helpdesk procedures
•
Validate Escalation procedures
Artifacts Inputs •
Training and Helpdesk Artifacts
•
User Documentation/ Manuals
•
Helpdesk procedures
•
User support and escalation procedures
Outputs •
Updated IV&V Issues/Risks Summary
•
Periodic Project Status Report
Desired outcomes: •
Adequacy of ongoing support, training, and knowledge transfer through the IV&V Project lifecycle
Execution and Control – Verify and Validate Business and Solution Alignment Tasks Requirements Analysis and Design Review requirements artifacts. Trace software requirements to functional requirements and functional requirements to software requirements. Verify that the identified requirements address business objectives. Validate that the detailed system design satisfies each software requirement. Verify that the solution design adheres to established software design standards. Validate that test plans address design elements.
Subtasks •
Review software Requirements
•
Review External Interfaces
•
Validate Infrastructure Adequacy for technical requirements
•
•
Trace design to requirements
•
Validate and Verify Test Plans to requirements and known risks being tracked
Validate that the proposed solution infrastructure adequately satisfies the non-functional or technical requirements. Identify any anomalies as issues and review with Accenture and Commonwealth teams to address
Review software Design and Architecture
•
Analyze Issues and Risks
Page 50 of 64
Artifacts Inputs •
Software and Interface Requirements Specification (SRS)
•
Business/Functional Requirements
•
Detailed System and Interface Design (DSD)
•
Inventory of all Interacting Systems
•
Design Standards
•
Test Plans
•
Configuration Management Plan and Process
•
Commonwealth and Accenture Meetings
Outputs •
Updated IV&V Issues/Risks Summary
Execution and Control – Verify and Validate Business and Solution Alignment Tasks them. Identify any risks to the completeness, correctness, consistency, and accuracy of artifacts and their mitigation strategies.
Subtasks
Artifacts •
Periodic status report
Desired outcomes: •
Ongoing forward and reverse traceability between requirements, design, test and deployment activities through the software development lifecycle
Construction/Implementation
•
Review source code
Review source code documentation and standards. Inspect program source code elements. Trace the source code unit to the design specification for that unit.
•
Trace source code to design specification
•
Detailed System and Interface Design (DSD)
•
Trace design specification to the requirement
•
Source Code and Documentation
•
Coding Standards
•
Component test plans, procedures and results
•
Commonwealth and Accenture Meetings
Verify that source code is adequately documented and consistent with documentation standards. Develop IV&V test cases for •
Component testing
•
Integration testing
•
System testing
•
Acceptance testing
Perform IV&V component testing. Analyze test results to validate that software correctly implements design. Verify that test results trace to test criteria established in the test planning documents. Use the IV&V component test results to validate that the software satisfies the IV&V test acceptance criteria. Document discrepancies between actual and expected test results.
•
Evaluate source code Documentation
•
Verify test case and test procedure generation
•
Perform IV&V Component Tests
•
Validate expected and actual results Analyze Issues and Risks
Identify any anomalies as issues and review with the Accenture and Commonwealth teams to address them. Identify any risks to the completeness, correctness, consistency, and accuracy of artifacts and their mitigation strategies. Desired outcomes: •
Ongoing forward and reverse traceability between requirements, design, test and deployment Page 51 of 64
Inputs
Outputs •
Updated IV&V Issues/Risks Summary
Periodic status report Test Results •
Execution and Control – Verify and Validate Business and Solution Alignment Tasks activities through the software development lifecycle Test Analyze relationships in the IV&V test plans, cases and procedures. Verify that all IV&V test procedures are traceable to IV&V test plans. Develop IV&V Acceptance Test procedures. For IV&V system tests, integration tests, and acceptance tests: perform testing, based on procedures developed during implementation. analyze test results to verify that the software components are integrated correctly.
Subtasks
Artifacts
•
Trace testing effort to implementation
•
Verify Acceptance Test Procedures
•
Verify Integration Test Execution
•
Verify System Test Execution
•
Verify Acceptance Test Execution
Analyze Issues and Risks
Verify that test results trace to test criteria established in the test planning documents. Document discrepancies between actual and expected test results.
Inputs •
Test Plans, Designs/Cases and procedures
•
Source and Executable Code
•
User Documentation
•
Test Results
•
Commonwealth and Accenture Meetings
Outputs •
Updated IV&V Issues/Risks Summary
•
Periodic status report
•
Test Results
•
Go/No-Go recommendation
Identify any anomalies as issues and review with the Accenture and Commonwealth teams to address them. Identify any risks to the completeness, correctness, consistency, and accuracy of artifacts and their mitigation strategies. Desired outcomes: •
Ongoing forward and reverse traceability between requirements, design, test and deployment activities through the software development lifecycle
Deployment
•
Verify that installed software corresponds to the software subjected to the IV&V procedures.
Validate Pilot and/or cutover plans
•
Audit Installation Configuration
•
Trace deployment to testing
Identify any anomalies as issues and review with the Accenture and Commonwealth teams to address them. Identify any risks to the completeness, correctness, consistency, and accuracy of artifacts and their mitigation strategies.
•
Installation Package
•
IV&V test results
•
Accenture pilot or cutover plans
•
Commonwealth and Accenture Meetings
Analyze Issues and Risks
Desired outcomes: •
Inputs
Ongoing forward and reverse traceability between requirements, Page 52 of 64
Outputs •
Updated IV&V Issues/Risks Summary
•
Periodic Status and Feedback
Execution and Control – Verify and Validate Business and Solution Alignment Tasks design, test and deployment activities through the software development lifecycle Operation and Maintenance Verify that operating procedures are consistent with user documentation and conform to system requirements. Assess software modifications for impacts to the deployed solution. Assess proposed changes (modifications, enhancements or additions) to determine the effect of changes to the deployed solution. Determine the extent to which IV&V tasks would be iterated.
Subtasks
Artifacts
•
Review Operating Plan and Procedures
•
Assess Migration and Retirement
•
Evaluate software transition Plans
•
Trace operations and maintenance to deployment
Analyze Issues and Risks
Inputs •
Operating Procedures and Problem Reports
•
Operations Manuals
•
Installation Package
•
Commonwealth and Accenture Meetings
Outputs •
Updated IV&V Issues/Risks Summary
•
Periodic Status and Feedback
Assess whether software implementation addresses specific migration requirements, migration tools, conversion of software products or data, and software archiving. Assess whether software installation package addresses software support, impact on existing systems and databases, and transition to a new software product. Identify any anomalies as issues and review with the Accenture and Commonwealth teams to address them. Identify any risks to the completeness, correctness, consistency, and accuracy of artifacts and their mitigation strategies. Desired outcomes: •
Effective transition to a stable operating environment.
Closure IV&V Closure Tasks Tasks
Subtasks
IV&V Project Transition Planning and Execution
•
Finalize Transition Plan
Establish and review criteria for
•
Implement Transition Page 53 of 64
Artifacts Inputs •
IV&V Work plan
•
Commonwealth and Accenture
IV&V Closure Tasks Tasks transition of the IV&V effort. Develop IV&V transition plan. Determine how IV&V procedures will be completed to closure. Identify transition team. Execute the IV&V transition plan.
Subtasks Plan
Artifacts Meetings Outputs •
IV&V Transition Plan
Desired outcomes: •
Practical schedule to execute IV&V closure/transition
Final Deliverable Review Review acceptance criteria for outstanding issues. Confirm issue resolution strategies, dispositions and schedule with Project stakeholders. Update IV&V transition plan as appropriate.
•
Review Outstanding Issues
•
Validation of completeness of IV&V Deliverables
Desired outcomes:
IV&V Deliverable List
•
IV&V Work Plan
•
Commonwealth and Accenture Meetings
•
Updated IV&V Deliverable List reflecting all completed deliverables
•
Updated IV&V Work Plan
•
Updated IV&V Transition Plan for outstanding Issues
Completion of IV&V deliverable reviews
IV&V Project Contract Closure
•
Close IV&V Contract
Review IV&V contractual obligations with Commonwealth to verify completion of ongoing services. Communicate closure of IV&V contract to stakeholders and close IV&V Contract.
Inputs •
IV&V Contract
•
IV&V Work Plan
•
Commonwealth and Accenture Meetings
Outputs
Desired outcomes: •
•
Outputs
Verify that all IV&V deliverables have been completed and reviewed by Commonwealth. Obtain Commonwealth approvals as appropriate. •
Inputs
•
Completion of IV&V transition activities
Page 54 of 64
Closed Contract
The following Exhibits are uploaded separately EXHIBIT B ACCENTURE PROJECT PLAN EXHIBIT C ACCENTURE RFQ AND RFQ RESPONSE - SYSTEM INTEGRATION SERVICE EXHIBIT D EXECUTED STATEMENT OF WORK BETWEEN ACCENTURE AND HED EXHIBIT E ACCENTURE TASK ORDERS 1, 2, AND 3 EXHIBIT F Deliberately Left Blank EXHIBIT G CURRENT WEEKLY PROJECT STATUS REPORT EXHIBIT H EXPORT OF ALL CURRENT RISKS AND ISSUES EXHIBIT I Deliberately Left Blank EXHIBIT J RECENT “GO/NO GO’’ MATRIX DECISION CHART EXHIBIT K Deliberately Left Blank EXHIBIT L DEFECT LIST, INCLUDING THE PRIORITY AND SEVERITY OF DEFECTS EXHIBIT M CURRENT LIST OF “HOT FIXES” CONTEMPLATED BY THE ESC
Page 55 of 64
EXHIBIT N COMMONWEALTH OF MASSACHUSETTS DIGITAL SERVICES PLAYBOOK This Exhibit is derived from the U.S. Digital Services Playbook, http://playbook.cio.gov. It provides guidelines for IT projects divided into thirteen (13) “plays.” The winning Bidder must use this Playbook to inform its work under this RFQ. Every Bidder must indicate in its RFQ Response how it plans to use the Playbook in performing the IV&V services required by this RFQ.
1 Understand what people need Digital projects begin by exploring and pinpointing the needs of the people who will use the service, and the ways the service will fit into their lives. Whether the users are members of the public or government employees, policy makers must include real people in their design process from the beginning. The needs of people — not constraints of government structures or silos — should inform technical and design decisions. Products need to be continually tested with real people to focus on what is important. checklist 1. Early in the project, spend time with current and prospective users of the service 2. Use a range of qualitative and quantitative research methods to determine people’s goals, needs, and behaviors; be thoughtful about the time spent 3. Test prototypes of solutions with real people, in the field if possible 4. Document the findings about user goals, needs, behaviors, and preferences 5. Share findings with the team and agency leadership 6. Create a prioritized list of tasks the user is trying to accomplish, also known as "user stories" 7. As the digital service is being built, regularly test it with potential users to ensure it meets people’s needs key questions Can the project identify and articulate the primary users? Can the project identify and articulate what user needs will this service address? Can the project identify and articulate why does the user want or need this service? Can the project identify and articulate which people will have the most difficulty with the service? Can the project identify and articulate which research methods were used? Can the project identify and articulate what were the key findings? Can the project identify and articulate how were the findings documented? Where can future team members access the documentation? Can the project identify and articulate how often are you testing with real people?
2 Address the whole experience, from start to finish The Commonwealth needs to understand the different ways people will interact with our services, including the actions they take online, through a mobile application, on a phone, or in person. Every encounter — whether it's online or offline — should move the user closer towards their goal. checklist Page 56 of 64
1. Can the project identify and articulate the different points at which people will interact with the service – both online and in person 2. Can the project identify and articulate the pain points in the current way users interact with the service, and prioritize these according to user needs 3. Has the project designed the digital parts of the service so that they are integrated with the offline touch points people use to interact with the service 4. Has the project designed developed metrics that will measure how well the service is meeting user needs at each step of the service key questions ● What are the different ways (both online and offline) that people currently accomplish the task the digital service is designed to help with? ● Where are user pain points in the current way people accomplish the task? ● Where does this specific project fit into the larger way people currently obtain the service being offered? ● What metrics will best indicate how well the service is working for its users?
3 Make it simple and intuitive The Commonwealth believes using a government service shouldn’t be stressful, confusing, or daunting. It’s our job to build services that are simple and intuitive enough that users succeed the first time, unaided. checklist 1. Has the project created or used an existing, simple, and flexible design style guide for the service 2. Has the project used the design style guide consistently for related digital services 3. Does the solution give users clear information about where they are in each step of the process 4. Does the solution Follow accessibility best practices to ensure all people can use the service 5. Does the solution provide users with a way to exit and return later to complete the process 6. Does the system use language that is familiar to the user and easy to understand 7. Does the solution use language and design consistently throughout the service, including online and offline touch points key questions ● What primary tasks are the user trying to accomplish? ● Is the language as plain and universal as possible? ● What languages is your service offered in? ● If a user needs help while using the service, how do they go about getting it? ● How does the service’s design visually relate to other government services?
4 Build the service using agile and iterative practices The Commonwealth believes in using an incremental, fast-paced style of software development to reduce the risk of failure. We want to get working software into users’ hands as early as possible to give the design and development team opportunities to adjust based on user feedback about the service. A critical capability is being able to automatically test and deploy the service so that new features can be added often and be put into production easily. checklist 1. Does the project focus on shipping a functioning “minimum viable product” (MVP) that solves a core user need as soon as possible, no longer than three months from the beginning of the project, using a “beta” or “test” period if needed?
Page 57 of 64
2. Does the project have plans in place to run usability tests frequently to see how well the service works and identify improvements that should be made 3. Does the project ensure the individuals building the service communicate closely (ex. techniques such as launch meetings, war rooms, daily standups, and team chat tools) 4. Does the project keep delivery teams small and focused; limit organizational layers that separate these teams from the business owners 5. Is there a plan to at least build features and improvements multiple times each month and release at least bi annually? 6. Has the project created a prioritized list of features and bugs, also known as the “feature backlog” and “bug backlog” 7. Does the project use a source code version control system 8. Does the project give the entire project team access to the issue tracker and version control system 9. Does the project use code reviews to ensure quality key questions ● How long did it take to ship the MVP? If it hasn't shipped yet, when will it? ● How long does it take for a production deployment? ● How many days or weeks are in each iteration/sprint? ● Which version control system is being used? ● How are bugs tracked and tickets issued? What tool is used? ● How is the feature backlog managed? What tool is used? ● How often do you review and reprioritize the feature and bug backlog? ● How do you collect user feedback during development? How is that feedback used to improve the service? ● At each stage of usability testing, which gaps were identified in addressing user needs?
5 Structure budgets and contracts to support delivery To improve the Commonwealth’s chances of success when contracting out development work, we need to work with experienced budgeting and contracting officers. In cases where we use third parties to help build a service, a well-defined contract can facilitate good development practices like conducting a research and prototyping phase, refining product requirements as the service is built, evaluating open source alternatives, ensuring frequent delivery milestones, and allowing the flexibility to purchase cloud computing resources. checklist 1. Does the project budget include research, discovery, and prototyping activities 2. Is the Project contract is structured to request frequent deliverables users can use or see, not multimonth milestones 3. Is the Project contract structured to hold vendors accountable to deliverables 4. Does the project contract give the government delivery team enough flexibility to adjust feature prioritization and delivery schedule as the project evolves 5. Does the project contract ensure open source solutions are evaluated when technology choices are made 6. Does the project contract specify that software and data generated by third parties remains under our control, and can be reused and released to the public as appropriate and in accordance with the law 7. Does the project contract allow the Commonwealth to use tools, services, and hosting from vendors with a variety of pricing models, including fixed fees and variable models like “pay-for-what-you-use” services 8. Does the project contract specify a warranty period where defects uncovered by the public are addressed by the vendor at no additional cost to the government Page 58 of 64
9. Does the project contract include a transition of services period and transition-out plan key questions ● What is the scope of the project? What are the key deliverables? ● What are the milestones? How frequent are they? ● What are the performance metrics defined in the contract (e.g., response time, system uptime, time period to address priority issues
6 Assign one leader and hold that person accountable The Commonwealth believes that a single product owner who has the authority and responsibility to assign tasks and work elements; make business, product, and technical decisions; and be accountable for the success or failure of the overall service. This product owner is ultimately responsible for how well the service meets needs of its users, which is how a service should be evaluated. The product owner is responsible for ensuring that features are built and managing the feature and bug backlogs. checklist 1. A product owner has been identified 2. All stakeholders agree that the product owner has the authority to assign tasks and make decisions about features and technical implementation details 3. The product owner has a product management background (not Project Management) or there is product management advisor with technical experience to assess alternatives and weigh tradeoffs 4. The product owner has a work plan that includes budget estimates and identifies funding sources, broken across useable deliverables, not milestones. 5. The product owner has a strong relationship with the contracting officer key questions ● Who is the product owner? ● What organizational changes have been made to ensure the product owner has sufficient authority over and support for the project? ● What does it take for the product owner to add or remove a feature from the service?
7 Bring in experienced teams The Commonwealth believes in talented people working in government who have experience creating modern digital services. This includes bringing in seasoned product managers, Product owners, developers, engineers, and designers. When outside help is needed, our teams should work with contracting officers who understand how to evaluate third-party technical competency so our teams can be paired with contractors who are good at both building and delivering effective digital services. The makeup and experience requirements of the team will vary depending on the scope of the project. checklist 1. Member(s) of the team have experience building popular, high-traffic digital services 2. Member(s) of the team have experience designing mobile and web applications 3. Member(s) of the team have experience using automated testing frameworks 4. Member(s) of the team have experience with modern development and operations (DevOps) techniques like continuous integration and continuous deployment 5. Member(s) of the team have experience securing digital services 6. A Commonwealth contracting manager/analyst is on the internal team or immediately available if a third party will be used for development work 7. A Commonwealth budget analyst is on the internal team or is a partner Page 59 of 64
8. The appropriate privacy, civil liberties, and/or legal advisor for the department or agency is a partner
8 Choose a modern technology stack The technology decisions the Commonwealth must make need to enable development teams to work efficiently and enable services to scale easily and cost-effectively. Our choices for hosting infrastructure, databases, software frameworks, programming languages and the rest of the technology stack should seek to avoid vendor lock-in and match what successful modern consumer and enterprise software companies would choose today. In particular, digital services teams should consider using open source, cloud-based, and commodity solutions across the technology stack, because of their widespread adoption and support by successful consumer and enterprise technology companies in the private sector. checklist 1. Has the project chosen a software frameworks that are commonly used by private-sector companies creating similar services 2. Has the project whenever possible, ensured that software can be deployed on a variety of commodity hardware types 3. Has the project ensured that each project partner has clear, understandable instructions for setting up a local development environment, and that team members can be quickly added or removed from projects 4. Has the project considered open source software solutions at every layer of the stack key questions ● What is your development stack and why did you choose it? ● Which databases are you using and why did you choose them? ● How long does it take for a new team member to start developing?
9 Deploy in a flexible hosting environment The Commonwealth suite of services should be deployed on flexible infrastructure, where resources can be provisioned in real-time to meet spikes traffic and user demand. checklist 1. Are Project hosting and technical resources are provisioned on demand 2. Are Project hosting and technical resources scalable based on real-time user demand 3. Are Project hosting and technical resources provisioned through an API 4. Are Project hosting and technical resources available multiple appropriate regions 5. Are Project hosting and technical resources only paying for resources they use or are they hedging on capacity promises. 6. Are Project hosting and technical static assets are served through a content delivery network 7. Are Project hosting and application is hosted on commodity hardware key questions ● Where is your service hosted? ● What hardware does your service use to run? ● What is the demand or usage pattern for your service? ● What happens to your service when it experiences a surge in traffic or load? ● How much capacity is available in your hosting environment? ● How long does it take you to provision a new resource, like an application server? ● How have you designed your service to scale based on demand? ● How are you paying for your hosting infrastructure (e.g., by the minute, hourly, daily, monthly, fixed)? Page 60 of 64
● ● ● ● ●
Is your service hosted in multiple regions, availability zones, or data centers? In the event of a catastrophic disaster to a datacenter, how long will it take to have the service operational? What would be the impact of a prolonged downtime window? What data redundancy do you have built into the system, and what would be the impact of a catastrophic data loss? How often do you need to contact a person from your hosting provider to get resources or to fix an issue?
10 Automate testing and deployments The Commonwealth stresses using automated performance tests which simulate surges in traffic to identify performance bottlenecks. While manual tests and quality assurance are still necessary, automated tests provide consistent and reliable protection against unintentional regressions, and make it possible for developers to confidently release frequent updates to the service. checklist 1. Has the project created automated tests that verify all user-facing functionality 2. Has the project created unit and integration tests to verify modules and components 3. Is the project using automated test to verify all modules and end to end functionality 4. Does the project run tests automatically as part of the build process 5. Does the project perform deployments automatically with deployment scripts, continuous delivery services, or similar techniques 6. Does the project conduct load and performance tests at regular intervals, including before public launch key questions ● What percentage of the code base is covered by automated tests? ● How long does it take to build, test, and deploy a typical bug fix? ● How long does it take to build, test, and deploy a new feature into production? ● How frequently are builds created? ● What test tools are used? ● Which deployment automation or continuous integration tools are used? ● What is the estimated maximum number of concurrent users who will want to use the system? ● How many simultaneous users could the system handle, according to the most recent capacity test? ● How does the service perform when you exceed the expected target usage volume? Does it degrade gracefully or catastrophically? ● What is your scaling strategy when demand increases suddenly?
11 Manage security and privacy through reusable processes Commonwealth digital services have to protect sensitive information and keep systems secure. This is typically a process of continuous review and improvement which should be built into the development and maintenance of the service. At the start of designing a new service or feature, the team lead should engage the appropriate privacy, security, and legal officer(s) to discuss the type of information collected, how it should be secured, how long it is kept, and how it may be used and shared. The sustained engagement of a privacy specialist helps ensure that personal data is properly managed. In addition, a key process to building a secure service is comprehensively testing and certifying the components in each layer of the technology stack for security vulnerabilities, and then to re-use these same pre-certified components for multiple services. The following checklist provides a starting point, but teams should work closely with their privacy specialist and security engineer to meet the needs of the specific service. Page 61 of 64
checklist Has the project determined, in consultation with a records officer, what data is collected and why, how it is used or shared, how it is stored and secured, and how long it is kept Has the project determined, in consultation with a privacy specialist, whether and how users are notified about how personal information is collected and used, including whether a privacy policy is needed and where it should appear, and how users will be notified in the event of a security breach Has the project determined whether the user should be able to access, delete, or remove their information from the service Does the project use deployment scripts to ensure configuration of production environment remains consistent and controllable key questions ● Does the service collect personal information from the user? How is the user notified of this collection? ● Does it collect more information than necessary? Could the data be used in ways an average user wouldn't expect? ● How does a user access, correct, delete, or remove personal information? ● Will any of the personal information stored in the system be shared with other services, people, or partners? ● How and how often is the service tested for security vulnerabilities? ● How can someone from the public report a security issue?
12 Use data to drive decisions At every stage of a project, The Commonwealth should measure how well our service is working for our users. This includes measuring how well a system performs and how people are interacting with it in realtime. Our teams and agency leadership should carefully watch these metrics to find issues and identify which bug fixes and improvements should be prioritized. Along with monitoring tools, a feedback mechanism should be in place for people to report issues directly. checklist 1. Does the project separate and record “Key Performance Metrics of the system, from Key Goal Metrics of the system? 2. For live modules does the project monitor system-level resource utilization in real time 3. For live modules does the project monitor system performance in real-time (e.g. response time, latency, throughput, and error rates) 4. For live modules does the project ensure monitoring can measure median, 95th percentile, and 98th percentile performance 5. For live modules does the project create automated alerts based on this monitoring 6. For live modules does the project track concurrent users in real-time, and monitor user behaviors in the aggregate to determine how well the service meets user needs 7. Does the project publish metrics internally 8. Does the project publish metrics externally 9. Does the project use an experimentation tool that supports multivariate testing in production to test its production use? key questions What are the key metrics for the service? How have these metrics performed over the life of the service? Which system monitoring tools are in place? Page 62 of 64
What is the targeted average response time for your service? What percent of requests take more than 1 second, 2 seconds, 4 seconds, and 8 seconds? What is the average response time and percentile breakdown (percent of requests taking more than 1s, 2s, 4s, and 8s) for the top 10 transactions? What is the volume of each of your service’s top 10 transactions? What is the percentage of transactions started vs. completed? What is your service’s monthly uptime target? What is your service’s monthly uptime percentage, including scheduled maintenance? Excluding scheduled maintenance? How does your team receive automated alert when incidents occur? How does your team respond to incidents? What is your post-mortem process? Which tools are in place to measure user behavior? What tools or technologies are used to for A/B testing? How do you measure customer satisfaction?
13 Default to open When the Commonwealth collaborates in the open and publish our data publicly, the Commonwealth believes we can improve Government together. By building services more openly and publishing open data, we simplify the public’s access to government services and information, allow the public to contribute easily, and enable reuse by entrepreneurs, nonprofits, other agencies, and the public. checklist 1. Does the system offer users a mechanism to report bugs and issues, and be responsive to these reports 2. Does the service provide datasets to the public, in their entirety, through bulk downloads and APIs (application programming interfaces) 3. Does the Commonwealth ensure that data from the service is explicitly in the public domain, and that rights are waived globally via an international public domain dedication, such as the “Creative Commons Zero” waiver 4. Has the project cataloged data in the agency’s enterprise data inventory and add any public datasets to the agency’s public data listing 5. Does the Project ensure that the Commonwealth maintain the rights to all data developed by third parties in a manner that is releasable and reusable at no cost to the public 6. Does the Project ensure that the Commonwealth maintain contractual rights to all custom software developed by third parties in a manner that is publishable and reusable at no cost 7. When appropriate, did the project create an API for third parties and internal users to interact with the service directly 8. When appropriate, did the project publish source code of projects or components online 9. When appropriate, did the project share development process and progress publicly key questions ● How are you collecting user feedback for bugs and issues? ● If there is an API, what capabilities does it provide? Who uses it? How is it documented? ● If the codebase has not been released under an open source license, explain why. ● What components are made available to the public as open source? ● What datasets are made available to the public?
Page 63 of 64
EXHIBIT O Elements of Benefits Realization
Elements of Benefits Realization typically include, but are not limited to, determining if each initiative/module has its expected outcomes clearly defined in terms of customer use or impact. For example,
do project outcomes create a measurable value to the organization? If so, how? do project outcomes support the achievement of clearly achievable strategic objectives? are expected outputs, outcomes and benefits described in the business case and approved at the beginning of the project and every year the project continues? are project outputs and outcomes frequently reviewed and re-aligned to current expectations? If so, how frequently and who signs off? do project outcomes adhere to the expected outcomes described in the business case? Is there a record of business-approved changes to outcomes? does the project scope include activities with the goal of ensuring the integration of project outputs into the regular business routine? does the organization work in a pre-planned and regular way to integrate project outputs into the business routine from the first delivery to the project’s closure? has the organization instituted a means to monitor project outcome after project closure in order to ensure the achievement of all benefits described in the business case?
Page 64 of 64
