This article provides a framework to beginning a Data Integration Project.

Data Integration Projects are unique in nature, and so a high-level guide of how to approach them is useful. Across all Data Integration projects, a well thought out, detailed planning process is crucial.

Planning and Leadership

Begin with policy decisions and involving key stakeholders from the start. It is best to work from the high level decisions down to granular ones, always keeping the end result in mind. Think about what information you want to know before deciding how you are going to get it.

  • Engage key stakeholders; those who will provide data, use data, pay for the project, or carry out the project.
  • Clearly define and document project vision and goals.
  • Develop a work plan for achieving the vision; designate a project champion to maintain forward momentum for the project.
  • Reach consensus on all key decisions; usually people are bigger barriers than the technology.
  • Address any obstacles as they come.


Documentation is key to a smooth data integration roll out. Be sure to thoroughly document all steps of the planning process and create agreements along the way. Here are some suggestions of useful documentation to create:

  • Meeting Agendas and Notes
  • Policies and Procedures
  • Interagency Agreement
  • Participation Agreement
  • Data Use Agreement

Do not underestimate the importance of defining and documenting roles and responsibilities for everyone involved in the project. The various roles and responsibilities can be defined across the different types of documents above as long as everyone understands and agrees with their expected role. Make sure that for every step of the data integration process, it is clearly stated who is responsible:

  • Data preparation, transfer and receipt
  • Data access rights
  • Data warehouse (or other system) management and administration
  • Data confidentiality and security
  • Data release, reporting, and publication

System Design Considerations

The following decisions should all be made to serve the broader policy goal. Remember that technology serves the policy, not the other way around. With this in mind, create consensus around the following system design variables to create buy-in with all potential stakeholders. Always keep in mind the end result and what the system must accomplish in order to be useful to the community as a whole.

  • Data Source; what data and from where it will be collected.
  • Data Format; XML, CSV, custom format.
  • Template for the data; Unique identifiers, naming protocol, de-duplication.
  • Software to handle the receipt and transmission of the portable data.
  • Analysis and Reporting Software; what do you want to report on and how often.
  • Security Features; Encryption, firewall, Secure Socket Layer connection.
  • Messaging protocol and interagency agreements; who owns data.

Data Specific Considerations

Even with specific data elements its is important to focus on the end result and work backwards. For example, the desired outcome of the system will inform what types of reports need to be created. The types of reports will inform what data points need to exported. The specific data elements exported will provide details on the required mapping, encryption, data format, and data source. See below a list of data specific considerations to answer before beginning the data integration project:

  • What is the data source(s)?
  • What is the data destination?
  • Is data just being exported from HMIS or other systems as well?
  • What is the technical level of staff working with the data?
  • What are the specific data elements?
  • Is Personally Identifying Information going to be included?
  • Is HUD data sufficient or is custom data necessary?
  • Should only updated data be exported or all data?
  • How often should data be exported/imported?

These questions can help inform what specific data format is best used for the project, and which of Clarity Human Services data integration tools is most suitable. Review the Export Tools Overview article.

Security Specific Considerations

In a Data Integration project, Data Privacy and Security must be protected in both a technical sense and legal sense. This means not only having secure hardware and software, but also having the right contractual agreements in place. The Department of Housing and Urban Development (HUD) outlines privacy and security standards in its HMIS Data and Technical Standards.

Data Privacy Agreements

Legal documentation must cover every step of the process in a Data Integration project. It must protect the data from its original source– the client– all the way to its final destination whether that be the HMIS system, a data warehouse, or any other possible systems. Data agreements may need to cover specific federal and state laws including but not limited to:

  • The Health Insurance Portability and Accountability Act, 45 CFR Parts 160, 162, and 164 (“HIPAA”)
  • The Health Information Technology for Economic and Clinical Health Act (“HITECH Act”), Pub. L. 111-5, Div. A, Title XIII, § 130001 et seq., Div. B, Title IV, § 4001 et seq., Feb. 17, 2009, 123 Stat. 226, 467, 42 U.S.C.A. § 300ii, et seq., and 4 U.S.C.A. § 17901, et seq.;
  • Code of Federal Regulations, at 42 CFR Part 2
  • Various state laws

Client Release of Information

A client consent form or Release of Information (ROI) should include how the data collected will be used and how it will be protected. If the data is being used beyond entry in HMIS, then that should be included in the ROI. It is also helpful to include the technical security details so that clients will feel comfortable consenting for their information to be entered into HMIS as well as used in the data integration projects.

End User HMIS Agreements

End users are people who will interact with the data entered into Clarity Human Services. They play a critical role in ensuring data security. It is crucial that End Users understand and agree to their responsibility as stewards of client’s confidential information. All people involved in the Data Integration projects with access to client-level data must be trained in data security protocol and sign an End User agreement or Data Use agreement.

Partner Agency Agreements

Partner Agency Agreements are signed by all participating agencies in the HMIS system. They typically cover general HMIS information, confidentiality protocol, data entry and use, required agency roles, reporting, insurance information, and standards terms and conditions. If the data integration project will be system-wide and include all agencies, then language about the project could be added into the Partner Agency Agreement in the Data Use section. Otherwise, if it is not a system-wide data integration and only certain agencies will choose to take part, a separate Participation Agreement could be utilized.

Participation Agreement

A Participation Agreement may be necessary if the Partner Agency Agreement does not cover the Data Integration project or if third-parties become involved in the Data Integration project. A Participation Agreement will be specific to the project and be at the Agency or organization level. It will encompass the scope of the project and its purpose. As well as outline the responsibilities of the data recipient in terms of data security and project roles. The Agreement will typically contain standard contractual terms and conditions.

Data Use Agreement

A Data Use Agreement may be necessary if people outside the general HMIS end users will have access to the HMIS data. These people would not have signed an HMIS End User Agreement, and therefore would need to agree to the security requirements of HMIS data.

Technical Security Requirements

Technical security requirements refer to the security of the software and hardware involved in the Data Integration project.

  • Use private wireless networks, only WiFi Protected Access (WPA) or WiFi Protected Access II (WPA2) security protocols are allowed.
  • Utilize Two-Factor Authentication (2FA) to access HMIS data.
  • Unique and secret User ID and Passwords to access HMIS data.
  • All connections to HMIS should be encrypted to HUD standards or higher- during data transmission and storage.
  • All computers accessing HMIS data be protected by a firewall and anti-virus software.
  • Computers and printers with access to HMIS data must be in secured location.