This article is a top level overview and comparison of the different tools Clarity Human Services has available to export data.

There are several purposes for which HMIS data might be exported from one system and imported to another. Clarity Human Services offers a range of data export tools to fit the requirements of the various types of export projects. Agencies may export:

  • Data that has been created within their agency in SCC HMIS
  • Profile data for clients enrolled into programs, or receiving assessments or services under their agencies, regardless of whether their agency or a Partner Agency created those profiles.

It is important to thoroughly review all of the options before determining what is the right method for your project. The following tools are organized from the most basic to the most sophisticated with a comparison chart at the end.

If you have further questions about any of the export methods below or want to request access to begin a Data Integration Project, please submit a Data Integration Request form to Bitfocus (found under the Technical Support Icon).

CSV and XML Exports

The HMIS CSV and HMIS XML formats are primarily intended to support migration from one HMIS application to another; warehousing of data from multiple HMIS implementations for analysis and reporting; and participation in a local HMIS implementation by regularly providing data entered and exported from an alternate database. Both the XML and CSV formats are based off the same HMIS Logical Model and contain the same source data definitions.

Both the CSV and XML Exports can be run using the [HUDX-111] HUD CSV / XML Program Data Export report stored under HUD Reports in the Report Library within Clarity Human Services. This is not an automated report, and requires manual entry of the report parameters each time the report should be run. The parameter options allow the user to specify the data that is included in the export and which format (CSV or XML).

CSV Export

After selecting the CSV format and other desired parameters, click Show Report. Depending on the parameters selected, this report could take more time downloading than a typical report in the Report Library.

The result will be a downloaded folder containing a folder with a set of multiple CSV files, as defined by HUD. In a CSV file, the different data elements are separated by a comma. CSV files are simple and generally considered easy to work with.

For more information about the fields and structure of the CSV data export, review the Current CSV Specifications provided by HUD.

XML Export

After selecting the XML format and other desired parameters, click Show Report. The result is a downloaded XML file.

The XML maintains a simple structure comprised of a relatively flat, single file. Almost all of the data types are one level below the Export data element with key references enforcing relationships between types. In an XML file, different data elements are surrounded by tags- Jane.

The key benefit of XML data is its ability to be validated and extended, meaning both systems sending and receiving data can independently and automatically check if the data matches the specifications.

For more detailed information and sample XML data, thoroughly review the Current XML Specifications provided by HUD.

Accessing the Export

To access the [HUDX-111] HUD CSV / XML Program Data Export report, click the Launchpad icon and click the Reports icon. The report is located in the HUD Reports section. Click the Run link to access the export parameters.

DIT_accessing_report.gif

Running the Export

Once the user has clicked Run on the HUDX-111 report, a series of parameters are displayed used to determine the range, amount, and format of data to be exported from the Clarity system. A description of each of the parameters is detailed below.

Parameter Description
Program Determines the program data to be included in the export. The user can choose a single program, multiple programs, or all programs.
CoC Enables the user to further filter the data based on CoC ID. The user can choose a single CoC, multiple CoCs, or all CoCs. The options available in this parameter are determined by the Program selected.
Date From This date field allows the user to specify the start date of the data to be included in the export.
Date To This date field allows the user to specify the end date of the data to be included in the export.
Encrypt Identifying Data With this setting active, the client’s First Name, Middle Name, Last Name and SSN fields will be replaced with a 64-character alphanumeric string leveraging the SHA-256 hash algorithm.
Format The user can choose to format the exported data in CSV or XML format.
Version The export format the CSV or XML is available in version 6.1.
Source Type Allows the user to identify what type of Clarity source data is being exported and described in the export.csv file. Options include Continuum-Operated HMIS, an Agency-Specific Database, a Data Warehouse, or Other. Note that an HMIS implementation operated jointly by more than one continuum should be identified as a data warehouse.
Include Deleted Data When active, the export process includes inactive/deleted Clarity records in the exported data.
Include Service Item ID When active, the export files are extended to include the specific Service Item IDs from Clarity.
Include Custom Services When active, the data export will include services that do not belong to specific HUD service categories. These records are still linked to programs, however will not include a record type or type provided per the HUD Data Standards.

Exporting the Data

Once the parameters have been configured, click the Show Report button. This will begin the export process. The Report Queue at the top of the screen will indicate that the export is running.

DIT_report__queue.png

Downloading the Export Data

Once the export has completed, a popup will appear indicating that the results are ready for download. Click the Open button to prompt the file download.

DIT_download.png

The file downloaded will be in ZIP format containing several program and client data files aligning with the configuration of the export parameters selected.

Automated CSV or XML Exports

While the CSV and XML exports from the Report Library are not automated, with Bitfocus intervention it is possible to utilize automated CSV or XML exports.

With a Statement of Work, Bitfocus can use our Automator tool to schedule CSV or XML exports to a secure FTP server designated by the Customer. These types of automated exports only include HUD data elements, and do not cover custom data.

Looker Exports

Using Looker is an alternative way to pull data out of Clarity for various uses. There are a few different ways to use Looker with varying degrees of complexity.

Manual Looker Exports

The most basic, involves the user creating a Look and then downloading the data as either a TXT, Excel spreadsheet, CSV, JSON, HTML, Markdown, or PNG. This can be done in the Data Analysis tab within Clarity Human Services or in a standalone Looker account. The data can include HUD data as well as any custom data in Clarity Human Services. Remember to follow privacy procedures and always protect client information. Never keep client identifying information saved to your computer or send it over insecure channels such as email or messaging systems.

This is a manual process which involves creating the Look (or running a saved Look) and downloading it. To learn more about building Looks and using Looker, consult the Learning Looker documentation.

Automated Looker Exports

Looker also has the ability to schedule automated, customer-defined exports using a the standalone Looker model (not Data Analysis tab). If you are interested in a standalone Looker account, but do not yet have access, submit a Data Integration Request Form under the Technical Assistance icon. In order to be granted standalone Looker access, there will need to be a specific reason why the Data Integration project cannot be successful using the Data Analysis tab within Clarity Human Services. It is a best practice to first try and make your project work given the level of access you already have in the system.

To do this, a Look first needs to built and saved with all of the required data, which can include HUD and custom data. Then the Look needs to be scheduled to deliver to either Amazon S3, email, SFTP, or Webhook. Remember that client Personally Identifying Information cannot be sent over email or unsecure channels.

These exports can be scheduled as often as every minute or as little as monthly. They can also be set up to only send if data has changed since the last run, depending on the amount of data.

Looker API

The most complex and robust option for pulling information from Looker is by using the Looker API. Looker provides the ability to programmatically pull the same type of information that is available from the Looker interface by making API calls directly. This will need both assistance from Bitfocus and a programmer familiar with API integration on the Customer side.

Using the Looker API may require an additional Looker license and configuration of API credentials, but it will allow you to write code to pull any data that can be obtained through your Looker interface. These API calls can be programmed by the Customer to pull at any interval that the Custom wishes by on business logic determined by the Customer.

Comparison Chart

CSV Export XML Export CSV/XML Automator Looker Export Looker API
Official HUD Data Format Yes Yes Yes No No
Automated No No Yes Yes Yes
Includes Custom Data No No No Yes Yes
Requires Approval from Bitfocus Yes Yes Yes Yes Yes
Requires Additional Cost No No Yes No Yes
Requires Programming Ability No No No No Yes
Agency-Specific Data Yes Yes Yes Yes Yes