Use the File System Onsite Connector

Connectors allow you to ingest documents into Haven OnDemand text indexes from various sources.

The File System Connector enables you to index all files in a local directory. It requires that you install a lightweight service on the local host to facilitiate the file transfer. This is available for the following operating systems and CPU architectures:

  • Windows x86 32-bit
  • Windows x86 64-bit
  • Linux x86 32-bit
  • Linux x86 64-bit

Create the Connector

To create a Conector for your File System, you must first call the Create Connector API. Calling this API creates a Connector of the type of your choice.

The required parameters are:

  • connector. The name of your Connector. You use this name for further updates, status checks, and deletion.

  • flavor. The type of Connector you want to use. For File System Onsite Connector, this value is filesystem_onsite.

  • config. A JSON configuration, relevant to the type of Connector.

  • destination. A JSON object to specify how to process the files returned by the Connector. Currently, this specifies the text index that you want to index into.

  • schedule. This parameter is an interval of at least 21600 seconds (six hours) between automatic launches of the Connector. It is required for an onsite Connector, because Haven OnDemand cannot start and stop it remotely. The schedule is incorporated into the local service.

/createconnector/v1?connector=fsf&flavor=filesystem_onsite
&config={ "directoryPathCSVs" : "/home/<me>/folder1", "directoryRecursive": true, "servicePort": 8002, "aciPort": 8000}
&destination={ "action" : "addtotextindex", "index" : "mydirectory" }
&schedule={ "frequency" : { "frequency_type" : "seconds", "interval" : 21600 } }

This configuration creates a File System Onsite Connector configuration on Haven OnDemand. The API returns a link to download the Connector installer.

You can also use the Download Links Discovery API to find the available Connector installer downloads.

{
  "message": "connector created",
  "connector": "fsf",
  "download_links": {
    "windows_x86": "http://cdn.havenondemand.com/FileSystemConnector_{VERSION}_WINDOWS_x86.exe",
    "windows_x86_64": "http://cdn.havenondemand.com/FileSystemConnector_{VERSION}_WINDOWS_x86_64.exe",
    "linux_x86": "http://cdn.havenondemand.com/FileSystemConnector_{VERSION}_LINUX_x86.bin",
    "linux_x86_64": "http://cdn.havenondemand.com/FileSystemConnector_{VERSION}_LINUX_x86_64.bin"
  }
}

Install the Connector

Installing with the Installer Wizard

On Microsoft Windows operating systems, or a Linux operating system with a Desktop environment, you can download and run the installer file from the link.

The installer requests the following information:

  1. License Agreement: Agree to the terms of the license to continue.
  2. Installation Directory: Specify the directory where you want to install the HOD Filesystem Connector.
    The Windows default is C:\HewlettPackardEnterprise\FileSystemConnector-<VERSION>.
    The Linux default starts from your home directory, for example /home/<me>/HewlettPackardEnterprise/FileSystemConnector--<VERSION>
  3. Haven OnDemand API Key: Enter your API key. You can copy and paste this from your Account page, under Manage your API Keys. Click Haven OnDemand API website to link to that page directly.
  4. HTTP Proxy configuration: If an HTTP proxy is needed to connect to the Haven OnDemand website, check the box HTTP proxy is needed and enter the details.
  5. Haven OnDemand Configuration: A list displays all the File System Connector configurations you have already created in Haven OnDemand. Select the one you want to use for this Connector. For the example configuration above, this value is fsf.
  6. Windows Service (for Windows configurations): The File System Connector service is named FileSystemConnector by default. If you have more than one File System Connector, change this name to distinguish it.
    By default, the Connector is configured and runs under the user Local System. If this user does not have at least read-access to the target file system, check the box Run Windows service as another account and enter the credentials of a user with sufficient rights.
  7. Start connector: This window lets you select if you want to start the HOD FileSystem Connector service automatically after the installation. If you select No, you must start the service manually in the Windows Services administration window when you want to run the Connector.
  8. Pre-Installation Summary: This window displays the product name and the installation directory. Review your settings and click Next.
  9. Completing the HOD FileSystem Connector Setup Wizard: When the installation has completed successfully, click Finish to exit the installer.

Installing from the Linux Command Line

If your Linux environment has only a shell interface, you can run the following command to download the Connector:

server:~# wget name_of_Linux_executable

substituting the name of the Linux executable for the one you downloaded after running the Create Connector API, for example:

server:~# wget http://cdn.havenondemand.com/HODFileSystemConnector_11.1.1_LINUX_X86_64.exe

Set this file to be executable and run it with the --mode text flag. For example:

server:~#chmod +x HODFileSystemConnector_11.1.1_LINUX_X86_64.exe
server:~#./HODFileSystemConnector_11.1.1_LINUX_X86_64 --mode text

Eventually, the installer asks for information about your API key and configuration. For example:

HOD Configuration
   Please enter the API key used for accessing your HOD configurations. If you
   do not have an API key, you can obtain it by visiting the HOD API website.
   https://www.havenondemand.com/account/api-keys.html:
HOD API Key (Default: ): my-api-key
HOD Configuration
   HOD Has the following configurations available for your Connector, please
   choose one:
HOD Connector Configuration
   0: filesys
   1: fsf
   : 1

After you provide the information, the Connector is installed. By default, it starts from your home folder, /home/<me>/HewlettPackardEnterprise/FileSystemConnector--<VERSION>.

Operate the Connector

Windows

The File System Connector runs as a Windows service and starts up with the Windows host. You can check running services in the Windows Services administration window. If the FileSystemConnector service is not running, right-click the service name and select Start in the popup menu.

Linux

To start the Connector, navigate to the installation directory and run the shell script. For example:

server:~#cd /home/<me>/HewlettPackardEnterprise/FileSsytemConnector-11.1.1
             server:~#./start-FileSystemConnector.sh
		

To stop the Connector:

server:~#cd /home/<me>/HewlettPackardEnterprise/FileSsytemConnector-11.1.1
			 server:~#./stop-FileSystemConnector.sh
		

Configure the Connector

In the previous Connector configuration example, the Connector points to a single directory. There are some other things that you might want to add to your configuration.

Multiple Folders

You can specify a comma-separated list of folder names in the directoryPathCSVs parameter to index content from multiple folders.

"directoryPathCSVs" : "/home/<me>/folder1,/home/<me>/folder2"
Turn off Directory Recursion

You can turn off directory recursion if you want to index only the files in the root folders that you specify in the directoryPathCSVs parameter.

"directoryRecursive": false

Update the Connector Configuration

After you create the Connector, you might want to change the configuration.

The Retrieve Config API takes a Connector name, and returns its configuration.

/retrieveconfig/v1?connector=fsf

After you check the configuration, you can change it by using the Update Connector API.

/updateconnector/v1?connector=fsf&config=NewConfig

You can also change the destination and the schedule. For example:

/updateconnector/v1?connector=fsf&destination=NewDestinationConfig
/updateconnector/v1?connector=fsf&schedule=NewConfig

You must restart your onsite Connector for these changes to take effect.

Track Connector Status or History

To get the status or history for an onsite flavor Connector, run the following command in a browser on the machine where the onsite Connector is running. The action port, called the aciPort, is the one you set when you create the Connector configuration (8000 by default; see File System Connector):

localhost:aciPort/action=QueueInfo&QueueAction=GetStatus&QueueName=Fetch

The above command returns the status of all previous Connector runs, with the most recent status returned first. The Connector status is in XML format.