Connector History

Returns connector status history information.

Connectors allow you to retrieve information from external systems and update it through Haven OnDemand APIs. You can use them to create an index of the data in the external repository, automate workflows, for notifications, or to facilitate migration to other locations.

The Connector History API allows you to check the history of your cloud flavor connectors and connector runs. It returns details of all connector runs, including scheduled runs and any runs that you started manually with the Start Connector API.

The optional parameters allow you to restrict the results to particular connectors, time ranges, or connector job statuses. You can also filter by one or more connector tokens, which identify a particular run of a connector. The Start Connector API returns this token when you start the connector.

Quick Start

The Connector History API retrieves historical information about a connector's jobs. For example:

/1/api/[async|sync]/connectorhistory/v1?connector=myconnector


{
  "history": [
    {
      "connector": "mycloudconnector",
      "status": "PROCESSING",
      "document_counts": {
        "errors": 0
      },
      "queued_time": "08/07/2016 10:08:03 +00:00",
      "time_in_queue": 3,
      "process_start_time": "08/07/2016 10:08:06 +00:00",
      "time_processing": 6,
      "start_time": "08/07/2016 10:08:03 +00:00",
      "token": "MTAuOC4xNi43OjcyMDA6RkVUQ0g6MTQ2Nzk3MjQ4MzA0NTAxNDE0OTM3NTQ4"
    },
    {
      "connector": "mycloudconnector",
      "status": "FINISHED",
      "document_counts": {
        "errors": 415
      },
      "queued_time": "08/07/2016 09:14:05 +00:00",
      "time_in_queue": 3,
      "process_start_time": "08/07/2016 09:14:08 +00:00",
      "time_processing": 1994,
      "process_end_time": "08/07/2016 09:47:22 +00:00",
      "start_time": "08/07/2016 09:14:05 +00:00",
      "token": "MTAuOC4xNi45OjcyMDA6RkVUQ0g6MTQ2Nzk2OTI0NTc3MS0xODM1MTYzNDIy"
    },
    {
      "connector": "mycloudconnector",
      "status": "STOPPED",
      "document_counts": {
        "errors": 0
      },
      "process_end_time": "08/07/2016 09:12:59 +00:00",
      "queued_time": "08/07/2016 09:01:09 +00:00",
      "time_in_queue": 2,
      "process_start_time": "08/07/2016 09:01:11 +00:00",
      "time_processing": 708,
      "start_time": "08/07/2016 09:01:09 +00:00",
      "token": "MTAuOC4xNi43OjcyMDA6RkVUQ0g6MTQ2Nzk2ODQ2OTAzNy0wNTY3MDI3MTY4"
    }
  ]
}

The API returns a list of all the statuses the connector has been through since its creation. These can be IDLE, QUEUED. PROCESSING, FINISHED, ERROR, STOPPING or STOPPED. The response also includes additional information, such as the number of documents added, the start time or the duration of the connector job (in seconds).

This API is available only for cloud connector flavors, such as the Web Cloud Connector. If you list one or more onsite flavor connectors in the connectors parameter, the API returns a warning, and displays results only for cloud flavor connectors that you specified.

Filtering

You have various filtering options to help you limit the history to only the information you want to see.
  • By status: If you want only one kind of status to appear in the list, for example, to view just the errors, set the statuses filter parameter.
  • By date:When many jobs are in the history, you can filter by date (and time) using the following parameters:
    • min_date. The minimum date for which results are returned.
    • max_date. The maximum date for which results are returned

    The following example returns only jobs that ran after 10:08:03 Greenwich Mean Time on the 7th of July 2016:

    min_date=08/07/2016 10:08:03 +00:00

    For more information on the date and time formats you can enter, see Parameter Date Formats.

  • Latest only: You can also set the Connector History API to show only the most recent status, by setting show_latest_only to true. This makes the API identical to the closely related API.
  • By connector run: When you start a run of the connector using the API, or when it starts by schedule, it receives a unique token which you can use to identify that specific run. Set the tokens parameter to filter the History to show only the actions in one or more runs.

Paging

When you have many jobs in the history, you can use the following extra parameters to page through the results:

  • start. The number of the result you want to start with. For example, start=7 starts at the seventh result.
  • max_page_results. The number of results to display in each page.
  • absolute_max_results. The total maximum number of statuses to return. When paging, set this parameter to at least start+max_page_results-1.

Getting the History for Onsite Connectors

To get the equivalent status or history for an onsite flavor connector, run the following command in a browser on the machine where the onsite connector is running. The action port, called the aciPort, is the one you set when you create the connector configuration (8000 by default; see File System Onsite Connector):

localhost:aciPort/action=QueueInfo&QueueAction=GetStatus&QueueName=Fetch

The above command returns, in XML format, the status of all previous connector runs, with the most recent status returned first.

Synchronous
https://api.havenondemand.com/1/api/sync/connectorhistory/v1
Asynchronous
https://api.havenondemand.com/1/api/async/connectorhistory/v1
Authentication

This API requires an authentication token to be supplied in the following parameter:

Parameter Description
apikey The API key to use to authenticate the API request.
Parameters

This API accepts the following parameters:

Optional
Name Type Description
absolute_max_results
number The absolute maximum number of statuses to return. Default value: 10000.
connectors
array<resource> List of the connector names. Maximum length of 100 characters.
max_date
string The latest date or time of a connector job to return. See Parameter Date Formats.
max_page_results
number The maximum page size from the absolute number of results returned. Default value: 5.
min_date
string The earliest date or time of a connector job to return. See Parameter Date Formats.
show_latest_only
boolean Set to true to return only the latest status history for each connector. Default value: false.
start
number The starting page position from the absolute number of results returned. Default value: 1.
statuses
array<enum> The statuses that you want to filter by.
tokens
array<string> List of start connector tokens to filter by. Start connector token is obtained from the response of startconnector api.
Enumeration Types

This API's parameters use the enumerations described below:

statuses
The statuses that you want to filter by.
idle Idle Status
Include connector jobs that are in the IDLE state.
queued Queued Status
Include connector jobs that are in the QUEUED state.
processing Processing Status
Include connector jobs that are in the PROCESSING state.
finished Finished Status
Include connector jobs that are in the FINISHED state.
error Error Status
Include connector jobs that are in the ERROR state.
stopping Stopping Status
Include connector job that are in the STOPPING state.
stopped Stopped Status
Include connector jobs that are in the STOPPED state.

This API returns a JSON response that is described by the model below. This single model is presented both as an easy to read abstract definition and as the formal JSON schema.

Asynchronous Use

Additional requests are required to get the result if this API is invoked asynchronously.

You can use /1/job/status/<job-id> to get the status of the job, including results if the job is finished.

You can also use /1/job/result/<job-id>, which waits until the job has finished and then returns the result.

Model
This is an abstract definition of the response that describes each of the properties that might be returned.
Connector History Response {
history ( array[History] , optional) List of connector history statuses.
warnings ( array[Warnings] , optional)
}
Connector History Response:History {
connector ( string ) The name of the connector.
document_counts ( Document_counts , optional) Information about the documents processed during the connector job.
error ( string , optional) Error message for this connector job.
failed ( string , optional) The failed reason for this connector job.
process_end_time ( string , optional) The time when the connector job finished processing. Format: DD/MM/YYYY HH:mm:ss Z.
process_start_time ( string , optional) The time when the connector job started being processed. Format: DD/MM/YYYY HH:mm:ss Z.
start_time ( string , optional) The last time when the connector job has been updated or processed. Format: DD/MM/YYYY HH:mm:ss Z.
queued_time ( string , optional) The time when the connector job has been put on the queue. Format: DD/MM/YYYY HH:mm:ss Z.
status ( string ) The current state of the connector job.
time_in_queue ( number , optional) The number of seconds the connector job spent in the queue.
time_processing ( number , optional) The number of seconds spent processing the connector job.
token ( string , optional) Start connector token.
}
Connector History Response:History:Document_counts {
added ( number , optional) The number of added documents.
deleted ( number , optional) The number of deleted documents.
errors ( number , optional) The number of documents errors.
ingest_added ( number , optional) The number of ingest-added documents.
ingest_deleted ( number , optional) The number of ingest-deleted documents.
ingest_failed ( number , optional) The number of ingest-failed documents.
}
Connector History Response:Warnings {
code ( integer , optional)
details ( object , optional)
}
Model Schema
This is a JSON schema that describes the syntax of the response. See json-schema.org for a complete reference.
{
    "properties": {
        "history": {
            "items": {
                "properties": {
                    "connector": {
                        "type": "string"
                    },
                    "document_counts": {
                        "properties": {
                            "added": {
                                "type": "number"
                            },
                            "deleted": {
                                "type": "number"
                            },
                            "errors": {
                                "type": "number"
                            },
                            "ingest_added": {
                                "type": "number"
                            },
                            "ingest_deleted": {
                                "type": "number"
                            },
                            "ingest_failed": {
                                "type": "number"
                            }
                        },
                        "type": "object"
                    },
                    "error": {
                        "type": "string"
                    },
                    "failed": {
                        "type": "string"
                    },
                    "process_end_time": {
                        "type": "string"
                    },
                    "process_start_time": {
                        "type": "string"
                    },
                    "start_time": {
                        "type": "string"
                    },
                    "queued_time": {
                        "type": "string"
                    },
                    "status": {
                        "type": "string"
                    },
                    "time_in_queue": {
                        "type": "number"
                    },
                    "time_processing": {
                        "type": "number"
                    },
                    "token": {
                        "type": "string"
                    }
                },
                "required": [
                    "connector",
                    "status"
                ],
                "type": "object"
            },
            "type": "array"
        },
        "warnings": {
            "type": "array",
            "items": {
                "type": "object",
                "additionalProperties": false,
                "properties": {
                    "code": {
                        "type": "integer"
                    },
                    "details": {
                        "type": "object"
                    }
                }
            }
        }
    },
    "type": "object",
    "additionalProperties": false
}


If you would like to provide us with more information then please use the box below:

We will use your submission to help improve our product.