Twitter Connector
Configuration settings for adjusting the Haven OnDemand Cloud Twitter Connector.

Twitter Cloud Connector

The Twitter Cloud connector retrieves content from a Twitter application and indexes it into Haven OnDemand. The connector connects to Twitter by using a Twitter application, and retrieves the content by searching using specified search filters.

This connector is a cloud connector that runs entirely in the Haven OnDemand environment.

Create a Twitter Application

To index content from Twitter, you must create a Twitter application to represent the connector and obtain an access token so that the connector can authenticate with Twitter.

Note: This procedure is subject to change. For information about creating a Twitter application and obtaining an OAuth Token, refer to the Twitter developer documentation.

To create a Twitter application and generate an OAuth Token

  1. Create an account on www.twitter.com, or use an existing account. The account must have a mobile phone number to create the application.
  2. Go to https://apps.twitter.com/ and ensure that you are logged in.
  3. Click Create New App.
  4. Complete the required application details and create the application. You can use any appropriate values for the application details. The connector does not use these values.
  5. After you have successfully created the application, select the Permissions tab.
  6. Change the permissions required to Read only.
  7. Select the Keys and Access Tokens tab.
  8. Copy the Consumer Key and Consumer Secret values. Use these values as the values for the app_key and app_secret parameters in your connector credentials configuration.
  9. Select Create my access token.
  10. Copy the Access Token and Access Token Secret values. Use these values as the values for the access_token and access_token_secret parameters in your connector credential configuration.

You can invalidate a previous token by generating a new token. In this case, you must use the Update Connector API to update the access_token and access_token_secret parameters for the connector.

Twitter Cloud Connector Configuration

This section outlines the configuration options that you can set for the twitter_cloud flavor connector. You can use these in the JSON object that you pass to the config parameter in the Create Connector API.

You configure the twitter_cloud flavor connector to retrieve tweets by using one of the options:

  • Search. You specify a search string that defines the kind of tweets you want to index.
  • Timeline. You specify the user name of a particular user, and index their timeline.
  • Stream. You specify a type of twitter stream, and the connector users the twitter streaming API to retrieve tweets. This option has further options to specify the type of tweets to index.

In your Twitter Cloud connector configuration, you must set exactly one of the search, timeline_screen_name, or stream_type parameters. The other options available depend on which value you choose.

Note: All configuration options are case sensitive.

Search Parameters

The following table outlines the configuration options to use to index tweets according to a search.

Parameter Type Description
search String The query string to use to retrieve tweets to index.
search_result_type Enum The type of results to return. You can use one of the following values:
  • recent. Search recent tweets. This is the default value.
  • popular. Search popular tweets.
  • mixed. Search recent tweets and popular tweets.
search_max_results Integer The maximum number of results to process in a single connector run. The default value is 0, which means that the connector retrieves all available search results.

Timeline Parameters

The following table outlines the configuration options to use to index tweets from the timeline of a particular user.

Parameter Type Description
timeline_screen_name String The screen name of the user whose timeline you want to index.

Stream Parameters

The following table outlines the configuration options to use to index tweets by using a stream.

Parameter Type Description
stream_type Enum The type of stream to use. You can use one of the following values:
  • statuses/filter. Stream filtered tweets. You must also set at least one of stream_track, stream_follow or stream_location to specify the results to filter.
  • statuses/sample. Stream a sample of tweets.
  • statuses/firehose. Stream all tweets. You must obtain special permission from Twitter to use this option.
stream_track Array<String> An array of keywords or phrases to track in your index stream. This option is available only when you set stream_type to statuses/filter.
stream_follow Array<String> An array of user names or IDs to follow in your index stream. This option is available only when you set stream_type to statuses/filter.
stream_location Array<Number> An array of longitude and latitude pairs that form bounding boxes of locations [W,S,E,N,...] that you want to include in your index steam. This option is available only when you set stream_type to statuses/filter.
stream_language Array<String> An array of BCP 47 language identifiers to restrict to (for example, en).

Example Configurations

{
	"search": "coffee",
	"search_result_type": "mixed"
}
{
	"timeline_screen_name": "bbcnews"
}
{
	"stream_type": "statuses/filter",
	"stream_location": [-0.6,51,0.4,52]
}

Twitter Cloud Connector Destination

This section outlines the options that you can set for the destination that the connector indexes into. You can use these in the JSON object that you pass to the destination parameter in the Create Connector API.

Note: All the options are case sensitive.

Required Parameters

Parameter Type Description
action Enum The action to take when indexing documents. You can use the following options:
  • addtotextindex. Add documents directly to a Haven OnDemand text index.

Parameters for Add to Text Index Action

The following parameters are required in the destination JSON object when action is set to addtotextindex

Parameter Type Description
index String The name of the text index that you want to index documents into. This index must already exist in Haven OnDemand (created by the Create Text Index API).

Example Destination

{
	"action": "addtotextindex",
	"index": "testindex"
}

Twitter Cloud Connector Schedule

This section outlines the options that you can set for the schedule that the connector runs on. You can use these in the JSON object that you pass to the schedule parameter in the Create Connector API.

Note: All the options are case sensitive.

Required Parameters

Parameter Type Description
frequency Object The frequency configuration that describes how often to run the connector.

The frequency object must contain the following parameter:

Parameter Type Description
frequency_type Enum The type of frequency configuration to use. This setting affects the other parameters that you must set in the frequency object. You can use one of the following values:
  • seconds. The connector frequency is set in seconds. You must also specify the interval parameter.

When you have set the frequency_type parameter to seconds, you must also set the following parameters:

Parameter Type Description
interval Integer The number of seconds between each connector run. This interval measures from the start of one connector run to the start of the next. The maximum interval is 31536000.
Note: The exact interval that the connector uses might vary by up to 30 minutes, depending on load on the system and the scheduler.

Optional Parameters

Parameter Type Description
occurrences Integer The number of times to attempt to schedule a connector run. If you do not set occurrences, the number of runs is unlimited. The schedule stops either after this number of runs, or when it reaches the configured end_date, whichever occurs first.
start_date String The date to start scheduling the connector. For a list of available date formats, see Date Formats for Parameters. If you do not set a start_date, the connector runs after the first interval elapses.
end_date String The date to stop scheduling the connector. For a list of available date formats, see Date Formats for Parameters. The schedule stops either after this date, or when it has run the number of times configured in occurrences, whichever occurs first.

Example Configuration

{
	"occurrences": 5,
	"start_date": "1",
	"end_date": "29/06/2015 12:00:00 -0600",
	"frequency": {
		"frequency_type": "seconds",
		"interval": 21600
	}
}

Schedule Errors

If an attempt to run a connector fails, for example because an error occurred on the system, the Connector Status and Connector History APIs return an error status in the response for the schedule. In this case, Haven OnDemand attempts to retry the connctor schedule up to three times. When a schedule fails, Haven OnDemand attempts to retry it the next time it scans the connector schedules (every minute).

If the schedule fails three times, Haven OnDemand stops the connector schedule. In this case, you must either use the Update Connector API to set a new schedule, or manually start the connector with the Start Connector API.

Twitter Cloud Connector Credentials

This section outlines the options that you can use to set credentials for the connector. You can use these in the JSON object that you pass to the credentials parameter in the Create Connector API. The credentials parameter is required for twitter_cloud flavor connectors.

Note: All the options are case sensitive.

Required Parameters

Parameter Type Description
app_key String The OAuth 1.0 application key value that the connector must use for the Twitter application.
app_secret String The OAuth 1.0 application secret value that the connector must use for the Twitter application.
access_token String The OAuth 1.0 access token that the connector must use for the Twitter application.
access_token_secret String The OAuth 1.0 access token secret that the connector must use for the Twitter application.

Example Connector Credentials

{
	"app_key": "GvPmjcLmMKETalUAvEBoy",
	"app_secret": "AoYnIKPrhXmw8FV2o5O97nXcqx5l7Mnqh7zQzvUdyp",
	"access_token": "216175673-0amdxmGfpg7mXgzJCZfIhblh0qISSmOxql1yiicUR",
	"access_token_secret": "AoYnIKPrhXmw8FV2o5O97nXcqx5l7EG7gks7kfGEB6"
}

Twitter Cloud Connector Credentials Policy

This section outlines the options that you can use to set the credentials policy for the connector. The credentials policy options define when the system can decrypt credentials. You can use these parameters in the JSON object that you pass to the credentials_policy parameter in the Create Connector API. The credentials_policy parameter is required if you have set credentials.

The credentials policy controls how Haven OnDemand manages decryption tokens for storing and decrypting the credentials that the connector uses to access the repository. You can obtain a decryption token from the Start Connector and Retrieve Config APIs, which require the decryption of the connector credentials. Haven OnDemand sends the decryption token to an email address that you specify in the credentials policy.

The credentials policy also specifies how long the decryption token is valid for. If you send an invalid token to one of the APIs that requires it, the API automatically generates and sends a new token to the email.

The credentials policy has its own expiration date. After this time, you must renew the policy with the Update Connector API.

Note: All the options are case sensitive.

Required Parameters

Parameter Type Description
notification_email String The email address to which to send information about connector activity.

Optional Parameters

Parameter Type Description Default
token_expiration Integer The number of seconds that a generated token remains valid. The expiration time is counted from the moment the token was generated. Every generated token is valid for the specified duration, and can be used for decryption a number of times specified by token_occurrences. After the token_expiration time, the token cannot be used, even if it has not had token_occurrences uses. When the token expires, a new token is generated, resetting the token_expiration time and token_occurrences. The minimum value is 1. 1800
token_occurrences Integer The number of times that a generated token can be used for decryption. Every generated token is valid for this number of uses, and can be used for a duration specified by token_expiration. After it has been used token_occurrences times, the token cannot be used, even if the token_expiration time has not been reached. When the number of uses are exhausted, a new token is generated, resetting the token_expiration time and token_occurrences. The minimum value is 1. 1
key_expiration String The duration that the credentials policy is valid for. When the key expires, the Haven OnDemand key management service returns an error stating that the policy has expired. For a list of available date formats, see Date Formats for Parameters. 3 months
notification_email_frequency Enum The frequency to use to send information about connector activity to the notification_email address. You can use the following values:
  • always. Always send email notifications for all connector activity.
  • on_decrypt. Send email notifications only when an attempt to decrypt connector credentials occurs.
  • on_failure. Send email notifications only when a failure occurs when using connector credentials.
  • never. Never send email notifications.
on_decrypt

Example Credentials Policy

{
	"notification_email": "test@example.com",
	"notification_email_frequency": "always",
	"key_expiration": "19/06/2015 11:25:00",
	"token_expiration": 3600,
	"token_occurrences": 10
}

Twitter Cloud Connector Limits

The twitter_cloud flavor Connector has the following limits:

Schedule

Property Max Limit
interval 31536000

Static_resource_unit_cost

It costs 1 static resource unit to create a twitter_cloud flavor connector.

Start_connector_unit_cost

It costs 5 start connector units to start a twitter_cloud flavor connector.