creating a small app with ES as backend

This is a series about a slack related app where we use ES as the backend for every event we would possibly like to store for analysis.

There are few things we need to consider and the first is related to the fields provided by the slack api that are related to dates and timestamps.

Most of them are ending with ts. By storing all events in ES we can then go and identify them.

date_fields="""
ts
event_ts
thread_ts
deleted_ts
timestamp
files.timestamp
files.created
root.ts
root.thread_ts
root.files.timestamp
attachments.ts
attachments.message_blocks.ts
previous_message.ts
previous_message.thread_ts
previous_message.root.ts
previous_message.root.thread_ts
previous_message.edited.ts
previous_message.attachments.ts
message.ts
message.thread_ts
message.root.thread_ts
message.root.ts
message.edited.ts
message.attachments.message_blocks.ts
message.attachments.ts
message.files.timestamp
message.files.created
message.latest_reply
"""

ES defaults to those will be type text which is not really useful. By looking at the elastic date link

mappings

The following will help construct mapping for the date fields:

mappings = {"properties":{}}
date_field = {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    }
for k in date_fields.split():
    mappings['properties'][k]=date_field

print(json.dumps(mappings, indent=2))

so it is

{
  "properties": {
    "ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "event_ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "thread_ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "deleted_ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "timestamp": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "files.timestamp": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "files.created": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "root.ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "root.thread_ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "root.files.timestamp": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "attachments.ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "attachments.message_blocks.ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "previous_message.ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "previous_message.thread_ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "previous_message.root.ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "previous_message.root.thread_ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "previous_message.edited.ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "previous_message.attachments.ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "message.ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "message.thread_ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "message.root.thread_ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "message.root.ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "message.edited.ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "message.attachments.message_blocks.ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "message.attachments.ts": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    },
    "message.files.timestamp": {
      "type": "date",
      "format": "strict_date_optional_time||epoch_second||epoch_millis"
    }
  }
}

reindexing

Whenever we do a mapping change we should reindex data from the source index to the one we changed the mapping in order for the data to be parsed according to the type of the filed we changed (in the destination index)

POST _reindex 
    {
      "source": {
        "index": "slack-messages"
      },
      "dest": {
        "index": "slack-messages_reindexed"
      }}

aliasing

It is expected to have mapping changes as you enhance or develop an app thus it is often recommended to get an alias in place to use when sending data. That will also provide the ability to perform changes without having to affect application and to minimise downtime

Have a look at mapping with 0 downtime to get an idea

kibana

So we have our data in es and we want to search them. Most of what we might want to do is already available.. For kibana to show the date histogram of our documents the timestamp field (here we marked ts field of slack messages for that) to be the one used for this purpose.

formating

Some slack messages or events contain urls in order to make them clickable in kibana you need to change the format of those fields so that are presented as links. Navigate to stack management (under kibana)index management <your index patterm> and find your field to do an edit. While in the “edit” of that field, enable set format and switch to Url.