Installing CellStore with Docker, Part 2: Database

These instructions are meant for the first installation only of CellStore. For updates, refer to the update instruction.

• 1: Docker
• 2: Database
• 3: Creating CellStore container
• 4: Starting CellStore
• 5: Creating CellStore service
• 6: Setup DTS cache
• 7: Additional Configuration and Optimization

CellStore uses an external database to store its data. Its architecture is generic and can be adapted to use a multitude of database products. For the best performance, the quality (latency and throughput) of the connection between the database and CellStore is important. Ideal conditions include a number of machines in the same LAN, Amazon Virtual Private Cloud, Amazon availability zone, or even the same machine.

This version of CellStore supports two Database systems:

• MongoDB v3.6 (Community or Enterprise Edition) or later.
• MarkLogic Server v9.
• MarkLogic Server v10.

The MongoDB setup can be extended with additional full-text search capabilities by using Elasticsearch (Version 6.x and later).

MongoDB

MongoDB 3.6 can be installed natively on:

• Amazon Linux 2013.03 and later
• Debian 7
• Debian 8
• RHEL/CentOS 6.2 and later
• SLES 11
• SLES 12
• Solaris 11 64-bit
• Ubuntu 12.04
• Ubuntu 14.04
• Ubuntu 16.04
• Windows Server 2008R2 and later
• Windows Vista and later
• OS X 10.7 and later

To install MongoDB natively, follow these installation instructions. Alternatively, you can use this Docker Container.

In order to support the search parameter for full-text search capabilities, you need to create text indexes to support these types of queries. We suggest to create the following indexes:

db.labels.createIndex(
  { "aspects.xbrl:value" : "text" },
  { default_language: "en" },
  { language_override: "aspects.iso2:lang" }
);
db.labelsGroupIndex.createIndex(
  { "aspects.xbrl:value" : "text" },
  { default_language: "en" },
  { language_override: "aspects.iso2:lang" }
);
db.facts.createIndex(
  { "aspects.xbrl:value" : "text" },
  { default_language: "en" },
  { language_override: "aspects.iso2:lang" }
);
db.factsGroupIndex.createIndex(
  { "aspects.xbrl:value" : "text" },
  { default_language: "en" },
  { language_override: "aspects.iso2:lang" }
);

Not all languages supported

Currently, mongodb only supports a limited number of languages. Switching to an explicit language for each request can be achieved by adding parameter locale to a request: e.g.: /cellstore/v1/facts?search=assets&locale=de.

Elasticsearch

Follow the installation documentation from the Elasticsearch documentation. No index mappings need to be setup. All of this is managed by CellStore automatically. Just follow the configuration section for Elasticsearch.

MarkLogic

MarkLogic can be installed natively on several supported platforms. Please follow the Installation Guide for the basic setup.

In order to prepare a database for CellStore you need to:

1. Open Marklogic Admin UI (most likely: http:// :8001/
2. Click Databases > Create
3. Use configuration (at least the ones that are mandatory) and click ok:
   • Name: (**mandatory**)
   • stemmed searches: basic (optional)
   • word searches: false (optional)
4. Click Forests > Create
5. Use configuration and click ok:
   • Name: (**mandatory**)
6. Click Databases > your database > Forests
7. Click attached for the forest you’ve created
8. Click Groups > Defaults > App Servers > Create HTTP
9. Use configuration and click ok:
   • Name: (**mandatory**)
   • root: / (mandatory)
   • port: , e.g. 8010 (**mandatory**)
   • database: choose (**mandatory**)
   • request timeout: 1200 (optional)
   • default time limit: 1200 (optional)
   • default error format: json (mandatory)
   • error handler: /MarkLogic/rest-api/error-handler.xqy (mandatory)
   • url rewriter: /MarkLogic/rest-api/rewriter.xml (mandatory)
   • rewrite resolves globally: true (mandatory)
10. Click Databases > > Field Range Indexes
11. Add an index for:
    • scalar type: string, localname: _gKey, collation: http://marklogic.com/collation/ Root Collation, range value positions: false, invalid values: reject
    • scalar type: string, localname: xbrl:instance, collation: http://marklogic.com/collation/ Root Collation, range value positions: false, invalid values: reject
    • scalar type: string, localname: xbrl:period, collation: http://marklogic.com/collation/ Root Collation, range value positions: false, invalid values: reject
    • scalar type: string, localname: xbrl:entity, collation: http://marklogic.com/collation/ Root Collation, range value positions: false, invalid values: reject
    • scalar type: string, localname: xbrl:concept, collation: http://marklogic.com/collation/ Root Collation, range value positions: false, invalid values: reject
    • scalar type: string, localname: xbrl:dts, collation: http://marklogic.com/collation/ Root Collation, range value positions: false, invalid values: reject
    • scalar type: string, localname: xbrl:elementName, collation: http://marklogic.com/collation/ Root Collation, range value positions: false, invalid values: reject
12. Increase the expanded tree cache to at least 2048 (mandatory)

Some helpful queries to update the timeouts and cache size (execute them from the Query Console - most likely http// :8000/qconsole):

xquery version "1.0-ml";

import module namespace admin = "http://marklogic.com/xdmp/admin"
		  at "/MarkLogic/admin.xqy";

let $server-name := "reportix-cellstore"

let $config := admin:get-configuration()
let $groupid := admin:group-get-id($config, "Default")
let $appserverId := admin:appserver-get-id($config, $groupid, $server-name)
let $newConfig := admin:appserver-set-request-timeout($config, $appserverId, 1200)
let $newConfig := admin:appserver-set-max-time-limit($newConfig, $appserverId, 3600)
let $newConfig := admin:appserver-set-session-timeout($newConfig, $appserverId, 3600)
let $newConfig := admin:appserver-set-default-time-limit($newConfig, $appserverId, 3600)
let $reportix-config := $newConfig/*:groups/*:group/*:http-servers/*:http-server[*:http-server-name eq $server-name]
return ( admin:save-configuration($newConfig), $reportix-config, $newConfig )

xquery version "1.0-ml";

import module namespace admin = "http://marklogic.com/xdmp/admin"
		  at "/MarkLogic/admin.xqy";

let $config := admin:get-configuration()
let $groupid := admin:group-get-id($config, "Default")
return (
  admin:save-configuration(admin:group-set-expanded-tree-cache-size($config, $groupid, 4096))
)

Only the database configured language supported for text search

Text search using the search parameter for the facts and labels endpoints works out-of-the-box with MarkLogic. However, MarkLogic supports only searches in the language that the database has been configured with. So you need to make sure to use the propper locale parameter for each search, e.g.: /cellstore/v1/facts?search=assets&locale=de (in case the database language is set to de).

On to Part 3 »