Content Manager 9.2 Nuggets

in

Here's a collection of changes introduced with 9.2....

IDOL is gone, sort of

For the end-users absolutely nothing has changed.  They will still search like they always have done.  However, document content indexing should now be implemented with ElasticSearch.  If you aren't on the bandwagon with that, you've probably invested in other uses for IDOL.  You can still configure 9.2 to use IDOL if you need.  However, for everyone else ElasticSearch is the future.  

2017-12-02_22-28-34.png

You can see that both are still referenced on the administration ribbon.  In my environment I have no IDOL components, but the toolbar options are still there.  I'll happily ignore the IDOL ones.  

Elasticsearch build-out

You can quickly find yourself unsure how to resource this new technology.  The course I took quickly rationalized 9 servers and over 128GB RAM for a database of just 1 million documents.  Below is the diagram used for that example.  Although entirely unrealistic I'm sure, it highlights what's meant by "elastic".  

2017-12-02_22-33-09.png

As time permits I'll post more about how to properly design an ElasticSearch infrastructure.  KeyView is still used to generate the content for the engine, and the engine is managed via rest commands.  However, unlike IDOL, there are really cool user interfaces for managing it (some at a minor cost).

Navigating via Keyboard 

When working with tree views you can no longer use arrow keys to expand/collapse a tree.   For example when trying to expand a category....

Navigating categories

Navigating categories

To expand this you can press + or -.  Alternatively, you can press control + asterisk on the numeric keypad.  Doing so will expand the structure recursively.  As noted in the comments, a bug report has been raised for this issue.

Webclient Record Nesting

You can now expand a container and see the contents.  You can even expand contents and continue drilling down. 

2017-12-02_20-32-08.png

Webclient Tabs

Tabs now appear as you perform actions.  You could have a search result in one tab, a new location in another, and saved searches in a third. I do not see any option to have tabs auto-open at startup (like the thick client has).

2017-12-02_20-21-56.png

Metadata Validation Rules

There's a new feature that aims to solve some of the validation challenges we have with metadata.  You can find the feature off the administration ribbon...

Right-click->New to create a new one

2017-12-02_16-23-35.png

Here's the configuration dialog...

2017-12-02_16-24-57.png

Silly me tries something silly....

Does not compute...

Does not compute...

The help file says...

2017-12-02_16-27-32.png

LookupSet Changes

Many new options with this one.  Here's the property page for a "States" lookup set.  Note how I've enabled both short and long values on the items.

2017-12-02_15-56-33.png

With this configuration the lookup item property page looks like below...

Both Short and Long Value

Both Short and Long Value

Here's what a few entries look like with comments enabled...

2017-12-02_16-01-41.png

Hierarchical Lookup Items

I created a lookup set name "Old Projects" and defined it as follows:

2017-12-02_20-45-17.png

Then I created a few entries, with a few of those having sub-entries.

2017-12-02_20-42-37.png

Here's how it appears to the end-users...

Error message shown if the user doesn't pick a "lowest level" item.

Error message shown if the user doesn't pick a "lowest level" item.

Once a valid value is selected, this is how it is saved into the field....

2017-12-02_20-48-58.png

Lookup Items in Custom Properties

You can now pick Lookup Item when selecting the format of a custom property, as shown below...

2017-12-02_16-03-25.png

Then you pick the lookup set from which the user will pick one item...

2017-12-02_16-04-47.png

Then you give it a name...

2017-12-02_16-06-13.png

For demonstration purposes I'll associate it with categories...

2017-12-02_16-07-55.png

Before demonstrating it, I created another custom property of type string.  I made this field use the same lookup set as its' source.  Then I also associated it with the category object.

2017-12-02_16-09-29.png

Next I selected Florida for both fields on an example category (the user experience was identical when selecting a value)...

2017-12-02_16-10-33.png

After saving, I can see one major difference in the view pane....

The Lookup Item is a hyperlink whereas the string is just text.  

The Lookup Item is a hyperlink whereas the string is just text.  

Regardless which route you select, modifications to the Lookup Item are reflected across the dataset.  In the screenshot below you can see how the selected state on the category shows a new value, even though I only updated the Lookup Item (and not the category).

Change is reflected even though I didn't modify category

Change is reflected even though I didn't modify category

SQL Database Changes

Schema Changes:

1 table removed, 3 views removed, 5 tables added, and two new encryption functions

1 table removed, 3 views removed, 5 tables added, and two new encryption functions

Across the board it appears all VARCHAR type columns have now been converted to NVARCHAR.  Below you can see some of the results of a schema comparison.  I've highlighted the example column.

2017-12-02_15-37-19.png

Check In Style Email Clean-up Option

In the previous version you could "Keep email in the mail system" and/or "Move deleted email to Deleted Items" (or neither).  Now you can only select one of these three options: Permanent Delete, Move to Deleted Items, Retain in Mail System.  

9.2 Options

9.2 Options

9.1 Options

9.1 Options

Check-in Styles Link off Administration Ribbon

Administrators can gain access to all check-in styles from here...

2017-12-02_14-58-57.png

Classification/Category Specific Record Types

Limit which record types can be associated with classification/category...

2017-12-02_14-33-36.png

If a user attempts to pick it they get an error...

2017-12-02_14-36-54.png

Automatically Declare as Final

I would think this makes sense for email, but here's what the help says....

“If the document store is enabled for SEC compliance then this option will be enabled, see About SEC Rule 17a-4 Compliance rules. Whenever a record is created that has a document attached, the record will be automatically finalized at the time it is saved to the database. If you attach a document to an existing record that doesn’t have a document attached, then that record will be finalized. The options for creating revisions are prevented.”

Here's the option on the record type:

2017-12-02_14-26-37.png

Default Copy Style for Referenced Access Controls

When creating a record type or classification you can define the behavior of referenced access controls. How you go about accessing the options has changed.  Clicking copy style shows you a popup. 

2017-12-02_14-18-22.png

In the previous version you accessed it from a second tab...

2017-12-02_14-22-19.png

Update Last Action Date when Viewing

In previous versions there was a global setting that controlled whether the last action date is to be updated when viewing a record.  This concept has now been moved to the record type level.  There is no longer a global setting for this behavior.

2017-12-02_14-12-35.png

Confirm Preview of Document

When enabled the user must click anywhere within the preview window, confirming their intention to view the record.  This was always a personal concern as an administrator may accidentally leave the preview pane enabled whilst they peruse search results (with no intention or desire to actually see the record).

2017-12-02_14-12-00.png

Toggle this setting on the record type's audit property page

2017-12-02_14-12-35.png

Feature Activation Required

If you attempt to enable the External Stores, Manage In Place, EMC Storage, or iTernity Content Addressable Storage features, you'll be prompted for an activation key for each...

2017-12-02_14-42-11.png

Render to PDF after save

This one has been here for a few versions I guess, but I just noticed it...

2017-12-02_14-30-18.png

Goodbye IDOL, Hello Elasticsearch

in

Yay!  As of 9.2, IDOL is no longer the default content index engine.  This is cool on many levels.  In its' place the developers have introduced Elasticsearch.  Time to dive in....

First I installed the Java Runtime Environment on the server...

2017-12-01_23-46-59.png
2017-12-01_23-47-16.png
2017-12-01_23-48-47.png

Then I installed ElasticSearch....

 
2017-12-01_23-51-19.png
 
 
2017-12-01_23-51-45.png
 
 
 
 
 
 
 

To test that it's working I use powershell as described in the documentation...

2017-12-02_0-03-39.png

Now over in the Enterprise Studio I can create a new content index.  Since it defaults to the current machine I should not need to change any settings.

2017-12-02_0-10-37.png

Next I need to enable the content indexing event....

2017-12-02_0-13-00.png

I dropped in all of the documentation from the source ISO and watched it process them almost immediately..

2017-12-02_0-17-17.png

I went back to the client and searched for a keyword and did receive results.  

2017-12-02_0-19-00.png

ElasticSearch is very cool!  Hopefully I'll get some time to post more about it.  I'm happy IDOL is gone in 9.2.

Correcting record types and numbers

There's an interesting question posted on the HPE Content Manager forum.  This post will be a response to that question....

DataPort can only update an existing record if it uses the expanded number (non-compressed record number) as the unique ID for searching.  If your spreadsheet contains the record number and you match it to the expanded number field, it will first attempt to update before creating the record.  This behavior removes any ability to change the record number property via DataPort.

I mocked up the environment with the record types described and was able to generate the picture below.  The goal, I believe, is to change the pink document records (Personal) to become teal document records (Corporate).

2017-12-01_15-51-11.png

To model this environment I created a powershell script that built out these records.  This assumes you already have the 4 described record types (or you've modified this script to accommodate your environment). 

WARNING: This script creates one folder with number "17/99999" and removes any records contained inside.  It is not intended to be run in a production environment.  It is only used to prove the final solution works correctly.

#Prepare the host and load the CM SDK/Database
Clear-Host
Add-Type -Path "d:\Program Files\Hewlett Packard Enterprise\Content Manager\HP.HPTRIM.SDK.dll"
$Database = New-Object HP.HPTRIM.SDK.Database
$Database.Connect()
 
#Load the various record types
$PersonalFolderRecordType = $Database.FindTrimObjectByName([HP.HPTRIM.SDK.BaseObjectTypes]::RecordType, "Personal Files")
$PersonalDocumentRecordType = $Database.FindTrimObjectByName([HP.HPTRIM.SDK.BaseObjectTypes]::RecordType, "Personal Document")
$CorporateFolderRecordType = $Database.FindTrimObjectByName([HP.HPTRIM.SDK.BaseObjectTypes]::RecordType, "Corporate Files")
$CorporateDocumentRecordType = $Database.FindTrimObjectByName([HP.HPTRIM.SDK.BaseObjectTypes]::RecordType, "Corporate Document")
 
#Prepare the example set
$FolderNumber = "17/99999"
$ExemplarFolder = $Database.FindTrimObjectByName([HP.HPTRIM.SDK.BaseObjectTypes]::Record, $FolderNumber)
if ( $ExemplarFolder -eq $null ) {
    $ExemplarFolder = New-Object HP.HPTRIM.SDK.Record -ArgumentList $Database, $CorporateFolderRecordType
    $ExemplarFolder.LongNumber = $FolderNumber    #Assumes pattern is "YY/GGGGG"
    $ExemplarFolder.TypedTitle = "test"
    $ExemplarFolder.Save()
    Write-Host "Created folder $($ExemplarFolder)"
} else {
    #purge any existing records
    $ExistingContents = New-Object HP.HPTRIM.SDK.TrimMainObjectSearch -ArgumentList $Database,Record
    $ExistingContents.SearchString = "container:[uri:$($ExemplarFolder.Uri)]"
    foreach ( $Result in $ExistingContents ) 
    {
        Write-Host "Removing existing record $(([HP.HPTRIM.SDK.Record]$Result).Number)"
        ([HP.HPTRIM.SDK.Record]$Result).Delete()
    }
}
 
#Create 4 records within the folder, the first 2 personal and the second 2 corporate
for ( $i = 0; $i -lt 4; $i++ ) {
    $ExemplarDocument = New-Object HP.HPTRIM.SDK.Record -ArgumentList $Database, $(If ( $i -lt 2 ) { $PersonalDocumentRecordType } Else { $CorporateDocumentRecordType })
    $ExemplarDocument.DateRegistered = (get-date).AddYears(-1*$i)
    $ExemplarDocument.DateCreated = (get-date).AddYears(-1*$i)
    $ExemplarDocument.TypedTitle = "test"
    $ExemplarDocument.SetContainer($ExemplarFolder,$true)
    $ExemplarDocument.Save()
    Write-Host "Created document $($ExemplarDocument.Number)"
}

When I run the script above I get one folder with four documents.  If I run it a second time the folder will be re-used, any existing documents will be removed, and the original sample will be re-created.

2017-12-01_23-29-25.png

In a new script I craft a high-level approach to resolving the problem.  First I find all corporate folders.  Then for each folder found, I search within for any personal documents.  I'll process each one and then continue until all found documents and folders are processed.

#Prepare the host and load the CM SDK/Database
Clear-Host
Add-Type -Path "d:\Program Files\Hewlett Packard Enterprise\Content Manager\HP.HPTRIM.SDK.dll"
$Database = New-Object HP.HPTRIM.SDK.Database
$Database.Connect()
 
#Load the various record types
$PersonalFolderRecordType = $Database.FindTrimObjectByName([HP.HPTRIM.SDK.BaseObjectTypes]::RecordType, "Personal Files")
$PersonalDocumentRecordType = $Database.FindTrimObjectByName([HP.HPTRIM.SDK.BaseObjectTypes]::RecordType, "Personal Document")
$CorporateFolderRecordType = $Database.FindTrimObjectByName([HP.HPTRIM.SDK.BaseObjectTypes]::RecordType, "Corporate Files")
$CorporateDocumentRecordType = $Database.FindTrimObjectByName([HP.HPTRIM.SDK.BaseObjectTypes]::RecordType, "Corporate Document")
 
#Find all corporate folders
$CorporateFolders = New-Object HP.HPTRIM.SDK.TrimMainObjectSearch -ArgumentList $Database, Record
$CorporateFolders.SearchString = "type:[uri:$($CorporateFolderRecordType.Uri)]"
foreach ( $CorporateFolder in $CorporateFolders ) {
    #Find and process all personal documents
    $PersonalDocuments = New-Object HP.HPTRIM.SDK.TrimMainObjectSearch -ArgumentList $Database, Record
    $PersonalDocuments.SearchString = "container:$($CorporateFolder.Uri) type:[uri:$($PersonalDocumentRecordType.Uri)]"
    Write-Host "Folder $($CorporateFolder.Number) has $($PersonalDocuments.Count) documents to fix"
    foreach ( $PersonalDocument in $PersonalDocuments ) {
        #Change record type and number
        Write-Host "Fixing Document $($PersonalDocument.Number)"
    }
}

Running this script gives me the following results...

2017-12-01_16-06-01.png

Sweet!  Now I just need some logic that actually fixes the record.  First I'll have it change the record type.  Then I'll figure out the appropriate prefix for the record's new record number (D17#, D16#, D11#, whatever).  With that in-hand I'll search for all records starting with that prefix, sorted to give me the last number first.  I grab that number and add one.  If there are no records from that year (which is the case in my scenario from the powershell script above) then I use 1 as the starting point.  

$PersonalDocument.RecordType = $CorporateDocumentRecordType
        #Figure out the correct number
        $Prefix = "D$($PersonalDocument.DateRegistered.Year.ToString().Substring(2))"
        $LastRecords = New-Object HP.HPTRIM.SDK.TrimMainObjectSearch -ArgumentList $database, Record
        $LastRecords.SearchString = "number:$($prefix)*"
        $LastRecords.SetSortString("number-")
        if ( $LastRecords.Count -eq 0 ) {
            $RecordNumber = "$($prefix)#1"
        } else {
            $Enumerator = $LastRecords.GetEnumerator()
            $Enumerator.MoveNext()
            $Record = [HP.HPTRIM.SDK.Record]$Enumerator.Current
            $LastNumber = $Record.Number.Replace("$($prefix)#","")
            $RecordNumber = "$($prefix)#$([convert]::ToInt32($LastNumber)+1)"
        }
        $PersonalDocument.LongNumber = $RecordNumber
        $PersonalDocument.Save()

After I run this I can see the changes within the client. Success!

2017-12-01_16-25-29.png

It's worth pointing out that in the newer versions of CM you can prevent users from storing personal documents into corporate containers.  

The complete script that fixes all records in the dataset is as follows:

#Prepare the host and load the CM SDK/Database
Clear-Host
Add-Type -Path "d:\Program Files\Hewlett Packard Enterprise\Content Manager\HP.HPTRIM.SDK.dll"
$Database = New-Object HP.HPTRIM.SDK.Database
$Database.Connect()
 
#Load the various record types
$PersonalFolderRecordType = $Database.FindTrimObjectByName([HP.HPTRIM.SDK.BaseObjectTypes]::RecordType, "Personal Files")
$PersonalDocumentRecordType = $Database.FindTrimObjectByName([HP.HPTRIM.SDK.BaseObjectTypes]::RecordType, "Personal Document")
$CorporateFolderRecordType = $Database.FindTrimObjectByName([HP.HPTRIM.SDK.BaseObjectTypes]::RecordType, "Corporate Files")
$CorporateDocumentRecordType = $Database.FindTrimObjectByName([HP.HPTRIM.SDK.BaseObjectTypes]::RecordType, "Corporate Document")
 
#Find all corporate folders
$CorporateFolders = New-Object HP.HPTRIM.SDK.TrimMainObjectSearch -ArgumentList $Database, Record
$CorporateFolders.SearchString = "type:[uri:$($CorporateFolderRecordType.Uri)]"
foreach ( $CorporateFolder in $CorporateFolders ) {
    #Find and process all personal documents
    $PersonalDocuments = New-Object HP.HPTRIM.SDK.TrimMainObjectSearch -ArgumentList $Database, Record
    $PersonalDocuments.SearchString = "container:$($CorporateFolder.Uri) type:[uri:$($PersonalDocumentRecordType.Uri)]"
    Write-Host "Folder $($CorporateFolder.Number) has $($PersonalDocuments.Count) documents to fix"
    foreach ( $PersonalDocument in $PersonalDocuments ) {
        #Change record type and number
        Write-Host "Fixing Document $($PersonalDocument.Number)"
        $PersonalDocument.RecordType = $CorporateDocumentRecordType
        #Figure out the correct number
        $Prefix = "D$($PersonalDocument.DateRegistered.Year.ToString().Substring(2))"
        $LastRecords = New-Object HP.HPTRIM.SDK.TrimMainObjectSearch -ArgumentList $database, Record
        $LastRecords.SearchString = "number:$($prefix)*"
        $LastRecords.SetSortString("number-")
        if ( $LastRecords.Count -eq 0 ) {
            $RecordNumber = "$($prefix)#1"
        } else {
            $Enumerator = $LastRecords.GetEnumerator()
            $Enumerator.MoveNext()
            $Record = [HP.HPTRIM.SDK.Record]$Enumerator.Current
            $LastNumber = $Record.Number.Replace("$($prefix)#","")
            $RecordNumber = "$($prefix)#$([convert]::ToInt32($LastNumber)+1)"
        }
        $PersonalDocument.LongNumber = $RecordNumber
        $PersonalDocument.Save()
    }
}