Repository inconsistency

Common causes of corruptions in the repository are incorrect shutdown while the repository is writing to the database due to power failure, process killed, Runtime.halt() called, VM crash, Failure to finish transactions due to JVM errors such as OOME or core dump. Here are symptoms that help you identify a corrupt repository and some remedies.

Symptoms

The following types of messages are symptomatic of corruptions in the repository:

Orphaned child
Error: NodeState CHILD_UUID references inexistent parent uuid PARENT_UUID
Resolution: Use Magnolia tools to remove the node.

Parent referencing inexistent child
Error: NodeState PARENT_UUID references inexistent child {}CHILD_NAME with id CHILD_UUID
Resolution: consistency fix

Child referencing invalid parent
Error: ChildNode has invalid parent uuid: INVALID_PARENT_UUID (instead of VALID_PARENT_UUID)
Resolution: Use Magnolia tools to remove the node.

Parent not referencing existing child
Error: javax.jcr.ItemNotFoundException: failed to build path of CHILD_UUID: PARENT_UUID has no child entry for CHILD_UUID
Resolution: Use Magnolia tools to remove the node.

Node already exists
Error: javax.jcr.ItemExistsException: <node_path> at org.apache.jackrabbit.core.NodeImpl.internalAddChildNode(NodeImpl.java:766)
Resolution: consistency check

Search index inconsistency
Error: WARN NodeIteratorImpl: Exception retrieving Node with UUID: 003171fe-e2e8-457b-a3af-f74eed12c1b9: javax.jcr.ItemNotFoundException: 003171fe-e2e8-457b-a3af-f74eed12c1b9
Resolution: recreate search index

Logging levels and consistency checks

To obtain a more detailed information about consistency checks in the logs, make sure you also have the appropriate setting in the log4j configuration file:

    <Logger name="org.apache.jackrabbit" level="info">
      <AppenderRef ref="console"/>
    </Logger>

Together with a consistency check for a workspace enabled (see below), the logger would print the following into the log file:

INFO org.apache.jackrabbit.core.RepositoryImpl 16.10.2018 09:48:32 -- initializing workspace 'website'...
INFO org.apache.jackrabbit.core.fs.local.LocalFileSystem 16.10.2018 09:48:32 -- LocalFileSystem initialized at path /m/mgnl-bundles/apache-tomcat/webapps/magnoliaAuthor/repositories/magnolia/workspaces/website/default
INFO org.apache.jackrabbit.core.persistence.bundle.ConsistencyCheckerImpl 16.10.2018 09:48:32 -- website: checked 601 bundles.
INFO org.apache.jackrabbit.core.query.lucene.SearchIndex 16.10.2018 09:48:32 -- Index initialized: /m/mgnl-bundles/apache-tomcat/webapps/magnoliaAuthor/repositories/magnolia/workspaces/website/index Version: 3
INFO org.apache.jackrabbit.core.RepositoryImpl 16.10.2018 09:48:32 -- workspace 'website' initialized

See especially line 3 in the result log.

Search index

You can run a search index consistency check for each workspace on startup. The workspace.xml files are available in /<CATALINA_HOME>/webapps/<contextPath>/repositories/magnolia/workspaces/*.

In the relevant workspace.xml file add two parameters in the <SearchIndex class="…"> element:

<param name="enableConsistencyCheck" value="true"/>
<param name="forceConsistencyCheck" value="true"/>

The autoRepair parameter controls whether errors should be repaired or only logged.

<param name="autoRepair" value="false"/> <!-- default is true -->

Example:

<SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
    <param name="path" value="${wsp.home}/index"/>
    <param name="enableConsistencyCheck" value="true"/>
    <param name="forceConsistencyCheck" value="true"/>
    <param name="autoRepair" value="true"/>
</SearchIndex>

Workspace

You can run a consistency check for each workspace on startup by modifying the individual workspace.xml files. The workspace.xml files are in /<CATALINA_HOME>/webapps/<contextPath>/repositories/magnolia/workspaces/*.

In the relevant workspace.xml file add the following parameter in the < PersistenceManager class="…"> element:

<param name="consistencyCheck" value="true" />

Restart Magnolia and the persistence manager will perform a consistency check and report the results in a log file.

Some inconsistencies can be fixed by adding the following parameter to workspace.xml file. This should fix missing children, but will not fix missing references or problems caused by missing versions.

<param name="consistencyFix" value="true" />

Clusters

In a clustered environment consistency checks and fixes can only run on one cluster node at a time. Don’t start other cluster nodes while the consistency fix is running. After it is finished, the other cluster nodes can be started.

Recreating search indexes

Search index folders are at /<CATALINA_OUT>/webapps/<contextPath>/repository/magnolia/workspaces/<workspace name>/index.

To recreate the search indexes:

Stop Magnolia.
Make a backup copy of all index folders.
Delete all index folders.
The indexes will be recreated on startup.

If a workspace performs a re-indexing on startup and finds an inconsistency, the re-indexing process will fail. Add the OnWorkspaceInconsistency parameter in the workspace.xml file to prevent the failure. The process will just log the inconsistency.

<param name="onWorkspaceInconsistency" value="log"/>

Feedback

DX Core