Server will NOT start. It announces itself, starts to load and crashes a few lines into the start-up.
After hours-and-hours of troubleshoot and more time w/IBM, we found that we were crashing dirman process. Dbdirman indexes / gathers the list of databases on the server for dbdirman.nsf.
Turns out that some of the temp files can be incompatible with dbdirman and crash it.
Moreover, before the crashes started, the temp files were causing the Files tab to take up to 5 minutes to load instead of 5-10 seconds. At first we and IBM thought that the shear number might be the issue, however doing the delete half and half again, found that as we halfed over-and-over what we left in the server, we were left with just a few files in the xsp and xspstate folders that caused the Domino server to crash.
It was decided to fix both issues to move the XPage temporary folders outside the /local/notesdata folder structure so that dbdirman and the files tab could not see them. (e.g. /local/xpagetmpfiles).
1. Create the new folder structure
# mkdir /local/xpagetmpfiles
# chown notes:notes -R /local/xpagetmpfiles
2. Copy the sample properties file to a new file and update the file.
# cd /local/notesdata/properties
# cp xsp.properties.sample xsp.properties
# vi xsp.properties
3. In the file, remove the # comment and set the following parameters:
Under the # FILE UPLOAD header, set/update the following line(s):
Note: The first line is the attachment upload size if you find the default too small.
Under the # JSF PERSISTENCE header, set/update the following lines:
4. Save the changes. (<esc> :wq)
5. Delete / clean-up the current temp folders.
6. Start the server again.
# service domino start