Had an absolute surprise when I landed at a client – “to just fine one policy”. The entire policy list was gone and the GUI would only give me a “failed to load policies” after 2 mins of loading the page.

This is on BIGIP v13.1.1.2. 

After a little clicking around the GUI – turns out, not just the policies but the network map page, the device management overview page where you sync the devices and the ASM event logs also failing to load. 

This looked like the classic ASM db corruption cases, so we followed the K14194: Troubleshooting the BIG-IP ASM MySQL database. 

The output I got for the steps there: 

Determining the status of the BIG-IP ASM process

This did not give me any down errors for the db.

Determining MySQL status by verifying the MySQL processes

This looked all normal as well.

Determining overall health of MySQL database and table contents

When you run this, just be patient as this took a while on my device

mysqlcheck -p`perl -MF5::Cfg -e 'print F5::Cfg::get_mysql_password(user => q{root})'` -A

Most of the checks were OKs, except for these two:

mysql.func OK
mysql.general_log
Error : You can't use locks with log tables.
status : OK
mysql.help_category OK
mysql.help_keyword OK
mysql.help_relation OK
mysql.help_topic OK
mysql.host OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.servers OK
mysql.slow_log
Error : You can't use locks with log tables.
status : OK
mysql.tables_priv OK

Repairing tables in the MySQL database

Then you run the repair, as all tables do not support it you will see a lot of  “note : The storage engine for the table doesn’t support repair”

mysqlcheck -p`perl -MF5::Cfg -e 'print F5::Cfg::get_mysql_password(user => q{root})'` -A -r

But the ones which had the problem were repaired alright, I thought this was it, but nope.

mysql.general_log OK
mysql.help_category OK
mysql.help_keyword OK
mysql.help_relation OK
mysql.help_topic OK
mysql.host OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.servers OK
mysql.slow_log OK
mysql.tables_priv OK

Checking the httpd log. 

The GUI pages still did not show up after all this, so in /var/log/httpd/httpd_errors I could see:

Jun 16 11:35:11 Internet_F5_Primary err httpd[29641]: [error] server reached MaxClients setting, consider raising the MaxClients setting

I googled and followed: K9588: Error Message: httpd: [error] server reached MaxClients setting

Restart the daemons which handle the GUI and the ASM.

Follow this order.

bigstart restart httpd tomcat
bigstart restart restjavad
bigstart restart restnoded

After this, I could see all the policies and the remaining pages just like before.

So hope this helps someone google brings here when asked for “bigip asm failed to load policies error“.