When an ASM database crashed for one of our clients and the database repair options failed, TAC recommended a full ASM DB reset on the lines of K6992

The KB article doesn’t really explain the full behavior.

As per the TAC, the policy name remains as it is, think of it as a “container”. This is because the virtual servers are referring to these policies hence the script keeps them intact but deletes all the “insides” of the policies, stuff like parameters, URLs and so on.

You then need to do a simple config sync from the peer unit to get all of that back.

However, once you run the script, the behavior is a bit tricky.

The policies all disappeared and over the next few minutes those policies keep coming up and the count goes up one by one and slooowwly

This looks a bit weird when you first see it, so just relax it all should be fine.

Our peer device had a count of 50 policies and the one on which I did the reset went up to 45 and stopped.

At this stage, I just synced from the peer unit, and again the policy count started going up one at a time and eventually reached 49. I am guessing it deleted some of the policies which were not assigned to a virtual server.

The F5 kb article indeed is very wanting on the expected behavior.

Had an absolute surprise when I landed at a client – “to just fine one policy”. The entire policy list was gone and the GUI would only give me a “failed to load policies” after 2 mins of loading the page.

This is on BIGIP v13.1.1.2. 

After a little clicking around the GUI – turns out, not just the policies but the network map page, the device management overview page where you sync the devices and the ASM event logs also failing to load. 

This looked like the classic ASM db corruption cases, so we followed the K14194: Troubleshooting the BIG-IP ASM MySQL database. 

The output I got for the steps there: 

Determining the status of the BIG-IP ASM process

This did not give me any down errors for the db.

Determining MySQL status by verifying the MySQL processes

This looked all normal as well.

Determining overall health of MySQL database and table contents

When you run this, just be patient as this took a while on my device

mysqlcheck -p`perl -MF5::Cfg -e 'print F5::Cfg::get_mysql_password(user => q{root})'` -A

Most of the checks were OKs, except for these two:

mysql.func OK
mysql.general_log
Error : You can't use locks with log tables.
status : OK
mysql.help_category OK
mysql.help_keyword OK
mysql.help_relation OK
mysql.help_topic OK
mysql.host OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.servers OK
mysql.slow_log
Error : You can't use locks with log tables.
status : OK
mysql.tables_priv OK

Repairing tables in the MySQL database

Then you run the repair, as all tables do not support it you will see a lot of  “note : The storage engine for the table doesn’t support repair”

mysqlcheck -p`perl -MF5::Cfg -e 'print F5::Cfg::get_mysql_password(user => q{root})'` -A -r

But the ones which had the problem were repaired alright, I thought this was it, but nope.

mysql.general_log OK
mysql.help_category OK
mysql.help_keyword OK
mysql.help_relation OK
mysql.help_topic OK
mysql.host OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.servers OK
mysql.slow_log OK
mysql.tables_priv OK

Checking the httpd log. 

The GUI pages still did not show up after all this, so in /var/log/httpd/httpd_errors I could see:

Jun 16 11:35:11 Internet_F5_Primary err httpd[29641]: [error] server reached MaxClients setting, consider raising the MaxClients setting

I googled and followed: K9588: Error Message: httpd: [error] server reached MaxClients setting

Restart the daemons which handle the GUI and the ASM.

Follow this order.

bigstart restart httpd tomcat
bigstart restart restjavad
bigstart restart restnoded

After this, I could see all the policies and the remaining pages just like before.

So hope this helps someone google brings here when asked for “bigip asm failed to load policies error“. 

In a recent case, with BigIP and Cisco webex servers we saw the SSL would break in the browser even while the configuration had no apparent errors.

The VIP on the BigIP hosted the webex over an SSL connection using a client-ssl profile and also used a server-ssl profile.

Cisco webex does not support the solution over HTTP port 80 and hence a server-ssl profile becomes a must. 

The perplexing part is also how different browsers dealt with the errors. 

Chrome gives you the option to accept the certificate error but then just refreshes the page and comes back to the same page. 

Mozilla Firefox just gives you a “secure connection failed” with nothing explained. 

Resolution

All this while we were trying to access the webex console in the browser using the IP address only, but in the end, we just decided to give a go with a URL and added a hostname entry to the windows hosts file. 

Luckily we chose the name “meeting.domain.com” and the webex console opened perfectly fine with a trusted certificate. Later on trying with the meet.domain.com, the connection started failing again as before.

I scoured the internet for some document which explains whether webex server requires the word “meeting” to be present in the hostname and didn’t find a thing.