f5 BIG-IP LTM – Allwyn Mascarenhas

Allwyn August 18, 2020

f5 BIG-IP LTM, F5 Certifications

Getting Started With F5 BIGIP LTM — Preparation for the 301a 301b LTM Specialist Exams

People often ask for a simple to follow start-up guide to get started on the F5 BIGIP product suite.

Here is an attempt to create one, covering the most basic features and concepts one comes across when working with the BIGIP LTM module.

This guide contains resources one can use to get a thorough understanding of the LTM module and prepare for the F5 301a and 301b exams. It covers, concepts one should know, and some troubleshooting advice if you run into trouble.

Start with A Video Course

Video tutorials are the easiest to start with as they give you a guided overview of the entire GUI with a decent enough introduction.

Start with a video course like the ones on Udemy or the CBT nuggets Keith Barker series.
Courses usually contain the introductory level and the easiest material only when it comes to monitors, SSL, HA, etc.

Monitors

Monitors are used to well monitor the actual backend nodes on an LTM device, there are many kinds of them, with specific rules on how to create them. These kb articles cover a lot of ground.

K2167: Constructing HTTP requests for use with the HTTP or HTTPS application health monitor
K12531: Troubleshooting health monitors
K3224: HTTP health checks may fail even though the node is responding correctly
K13898: Determining which monitor triggered a change in the availability of a node or pool member (11.x)
How self-IPs and floating-IPs work with monitors

SSL Concepts

Learn about certificates and PKI. Terms like CSR, certificate signing, root certificates, and certificate formats.

Create a CSR on BIGIP and sign certificates using this simple tool. A windows CA is recommended so you can create user certificates as well.
Deploy client SSL profiles on bigip. This is the simple SSL offloading with the server-side on plain text.
Modify ciphers on the client SSL, a very common cause for failures with legacy browsers and operating systems.
Install zenmap for windows and view the ciphers configured in the client-SSL profile.
Configure client-side SSL authentication on a virtual server by importing a user certificate in the client-SSL profile. Requires a Windows PKI to create a user certificate
Remember to always use SAN names in the certificates. The SSL tool supports it too.
Using decryption irules with tcpdump to decrypt traffic captured on the bigip, often required for troubleshooting in application issues which cannot be seen if the traffic is encrypted.

Server-side SSL Concepts

The F5 acts like a web browser with the backend server and this achieve s end-to-end encryption from user to the webserver

Configure the SSL server profile to encrypt traffic from f5 to the backend SSL node.
Remember requirements like Server Name Indication, which are used to pull the correct certificate from multiple certificates.
The decryption irule also contains a server-side decryption method. Do it.
Use curl and OpenSSL to check for issues with the SSL configuration of the backend node, so that f5 can create a proper TLS connection.
- Try using OpenSSL and curl to send requests with SNI, SAN names, etc to learn the proper responses and detect issues.

Troubleshooting SSL

SSL debug and OpenSSL are inbuilt tools on the bigip to check for various issues.
K9812: Overview of BIG-IP TCP RST behavior — BIGIP usually responds to any SSL or network failure with a TCP resets, knowing how they work on the bigip is essential.
K15292: Troubleshooting SSL/TLS handshake failures
K15475: Troubleshooting SSL/TLS renegotiation

HA Concepts

Probably one of the most important concepts to understand if you are just starting with BIGIP. Has multiple new concepts and terms. The best way to get them down is to configure up to 3 devices in an HA pair in and lab environment.

The most often used failover type is a network failover.

This video is a good review of how to configure the HA.
K95002127: Troubleshooting BIG-IP failover events. Test these commands after initiating different types of failover. Also contains multiple failover features, most often used is the HA ordered list.
K13946: Troubleshooting ConfigSync and device service clustering issues

Log Files

The /var/log/ltm file is the most important when working with just the LTM module. These kb articles explain how to search for logs using Linux commands like grep.

Another easier way to view logs is by uploading a qkview to the f5 ihealth website, try it in your lab device.

Allwyn September 18, 2019

f5 BIG-IP LTM

The BIGIP Auto-Backup iApp and FTP 550 Filename Invalid Error

The detailed post title helps with google hits.

This user created iApp works perfect to setup auto-backup on the BIGIP and you can save the gazillion dollars required to get a BIGIQ.

However, when using this with an FTP server a minor misunderstanding of how the directories work might waste you some time and effort.

It goes like this.

When you give it the directory /ftpsrv in my example, you must only mention the root of the directory on your FTP server or else you will keep getting this below!

550 File Name invalid

Here’s how you add the directory in FileZilla for it to work.

I have the folder ftpsrv inside C: and hence I must only add C: to the user on filezilla while assigning the directory.

This is because the iApp sends the /ftpsrv along with the entire UCS file name as it’s already added in the iApp config as in the first pic of this post. See:

This is the reason for those path errors.

Kinda silly but it did a take a while to get this done and move with my life.

Hope it helps someone out there!

Allwyn September 7, 2019

f5 BIG-IP LTM

BIGIP ASM Database Reset to Default

When an ASM database crashed for one of our clients and the database repair options failed, TAC recommended a full ASM DB reset on the lines of K6992

The KB article doesn’t really explain the full behavior.

As per the TAC, the policy name remains as it is, think of it as a “container”. This is because the virtual servers are referring to these policies hence the script keeps them intact but deletes all the “insides” of the policies, stuff like parameters, URLs and so on.

You then need to do a simple config sync from the peer unit to get all of that back.

However, once you run the script, the behavior is a bit tricky.

The policies all disappeared and over the next few minutes those policies keep coming up and the count goes up one by one and slooowwly

policy count on bigip asm — ASM policy count

This looks a bit weird when you first see it, so just relax it all should be fine.

Our peer device had a count of 50 policies and the one on which I did the reset went up to 45 and stopped.

At this stage, I just synced from the peer unit, and again the policy count started going up one at a time and eventually reached 49. I am guessing it deleted some of the policies which were not assigned to a virtual server.

The F5 kb article indeed is very wanting on the expected behavior.

Allwyn June 18, 2019

f5 BIG-IP LTM

F5 BigIP ASM Failed to Load Policies Error

Had an absolute surprise when I landed at a client – “to just fine one policy”. The entire policy list was gone and the GUI would only give me a “failed to load policies” after 2 mins of loading the page.

This is on BIGIP v13.1.1.2.

After a little clicking around the GUI – turns out, not just the policies but the network map page, the device management overview page where you sync the devices and the ASM event logs also failing to load.

This looked like the classic ASM db corruption cases, so we followed the K14194: Troubleshooting the BIG-IP ASM MySQL database.

The output I got for the steps there:

Determining the status of the BIG-IP ASM process

This did not give me any down errors for the db.

Determining MySQL status by verifying the MySQL processes

This looked all normal as well.

Determining overall health of MySQL database and table contents

When you run this, just be patient as this took a while on my device

mysqlcheck -p`perl -MF5::Cfg -e 'print F5::Cfg::get_mysql_password(user => q{root})'` -A

Most of the checks were OKs, except for these two:

mysql.func OK mysql.general_log Error : You can't use locks with log tables. status : OK mysql.help_category OK mysql.help_keyword OK mysql.help_relation OK mysql.help_topic OK mysql.host OK mysql.ndb_binlog_index OK mysql.plugin OK mysql.proc OK mysql.procs_priv OK mysql.servers OK mysql.slow_log Error : You can't use locks with log tables. status : OK mysql.tables_priv OK

Repairing tables in the MySQL database

Then you run the repair, as all tables do not support it you will see a lot of “note : The storage engine for the table doesn’t support repair”

mysqlcheck -p`perl -MF5::Cfg -e 'print F5::Cfg::get_mysql_password(user => q{root})'` -A -r

But the ones which had the problem were repaired alright, I thought this was it, but nope.

mysql.general_log OK
mysql.help_category OK
mysql.help_keyword OK
mysql.help_relation OK
mysql.help_topic OK
mysql.host OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.servers OK
mysql.slow_log OK
mysql.tables_priv OK

Checking the httpd log.

The GUI pages still did not show up after all this, so in /var/log/httpd/httpd_errors I could see:

Jun 16 11:35:11 Internet_F5_Primary err httpd[29641]: [error] server reached MaxClients setting, consider raising the MaxClients setting

I googled and followed: K9588: Error Message: httpd: [error] server reached MaxClients setting

Restart the daemons which handle the GUI and the ASM.

Follow this order.

bigstart restart httpd tomcat
bigstart restart restjavad
bigstart restart restnoded

After this, I could see all the policies and the remaining pages just like before.

So hope this helps someone google brings here when asked for “bigip asm failed to load policies error“.

Allwyn May 27, 2019

f5 BIG-IP LTM

F5 BIGIP SSL Errors When Fronting Cisco Webex Solution

In a recent case, with BigIP and Cisco webex servers we saw the SSL would break in the browser even while the configuration had no apparent errors.

The VIP on the BigIP hosted the webex over an SSL connection using a client-ssl profile and also used a server-ssl profile.

Cisco webex does not support the solution over HTTP port 80 and hence a server-ssl profile becomes a must.

The perplexing part is also how different browsers dealt with the errors.

Chrome gives you the option to accept the certificate error but then just refreshes the page and comes back to the same page.

Mozilla Firefox just gives you a “secure connection failed” with nothing explained.

Image result for mozilla secure connection failed

Resolution

All this while we were trying to access the webex console in the browser using the IP address only, but in the end, we just decided to give a go with a URL and added a hostname entry to the windows hosts file.

Luckily we chose the name “meeting.domain.com” and the webex console opened perfectly fine with a trusted certificate. Later on trying with the meet.domain.com, the connection started failing again as before.

I scoured the internet for some document which explains whether webex server requires the word “meeting” to be present in the hostname and didn’t find a thing.

Allwyn January 25, 2019

f5 BIG-IP LTM, LABS, Wireshark

F5 BigIP SSL TLS Traffic Decryption Methods and Notes

There are three methods to decrypt SSL encrypted packets on the BigIP:

From Jim Shaver’s blog, using your browser
Using the SSL Decryption Irule
Using the SSLDUMP on the BigIP platform

Jim Shaver’s blog, using your browser

Adding the SSLKEYLOGFILE to environmental setingd

This method simply involves adding a SSLKEYLOG variable to your windows settings and both chrome and firefox will start dumping all the SSL session keys there.

Then you import the file to wireshark under edit > preferences > SSL pre-master session keys.

Note: This only work on Chrome and FF and not on the IE browser.

Using the SSL Decryption Irule

K16700: Decrypting SSL traffic using the SSL::sessionsecret iRules command

You use the below irule on the virtual server and you get the RSA and Master-Key. This decrypts the client of the traffic only.

when CLIENTSSL_HANDSHAKE {
if {[IP::addr [IP::client_addr] equals <client_IP_addr>] } {
log local0. "[TCP::client_port] :: RSA Session-ID:[SSL::sessionid] Master-Key:[SSL::sessionsecret]"
}
}

Once you have the keys you put them in a .pms file like:

sed -e 's/^.*\(RSA Session-ID\)/\1/;tx;d;:x' /var/log/ltm > /var/tmp/sessionsecrets.pms

Then you import them to the same place as with the first method.

Notes:

If you have route domains on your BigIP then don’t forget to add the route domain to the client IP in the irule, if you miss it the traffic will never match the irule and you won’t see any keys being written to the ltm log.

Using the SSLDUMP on the BigIP CLI

This method requires the use of the private key used to encrypt the session and the pcap file to generate the Pre-Master Secret keys.

Once you capture traffic using TCPDUMP on the BigIP and you want to decrypt the client side of the traffic, you just use:

ssldump -r /path/to/capture_file -k /path/to/private_key -M /path/to/pre-master-key_log_file

For example:

ssldump -r /var/tmp/www-ssl-client1.cap -k /config/filestore/files_d/Common_d/certificate_key_d/\:Common\:test.org.key_1 -M /var/tmp/client1.pms

Now you can get the pcap file and the pms using WinSCP and load them into Wireshark 1.6 and later to view the decrypted traffic.

Wireshark Notes

If you are searching for a string match in Wireshark like a name/password use edit > Find packet > set the option to packet details and string.

Footnotes:

UPDATED: 25 JAN 2019.

Allwyn May 18, 2017

f5 BIG-IP LTM, Web Security

F5 BIG-IP LTM Persistence Cookie Shows Wrong Timezone

As I’m working through the CBTnuggets course for f5 BIG-IP LTM I have set up the following lab as required in the course.

Chapter/Video 15 focuses on the concept of persistenceHere’s persistence from the f5 manual;

Using BIG-IP Local Traffic Manager, you can configure session persistence. When you configure session persistence, Local Traffic Manager tracks and stores session data, such as the specific pool member that serviced a client request. The primary reason for tracking and storing session data is to ensure that client requests are directed to the same pool member throughout the life of a session or during subsequent sessions.

Persistence is simply the BIG-IP tracking the client/user session in a way that the client is sent to the same server till that session expires or till the persistence timeout as set on the BIG-IP expires. For e.g. one type of persistence explained in the course is cookie persistence, which tracks the user by storing an HTTP cookie in the user’s browser.

This cookie is obviously added by the BIG-IP in the HTTP response. We as BIG-IP admins have the option to add a timer in mins for which the cookie remains active on the client and then the cookie expires.

A example would be a user hitting the BIG-IP public IP to fill a form. The first initially selection of the server from the back-end pool is done by the BIG-IP based on the load balancing method in place. Let’s say server green is chosen, once the user hits green server, BIG-IP then adds an HTTP cookie to the HTTP response and from then onwards the remaining requests from the user will only be sent to the green server instead of being load balanced, this can go on till the cookie time as set on the BIG-IP expires or till the session expires—whichever is configured.

In my lab above the issue is that the cookie persistence profile was simply not taking effect and the client kept load balancing to all the three back-end servers in the pool.

f5 troubleshooting manuals strongly recommended setting the correct time on the BIG-IP. I did that rebooted.

[root@big-ip-test:Active:Standalone] config # date
Thu May 18 19:14:31 GST 2017
[root@big-ip-test:Active:Standalone] config #

Wireshark shows that the cookie was being added correctly to the HTTP response but expiry time was wrong. I am adding the cookie at 18:40 GST and the cookie expiry is set to 14:29.

Which means by the time I test this, the cookie has already expired!

This is strange. Also the cookie timezone in wireshark is GMT while I’m in Dubai and the BIG-IP is set to use Asia/Dubai timezone which is GST.

To be updated as I troubleshoot this.

UPDATE – 21 May 2017

DISCLAIMER: It turns out this could simply be a issue with my lab as I’m using my own windows host machine to test the load balancing. The issue was resolved when I created a different VM on virutal box and used GNS3 to connect it to the BIGIP external network and tested the persistence again.

But the troubleshooting is still worth it!

The first thing to get out of the way is the time issue. We can do this using perl.

Download and install strawberry perl for windows and simply find the scaler time.

root@(big-ip-test)(cfg-sync Standalone)(Active)(/Common)(tmos)# run /util bash
[root@big-ip-test:Standalone] # perl -le ‘print scalar time()’
1495179814
root@big-ip-test:Standalone] # exit

[root@big-ip-test:Active:Standalone] config # perl -le ‘print scalar time()’
1495183038

C:Usersallwyn>perl -le “print scalar time”
1495183035

That looks good, so no issues here.

Also the timezone for the cookie is always recommended to be in the GMT timezone since a website has to serve an global audience, I need to do more reading on this though. For next time.

So the packet in the above wireshark pcap is the HTTP response packet which has the cookie inserted by the BIGIP after the initial load balancing. This cookie is then saved by the browser and presented to the BIGIP for the remainder of the session or the cookie timeout as configured on the BIGIP.

Now we know that the BIGIP is inserting the cookie but whether it’s getting back needs to be verified, and the following iRule helps to log this.

when HTTP_REQUEST {
if { [HTTP::header exists Cookie] } {
log local0. “Client: [IP::client_addr] : Incoming Cookie header: [HTTP::header values Cookie]”
} else {
log local0. “Client: [IP::client_addr] : Incoming Cookie header: None”
}
}
when LB_SELECTED {
log local0. “client addr : [client_addr] : Selected pool member: [LB::server addr]”
}

Then to view the logs use this guide. You have to enter the tmsh utility and filter the log events with a pipe |.

[root@big-ip-test:Active:Standalone] config # tmsh
root@(big-ip-test)(cfg-sync Standalone)(Active)(/Common)(tmos)# show /sys log ltm | grep cookie

Test-iRule <lb_selected>: client addr : 192.168.1.5 : Selected pool member: 10.2.0.22
Fri May 19 15:28:00 GST 2017 info big-ip-test tmm[10349] Rule /Common/Cookie-Test-iRule <http_request>: Client: 192.168.1.5 : Incoming Cookie header: None
Fri May 19 15:28:02 GST 2017 info big-ip-test tmm[10349] Rule /Common/Cookie-Test-iRule <lb_selected>: client addr : 192.168.1.5 : Selected pool member: 10.2.0.33</lb_selected></http_request></lb_selected>

You can also view the same logs from in GUI at system – logs – LTM and filter the events.

We see here clearly that the incoming (from client to BIGIP) http_request does not have the cookie set and hence the BIGIP keeps load balancing across the servers.

At this stage, as I already mentioned I just created a new VM and used GNS3 to connect it to the VMware network and tested the persistence.

Here’s the log which then shows the incoming cookie.

ltm 05-19 18:43:13 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <HTTP_REQUEST>: Client: 192.168.1.192 : Incoming Cookie header: BIGipServerour-http-pool=184549898.20480.0000
ltm 05-19 18:43:13 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <LB_SELECTED>: client addr : 192.168.1.192 : Selected pool member: 10.2.0.11

ltm 05-19 18:43:13 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <HTTP_REQUEST>: Client: 192.168.1.192 : Incoming Cookie header: BIGipServerour-http-pool=184549898.20480.0000

ltm 05-19 18:52:05 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <LB_SELECTED>: client addr : 192.168.1.192 : Selected pool member: 10.2.0.22

ltm 05-19 18:52:05 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <HTTP_REQUEST>: Client: 192.168.1.192 : Incoming Cookie header: BIGipServerour-http-pool=369099274.20480.0000

ltm 05-19 18:52:06 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <LB_SELECTED>: client addr : 192.168.1.192 : Selected pool member: 10.2.0.22

The BIGIP then halts the load balancing and sends all requests to just one back-end server.