Allwyn Mascarenhas
infosec and design thoughts
  • About
  • Contact
  • Articles
Allwyn
Allwyn August 18, 2020
f5 BIG-IP LTM
0

Getting Started With F5 BIGIP LTM — Preparation for the 301a 301b LTM Specialist Exams

People often ask for a simple to follow start-up guide to get started on the F5 BIGIP product suite.

Here is an attempt to create one, covering the most basic features and concepts one comes across when working with the BIGIP LTM module.

This guide contains resources one can use to get a thorough understanding of the LTM module and prepare for the F5 301a and 301b exams. It covers, concepts one should know, and some troubleshooting advice if you run into trouble. 

Start with A Video Course

Video tutorials are the easiest to start with as they give you a guided overview of the entire GUI with a decent enough introduction.

  • Start with a video course like the ones on Udemy or the CBT nuggets Keith Barker series.
  • Courses usually contain the introductory level and the easiest material only when it comes to monitors, SSL, HA, etc. 

Monitors

Monitors are used to well monitor the actual backend nodes on an LTM device, there are many kinds of them, with specific rules on how to create them. These kb articles cover a lot of ground.

  • K2167: Constructing HTTP requests for use with the HTTP or HTTPS application health monitor
  • K12531: Troubleshooting health monitors
  • K3224: HTTP health checks may fail even though the node is responding correctly
  • K13898: Determining which monitor triggered a change in the availability of a node or pool member (11.x)
  • How self-IPs and floating-IPs work with monitors

SSL Concepts

Learn about certificates and PKI. Terms like CSR, certificate signing, root certificates, and certificate formats. 

  • Create a CSR on BIGIP and sign certificates using this simple tool. A windows CA is recommended so you can create user certificates as well.
  • Deploy client SSL profiles on bigip. This is the simple SSL offloading with the server-side on plain text.
  • Modify ciphers on the client SSL, a very common cause for failures with legacy browsers and operating systems.
  • Install zenmap for windows and view the ciphers configured in the client-SSL profile.
  • Configure client-side SSL authentication on a virtual server by importing a user certificate in the client-SSL profile. Requires a Windows PKI to create a user certificate
  • Remember to always use SAN names in the certificates. The SSL tool supports it too.
  • Using decryption irules with tcpdump to decrypt traffic captured on the bigip, often required for troubleshooting in application issues which cannot be seen if the traffic is encrypted.

Server-side SSL Concepts

The F5 acts like a web browser with the backend server and this achieve s end-to-end encryption from user to the webserver

  • Configure the SSL server profile to encrypt traffic from f5 to the backend SSL node.
  • Remember requirements like Server Name Indication, which are used to pull the correct certificate from multiple certificates.
  • The decryption irule also contains a server-side decryption method. Do it.
  • Use curl and OpenSSL to check for issues with the SSL configuration of the backend node, so that f5 can create a proper TLS connection.
    • Try using OpenSSL and curl to send requests with SNI, SAN names, etc to learn the proper responses and detect issues. 

Troubleshooting SSL

  • SSL debug and OpenSSL are inbuilt tools on the bigip to check for various issues.
  • K9812: Overview of BIG-IP TCP RST behavior — BIGIP usually responds to any SSL or network failure with a TCP resets, knowing how they work on the bigip is essential. 
  • K15292: Troubleshooting SSL/TLS handshake failures
  • K15475: Troubleshooting SSL/TLS renegotiation

HA Concepts

Probably one of the most important concepts to understand if you are just starting with BIGIP. Has multiple new concepts and terms. The best way to get them down is to configure up to 3 devices in an HA pair in and lab environment.

The most often used failover type is a network failover. 

  • This video is a good review of how to configure the HA.
  • K95002127: Troubleshooting BIG-IP failover events. Test these commands after initiating different types of failover. Also contains multiple failover features, most often used is the HA ordered list.
  • K13946: Troubleshooting ConfigSync and device service clustering issues

Log Files 

The /var/log/ltm file is the most important when working with just the LTM module. These kb articles explain how to search for logs using Linux commands like grep. 

Another easier way to view logs is by uploading a qkview to the f5 ihealth website, try it in your lab device. 

  • K16197: Reviewing BIG-IP log files
  • Advanced grep filters for F5 logs

 

Allwyn
Allwyn May 2, 2020
Palo Alto Firewalls
0

URL Filtering Based On URIs on Palo Alto Firewalls

We get some requests such as blocking the base domain and to only allow certain pages on websites based on the URIs — that which comes after the “/” e.g.

hostname.com – block
hostname.com/page2 – allow

You need the URL filtering license to be able to do this. 

Initially, I tried it with a single policy which failed, then using two policies you can get the exact filtering. 

I am using Daniel Miessler’s blog for demonstration. We will allow /blog, /popular, and /study and block the base hostname.

danielmiessler.com – block

And these to be allowed with everything being blocked. 

danielmiessler.com/blog 
danielmiessler.com/popular
danielmiessler.com/study

Note: open the images in a new tab for larger sizes

Create a custom object URL category of paths to allow

panos-url-category-paths-allowed

 

Repeat this for the base domain to be blocked

 

panos base domain to block

 

Add them to URL filtering objects and set the proper Actions. 

Note the actions, getting them right is the most important here.

 

panos url filtering allowed paths
this goes in the first security policy, the paths category object is set to allow

 

panos url base domain block
this goes in the second security policy with action deny, the block base domain category object is set to block with paths set to none

 

Create both security policies to allow the paths and to deny the base domain

panos path allowed

panos block base
ignore the blurred part and just use the block domain URL filtering which you created before

 

The final policy order with allow rule before the deny one. 

panos url filtering rule order

 

And that’s all there is to it folks! You can try other variations of this, I’ll update the post if I come across something.

Allwyn
Allwyn September 18, 2019
f5 BIG-IP LTM
0

The BIGIP Auto-Backup iApp and FTP 550 Filename Invalid Error

The detailed post title helps with google hits. 

This user created iApp works perfect to setup auto-backup on the BIGIP and you can save the gazillion dollars required to get a BIGIQ.

However, when using this with an FTP server a minor misunderstanding of how the directories work might waste you some time and effort. 

It goes like this.

iapp config
see the directory

 

When you give it the directory /ftpsrv in my example, you must only mention the root of the directory on your FTP server or else you will keep getting this below!

550 File Name invalid

notice the dir setting
notice the dir setting

 

Here’s how you add the directory in FileZilla for it to work. 

 

Filezilla config for BIGIP iApp
Filezilla config for BIGIP iApp

I have the folder ftpsrv inside C: and hence I must only add C: to the user on filezilla while assigning the directory. 

This is because the iApp sends the /ftpsrv along with the entire UCS file name as it’s already added in the iApp config as in the first pic of this post. See: 

BIGIP FTP backup iApp config
BIGIP FTP backup iApp config

This is the reason for those path errors. 

Kinda silly but it did a take a while to get this done and move with my life.

Hope it helps someone out there! 

Allwyn
Allwyn September 7, 2019
f5 BIG-IP LTM
0

BIGIP ASM Database Reset to Default

When an ASM database crashed for one of our clients and the database repair options failed, TAC recommended a full ASM DB reset on the lines of K6992

The KB article doesn’t really explain the full behavior.

As per the TAC, the policy name remains as it is, think of it as a “container”. This is because the virtual servers are referring to these policies hence the script keeps them intact but deletes all the “insides” of the policies, stuff like parameters, URLs and so on.

You then need to do a simple config sync from the peer unit to get all of that back.

However, once you run the script, the behavior is a bit tricky.

The policies all disappeared and over the next few minutes those policies keep coming up and the count goes up one by one and slooowwly

policy count on bigip asm
ASM policy count

This looks a bit weird when you first see it, so just relax it all should be fine.

Our peer device had a count of 50 policies and the one on which I did the reset went up to 45 and stopped.

At this stage, I just synced from the peer unit, and again the policy count started going up one at a time and eventually reached 49. I am guessing it deleted some of the policies which were not assigned to a virtual server.

The F5 kb article indeed is very wanting on the expected behavior.

Allwyn
Allwyn June 18, 2019
f5 BIG-IP LTM
0

F5 BigIP ASM Failed to Load Policies Error

Had an absolute surprise when I landed at a client – “to just fine one policy”. The entire policy list was gone and the GUI would only give me a “failed to load policies” after 2 mins of loading the page.

This is on BIGIP v13.1.1.2. 

After a little clicking around the GUI – turns out, not just the policies but the network map page, the device management overview page where you sync the devices and the ASM event logs also failing to load. 

This looked like the classic ASM db corruption cases, so we followed the K14194: Troubleshooting the BIG-IP ASM MySQL database. 

The output I got for the steps there: 

Determining the status of the BIG-IP ASM process

This did not give me any down errors for the db.

Determining MySQL status by verifying the MySQL processes

This looked all normal as well.

Determining overall health of MySQL database and table contents

When you run this, just be patient as this took a while on my device

mysqlcheck -p`perl -MF5::Cfg -e 'print F5::Cfg::get_mysql_password(user => q{root})'` -A

Most of the checks were OKs, except for these two:

mysql.func OK
mysql.general_log
Error : You can't use locks with log tables.
status : OK
mysql.help_category OK
mysql.help_keyword OK
mysql.help_relation OK
mysql.help_topic OK
mysql.host OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.servers OK
mysql.slow_log
Error : You can't use locks with log tables.
status : OK
mysql.tables_priv OK

Repairing tables in the MySQL database

Then you run the repair, as all tables do not support it you will see a lot of  “note : The storage engine for the table doesn’t support repair”

mysqlcheck -p`perl -MF5::Cfg -e 'print F5::Cfg::get_mysql_password(user => q{root})'` -A -r

But the ones which had the problem were repaired alright, I thought this was it, but nope.

mysql.general_log OK
mysql.help_category OK
mysql.help_keyword OK
mysql.help_relation OK
mysql.help_topic OK
mysql.host OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.servers OK
mysql.slow_log OK
mysql.tables_priv OK

Checking the httpd log. 

The GUI pages still did not show up after all this, so in /var/log/httpd/httpd_errors I could see:

Jun 16 11:35:11 Internet_F5_Primary err httpd[29641]: [error] server reached MaxClients setting, consider raising the MaxClients setting

I googled and followed: K9588: Error Message: httpd: [error] server reached MaxClients setting

Restart the daemons which handle the GUI and the ASM.

Follow this order.

bigstart restart httpd tomcat
bigstart restart restjavad
bigstart restart restnoded

After this, I could see all the policies and the remaining pages just like before.

So hope this helps someone google brings here when asked for “bigip asm failed to load policies error“. 

Allwyn
Allwyn May 27, 2019
f5 BIG-IP LTM
0

F5 BIGIP SSL Errors When Fronting Cisco Webex Solution

In a recent case, with BigIP and Cisco webex servers we saw the SSL would break in the browser even while the configuration had no apparent errors.

The VIP on the BigIP hosted the webex over an SSL connection using a client-ssl profile and also used a server-ssl profile.

Cisco webex does not support the solution over HTTP port 80 and hence a server-ssl profile becomes a must. 

The perplexing part is also how different browsers dealt with the errors. 

Chrome gives you the option to accept the certificate error but then just refreshes the page and comes back to the same page. 

Mozilla Firefox just gives you a “secure connection failed” with nothing explained. 

Image result for mozilla secure connection failed

Resolution

All this while we were trying to access the webex console in the browser using the IP address only, but in the end, we just decided to give a go with a URL and added a hostname entry to the windows hosts file. 

Luckily we chose the name “meeting.domain.com” and the webex console opened perfectly fine with a trusted certificate. Later on trying with the meet.domain.com, the connection started failing again as before.

I scoured the internet for some document which explains whether webex server requires the word “meeting” to be present in the hostname and didn’t find a thing. 

Allwyn
Allwyn February 23, 2019
SSL, Tools
0

OpenSSL – WIndows Installation and Certificate Operation Commands

For some reason OpenSSL packages are hard to find, most of them have some missing binaries etc. and things don’t work correctly.

In the end I found a proper working installer here.

Using it from Windows Command Line CMD

Once installed, you should add it to your environmental variables so you can invoke it from CMD straight away.

Open Start -> System and environmental variables.

 

Windows openssl environmental variabled
Windows openssl environmental variables

OpenSSL Files Directory

If you create a new file with openssl it goes to your users directory on windows: C:\Users\<username> 

To output them to a specific folder you can add path with “path” like:

OpenSSL> genrsa -des -out "C:\anypath\testopenssl.key"

Certificate format change operations

One of the most common uses of having openssl installed is to convert and combine all those different(lord knows why there are no standards!) SSL certificates.

PFX to CRT and KEY

PFX/PKCS12 is a format which combines the certificate and the public key into one file with a .pfx extension. Microsoft AD CA gives you this file. You can divide into the cert and key with:

Then to extra the cert:

openssl pkcs12 -in yourfile.pfx -nokeys -out keyfile-encrypted.pem

To extract the key:

openssl pkcs12 -in yourfile.pfx -nocerts -out keyfile-encrypted.key

Once entered you need to type in the import password of the .pfx file.

This is the password that you used to protect your keypair when you created your .pfx file. If you cannot remember it anymore you can just throw your .pfx file away, cause you won’t be able to import it again, anywhere!.

Once you entered the import password OpenSSL requests you to type in another password, twice!. This new password will protect your .key file.

PEM (.pem, .crt, .cer) to PFX/PKCS12

openssl pkcs12 -export -out certificate.pfx -inkey privateKey.key -in certificate.crt -certfile more-int-certs.crt 

  • -certfile more-int-certs.crt  > optional, only required if you need to import intermediate certificates too. 

This will ask you for a password. To use no pass use:

openssl pkcs12 -export -out certificate.pfx -inkey privateKey.key -in certificate.crt -nopass

PKCS#7/P7B (.p7b, .p7c) to PFX

P7B files cannot be used to directly create a PFX file. P7B files must be converted to PEM. Once converted to PEM, follow the above steps to create a PFX file from a PEM file.

openssl pkcs7 -print_certs -in certificate.p7b -out certificate.crt

PKCS7 and PKCS12

These 

Testing Client based SSL auth with s_client

If you are using certificates to do client auth then you need to use:

openssl s_client -connect website:443 -cert user-cert.cert -key user-key.key 

If it fails then the you will see an error with the SSL Handshake failure. 

OpenSSL> s_client -connect auc.akmlab.local:443
CONNECTED(00000274)
depth=0 C = US, ST = Dubai, L = dubai, O = orgname, OU = IT, CN = *.akmlab.local, emailAddress = allwyn.mascarenhas@domain.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = Dubai, L = dubai, O = orgname, OU = IT, CN = *.akmlab.local, emailAddress = allwyn.mascarenhas@domain.com
verify error:num=21:unable to verify the first certificate
verify return:1
8732:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:ssl\record\rec_layer_s3.c:1528:SSL alert number 40

Verifying Certificate Trust

Recently got a case where there seems to be confusion with user certificates and the signing CA certificate, so in these situations you can just quickly verify the trust instead of importing the certs to production only to find out they do not work. 

This also helps a lot with intermediate certificates in case you using those. 

To verify the trust between an intermediate and CA signing root cert:

OpenSSL> verify -CAfile "C:<path>\akmdc-root.pem" "C:<path>\akmdc-root-intermediate-21feb19.pem"
C:<path>\akmdc-root-intermediate-21feb19.pem: OK

If you using a user cert for SSL client auth along with an intermediate cert and root then use this to check the chain trust all the way to the root:

OpenSSL> verify -CAfile "C:<path>\akmdc-root.pem" "C:<path>\akmdc-root-intermediate-21feb19.pem" "C:<path>\user-cert.pem"
C:<path>\akmdc-root-intermediate-21feb19.pem: OK
C:<path>\user-cert.pem: OK

Notes:

  1. UPDATED: 23 Feb 2019
Allwyn
Allwyn January 26, 2019
Symantec Bluecoat ProxyGS
0

Symantec/Bluecoat ProxySG Doesn’t Trust RapidSSL Intermediate Certificate

When SSL interception is configured on a full proxy, these errors are quite common mostly due to some websites having expired certificates or the CN in the certificate not matching the actual hostname in the browser.

This is what you see in the browser when the proxysg fails SSL verification of the OCS – original content server.


Untrusted SSL Server Certificate (ssl_server_cert_untrusted_issuer) 


Proxysg error Untrusted SSL Server Certificate (ssl_server_cert_untrusted_issuer)
Browser error when proxy fails ssl cert validation

This case was a bit different as the certificate on the site was fine but the proxysg did not trust the intermediate cert in the chain. 

Here you can see the RapidSSL intermediate cert when accessing the website directly without the proxy.

rapidssl intermediate cert
browser showing the rapidssl intermediate cert

 

If you look at the chrome certificate store the RapidSSL intermediate cert is present under intermediate certs and the Digicert root cert under trusted root certs.

Chrome certificate store
Chrome cert store

However the cert store on the proxysg only contains the Digicert root cert in its certificate store.

proxysg certificate store
proxysg certificate store

 

And hence why the connection from the proxy to the server breaks as the proxy doesn’t trust the certificate!

The way out of this is to either use a VPM policy by disabling the server certificate validation in an SSL Access Layer.

vpn policy server cert validation

 

..or by exporting the RapidSSL intermediate certificate from your browser and importing it to the proxysg. Always apply changes in proxysg for them to take effect.

importing cert to proxysg CA

To import you just open the cert in notepad and paste it using the paste from clipboard option as in the image. 

It’s not done, the new cert needs to be added to browser trusted for it to work. 

add cert to browser trusted

 

Once you do either of these the browser should load the page correctly through the proxysg!

 

 

 

Allwyn
Allwyn January 25, 2019
f5 BIG-IP LTM, LABS, Wireshark
0

F5 BigIP SSL TLS Traffic Decryption Methods and Notes

There are three methods to decrypt SSL encrypted packets on the BigIP:

  • From Jim Shaver’s blog, using your browser
  • Using the SSL Decryption Irule
  • Using the SSLDUMP on the BigIP platform

Jim Shaver’s blog, using your browser

Adding the SSLKEYLOGFILE to environmental setingd

 

This method simply involves adding a SSLKEYLOG variable to your windows settings and both chrome and firefox will start dumping all the SSL session keys there.

Then you import the file to wireshark under edit > preferences > SSL pre-master session keys.

Note: This only work on Chrome and FF and not on the IE browser.

Using the SSL Decryption Irule

  • K16700: Decrypting SSL traffic using the SSL::sessionsecret iRules command

You use the below irule on the virtual server and you get the RSA and Master-Key. This decrypts the client of the traffic only.

when CLIENTSSL_HANDSHAKE {
if {[IP::addr [IP::client_addr] equals <client_IP_addr>] } {
log local0. "[TCP::client_port] :: RSA Session-ID:[SSL::sessionid] Master-Key:[SSL::sessionsecret]"
}
}


Once you have the keys you put them in a .pms file like:

sed -e 's/^.*\(RSA Session-ID\)/\1/;tx;d;:x' /var/log/ltm > /var/tmp/sessionsecrets.pms


Then you import them to the same place as with the first method.

Notes:

  • If you have route domains on your BigIP then don’t forget to add the route domain to the client IP in the irule, if you miss it the traffic will never match the irule and you won’t see any keys being written to the ltm log.

Using the SSLDUMP on the BigIP CLI

This method requires the use of the private key used to encrypt the session and the pcap file to generate the Pre-Master Secret keys.

Once you capture traffic using TCPDUMP on the BigIP and you want to decrypt the client side of the traffic, you just use:

ssldump -r /path/to/capture_file -k /path/to/private_key -M /path/to/pre-master-key_log_file


For example:

ssldump -r /var/tmp/www-ssl-client1.cap -k /config/filestore/files_d/Common_d/certificate_key_d/\:Common\:test.org.key_1 -M /var/tmp/client1.pms


Now you can get the pcap file and the pms using WinSCP and load them into Wireshark 1.6 and later to view the decrypted traffic.

Wireshark Notes

If you are searching for a string match in Wireshark like a name/password use edit > Find packet > set the option to packet details and string.

Footnotes:

  1. UPDATED: 25 JAN 2019.
Allwyn
Allwyn October 4, 2018
Cybersecurity labs cybersecurity labs, f5 labs, paloalto labs, vmware
0

Understanding VMNETs in VMWARE Workstation

If you are going to learn cybersecurity then the first thing you realize is how you don’t really understand anything till you do it. Good job to whoever came up with;

“I hear, and I forget

I see, and I remember

I do, and I understand.“

— not Confucius

And since most of us don’t have data centers lying around at home, the amazing power of virtualization is all we have.

Initially setting these up gets messy because a lot of “network” guys go almost their entire lives never done anything related to servers as they are not the “server” guy. Well, after a few CBT Nugget videos this thing becomes very straightforward.

Now I will stop rambling and get to the point.

Every VMnet Is Like A Network/Subnet

If you look at my setup below.

VMnet settings on my vmware setup
my setup

This means VMnet1 is a network in the range 10.128.1.0/24. Think of it like a switch and the gateway is your own machine which has a VMnet1 adapter with the IP 10.128.1.1/24.

Quick explanation of the highlighted parts:

  • I use VMnet1 as management network or OOB network.
  • Host only – means it doesn’t deal with your host’s IP addresses in anyway and is totally a separate network.
  • Connect a host virtual adapted to this network option creates a vmnet1 adapter on the host machine, which is your PC and the address can be anything in the subnet range but by default is .1, like mine is 10.128.1.1/24. This is like a cable from your machine to the vmnet1 switch and now your machine can access everything connected to this vmnet1.
goto run -> type: ncpa.cpl

  • DHCP – I disable it as most devices just pick up a random IP then it’s difficult to find, just better assign them manually.
  • Subnet – For the love of your God, plan your subnets and IP ranges well, or else you will just stop doing labs when you keep forgetting what IP is where, like happened to me for the first 2 years. 

A NAT VMnet For Internet Access

This is maybe just my setup. I did it this way for the CBT Nuggets F5 labs.

  • The only difference is this is a NAT adapter and not host only, so it will NAT the 10.128.10.0/24 range with your host’s IP address and so anything assigned this adapter can go to the internet through 10.128.10.2
  • Notice the gateway next-hop you have to point to is 10.128.10.2 as .1 is assigned to the vmnet2 adapter on your own host PC.
  • I use this this as my external vlan, where the virtual servers live in f5 labs and also to give a default route to the f5 to the internet through 10.128.10.2.
  • You can use it to give internet access to a firewall too, but remember to NAT your internet inside zone LAN to 10.128.10.2 first as that will be a separate private IP address space.
  • This also puts a vmnet2 adapter on your host machine as explained, which is what lets you connect to the virtual servers if you use this as an external vlan.

One Completely Isolated VMnet

this one’s all by himself

This VMnet cannot be accessed directly from your host machine as it does not “connect a host virtual adapter”. You can only reach him or it to you via a routing device like a firewall or f5.

This is perfect to simulate an inside network in firewall labs and for the internal vlan in f5 labs where you put the back end nodes/servers.

1 2 3
© Allwyn Mascarenhas 2022
Powered by WordPress • Themify WordPress Themes