Allwyn – Page 2 – Allwyn Mascarenhas

Allwyn January 26, 2019

Symantec Bluecoat ProxyGS

Symantec/Bluecoat ProxySG Doesn’t Trust RapidSSL Intermediate Certificate

When SSL interception is configured on a full proxy, these errors are quite common mostly due to some websites having expired certificates or the CN in the certificate not matching the actual hostname in the browser.

This is what you see in the browser when the proxysg fails SSL verification of the OCS – original content server.


Untrusted SSL Server Certificate (ssl_server_cert_untrusted_issuer)

Proxysg error Untrusted SSL Server Certificate (ssl_server_cert_untrusted_issuer) — Browser error when proxy fails ssl cert validation

This case was a bit different as the certificate on the site was fine but the proxysg did not trust the intermediate cert in the chain.

Here you can see the RapidSSL intermediate cert when accessing the website directly without the proxy.

If you look at the chrome certificate store the RapidSSL intermediate cert is present under intermediate certs and the Digicert root cert under trusted root certs.

However the cert store on the proxysg only contains the Digicert root cert in its certificate store.

And hence why the connection from the proxy to the server breaks as the proxy doesn’t trust the certificate!

The way out of this is to either use a VPM policy by disabling the server certificate validation in an SSL Access Layer.

vpn policy server cert validation

..or by exporting the RapidSSL intermediate certificate from your browser and importing it to the proxysg. Always apply changes in proxysg for them to take effect.

importing cert to proxysg CA

To import you just open the cert in notepad and paste it using the paste from clipboard option as in the image.

It’s not done, the new cert needs to be added to browser trusted for it to work.

add cert to browser trusted

Once you do either of these the browser should load the page correctly through the proxysg!

Allwyn January 25, 2019

f5 BIG-IP LTM, LABS, Wireshark

F5 BigIP SSL TLS Traffic Decryption Methods and Notes

There are three methods to decrypt SSL encrypted packets on the BigIP:

From Jim Shaver’s blog, using your browser
Using the SSL Decryption Irule
Using the SSLDUMP on the BigIP platform

Jim Shaver’s blog, using your browser

Adding the SSLKEYLOGFILE to environmental setingd

This method simply involves adding a SSLKEYLOG variable to your windows settings and both chrome and firefox will start dumping all the SSL session keys there.

Then you import the file to wireshark under edit > preferences > SSL pre-master session keys.

Note: This only work on Chrome and FF and not on the IE browser.

Using the SSL Decryption Irule

K16700: Decrypting SSL traffic using the SSL::sessionsecret iRules command

You use the below irule on the virtual server and you get the RSA and Master-Key. This decrypts the client of the traffic only.

when CLIENTSSL_HANDSHAKE {
if {[IP::addr [IP::client_addr] equals <client_IP_addr>] } {
log local0. "[TCP::client_port] :: RSA Session-ID:[SSL::sessionid] Master-Key:[SSL::sessionsecret]"
}
}

Once you have the keys you put them in a .pms file like:

sed -e 's/^.*\(RSA Session-ID\)/\1/;tx;d;:x' /var/log/ltm > /var/tmp/sessionsecrets.pms

Then you import them to the same place as with the first method.

Notes:

If you have route domains on your BigIP then don’t forget to add the route domain to the client IP in the irule, if you miss it the traffic will never match the irule and you won’t see any keys being written to the ltm log.

Using the SSLDUMP on the BigIP CLI

This method requires the use of the private key used to encrypt the session and the pcap file to generate the Pre-Master Secret keys.

Once you capture traffic using TCPDUMP on the BigIP and you want to decrypt the client side of the traffic, you just use:

ssldump -r /path/to/capture_file -k /path/to/private_key -M /path/to/pre-master-key_log_file

For example:

ssldump -r /var/tmp/www-ssl-client1.cap -k /config/filestore/files_d/Common_d/certificate_key_d/\:Common\:test.org.key_1 -M /var/tmp/client1.pms

Now you can get the pcap file and the pms using WinSCP and load them into Wireshark 1.6 and later to view the decrypted traffic.

Wireshark Notes

If you are searching for a string match in Wireshark like a name/password use edit > Find packet > set the option to packet details and string.

Footnotes:

UPDATED: 25 JAN 2019.

Allwyn October 4, 2018

Cybersecurity labs cybersecurity labs, f5 labs, paloalto labs, vmware

Understanding VMNETs in VMWARE Workstation

If you are going to learn cybersecurity then the first thing you realize is how you don’t really understand anything till you do it. Good job to whoever came up with;

“I hear, and I forget
I see, and I remember
I do, and I understand.“
— not Confucius

And since most of us don’t have data centers lying around at home, the amazing power of virtualization is all we have.

Initially setting these up gets messy because a lot of “network” guys go almost their entire lives never done anything related to servers as they are not the “server” guy. Well, after a few CBT Nugget videos this thing becomes very straightforward.

Now I will stop rambling and get to the point.

Every VMnet Is Like A Network/Subnet

If you look at my setup below.

VMnet settings on my vmware setup — my setup

This means VMnet1 is a network in the range 10.128.1.0/24. Think of it like a switch and the gateway is your own machine which has a VMnet1 adapter with the IP 10.128.1.1/24.

Quick explanation of the highlighted parts:

I use VMnet1 as management network or OOB network.
Host only – means it doesn’t deal with your host’s IP addresses in anyway and is totally a separate network.
Connect a host virtual adapted to this network option creates a vmnet1 adapter on the host machine, which is your PC and the address can be anything in the subnet range but by default is .1, like mine is 10.128.1.1/24. This is like a cable from your machine to the vmnet1 switch and now your machine can access everything connected to this vmnet1.

DHCP – I disable it as most devices just pick up a random IP then it’s difficult to find, just better assign them manually.
Subnet – For the love of your God, plan your subnets and IP ranges well, or else you will just stop doing labs when you keep forgetting what IP is where, like happened to me for the first 2 years.

A NAT VMnet For Internet Access

This is maybe just my setup. I did it this way for the CBT Nuggets F5 labs.

The only difference is this is a NAT adapter and not host only, so it will NAT the 10.128.10.0/24 range with your host’s IP address and so anything assigned this adapter can go to the internet through 10.128.10.2
Notice the gateway next-hop you have to point to is 10.128.10.2 as .1 is assigned to the vmnet2 adapter on your own host PC.
I use this this as my external vlan, where the virtual servers live in f5 labs and also to give a default route to the f5 to the internet through 10.128.10.2.
You can use it to give internet access to a firewall too, but remember to NAT your internet inside zone LAN to 10.128.10.2 first as that will be a separate private IP address space.
This also puts a vmnet2 adapter on your host machine as explained, which is what lets you connect to the virtual servers if you use this as an external vlan.

One Completely Isolated VMnet

This VMnet cannot be accessed directly from your host machine as it does not “connect a host virtual adapter”. You can only reach him or it to you via a routing device like a firewall or f5.

This is perfect to simulate an inside network in firewall labs and for the internal vlan in f5 labs where you put the back end nodes/servers.

Allwyn September 21, 2018

Bluecoat, Symantec

Changing the Management Certificate on Symantec/Bluecoat ProxySG/ASG

I like the nice green lock which most browsers show up with when using a proper HTTPS config with a good certificate.

I like it so much that I have a trusted cert for all my lab devices. Once of which is a proxysg.

And it took me a good one hour to find the steps to assign a certificate to the proxysg management interface since for some reason it’s not in any of the documentation I looked into.

In the end I found the option just by manually looking at all the options in the GUI. The certificate import method to begin with is weird.

It doesn’t just accept the .pfx file but needs you to import .pem files and you must import the pvt key first.

Here’s the details:

Goto the config > SSL > keyrings > create
Name and import the private key first using the “pasting from clipboard” option
Apply the changes
Then click the pvt key and import the certificate for it again by using the “pasting from clipboard” option
Ref the screenshots below 😀

And once you have the keypair imported in the keyrings, the below screenshot shows you where you assign it to your management interface.

And with that just make sure you have the root certificate in your browser installed and you have nice green secured connection to the proxysg management.

Allwyn September 21, 2018

Bluecoat, Symantec

Symantec/Bluecoat ProxyASG Upgrade to 6.7.3.10 CAS Errors

Had an absolute nightmare of an upgrade, and all this after following the Symantec TAC recommended upgrade path to go to 6.6.5.13 > 6.7.3.2 > 6.7.3.10.

And I also upgraded the proxyasg BMC to the latest version 3.1.2.1 as well, following the steps here.

So the BMC upgrade was done first and then the firmware.

Now I was already aware of some known issues with CAS in these upgrades, but since the TAC recommended the upgrade, I was expecting a smooth sailing — I had plans and God was laughing.

Right after the upgrade I checked the CAS and saw these hideous errors which have no documentation anywhere.

Under Content Analysis > AV Patterns

"error no such interface tap0"

and I saw “unavailable” for services that should be working:

And any traffic going through this box was blocked with this:

“icap_error” an error occured while perfoming an ICAP operation. Uknown error (16:0x0); nonname; Sub File: ; Vendor: Kaspersky Labs; Engine version: unknown; Pattern version: Unknown; Pattern data: Unknown

The box did not do HTTPS inspection so it was only the HTTP websites like bbc.com and example.com which were being blocked. The error makes sense now, since it looks like the box cannot find any icap engine to do the content scanning and due to the “block if not able to scan” was enabled it rightfully did so.

Since there was no information online opening a TAC case was all there was to do.

And that also was for nothing, since the support did not have any idea on what was happening.

With time I think one from their engineering teams joined the session and a restart of ICAP services was recommended.

The restart options are in Content Analysis > Utilities.

After that, all the ugly errors were gone and we saw the AV patterns in Under “Services > AV Patters” were still downloading. Once they downloaded, then the traffic started working through the device.

Later after Symantec log analysis for an RCA, it seems we hit a bug in the 6.7.2.1 and a better upgrade path would have been 6.6.5.13 > 6.7.3.2 > 6.7.3.10.

However, I thought I will just record this info here, in case someone somewhere in a far away galaxy will come looking for these errors at 1AM early morning and will probably save some time.

Allwyn June 14, 2017

Open Source, XAMPP

OsTicket — An open source helpdesk ticketing platform

My friend is looking for a ticketing platform for her place of work in a medium scale advertising, marketing firm.

I evaluated LimeSurvery but seems development on it has stalled and things are little bit difficult to figure out.

Google forms would have worked for this use case as the only requirement is to capture responses to a spreadsheet, but as with most firms, all cloud apps are blocked to prevent data leak and malicious downloads etc.

Now I am reviewing OsTicket.

I have installed it using XAMPP which is a very easy and simply way to run a powerful webserver with Apache, MySQL and Perl on your local workstation. It will easy handle a good amount of traffic for most needs.

Some quick notes to keep in mind while installing OsTicket:

Use this XAMPP installation guide on YouTube.
It’s good to change the default port numbers for your apps where ever you can, but it depends on other considerations too.
To install OsTicket on XAMPP use this video on Youtube.

When creating the MySQL Database and User, you might get something like this:

Error: 1018 SQLSTATE: HY000 (ER_CANT_READ_DIR)
Message: Can’t read dir of ‘%s’ (errno: %d – %s)
this Message takes me to SQL Query in the PHPmyadmin server which shows this,
SHOW PLUGINS SONAME LIKE ‘%_password_check%’

Just create a directory as stated in the answer:

If you installed XAMPP in the default folder C:xampp go and manually create the folder C:xamppmysqllibplugin in Explorer. This should fix the user creation issue

You might get a blank page after you install OsTicket. Like I did and also reported here. Do this:

change
max_execution_time=30

to
max_execution_time=180

in php.ini file and restart all services

Should work now

You find this file in C:xamppphp

You can create a free gmail account and achieve email notifications from the OsTicketing server. Your ticket creating users also get canned responses after creating a ticket!

Some limitations in OsTicket:

If you want to export ticket data to excel, it doesn’t here support selecting the fields to export.
Also it doesn’t allow you to export the internal custom fields which you might have added to your ticket

These features are already in the pipeline.

This is a very easy to use and powerful ticketing platform rivaling the best of support systems out there.

Give it a shot in your lab environment and evaluate it at least once!

Allwyn May 23, 2017

Cisco ASA, IPsec VPN

ASA – IOS Router IPsec VPN Config And Troubleshooting

We will use this network diagram for this lab.

The IPsec VPN will be created between Outside-R and the ASAv.

Traffic from 192.168.20.0/24 and 10.1.1.0/24 subnets will be encrypted using the tunnel.

The tricky part on the ASA is that ASDM doesn’t allow you to remove or let’s say limit the number of cipher suits to be used with the IPsec tunnel which is the phase-2 of the IPsec IKEv1. It’ll be clear with the CLI below.

Here’s the VPN config on the IOS router.

Outside-R#show running-config | section crypto
crypto isakmp policy 1
encr aes
hash md5
authentication pre-share
group 2
crypto isakmp key cisco address 0.0.0.0 0.0.0.0
crypto ipsec transform-set ourset esp-aes
crypto map ourmap 1 ipsec-isakmp
set peer 192.168.2.1
set transform-set ourset
match address 100
crypto map ourmap

For the ASA config it’s highly recommended to just use the IPsec VPN wizard from the wizards menu.

Here’s the ASA config from the CLI anyway.

ciscoasa# show running-config cryptocrypto ipsec security-association pmtu-aging infinite
crypto map outside_map 1 match address outside_cryptomap
crypto map outside_map 1 set peer 192.168.2.2
crypto map outside_map 1 set ikev1 transform-set ESP-AES-128-MD5 ESP-AES-128-MD5-TRANS
crypto map outside_map interface outside
crypto ca trustpool policy
crypto ikev1 enable outside
crypto ikev1 policy 1
authentication pre-share
encryption aes
hash md5
group 2
lifetime 86400

I have edited the ASA output to exclude a huge section which is unnecessary and which actually causes the problem.

Let’s see the IOS router debug first.

Outside-R# debug crypto isakmp
*Mar 1 03:15:12.955: ISAKMP:(1001): retransmitting phase 2 QM_IDLE 1349190959 ...
*Mar 1 03:15:12.955: ISAKMP (0:1001): incrementing error counter on node, attempt 2 of 5: retransmit phase 2
*Mar 1 03:15:12.955: ISAKMP (0:1001): incrementing error counter on sa, attempt 2 of 5: retransmit phase 2
*Mar 1 03:15:12.955: ISAKMP:(1001): retransmitting phase 2 1349190959 QM_IDLE
*Mar 1 03:15:12.959: ISAKMP:(1001): sending packet to 192.168.2.1 my_port 500 peer_port 500 (R) QM_IDLE
*Mar 1 03:15:12.959: ISAKMP:(1001):Sending an IKE IPv4 Packet.
*Mar 1 03:15:12.975: ISAKMP (0:1001): received packet from 192.168.2.1 dport 500 sport 500 Global (R) QM_IDLE
*Mar 1 03:15:12.975: ISAKMP: set new node -619099427 to QM_IDLE
*Mar 1 03:15:12.975: ISAKMP:(1001): processing HASH payload. message ID = -619099427
*Mar 1 03:15:12.975: ISAKMP:(1001): processingNOTIFY PROPOSAL_NOT_CHOSEN protocol 3
spi 0, message ID = -619099427, sa = 66772820
*Mar 1 03:15:12.975: ISAKMP:(1001):deleting node -619099427 error FALSE reason "Informational (in) state 1"
*Mar 1 03:15:12.975: ISAKMP:(1001):Input = IKE_MESG_FROM_PEER, IKE_INFO_NOTIFY
*Mar 1 03:15:12.975: ISAKMP:(1001):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE

The highlighted part in the last line show us that Phase-1 is complete.

Which means NOTIFY PROPOSAL_NOT_CHOSEN is a phase-2 problem.

And the reason is a mismatch between the ciphers used for the phase 2 negotiation.

Refer back to the config lines on both the devices we see:

crypto ipsec transform-set ourset esp-aes - router

crypto map outside_map 1 set ikev1 transform-set ESP-AES-128-MD5 - ASA

These lines show control the phase-2 cipher negotiation and both do look the same, 
here the part which I omitted from the ASA config comes in to play.

Here’s the complete ASA config from the CLI

ciscoasa# show running-config crypto
crypto ipsec ikev1 transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-AES-128-SHA-TRANS esp-aes esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-AES-128-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-128-MD5-TRANS esp-aes esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-AES-128-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-192-SHA-TRANS esp-aes-192 esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-AES-192-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-192-MD5-TRANS esp-aes-192 esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-AES-192-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS esp-aes-256 esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-256-MD5-TRANS esp-aes-256 esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-AES-256-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-3DES-SHA-TRANS esp-3des esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-3DES-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-3DES-MD5-TRANS esp-3des esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-3DES-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-DES-SHA esp-des esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-DES-MD5 esp-des esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-DES-SHA-TRANS esp-des esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-DES-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-DES-MD5-TRANS esp-des esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-DES-MD5-TRANS mode transport
crypto ipsec security-association pmtu-aging infinite
crypto map outside_map 1 match address outside_cryptomap
crypto map outside_map 1 set peer 192.168.2.2
crypto map outside_map 1 set ikev1 transform-set ESP-AES-128-MD5 ESP-AES-128-MD5-TRANS
crypto map outside_map interface outside
crypto ca trustpool policy
crypto ikev1 enable outside
crypto ikev1 policy 1
authentication pre-share
encryption aes
hash md5
group 2
lifetime 86400

Now see the highlighted line above, that’s the only cipher we need – because that’s the one in use on the router or else the phase 2 negotiation won’t match and we keep getting the error as shown in the debug.

The whole confusion is that the ASDM doesn’t properly display this and the only thing you see there is just a single cipher.

Maybe this is a bug or I missed something. But for now to get this VPN to work you simply have to remove all the other non-matching ciphers with a “no” command in the ASA CLI.

That is just keep the one cipher which is highlighted or whichever you use in your case.

no crypto ipsec ikev1 transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac
no crypto ipsec ikev1 transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac
no crypto ipsec ikev1 transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac
no crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac
no crypto ipsec ikev1 transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac
no crypto ipsec ikev1 transform-set ESP-AES-128-SHA-TRANS esp-aes esp-sha-hmac
no crypto ipsec ikev1 transform-set ESP-AES-128-SHA-TRANS mode transport
no crypto ipsec ikev1 transform-set ESP-AES-128-MD5-TRANS esp-aes esp-md5-hmac
no crypto ipsec ikev1 transform-set ESP-AES-128-MD5-TRANS mode transport
no crypto ipsec ikev1 transform-set ESP-AES-192-SHA-TRANS esp-aes-192 esp-sha-hmac
no crypto ipsec ikev1 transform-set ESP-AES-192-SHA-TRANS mode transport
no crypto ipsec ikev1 transform-set ESP-AES-192-MD5-TRANS esp-aes-192 esp-md5-hmac
no crypto ipsec ikev1 transform-set ESP-AES-192-MD5-TRANS mode transport
no crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS esp-aes-256 esp-sha-hmac
no crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS mode transport
no crypto ipsec ikev1 transform-set ESP-AES-256-MD5-TRANS esp-aes-256 esp-md5-hmac
no crypto ipsec ikev1 transform-set ESP-AES-256-MD5-TRANS mode transport
no crypto ipsec ikev1 transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
no crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
no crypto ipsec ikev1 transform-set ESP-3DES-SHA-TRANS esp-3des esp-sha-hmac
no crypto ipsec ikev1 transform-set ESP-3DES-SHA-TRANS mode transport
no crypto ipsec ikev1 transform-set ESP-3DES-MD5-TRANS esp-3des esp-md5-hmac
no crypto ipsec ikev1 transform-set ESP-3DES-MD5-TRANS mode transport
no crypto ipsec ikev1 transform-set ESP-DES-SHA esp-des esp-sha-hmac
no crypto ipsec ikev1 transform-set ESP-DES-MD5 esp-des esp-md5-hmac
no crypto ipsec ikev1 transform-set ESP-DES-SHA-TRANS esp-des esp-sha-hmac
no crypto ipsec ikev1 transform-set ESP-DES-SHA-TRANS mode transport
no crypto ipsec ikev1 transform-set ESP-DES-MD5-TRANS esp-des esp-md5-hmac
no crypto ipsec ikev1 transform-set ESP-DES-MD5-TRANS mode transport

That’s it. Now your beautiful VPN tunnel will come up the moment you send the interesting traffic.

This entire confusion is because of the way ASDM displays the ciphers.

I hope to find more info on that and update this further.

Allwyn May 18, 2017

f5 BIG-IP LTM, Web Security

F5 BIG-IP LTM Persistence Cookie Shows Wrong Timezone

As I’m working through the CBTnuggets course for f5 BIG-IP LTM I have set up the following lab as required in the course.

Chapter/Video 15 focuses on the concept of persistenceHere’s persistence from the f5 manual;

Using BIG-IP Local Traffic Manager, you can configure session persistence. When you configure session persistence, Local Traffic Manager tracks and stores session data, such as the specific pool member that serviced a client request. The primary reason for tracking and storing session data is to ensure that client requests are directed to the same pool member throughout the life of a session or during subsequent sessions.

Persistence is simply the BIG-IP tracking the client/user session in a way that the client is sent to the same server till that session expires or till the persistence timeout as set on the BIG-IP expires. For e.g. one type of persistence explained in the course is cookie persistence, which tracks the user by storing an HTTP cookie in the user’s browser.

This cookie is obviously added by the BIG-IP in the HTTP response. We as BIG-IP admins have the option to add a timer in mins for which the cookie remains active on the client and then the cookie expires.

A example would be a user hitting the BIG-IP public IP to fill a form. The first initially selection of the server from the back-end pool is done by the BIG-IP based on the load balancing method in place. Let’s say server green is chosen, once the user hits green server, BIG-IP then adds an HTTP cookie to the HTTP response and from then onwards the remaining requests from the user will only be sent to the green server instead of being load balanced, this can go on till the cookie time as set on the BIG-IP expires or till the session expires—whichever is configured.

In my lab above the issue is that the cookie persistence profile was simply not taking effect and the client kept load balancing to all the three back-end servers in the pool.

f5 troubleshooting manuals strongly recommended setting the correct time on the BIG-IP. I did that rebooted.

[root@big-ip-test:Active:Standalone] config # date
Thu May 18 19:14:31 GST 2017
[root@big-ip-test:Active:Standalone] config #

Wireshark shows that the cookie was being added correctly to the HTTP response but expiry time was wrong. I am adding the cookie at 18:40 GST and the cookie expiry is set to 14:29.

Which means by the time I test this, the cookie has already expired!

This is strange. Also the cookie timezone in wireshark is GMT while I’m in Dubai and the BIG-IP is set to use Asia/Dubai timezone which is GST.

To be updated as I troubleshoot this.

UPDATE – 21 May 2017

DISCLAIMER: It turns out this could simply be a issue with my lab as I’m using my own windows host machine to test the load balancing. The issue was resolved when I created a different VM on virutal box and used GNS3 to connect it to the BIGIP external network and tested the persistence again.

But the troubleshooting is still worth it!

The first thing to get out of the way is the time issue. We can do this using perl.

Download and install strawberry perl for windows and simply find the scaler time.

root@(big-ip-test)(cfg-sync Standalone)(Active)(/Common)(tmos)# run /util bash
[root@big-ip-test:Standalone] # perl -le ‘print scalar time()’
1495179814
root@big-ip-test:Standalone] # exit

[root@big-ip-test:Active:Standalone] config # perl -le ‘print scalar time()’
1495183038

C:Usersallwyn>perl -le “print scalar time”
1495183035

That looks good, so no issues here.

Also the timezone for the cookie is always recommended to be in the GMT timezone since a website has to serve an global audience, I need to do more reading on this though. For next time.

So the packet in the above wireshark pcap is the HTTP response packet which has the cookie inserted by the BIGIP after the initial load balancing. This cookie is then saved by the browser and presented to the BIGIP for the remainder of the session or the cookie timeout as configured on the BIGIP.

Now we know that the BIGIP is inserting the cookie but whether it’s getting back needs to be verified, and the following iRule helps to log this.

when HTTP_REQUEST {
if { [HTTP::header exists Cookie] } {
log local0. “Client: [IP::client_addr] : Incoming Cookie header: [HTTP::header values Cookie]”
} else {
log local0. “Client: [IP::client_addr] : Incoming Cookie header: None”
}
}
when LB_SELECTED {
log local0. “client addr : [client_addr] : Selected pool member: [LB::server addr]”
}

Then to view the logs use this guide. You have to enter the tmsh utility and filter the log events with a pipe |.

[root@big-ip-test:Active:Standalone] config # tmsh
root@(big-ip-test)(cfg-sync Standalone)(Active)(/Common)(tmos)# show /sys log ltm | grep cookie

Test-iRule <lb_selected>: client addr : 192.168.1.5 : Selected pool member: 10.2.0.22
Fri May 19 15:28:00 GST 2017 info big-ip-test tmm[10349] Rule /Common/Cookie-Test-iRule <http_request>: Client: 192.168.1.5 : Incoming Cookie header: None
Fri May 19 15:28:02 GST 2017 info big-ip-test tmm[10349] Rule /Common/Cookie-Test-iRule <lb_selected>: client addr : 192.168.1.5 : Selected pool member: 10.2.0.33</lb_selected></http_request></lb_selected>

You can also view the same logs from in GUI at system – logs – LTM and filter the events.

We see here clearly that the incoming (from client to BIGIP) http_request does not have the cookie set and hence the BIGIP keeps load balancing across the servers.

At this stage, as I already mentioned I just created a new VM and used GNS3 to connect it to the VMware network and tested the persistence.

Here’s the log which then shows the incoming cookie.

ltm 05-19 18:43:13 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <HTTP_REQUEST>: Client: 192.168.1.192 : Incoming Cookie header: BIGipServerour-http-pool=184549898.20480.0000
ltm 05-19 18:43:13 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <LB_SELECTED>: client addr : 192.168.1.192 : Selected pool member: 10.2.0.11

ltm 05-19 18:43:13 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <HTTP_REQUEST>: Client: 192.168.1.192 : Incoming Cookie header: BIGipServerour-http-pool=184549898.20480.0000

ltm 05-19 18:52:05 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <LB_SELECTED>: client addr : 192.168.1.192 : Selected pool member: 10.2.0.22

ltm 05-19 18:52:05 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <HTTP_REQUEST>: Client: 192.168.1.192 : Incoming Cookie header: BIGipServerour-http-pool=369099274.20480.0000

ltm 05-19 18:52:06 info big-ip-test tmm[10349]: Rule /Common/Cookie-Test-iRule <LB_SELECTED>: client addr : 192.168.1.192 : Selected pool member: 10.2.0.22

The BIGIP then halts the load balancing and sends all requests to just one back-end server.

Allwyn May 10, 2017

Juniper

All Juniper Posts — Notes, CLI and GNS3 Lab

These are some posts I wrote while preparing for the JNCIA. They’ll help you setup a lab and work with it and prepare for the certification or just learn juniper – it’s fun!

To be updated. .

Allwyn May 10, 2017

Juniper

GNS3 Juniper SRX Lab And CLI Commands – Part 2

Part-1 is here.

We will continue with our network diagram, just putting it here for ref.

In part-1 we created policies to allow traffic from the trusted to the untrusted zone without NAT. Let’s do NAT and confirm it on the destination test router Outside-R as well.

We will start with Source NAT using the egress interface, then do it using a pool of IP addresses and also see a weird situation I came across.

Just jump to the config mode and then go to the nat hierarchy:

[edit]
admin@SRX1# edit security nat source rule-set IN-OUT

#”IN-OUT is the rule-set name

[edit security nat source rule-set IN-OUT]
admin@SRX1# set from zone trust

[edit security nat source rule-set IN-OUT]
admin@SRX1# set to zone untrust

[edit security nat source rule-set IN-OUT]
admin@SRX1# edit rule IN-OUT

#the rule name can be same or different, comes under the rule-set

[edit security nat source rule-set IN-OUT rule IN-OUT] #now under “rule” not “rule-set”
admin@SRX1# set match source-address 192.168.3.0/24

[edit security nat source rule-set IN-OUT rule IN-OUT]
admin@SRX1# set match destination-address all

[edit security nat source rule-set IN-OUT rule IN-OUT]
admin@SRX1# set then source-nat interface

#the interface is chosen based on the “to zone” – we’re using “untrusted”

[edit security nat source rule-set IN-OUT rule IN-OUT]
admin@SRX1# commit check
configuration check succeeds

The same can be as below, you can copy this by just pasting to notepad and making your changes and then paste to the CLI

[edit]
admin@SRX1# edit security nat source

[edit security nat source] #make sure you are here in the hierarchy
set rule-set IN-OUT from zone trust
set rule-set IN-OUT to zone untrust
set rule-set IN-OUT rule IN-OUT match source-address 192.168.3.0/24
set rule-set IN-OUT rule IN-OUT match destination-address 0.0.0.0/0
set rule-set IN-OUT rule IN-OUT then source-nat interface

Now goto the Inside-R router to confirm this.

Inside-R#ssh -l bob 192.168.2.2

Password:

Outside-R#who
Line       User       Host(s)              Idle       Location
* 99 vty 1     bob        idle                 00:00:00 192.168.2.1 #the egress interface IP

Interface    User               Mode         Idle     Peer Address

We see our incoming connection on the destination router. This is one reason why GNS3 is so much fun, allows you instant confirmation of your config.

Source NAT Using A Pool

Now instead of using the egress interface we will NAT the source IP address using a per-defined pool of addresses.

We just change the NAT rule on top for this, but first create the pool.

[edit security nat] #in the security NAT hierarchy
admin@SRX1# set source pool IN-OUT-POOL address 192.168.2.50 to 192.168.2.100

Now go to the above NAT rules and just add pool instead of the interface. It automatically replaces it in the config.

[edit security nat source]
admin@SRX1# edit rule-set IN-OUT

[edit security nat source rule-set IN-OUT]
admin@SRX1# edit rule IN-OUT

[edit security nat source rule-set IN-OUT rule IN-OUT]
admin@SRX1# set then source-nat pool IN-OUT-POOL

#using pool instead of the interface in the last line

[edit security nat source rule-set IN-OUT rule IN-OUT]
admin@SRX1# commit
commit complete

Always, always remember to commit!

Doing the SSH test from Inside-R to the Outside-R again.

IT DOESN’T WORK!

Then I had to troubleshoot and found a post on Juniper forums. It said proxy-arp is not required if the IP pool range is the same as the range of the egress interface, but a different NAT troubleshooting guide yet recommends configuring proxy arp for the pool. In our case 192.168.2.50 to 192.168.2.100.

As below.

[edit]
admin@SRX1# edit security nat proxy-arp

[edit security nat proxy-arp]
admin@SRX1# set interface ge-0/0/0 address 192.168.2.50 to 192.168.2.100 #our egress

And then testing again from the Inside-R

Inside-R#ssh -l bob 192.168.2.2

Password:

Outside-R#who
Line       User       Host(s)              Idle       Location
* 99 vty 1     bob        idle                 00:00:00 192.168.2.62

Interface    User               Mode         Idle     Peer Address

And the weird situation I saw here, which I don’t yet have a reason for is SSH worked but ping doesn’t work!

Inside-R#ping 192.168.2.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.2, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)

This only happens with source NAT using a pool and not when using the egress interface.

I will definitely get to this and update in the future. Do leave a comment if you have an answer.

This could very be a GNS3 issue because of the proxy-arp or something and have nothing to do with anything else but still SSH has worked, so it’s worth figuring out.

This is too long now and we will do destination NAT in a new post.

1 2 3

About Allwyn

Posts by :