Jump to content

About This Club

F5 gets its own club just because there is so much to cover.
  1. What's new in this club
  2. How to add user How to verify.. verify what IP address SNMP query will see run the following on F5 to find out (10.47.178.99 is the SNMP source that is querying the F5) ip route get 10.47.178.99 10.47.178.99 via 10.47.200.1 dev mgmt src 10.47.200.203 cache In the above example: the source (10.47.178.99) needs to be querying the F5 at 10.47.200.203 verify F5 is seeing snmp queries with snmp v3 user c0wboy On F5 Device: tcpdump -ni 0.0:nnn -s0 port 161 | grep "c0wboy" On Jumpbox (or device in SNMP allowed list) run: SNMP v3: snmpget -v 3 -u c0wboy -a SHA -A snmp\!auth\-pwd\! -x AES -X snmpprivpwd -l authPriv 10.47.200.203 sysSystemUptime.0 SNMP v2: snmpwalk -v 2c -c Public 10.47.200.203 F5-BIGIP-SYSTEM-MIB::sysSystemUptime.0 NOTE: password must use \ to escape special characters. ..
  3. Another good F5 article that references: zrd process exits upon the 'journal rollforward failed: journal out of sync with zone'
  4. This forum topic is my journey figuring out why one of our BIG-IP GTM's (BIG-IP DNS) was disconnected from the sync group These broadcast messages were blowing up the cli Broadcast message from systemd-journald@usfnt1slbgtm99.thezah.corp (Wed 2022-02-23 12:18:52 CST): logger[16928]: Re-starting zrd The BIG-IP GTM system may occasionally log the following error message to the console and to the /var/log/daemon.log file: Re-starting zrd tmsh show sys service zrd zrd finish (pid 26149) 0 seconds, 21 starts, 6209 restarts The zrd process is the ZoneRunner daemon. The ZoneRunner utility is used to create, manage, and maintain DNS files on the BIG-IP GTM system. Various issues may cause the zrd process to restart. The most common issues include the following: Syntax errors in the BIND configuration file Syntax errors in the BIND zone files ZoneRunner records do not match BIND zone files To determine why the zrd process is restarting, perform the following procedure: Log in to the command line. If the Re-starting zrd error message is repeatedly logged to the console, you can prevent the zrd process from attempting to restart by typing the following command: bigstart stop zrd After the zrd process halts, use a text reader such as the less command to review the the /var/log/gtm file for error messages related to the named.conf file, or specific zone file. For example: less /var/log/gtm For example, the following log entry indicates the journal file for the test.com zone is out of sync with the zone file. The error message also indicates that the test.com zone was not found. This behavior is caused by the errant space in the zone name in the named.conf file: zrd[26232]: 0115020b:3: Errors in config file zone siterequest.com/IN: loaded serial 55. zone m1.test.net/IN: loaded serial 2009020301. db.external.test.com.:3: ignoring out-of-zone data (test.com). zone test.\032com/IN: journal rollforward failed: journal out of sync with zone. externa. l/test. com./IN: not found. Depending on the error message found in the /var/log/gtm file, you may need to refer to one of the following articles: If an error message indicates a syntax issue with the named.conf file, refer to K6963: Managing the BIG-IP BIND configuration file. If an error message indicates a syntax issue with a BIND zone file, refer to K7032: Freezing zone files to allow manual update to ZoneRunner-managed zone files. If an error message indicates that ZoneRunner records do not match BIND zone files, refer to K5738: ZoneRunner records do not match BIND zone files. After you have fixed the source of the error, restart the zrd process by typing the following command: bigstart start zrd error: zone domain.com/IN/external: journal rollforward failed: journal out of sync with zone The zrd process exits when it encounters this error message. Additionally, you cannot restart the zrd process until you resolve the error condition. Manually deleting the zone journal files from the BIG-IP DNS/GTM system FIRST identify what zones you have issues with: grep named /var/log/daemon.log Feb 23 12:00:49 usfnt1slbgtm99.thezah.corp err named[13617]: 23-Feb-2022 12:00:49.713 general: error: zone thezah.com/IN/external: journal rollforward failed: journal out of sync with zone Feb 23 12:00:49 usfnt1slbgtm99.thezah.corp err named[13617]: 23-Feb-2022 12:00:49.720 general: error: zone thezah.com/IN/external: not loaded due to errors. Feb 23 12:00:49 usfnt1slbgtm99.thezah.corp err named[13617]: 23-Feb-2022 12:00:49.723 general: error: zone int.thezah.com/IN/external: journal rollforward failed: journal out of sync with zone Feb 23 12:00:49 usfnt1slbgtm99.thezah.corp err named[13617]: 23-Feb-2022 12:00:49.723 general: error: zone int.thezah.com/IN/external: not loaded due to errors. NOTE: You may have too many errors that filled up the latest daemon.log so you may have to search the older files which you can do with: zgrep named /var/log/daemon* In the above example you can see the zones that are having issues are thezah.com and int.thezah.com Location zone files: ls -ltrh /var/named/config/namedb ls -ltrh /var/named/config/namedb total 128K -rw-r--r--. 1 named named 337 Oct 12 2014 db.external.abc. -rw-r--r--. 1 named named 350 Sep 12 2018 db.external.int.thezah.com.https. -rw-r--r--. 1 named named 14K Nov 15 19:48 db.external.int.thezah.com.~ -rw-r--r--. 1 named named 3.3K Nov 15 19:49 db.external.thezah.com.~ -rw-r--r--. 1 named named 984 Nov 15 19:53 db.external.eventguyz.com..jnl -rw-r--r--. 1 named named 7.5K Nov 15 19:53 db.external.thezah.corp..jnl -rw-r--r--. 1 named named 1.7K Nov 15 19:53 db.external.cowboydenny.com..jnl -rw-r--r--. 1 named named 4.1K Nov 15 20:04 db.external.thezah.corp. -rw-r--r--. 1 named named 469 Nov 15 20:05 db.external.eventguyz.com. -rw-r--r--. 1 named named 881 Nov 15 20:05 db.external.cowboydenny.com. -rw-r--r--. 1 named named 22K Nov 16 21:22 db.external.int.thezah.com..jnl -rw-r--r--. 1 named named 11K Nov 16 21:22 db.external.thezah.com..jnl -rw-r--r--. 1 named named 3.4K Nov 16 21:33 db.external.thezah.com. -rw-r--r--. 1 named named 14K Nov 16 21:34 db.external.int.thezah.com. Stop ZonerRunner: tmsh stop /sys service zrd Stop named: tmsh stop /sys service named Remove the journal files (.jnl) for the affected zones rm /var/named/config/namedb/db.external.thezah.com..jnl rm /var/named/config/namedb/db.external.int.thezah.com..jnl Start services backup tmsh start /sys service named tmsh start /sys service zrd
  5. The BIG-IP is a 'full' proxy. This means there are 2 separate and independent connections that are managed by the BIG-IP. We refer to those as the Clientside (incoming traffic) and Serverside (outgoing traffic). Whether the traffic originates on your external or internal side is irrelevant to a BIG-IP. It is where the connection originates and hits that virtual server (clientside) and exits the BIG-IP (serverside). So, in most cases to track an issue or resolve a question you need the traffic for both connections and then be able to align that traffic. To do that there are 2 things that will help. One is referred to as the 'p' flag. This will instruct the BIG-IP to catch the flow on both sides of the BIG-IP. The other is to drop the capture into Wireshark and look at the F5 Ethernet headers. To gather those you need the interface modifier :nnn. When using a 'p' flag in the capture syntax, and running the capture on the BIG-IP, it will instruct the BIG-IP to capture the traffic on both sides of the BIG-IP. The syntax is fairly simple to construct. For example, you want to track the traffic for a specific virtual server and its pool members, and it would look something like this... tcpdump -nni 0.0:nnnp -s0 --f5 ssl host 10.17.199.72 or host 10.17.21.59 or host 10.17.21.61 or host 10.17.26.87 or 10.17.24.74 or 10.17.24.75 -vw /var/tmp/tcpdump_VS-POOL_$(date +%d_%b_%H_%M_%S)_$HOSTNAME.pcap Lets break that down... -s0 is an Unlimited Snaplen. SnapLen, Snap Length, or snapshot length is the amount of data for each frame that is actually captured by the network capturing tool and stored into the CaptureFile. This will provide the most data. -nn Don’t convert host addresses to names. This is used to avoid DNS lookups. i 0.0 Capture the traffic on interface 0.0 which tells the BIG-IP to use 'any' interface to gather this traffic on. :nnnp Here you see the 'p' flag and what we call "full noise" by the use of the 'nnn'. This will create the information for the F5 Ethernet Trailers and the ‘p’ the traffic on both sides of the proxy. ‘host’ The IP of the virtual server or source IP. If the virtual server destination is 0.0.0.0:0 (referred to as any:any) then the source IP is what will be needed. ‘port’ the specific port used by the virtual server. This helps to reduce the size of the capture. A virtual IP address can be used many times but must be used in conjunction with a port number to form a websocket. So, if you are using 192.168.1.1:443 for ssl and then using it as 192.168.1.1:161 for SNMP and 192.168.1.1:25 for SMTP you would want to filter out the traffic you do not need to look at by specifying the port. -v will add verbosity and provide and screen counter so you see if packets are being caught, how many, and how fast. -w this will send it to the file location. /var/tmp/hostname the path to the location and the file name. You can name it what you want. But remember it is a Linux system and there are format rules. The file does not need to be created before the capture is started as the system will do it for you. .pcap is the file type. (.cap is still used but is not quite as effective and pcapng is the newest form. In most cases all will work) Important: When using the 'p' flag simply typing Ctrl + c will not stop some background functions of the 'p' flag and this can cause some issues. To completely stop it you can kill the PID or simply stop it by typing... killall tcpdump Note: There are some times in which a 'p' flag cannot be used, such as when HTTP/2 and some other protocols are in play. In that case this article should help you deal with those circumstances...K87524842: The tcpdump 😛 option does not capture peer traffic on some types of flows Note: Beginning in Wireshark 2.6.0, the f5ethtrailer dissector is built into the utility. To display TMM information in Wireshark 2.6.0 and later, navigate to Analyze > Enabled Protocols and search for f5ethtrailer. Click the options to enable the F5 Ethernet trailer. If you are using a Wireshark version before 2.6.0, F5 recommends upgrading your Wireshark to the latest version to make use of this feature. You can also use the TMM information to filter the dump using some additional F5 details. For example, the following Wireshark filter string shows traffic to and from TMM0 on Slot1: f5ethtrailer.slot == 1 and f5ethtrailer.tmm == 0 A list of all F5 filters appears in Wireshark within Filter Expression. Pre-REQ for SSL tmsh modify sys db tcpdump.sslprovider value enable Identify Client IP Address Identify Virtual Server IP Address Identify Pool Members BEGIN the capture If you know the client ip address: tcpdump -ni 0.0:nnnp -s0 --f5 ssl host [client ip address] -w /var/tmp/api-qa_tcpdump_client_$(date +%d_%b_%H_%M_%S)_$HOSTNAME.pcap Otherwise, filter for the virtual ip and pool member ip(s): tcpdump -ni 0.0:nnn -s0 --f5 ssl host [virtual server ip] or host [pool member ip] or host [pool member ip] -w /var/tmp/api-qa_tcpdump_VS_$(date +%d_%b_%H_%M_%S)_$HOSTNAME.pcap When you open the capture file... How to Filter HTTP Traffic in Wireshark Filtering HTTP traffic in Wireshark is a fairly trivial task but it does require the use of a few different filters to get the whole picture. Many people think the http filter is enough, but you end up missing the handshake and termination packets. Wireshark HTTP Protocol Filter To display packets using the HTTP protocol you can enter the following filter in the Display Filter Toolbar: http You’ll notice that all the packets in the list show HTTP for the protocol. The unfortunate thing is that this filter isn’t showing the whole picture. You’re missing the setup handshakes and termination tcp packets. To display all the HTTP traffic you need to use the following protocol and port display filter: tcp.dstport == 443 Now you’ll see all the packets related to your browsing of any HTTPS sites you browsed while capturing. Filtering HTTP Traffic to and from Specific IP Address in Wireshark If you want to filter for all HTTP traffic exchanged with a specific you can use the “and” operator. If, for example, you wanted to see all HTTPS traffic related to a server with an IP of 10.17.199.72 you could use the following filter: tcp.dstport == 443 and ip.addr == 10.17.199.72 Notice only packets with 10.17.199.72 in either the source or destination columns is shown. You can also use the OR or || operators to create an “either this or that” filter. tcp.dstport == 443 || ip.addr == 10.17.199.72 Wireshark HTTP Method Filter If you want to dig into your HTTP traffic you can filter for things like GET, PUT, POST, DELETE, HEAD, OPTIONS, CONNECT, and TRACE. To filter for these methods use the following filter syntax: http.request.method == "requestmethod" For example, if you wanted to filter for just the GET requests, enter the following filter in the Display Filter toolbar: http.request.method == "GET" Now you’re left with all of the GET requests for assets from the website. Viewing HTTP Packet Information in Wireshark Working with the GET Method Filter displayed above, click on a packet in the Packet List Pane and then look at the information in the Packet Details Pane. Expand the Hypertext Transfer Protocol detail: Wireshark HTTP Response Filter One of the many valuable bits of information in a HTTP conversation is the response. This is the code a website returns that tells the status of the asset that was requested. You’ve probably seen things like Error 404 (Not Found) and 403 (Forbidden). These are HTTP responses and only a couple of the many that exist. To filter for all responses enter the following display filter: http.response Notice to the right of the protocol version information there is a column of numbers. These are your response codes. If you see 200 in this example which means the HTTP request was successful. To filter for a specific response, such as a HTTP 200 (OK), HTTP 301 (Moved Permanently), or HTTP 404 (Not Found) use the following display filter: http.response.code == 200 Change 200 to another code to search for that code. Follow the Full HTTP Stream to Match Get Requests with Responses A very handy feature of Wireshark is the ability to view streams in a human readable format from beginning to end. To this, pick a HTTP protocol packet such as the packet containing the 200 response that we saw earlier and right click on it. Click on Follow -> HTTP Stream. You’ll now be presented with a window that shows the entire stream including the GET (red) and HTTP/1.1 200 OK (Blue) As you can see, there is a lot to HTTP traffic and just filtering for the HTTP protocol doesn’t cut it. If you really want to put the whole picture together when troubleshooting problems with accessing websites you have to take a multi-pronged approach.
  6. Many times I removed a device before removing it from bigIQ then I have to scramble on how to figure out how to fix it. This time I am putting down the steps here so I can reference it later since its more common than I thought. In order to force the removal of the BIG-IPs from the BIG-IQ, please follow the below Plan of Action. Note: please ensure you insert your own User and Password where prompted. This will display all discovered BIG-IPs associate to the BIG-IQ: restcurl -u admin:YourPassword /mgmt/cm/system/machineid-resolver | jq '.items[] | {uuid,hostname,version,product,managementAddress,selfLink}' --This will display similar to below: "uuid": "11111111-2222-3a3b-4c4d-555555abcdef", "hostname": "BIG-IP-1", "version": "15.1.2.1", "product": "BIG-IP", "managementAddress": "10.0.0.1", "selfLink": "https://localhost/mgmt/cm/system/machineid-resolver/11111111-2222-3a3b-4c4d-555555abcdef" Record the UUID of the device you need to remove. Run the following command with the UUID to force the BIG-IP to be removed from the BIG-IQ: restcurl -u <user:password> -X GET /mgmt/shared/resolver/device-groups/ |grep selfLink | grep -o "/mgmt/.*[[:alnum:]]" | xargs -I {} restcurl -u <user:password> -X DELETE {}/devices/<UUID> Note: This command requires to replace your credentials twice Example: restcurl -u admin:YourPassword -X GET /mgmt/shared/resolver/device-groups/ |grep selfLink | grep -o "/mgmt/.*[[:alnum:]]" | xargs -I {} restcurl -u admin:YourPassword -X DELETE {}/devices/11111111-2222-3a3b-4c4d-555555abcdef Running this will display and scroll several lines to the screen. Let this run to completion. Once the command is complete, the BIG-IP device should be fully removed from the BIG-IQ GUI. REFERENCE: https://support.f5.com/csp/article/K45451482
  7. I added new devices to bigIQ and then I tried to import services (all done via the add multiple devices where you tell it to discover services automatically). It hung for a long time and then I went to do it manually and it gives me the following error: So the solution I found here which basically says: On the device you are having issues, under services click Import You have to look at the /var/log/restjavad.0.log and look for the task that is getting stuck: grep "mport task" /var/log/restjavad.0.log Then delete the task via REST: restcurl -X DELETE '/cm/adc-core/tasks/declare-mgmt-authority/bfcf5974-44be-4805-8e98-786427c57651' Do the same for all the pending tasks that you see in the logs. Now, from BigIQ CM's GUI, remove all services associated with BigIP and eventually remove BigIP. Next, re-add the same BigIP and verify if discovery and import works.
  8. This is a discussion about the differences on Health Monitors for your LTM Pool and how to test them before you rely on them. Simple monitors Simple monitors only monitor the node address itself, not individual protocols, services, or applications on a node. gateway_icmp, icmp, tcp_echo, tcp_half_open Extended content verification (ECV) monitors ECV monitors use Send and Receive string settings to retrieve content from pool members or nodes. tcp, http, https, and https_443 # time curl https://10.72.54.184 curl: (51) SSL: certificate subject name '*.services.hosangit.com' does not match target host name '10.72.54.184' real 0m0.259s user 0m0.093s sys 0m0.050s
  9. Here is a nice video / tutorial that you may also find helpful And this video might be helpful as well
  10. There has been a few times when an upgrade broke an application but I didn't know since logs looked good and everything appeared to be good. So now I run a before (B4) and an after test and then I compare to get more details on status changes and what is affected vs just knowing the number of available WideIPs isn't the same. GTM CHANGE B4 rm -rf /var/tmp/*B4.txt rm -rf /var/tmp/*AFTER.txt tmsh save sys config tmsh load sys config verify tmsh save sys ucs /var/tmp/$HOSTNAME"."$(date +%Y%m%d)B4.ucs qkview -s0 date > /var/tmp/$HOSTNAME"."$(date +%Y%m%d)B4.txt echo "F5 MasterKey:" >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)B4.txt f5mku -K >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)B4.txt echo "." echo "Number of Available WideIPs:" >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)B4.txt tmsh show /gtm wideip | egrep 'Gtm::|WideIP|Availability|State|Reason' | grep -c ': available' >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)B4.txt tmsh show /gtm wideip | egrep 'Gtm::|WideIP|Availability|State|Reason' > /var/tmp/$HOSTNAME"."$(date +%Y%m%d)wideipB4.txt echo "." echo "Number of Available Pools:" >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)B4.txt tmsh show /gtm pool | egrep 'Gtm::|Pool|Availability|State|Reason' | grep -c ': available' >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)B4.txt tmsh show /gtm pool | egrep 'Gtm::Pool|Availability|Reason' > /var/tmp/$HOSTNAME"."$(date +%Y%m%d)poolsB4.txt echo "." echo "Number of Available Servers:" >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)B4.txt tmsh show /gtm server all | egrep 'Gtm::|Server|Availability|State|Reason' | grep -c ': available' >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)B4.txt tmsh show /gtm server all | egrep 'Gtm::Server|Availability|Reason' > /var/tmp/$HOSTNAME"."$(date +%Y%m%d)serversB4.txt echo "." echo "iQuery:" tmsh show /gtm iquery | egrep 'Gtm::IQuery|Server|State' | grep -c 'connected' >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)B4.txt tmsh show /gtm iquery | egrep 'Gtm::IQuery|Server|State' > /var/tmp/$HOSTNAME"."$(date +%Y%m%d)iQueryB4.txt echo "." echo "Number of Available Data Centers:" >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)B4.txt tmsh show gtm datacenter all | egrep 'Gtm::|Datacenter|Availability|State|Reason|Connections' | grep -c ': available' >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)B4.txt tmsh show gtm datacenter all | egrep 'Gtm::|Datacenter|Availability|State|Reason|Connections' > /var/tmp/$HOSTNAME"."$(date +%Y%m%d)dcB4.txt cat /var/tmp/$HOSTNAME"."$(date +%Y%m%d)B4.txt GTM CHANGE AFTER date > /var/tmp/$HOSTNAME"."$(date +%Y%m%d)AFTER.txt echo "F5 MasterKey:" >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)AFTER.txt f5mku -K >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)AFTER.txt echo "." echo "Number of Available WideIPs:" >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)AFTER.txt tmsh show /gtm wideip | egrep 'Gtm::|WideIP|Availability|State|Reason' | grep -c ': available' >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)AFTER.txt tmsh show /gtm wideip | egrep 'Gtm::|WideIP|Availability|State|Reason' > /var/tmp/$HOSTNAME"."$(date +%Y%m%d)wideipAFTER.txt tmsh reset-stats gtm wideip echo "." echo "Number of Available Pools:" >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)AFTER.txt tmsh show /gtm pool | egrep 'Gtm::|Pool|Availability|State|Reason' | grep -c ': available' >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)AFTER.txt tmsh show /gtm pool | egrep 'Gtm::Pool|Availability|Reason' > /var/tmp/$HOSTNAME"."$(date +%Y%m%d)poolsAFTER.txt tmsh reset-stats gtm pool echo "." echo "Number of Available Servers:" >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)AFTER.txt tmsh show /gtm server all | egrep 'Gtm::|Server|Availability|State|Reason' | grep -c ': available' >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)AFTER.txt tmsh show /gtm server all | egrep 'Gtm::Server|Availability|Reason' > /var/tmp/$HOSTNAME"."$(date +%Y%m%d)serversAFTER.txt tmsh reset-stats gtm server echo "." echo "iQuery:" tmsh show /gtm iquery | egrep 'Gtm::IQuery|Server|State' | grep -c 'connected' >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)AFTER.txt tmsh show /gtm iquery | egrep 'Gtm::IQuery|Server|State' > /var/tmp/$HOSTNAME"."$(date +%Y%m%d)iQueryAFTER.txt tmsh reset-stats gtm iquery echo "." echo "Number of Available Data Centers:" >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)AFTER.txt tmsh show gtm datacenter all | egrep 'Gtm::|Datacenter|Availability|State|Reason|Connections' | grep -c ': available' >> /var/tmp/$HOSTNAME"."$(date +%Y%m%d)AFTER.txt tmsh show gtm datacenter all | egrep 'Gtm::|Datacenter|Availability|State|Reason|Connections' > /var/tmp/$HOSTNAME"."$(date +%Y%m%d)dcAFTER.txt tmsh reset-stats gtm datacenter cat /var/tmp/$HOSTNAME"."$(date +%Y%m%d)AFTER.txt GTM COMPARE diff /var/tmp/*wideipB4.txt /var/tmp/*wideipAFTER.txt diff /var/tmp/*poolsB4.txt /var/tmp/*poolsAFTER.txt diff /var/tmp/*serversB4.txt /var/tmp/*serversAFTER.txt diff /var/tmp/*iQueryB4.txt /var/tmp/*iQueryAFTER.txt diff /var/tmp/*dcB4.txt /var/tmp/*dcAFTER.txt
  11. Cowboy Denny

    Velos

    Performance and reliability of hardware. Agility and scale of a modern architecture.
  12. Cowboy Denny

    rSeries

    F5 rSeries is a rearchitected, next-generation hardware platform that scales application delivery performance and automates application services to address many of today’s most critical business challenges.
  13. Cowboy Denny

    iSeries

    BIG-IP iSeries appliances accelerate app delivery in regulated environments with quick and easy programmability, multi-vendor service orchestration, software-defined hardware and line-rate performance.
  14. Cowboy Denny

    Viprion

    Here are images / videos of everything to do with F5 Viprions
  15. This is my next attempt to get the upgrade of BIG-IQ from 8.1.0.1 to 8.2.0 1) Create new install volume using addvol addvol HD1.2 /shared/images/BIG-IQ-8.2.0-0.0.310.iso 2) Verify the newly created /var is at least 40 GB (based on anticipated resulting /var size of 20 GB) lvscan | grep _var ACTIVE '/dev/vg-db-sda/set.1._var' [75.00 GiB] inherit ACTIVE '/dev/vg-db-sda/set.2._var' [40.00 GiB] inherit <-- This should show 40 GB or more If the size of set.2._var is less than 40 GB, resize it to 40G resizevol /var 41943040 3) Install V8.2 with no reboot tmsh install sys software image BIG-IQ-8.2.0-0.0.310.iso volume HD1.2 create-volume 4) Mount the new volume volumeset -f mount HD1.2 cd /mnt/HD1.2 a) Make changes to allow appiq daemons to log to the proper location: find var/config/appiq -name "log4j2.xml" -exec vi {} \; Modify the RollingFile name= line to include /var/log/appiq/ for the fileName and filePattern Under <Root level = "info"> add <AppenderRef ref="RollingFile"/> For postaggregator only, modify the following line: From: <DefaultRolloverStrategy max="10"/> To: <DefaultRolloverStrategy max="10"> Or Follow the work-around steps in https://cdn.f5.com/product/bugtracker/ID1117597.html and download the gz to /shared/tmp and extract the files then run the following i. yes | cp /shared/tmp/agentmanager_log4j2.xml /mnt/HD1.2/var/config/appiq/agentmanager/config/log4j2.xml ii. yes | cp /shared/tmp/configserver_log4j2.xml /mnt/HD1.2/var/config/appiq/configserver/config/log4j2.xml iii. yes | cp /shared/tmp/queryservice_log4j2.xml /mnt/HD1.2/var/config/appiq/queryservice/config/log4j2.xml iv. yes | cp /shared/tmp/postaggregator_log4j2.xml /mnt/HD1.2/var/config/appiq/postaggregator/config/log4j2.xml b) Make change to /mnt/HD1.2/usr/share/rest/tokumon/config/modules/global.js }, "declaration": { "enabled": false }, <--- Add "body": { <--- Add "enabled": false <--- Add } c) Make changes to /mnt/HD1.2/etc/biq_daemon_provision.json to change tokumond find.oplog and send.oplog. Under big_iq.tokumond, modify the batch_size for send_find_buffer_allocation.SYS_32GB and send_oplog_buffer_allocation.SYS_32GB to 324 (match SYS_16GB) I discussed the changes that were made to big_iq.elasticsearch (SYS_23GB increased to 4000m) and big_iq.appiqpostaggregator (SYS_32GB increased to 1000m) on September first during an online session for https://tron.f5net.com/sr/1-8494851811 / https://f5.my.salesforce.com/5001T00001kD2B4). There were no specific notes on this change that I could find, so I decided the change was likely an attempt to fix the UI not coming online. It was later determined that the UI was not coming online because of tokumond, so these changes ultimately had no affect getting the system to function. Because of this I'm omitting these changes. d) Move /mnt/HD1.2/etc/cron.daily/update-top-pg-tables to /etc/cron.d mv /mnt/HD1.2/etc/cron.daily/update-top-pg-tables /mnt/HD1.2/etc/cron.d e) chmod 644 /mnt/HD1.2/etc/cron.d/update-top-pg-tables 5) Reboot into the new volume, which will initiate the remainder of the upgrade. tmsh reboot volume HD1.2 6) Monitoring the upgrade progress: Bootstrap (took about 10 minutes): This is the process initiated following the reboot that gets the database loaded and upgraded to the current version. The progress can be monitored using the following command: tail -f /var/log/bootstrap/bootstrap-<dateTime>.out The bootstrap process is finished when we see the following message: ==== BOOTSTRAP SUCCESSFUL ==== RBAC Reset (took about 9 hours): This process is initiated following the database upgrade that causes all of the role based access controls to be rebuilt. This is the process that can cause /var to grow rapidly. The progress can be monitored using the following command: psql -U postgres -d bigiq_db -c "select count(*) from bigiqauthorization.shared_authorization_journals;" The RBAC Reset is complete when the count goes to zero. It will jump up to 7K or so within the first few minutes of the process. You can create a short script to monitor the progress at an interval (change the sleep value at the end to control that): while true ; do date ; psql -U postgres -d bigiq_db -c "select count(*) from bigiqauthorization.shared_authorization_journals;" ; sleep 60 ; done Tokumon Database (took about 10 minutes): This process builds the cross reference tables used by the UI. The progress can be monitored using the following command: While waiting for RBAC Reset to complete, the tokumon logs will show the following messages: tailf /var/log/tokumon/current [INFO] postgres_configurator: configurePostgresqlIsReady:readWriteReady. ready:true [INFO] postgres_configurator: configurePostgresqlIsReady:replicationReady. ready:true [INFO] postgres_configurator: configurePostgresqlIsReady:rbacReady. ready:false [INFO] postgres_configurator: is db ready. dbIsReady:false [WARNING] tokumon: isDBReady db is NOT ready. Once the RBAC Reset is complete, the tokumon database rebuild will actually start. tail -f /var/log/tokumon/current | grep Progress: Once the Progress bar shows percentComplete:98., check to see that it saves the checkpoint indicating it finished completely: [INFO] mode: setting Silent to false [INFO] tokumon: Saving Checkpoing.... Checkpoint{ _id: '649/5649DC58', After the above messages, the logs will begin to show messages similar to these: [INFO] logical-replication: Acknowledge 649/5C1E5918 [INFO] logical-replication: Acknowledge 649/5C4FC188 [INFO] logical-replication: Acknowledge 649/5CFCEBA8 guiserver (seconds after tokumon database is finished): This is responsible for the UI. The progress can be monitored using the following command: tail -f /var/log/guiserver/current While waiting for RBAC Reset to finish, we will see messages similar to the following: info: f5IndexingStatus: checking status of index... info: f5IndexingStatus: received 404 response, indicating initial indexing is not yet complete info: initial indexing not yet complete, reason: no indexing status document found While waiting for Tokumon Database to be rebuilt, we will see messages similar to the following: info: f5IndexingStatus: checking status of index... info: initial indexing not yet complete, reason: indexing mode is find The UI should be online and accessible when we see the following message: info: initial indexing is complete
  16. How I change the admin password from the command line is just running the following command tmsh modify auth user admin password yourAdminPassword
  17. Running 15.1.2.1 and when you first bootup you get prompted to change your root password via cli so you do and it changes the admin default password to that new root password as well. Then you try and log in via the GUI and it prompts you to change your admin password which you use the new root password as current and you add a new password for admin but it just won't take. Keeps erroring with
  18. We get asked all the time to take a tcpdump of the app on the F5 since they always think its the F5 (which maybe 1% of the time it is) So this is how I do it (which may be the wrong way but it works for me) Step 1: Enable ssl decrypt (ONLY IF TROUBLESHOOTING SSL ISSUES) It is often necessary to create a decrypted capture in order to track down an issue while troubleshooting. DB Variable tcpdump.sslprovider has been introduced and will cause the LTM to save TLS Master Secret, Client Random and Server Random data to the end of each TLS Packet. In order to leverage the new functionality, add --f5 ssl to the tcpdump flags. tmsh modify sys db tcpdump.sslprovider value enable Important NOTICE: When you perform a tcpdump capture with tcpdump.sslprovider enabled, understand that the TLS master secret will be written to the tcpdump capture itself. Be careful with whom you share the capture file. Using the "ssl" option captures additional information related to the SSL/TLS connections, such as master secrets. This enables some packet capture analysis tools to decrypt the SSL/TLS payload in the captured packets. Use only as needed for troubleshooting purposes, and handle captured data with caution. Step 2: Create datagroup with Source IP's and/or Subnets To much traffic to capture if you don't limit/filter down what you are capturing and you don't ever want to fill up your /shared partition because bad things happen. Easiest way is to have a , seperated txt file with you IP Addresses or subnets.. below is an example of the different txt files. Since datagroup can only be IP Address, String or Integer (never combined) then which ever one you are building (in my case it was IP Address) you have an example below. Note every entry has key, value (note address you don't need a value but you can) and key and value are seperated in your text file with := Address # cat ext_dg_address.txt host 192.168.1.1, host 192.168.1.2 := "host 2", network 192.168.2.0/24, network 192.168.3.0 mask 255.255.255.0 := "network 3", network 192.168.4.0 prefixlen 24, IMPORT using: tmsh create /sys file data-group ext_dg_address_file separator ":=" source-path file:/var/tmp/ext_dg_address.txt type ip String # cat ext_dg_string.txt "name1" := "value1", "name2" := "value2", "name3" := "value3", IMPORT using: tmsh create /sys file data-group ext_dg_string_file separator ":=" source-path file:/var/tmp/ext_dg_string.txt type string Integer # cat ext_dg_integer.txt 1 := "test 1", 2 := "test 2", 3 := "test 3", IMPORT using: tmsh create /sys file data-group ext_dg_integer_file separator ":=" source-path file:/var/tmp/ext_dg_integer.txt type integer Step 3: Create an iRule to decrypt the data First you need to identify if you are using ssl cache (if its enabled) which more than likely it is. when you run the tmsh command below and you get a cache-size more than 0 then cache is enabled. tmsh list ltm profile client-ssl <ssl client profile> cache-size ltm profile client-ssl <ssl client profile> { cache-size 262144 <—DEFAULT but can be 262,144 sessions to 4,194,304 sessions } Your iRule should look like this (based on the datagroup you created above for address named ext_dg_address_file) when CLIENTSSL_HANDSHAKE { if {[class match [getfield [IP::client_addr] "%" 1] equals ext_dg_address_file] } { log local0. "CLIENT_Side_IP:TCP source port: [IP::client_addr]:[TCP::remote_port]" log local0. "CLIENT_RANDOM [SSL::clientrandom] [SSL::sessionsecret]" log local0. "RSA Session-ID:[SSL::sessionid] Master-Key:[SSL::sessionsecret]" } } when SERVERSSL_HANDSHAKE { if {[class match [getfield [IP::client_addr] "%" 1] equals ext_dg_address_file] } { log local0. "CLIENT_Side_IP:TCP source port: [IP::client_addr]:[TCP::remote_port]" log local0. "CLIENT_RANDOM [SSL::clientrandom] [SSL::sessionsecret]" log local0. "RSA Session-ID:[SSL::sessionid] Master-Key:[SSL::sessionsecret]" } } Now you are prepped for Implementation/Testing. Is your application tester ready to begin testing the application that you would like to capture an error or the traffic? If yes, then proceed Add your newly created iRule to the Virtual Server you want to decrypt and capture traffic Then Start your capture Run tcpdump (remove -f5 ssl from commands below if you don't need the SSL traffic decrypted) tcpdump -ni 0.0:nnn -s0 --f5 ssl host [virtual server IP] or host [pool member 1 IP] or host [pool member 2 IP] -W 10 -C 100 -w /var/tmp/app_tcpdump_VS_$(date +%d_%b_%H_%M_%S)_$HOSTNAME.pcap -W 10 (means rotate up to 10 files) -C 100 (means each file is up to 100MB in size) -w write output of capture to that file Alternative1 command for Step2 that should show packets on the screen while also sending to the .pcap file tcpdump -ni 0.0:nnn -s0 --f5 ssl host [virtual server IP] or host [pool member 1 IP] or host [pool member 2 IP] -U -w - | tee /var/tmp/app_tcpdump_VSandMEMBERS_$(date +%d_%b_%H_%M_%S)_$HOSTNAME.pcap | tcpdump -r - Alternative2, an additional command for Step2 that could be used to see packets on the screen while also writing to a pcap file tail -F -n+0 $pcapfile | tcpdump -r - AFTER you capture the error and press CTRL + C to break out of tcpdump you have a few things left to do capture the pre-master shared keys by running sed -e 's/^.*\(RSA Session-ID\)/\1/;tx;d;:x' /var/log/ltm > /var/tmp/sessionsecrets_$(date +%d_%b_%H_%M_%S)_$HOSTNAME.pms Download the .pms file you just created NEXT merge all the pcap files since you should have no more than 10 pcap files that end with pcap0, pcap1, pcap2, etc.. and you merge these files by running a command like this mergecap -w /var/tmp/app_tcpdump_VS_10_Nov_20_53_17__server1.hosangit.com.pcap $(ls /var/tmp/app_tcpdump_VS_10_Nov_20_53_17__server1.hosangit.com.pcap*|xargs) Download the .pcap file you just created Final step on the F5 is to remove the iRule you created above from the Virtual Server. HOW TO VIEW Capture in Wireshark Open the .pcap you downloaded and click on Wireshark - Preferences - Protocols and scroll down until you find TLS Now while you are on TLS you should see (Pre)-Master-Secret log filename click Browse to the right of that and click on the .pms file you downloaded and click OK
  19. Next I have to try what is stated in this article https://support.f5.com/csp/article/K14532110 In short... Download the BIG-IQ 8.2.0 disk-sizing tools from F5 Extract the tools: tar xpf /shared/F5_Networks_Disk_Size_Tools.tar -C /shared/. Discover what size disk BIG-IQ 8.2.0 needs: /shared/F5_Networks_Disk_Size_Tools/imageplan /shared/images/BIG-IQ-8.2.0-0.0.310.iso Standard plan (500GB HDD) Mount point: /, Size: 450000k Mount point: /usr, Size: 10485760k Mount point: /config, Size: 3320000k Mount point: /var, Size: 78643200k Tiny plan (95GB) Mount point: /, Size: 450000k Mount point: /usr, Size: 10485760k Mount point: /config, Size: 500000k Mount point: /var, Size: 26214400k Utilize vgdisplay to see what size your HDD drive is # vgdisplay --- Volume group --- VG Name vg-db-sda System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 188 VG Access read/write VG Status resizable MAX LV 0 Cur LV 12 Open LV 7 Max PV 0 Cur PV 1 Act PV 1 VG Size 498.50 GiB PE Size 4.00 MiB Total PE 127617 Alloc PE / Size 84563 / 330.32 GiB Free PE / Size 43054 / 168.18 GiB VG UUID niI0Pt-8xVm-GghP-hXHK-X1YG-5Bm6-wWFAII lvs create volume manually: addvol 1.2 /shared/images/BIG-IQ-8.2.0-0.0.310.iso # addvol 1.2 /shared/images/BIG-IQ-8.2.0-0.0.310.iso create volume 1.2 for image /shared/images/BIG-IQ-8.2.0-0.0.310.iso product BIG-IQ version 8.2.0 build 0.0.310 (BIGIQ820) selected Creating new location sda, 2... warning: tm_install::VolumeSet::choose_filesystem_plan -- Selected standard plan (disk = 524288000, sz_standard = 92898960) warning: tm_install::VolumeSet::choose_filesystem_plan -- Selected standard plan (disk = 524288000, sz_standard = 92898960) reboot Verify HD1.2 exists (software is not installed to it yet) : tmsh show sys software status when you run lvs you will see the volumes are not large enough and you will need to resize the volumes first Normally the only volume that is wrong is the /var volume (set to 75GB instead of 120GB) so you need to resize it by running: resizevol /var 125829120 (NOTE: it does it for all logical volumes regardless of what partition you are on (HD1.1, HD1.2, HD1.3) but it doesn't affect anything.) NOW the partition is prepared, install the OS to it. cd /shared/images tmsh install /sys software image BIG-IQ-8.2.0-0.0.310.iso volume HD1.2 tmsh show /sys software status Confirm that /var is still set to 120GB and not 75GB still: lvs When ready to boot to the new volume run: tmsh reboot volume HD1.2 ROLLBACK is super easy tmsh reboot volume HD1.1 (original working volume) tmsh delete /sys software volume HD1.2 (new volume you created and booted to unsuccessfully)
  20. Here are some notes I have collected to help troubleshoot DNS issues on the F5 GTM in hopes it helps someone. If you have some additional helpful notes, please share. check out the logs for zone transfers AXFR IXFR
    • One Right Answer
    • 5 minutes
    • 10 Questions
    • 0 Players
    This quiz briefly covers troubleshooting Layer 1 Connectivity Layer 2 Connectivity Layer 3 Connectivity HTTP Troubleshooting BIG-IP Troubleshooting
    • One Right Answer
    • 5 minutes
    • 12 Questions
    • 0 Players
    This quiz covers the following topics HTTP TLS - SSL VPN DNS NTP Syslog SNMP
    • One Right Answer
    • 5 minutes
    • 10 Questions
    • 0 Players
    Here are the topics this quiz will cover Monitoring Device and Software Upgrade Traffic Flow iHealth
    • One Right Answer
    • 5 minutes
    • 13 Questions
    • 0 Players
    This quiz is focused on the following Load Balancing Technology Concepts Configuring Load Balancing in ADC Health Monitors Profiles Persistance iRules High Availability
    • One Right Answer
    • 5 minutes
    • 16 Questions
    • 0 Players
    This quiz is about F5 Networking Basics which covers Switching Concepts IP Addressing and Subnetting Routing Concepts Configuring ADC Networking Transport Layer Network Address Translation (NAT) Dynamic Host Configuration Protocol (DHCP)
  21. If you installed the new version on HD1.2 and the original version is on HD1.1 to rollback you just type: tmsh reboot volume HD1.1 Then when it finally comes back online delete HD1.1 tmsh delete /sys software volume HD1.2 Before you reinstall 8.2.0 on HD1.2 make sure your cluster is green curl -s localhost:9200/_cluster/health?pretty { "cluster_name" : "a6f1a0d6-c57a-40bf-8037-eef75a6b7f5b", "status" : "green", "timed_out" : false, "number_of_nodes" : 9, "number_of_data_nodes" : 8, "active_primary_shards" : 1567, "active_shards" : 3140, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } If it's green, then continue to reinstall the image cd /shared/images tmsh install /sys software image BIG-IQ-8.2.0-0.0.310.iso volume HD1.2 create-volume tmsh show sys software status Once it shows fully installed, reboot to the new image tmsh reboot volume HD1.2 If you get stuck with Waiting for BIG-IQ services to become available then check for errors tailf /var/log/tokumon/current 2022-09-30_12:42:29.95716 [INFO] postgres_configurator: configurePostgresqlIsReady:readWriteReady. ready:true 2022-09-30_12:42:29.95725 [INFO] postgres_configurator: configurePostgresqlIsReady:replicationReady. ready:true 2022-09-30_12:42:29.95750 [INFO] postgres_configurator: configurePostgresqlIsReady:rbacReady. ready:false 2022-09-30_12:42:29.95758 [INFO] postgres_configurator: is db ready. dbIsReady:false 2022-09-30_12:42:29.95778 [WARNING] tokumon: isDBReady db is NOT ready Then try this solution article: https://support.f5.com/csp/article/K61023744
  22.  
×
×
  • Create New...