Using Bro to Explore Your Networks, like an AWS WordPress Blog

Home » Sled Meta Blog » Security Research » Using Bro to Explore Your Networks, like an AWS WordPress Blog
My personal WordPress blog was hacked a few weeks ago. I hadn’t checked my simple AWS micro instance in a while, and fortunately, I happened to look at the site just a day after it was compromised. WordPress makes it easy to stand up a pretty website if you don’t want to spend time crafting a front-end yourself, but security is a major question mark. If you keep it up to date and don’t load it with tons of third party content, it’s probably OK for a personal blog. However, I had several WP plugins and themes that were a few weeks behind the last update, as was WP Core. This is likely how the attacker was able to gain write-access to my one and only post on my blog and change the links to point at some sketchy-looking foreign domain websites to improve his SEO. Interestingly enough, the attacker also made some edits in the article, including grammatical fixes and clauses that were consistent with the content (environmental regulations).

Regardless, I updated everything, removed unnecessary plugins, added a plugin to record more detailed WordPress history, secured my active accounts, and reverted the changes (except for one grammatical improvement – thank you attacker). Good WordPress security and statistics plugins are difficult to come by, and they most likely increase the attack surface of the website, so I decided to opt for the tool I use and develop every day: Bro IDS. The visibility provided by Bro goes beyond threat detection and can be used as a more powerful version of netflow, showing what’s running on a system in significant detail.

Monitoring a single web server instance facing external http requests behind a pretty limiting firewall (inbound tcp/80 only, or tcp/22 from my static IP address only) is not our typical deployment for Bro here at Packetsled — limited to basic web requests, and with no honeypot, I didn’t expect a whole lot of interesting attack vectors. However, there were some takeaways after letting Bro run for a few weeks, mostly regarding how my network operates and what kind of requests my site typically sees.

The blog uses basic unverified HTTP (I’m too lazy and cheap to set it up right), so I was unsure why I had at least one ssl log entry every hour:
#fields id.orig_h id.orig_p id.resp_h id.resp_p version cipher curve server_name resumed last_alert next_protocol established cert_chain_fuids client_cert_chain_fuids subject issuer client_subject client_issuer validation_status

172.31.40.119 59718 52.201.183.155 443 TLSv12 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 secp256r1 - F - - F Fb0mOL2796DAoCkRa3 (empty) - - - - -
172.31.40.119 41499 66.155.40.189 443 TLSv12 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 secp256r1 - F - http/1.1 F Fr4ISl3di8C0feHNC4,F2sfO14ZkvhwHhaNb4,F10O3d278YkNUhURTf (empty) - - - - -
Ah of course — after opening up the corresponding x509 certificate log, it’s clear that my device is reaching out to WordPress about twice a day, looking for updates from “CN=*.wordpress.org,OU=Domain Control Validated CN=Go Daddy Secure Certificate Authority – G2,OU=http://certs.godaddy.com/repository/,O=GoDaddy.com.” More surprisingly though, my instance periodically was reaching out to an Amazon-based address every hour on my exposed network (as opposed to the back channels AWS uses to control instances):
pi@raspberrypi:~ $ whois 52.201.183.155
NetRange: 52.192.0.0 - 52.223.255.255 
CIDR: 52.192.0.0/11 
... 
Organization: Amazon Technologies Inc. (AT-88-Z) 
x509.log
#fields certificate.version certificate.serial certificate.subject certificate.issuer certificate.not_valid_before certificate.not_valid_after certificate.key_alg certificate.sig_alg certificate.key_type certificate.key_length certificate.exponent certificate.curve san.dns san.uri san.email san.ip basic_constraints.ca basic_constraints.path_len
3 07 emailAddress=info@bitrock.com,CN=stats.bitnami.com,O=BitRock Update Services,L=Seville,ST=Seville,C=ES emailAddress=info@bitrock.com,CN=BitRock Update Services,O=BitRock Update Services,L=Seville,ST=Seville,C=ES 1284478570.000000 1599838570.000000 rsaEncryption sha1WithRSAEncryption rsa 1024 65537 - - - - - F -
3 280392F6950C4C CN=*.wordpress.org,OU=Domain Control Validated CN=Go Daddy Secure Certificate Authority - G2,OU=http://certs.godaddy.com/repository/,O=GoDaddy.com\\, Inc.,L=Scottsdale,ST=Arizona,C=US 1417824798.000000 1513368681.000000 rsaEncryption sha256WithRSAEncryption rsa 2048 65537 - *.wordpress.org,wordpress.org - - - F -
Again using Bro’s x509 log to extract more information about the certificate owner, this is Bitnami, whose VM and tech stack I used to set up my WordPress instance. It resolves to stats.bitnami.org. That must have been something I missed in the ‘Terms and Conditions’.

The dhcp.log was about what you would expect with a rarely changing IP, but some of the DNS queries my machine made were interesting. Most of them are pretty clear: ‘security.ubuntu.com’ for updates, ‘stats.bitnami.org’ for my newly discovered stats reporting mechanism, and plenty of internal AWS domain server lookups. Others were less clear, such as a large number of queries for my own website name www.leolinsky.com (never figured out that one, but I assume it’s related to one of my WordPress plugins), and a Google lookup ( crawl-66-249-79-106.googlebot.com — also related to one of my statistics and monitoring plugins, I believe)
dns.log
#path dns
#fields id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id rtt query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected
172.31.40.119 59480 172.31.0.2 53 udp 2863 - 254.118.225.73.in-addr.arpa - - - - 0 NOERROR F F F T 0 c-73-225-118-254.hsd1.wa.comcast.net 60.000000 F  
172.31.40.119 52552 172.31.0.2 53 udp 42576 - 2.0.31.172.in-addr.arpa - - - - 0 NOERROR F F F T 0 ip-172-31-0-2.us-west-2.compute.internal 60.000000 F 
172.31.40.119 60949 172.31.0.2 53 udp 48886 - 114.79.249.66.in-addr.arpa - - - - 0 NOERROR F F F T 0 crawl-66-249-79-114.googlebot.com 60.000000 F 
172.31.40.119 36634 172.31.0.2 53 udp 35400 - www.leolinsky.com - - - - 0 NOERROR F F F T 0 35.162.195.120 60.000000 F 
172.31.40.119 49864 172.31.0.2 53 udp 30057 - stats.bitnami.org - - - - 0 NOERROR F F F T 0 52.201.183.155 60.000000 F 
172.31.40.119 44066 172.31.0.2 53 udp 39877 - security.ubuntu.com - - - - 0 NOERROR F F F T 0
The http log is where it quickly became clear that no one visits my website except bots, malicious entities, and myself. Note that my instance’s internal IP is 172.31.40.119.
http.log
#fields id.orig_h id.orig_p id.resp_h id.resp_p trans_depth method host uri referrer version user_agent request_body_len response_body_len status_code status_msg info_code info_msg tags username password proxied orig_fuids orig_filenames orig_mime_types resp_fuids resp_filenames resp_mime_types
35.162.195.120 39028 172.31.40.119 80 1 POST leolinsky.com /wp-cron.php?doing_wp_cron=1488229413.0440499782562255859375 http://leolinsky.com/wp-cron.php?doing_wp_cron=1488229413.0440499782562255859375 - WordPress/4.7.2; http://leolinsky.com 0 0 - - - - (empty) - - - -- - - - -
172.31.40.119 49929 169.254.169.254 80 1 - - - - 1.0 - 0 8 200 OK - - (empty) - - - - - - FIOPkb38ZkF8rKEOel - -
172.31.40.119 49930 169.254.169.254 80 1 - - - - 1.0 - 0 345 404 Not Found - - (empty) - - - - - - FZXQ67dF6Wx9MbUrh - text/html
35.162.195.120 39047 172.31.40.119 80 1 POST www.leolinsky.com /wp-cron.php?doing_wp_cron=1488230712.3863990306854248046875 http://www.leolinsky.com/wp-cron.php?doing_wp_cron=1488230712.3863990306854248046875 - WordPress/4.7.2; http://www.leolinsky.com 0 0 - - - - (empty) - -- - - - - - -
66.249.79.110 56031 172.31.40.119 80 1 GET www.leolinsky.com /2016/11/ - - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 0 0 - -- - (empty) - - - - - - - - -
180.76.15.27 50507 172.31.40.119 80 1 GET www.leolinsky.com /tag/non-aggression/ - - Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) 0 0- - - - (empty) - - - - - - - - -
92.85.68.75 57131 172.31.40.119 80 4 POST www.leolinsky.com /wp-login.php - - Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1 51 0 - -- - (empty) - - - FgaxLN10GdbeKnj1db - text/plain - - -
81.11.190.250 65495 172.31.40.119 80 2 POST www.leolinsky.com /xmlrpc.php - - Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1 216 0 - -- - (empty) - - - FvY2Ce1KqOaUXXiiTf - application/xml - - -
81.11.190.250 65495 172.31.40.119 80 3 GET www.leolinsky.com /wp-login.php - - Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1 0 0 - -- - (empty) - - - - - - - - -
81.11.190.250 65495 172.31.40.119 80 5 GET www.leolinsky.com /wp-login.php - - Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1 0 0 - -- - (empty) - - - - - - - - -
81.11.190.250 65495 172.31.40.119 80 1 POST www.leolinsky.com /xmlrpc.php - - Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1 216 0 - -- - (empty) - - - F3EeSQyDVfg8BS47k - application/xml - - -
81.11.190.250 65495 172.31.40.119 80 4 POST www.leolinsky.com /wp-login.php - - Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1 50 0 - -- - (empty) - - - Ftq19Q1J1z3ik6f3N4 - text/plain - - -- - -
172.31.40.119 38934 35.162.195.120 80 1 - - - - 1.1 - 0 0 200 OK - - (empty) - - - - - - - - -
89.211.211.103 49616 172.31.40.119 80 2 POST www.leolinsky.com /xmlrpc.php - - Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1 217 0 - -- - (empty) - - - FKL6zw4tA23qNtOcK5 - application/xml - - -
195.230.145.22 47823 172.31.40.119 80 1 GET 35.162.195.120 / -- curl/7.17.1 (mips-unknown-linux-gnu) libcurl/7.17.1 OpenSSL/0.9.8i zlib/1.2.3 0 0 - - - - (empty) - - - - - - - - -
We can see how Chrome identifies itself from my system via the user-agent string while I was logged into my website ( this can be linked to a Chrome version; the OS information is irrelevant ).
x.x.x.x 48144 172.31.40.119 80 1 GET www.leolinsky.com /wp-admin/admin-ajax.php?action=SimpleHistoryNewRowsNotifier&apiArgs[since_id]=146&apiArgs[dates]=lastdays:30&ip-geo-block-auth-nonce=f8bc15686e http://www.leolinsky.com/wp-admin/ - Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 0 0 - - - - (empty)-- - - - - - - -
Here’s an example of an IP that attempted to login to my site, according to a WP history plugin:
99.20.91.64 52001 172.31.40.119 80 1 POST www.leolinsky.com /xmlrpc.php - - Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1 215 0 - - - - (empty) - - - Fsqr162lYxheinQ95g - application/xml - - -
This attacker also triggered some “Weird” logs with malformed TCP flows, perhaps attempting known out-dated WP vulnerabilities to gain access:
99.20.91.64 52001 172.31.40.119 80 bad_TCP_checksum
99.20.91.64 52001 172.31.40.119 80 data_before_established
99.20.91.64 52001 172.31.40.119 80 inappropriate_FIN
99.20.91.64 52049 172.31.40.119 80 data_before_established
99.20.91.64 52049 172.31.40.119 80 inappropriate_FIN
I noticed a large number of traffic from the unroutable address 169.254.169.254, which was interesting because it included HTTP requests, even triggering 404’s. I’m not sure what Amazon was doing there, although it used a second Amazon-owned IP as well (35.162.195.120). Amazon is known to alert users when Bitcoin mining is detected on their instances (a common use case of a compromised account), so it’s certainly possible that they do other web monitoring as well.

There were also a number of foreign-originated tcp sessions, most of which seemed like bots (the patterns become obvious when plotted): In summary, there’s Bro’s conn log with its exhaustive list of netflow-like entries. I was interested in the longest flow (sorted with ` awk ‘NR > 4′ < conn.log | sort -t$’\t’ -k 9 -n ` from bro.org) which was a 30 second long AWS tcp flow which generated all kinds of weird logs.
x.x.x.x 44675 172.31.40.119 80 tcp http 4.120405 1186 0 SH F T 0ScDAF 6 1438 0 0 (empty) x.x.x.x 41411 172.31.40.119 80 tcp http 4.937924 1186 0 SH F T 0ScDAF 6 1438 0 0 (empty) 172.31.40.119 60090 91.189.88.162 80 tcp - 7.811027 0 984617 SHR T F 0^hCacdf 0 0 43 946737 (empty) 172.31.40.119 39107 54.71.119.81 80 tcp - 30.188989 0 2746583 SHR T F 0^hCdcaf 0 0 884 2535428 (empty)
bitnami@ip-172-31-40-119:/usr/local/bro/logs/current$ bitnami@ip-172-31-40-119:/usr/local/bro/logs/current$ grep -rnw "54.71.119.81"
files.log:18: 54.71.119.81 172.31.40.119 Cw2Zz81Z7gWRzH8Lph HTTP 0 MD5,SHA1 text/html - 0.000000 F F 325 325 0 0 F - 3db7e12da4a623a3a10784b27db3d48f b785d991b655989751ce05841301484d0c636f1a - - files.log: 54.71.119.81 172.31.40.119 Cw2Zz81Z7gWRzH8Lph HTTP 0 MD5,SHA1 text/plain - 0.000000 F F 31562 65876 0 0 F - 0b132e685181d75acc5ec0623996868c 04f782fff2f7a5e194cba0b147399923e44dbc54 - - conn.log: 172.31.40.119 39107 54.71.119.81 80 tcp - - - - OTH T F 0C 0 0 0 0 (empty) conn.log: 172.31.40.119 39107 54.71.119.81 80 tcp - 30.188989 0 2746583 SHR T F0 ^hCdcaf 0 0 884 2535428 (empty) http.log: 172.31.40.119 39107 54.71.119.81 80 1 - - - - 1.1 - 0 325 404 Not Found - - (empty) - - - - - - FufUJjC7AjvIX9Xh3 - text/html http.log: 172.31.40.119 39107 54.71.119.81 80 2 - - - - 1.1 - 0 31562 200 OK - - (empty) - - - - - - FGd2Jb1Y3F3Pa6y9va - text/plain weird.log: 172.31.40.119 39107 54.71.119.81 80 active_connection_reuse - F bro weird.log: 172.31.40.119 39107 54.71.119.81 80 truncated_tcp_payload - F bro weird.log: 172.31.40.119 39107 54.71.119.81 80 above_hole_data_without_any_acks - F bro
And connection summary put out by Bro:
>== Total === 2017-02-27-00-00-03 - 2017-02-27-00-58-20
- Connections 51.0 - Payload 12.6k -
Ports | Sources | Destinations | Services | Protocols | States |
80 62.7% | 172.31.40.119#1 80.4% | 169.254.169.254#2 31.4% | - 49.0% | 6 66.7% | SHR 56.9% |
53 27.5% | 35.162.195.120#3 5.9% | 172.31.0.2#4 27.5% | dns 27.5% | 17 33.3% | OTH 23.5% |
67 5.9% | 79.31.239.12#5 3.9% | 172.31.40.119#6 19.6% | http 17.6% | | SH 17.6% |
443 3.9% | 66.249.64.110#7 3.9% | 35.162.195.120#8 11.8% | dhcp 5.9% | | S0 2.0% |
| 68.180.230.223#9 2.0% | 172.31.32.1#10 5.9% | | | |
| 66.249.64.163#11 2.0% | 52.201.183.155#12 3.9% | | | |
| 66.249.64.114#13 2.0% | | | | |
#1=ip-172-31-40-119.us-west-2.compute.internal
#2=instance-data.us-west-2.compute.internal #3=ec2-35-162-195-120.us-west-2.compute.amazonaws.com
#4=ip-172-31-0-2.us-west-2.compute.internal #5=host12-239-dynamic.31-79-r.retail.telecomitalia.it #6=ip-172-31-40-119.us-west-2.compute.internal
#7=crawl-66-249-64-110.googlebot.com #8=ec2-35-162-195-120.us-west-2.compute.amazonaws.com
#9=b115511.yse.yahoo.net
#10=ip-172-31-32-1.us-west-2.compute.internal
#11=crawl-66-249-64-163.googlebot.com #12=ec2-52-201-183-155.compute-1.amazonaws.com
#13=crawl-66-249-64-114.googlebot.com

Bro is a powerful tool. However, the raw data can still be overwhelming — many of these log excerpts were already heavily redacted for readability. At Packetsled, we build and rank our detections to filter out the noise and further decorate Bro’s statistics, allowing both visibility and real-time threat detection.
in Security Research by Leo Linsky Comments are off

© 2017 PacketSled, Inc.