# Media

url: https://www.youtube.com/watch?v=4lQ2GnmTFkY
title: DEF CON 32 - Leveraging private APNs for mobile network traffic analysis - Aapo Oksman
type: video
author: DEFCONConference
ingestion_date: 2024-11-10
transcription_model: large-v2
chunking_status: completed
content:
{
  "webpage_url": "https://www.youtube.com/watch?v=4lQ2GnmTFkY",
  "title": "DEF CON 32 - Leveraging private APNs for mobile network traffic analysis - Aapo Oksman",
  "description": "Knowing where and how your mobile and IoT devices communicate on the Internet is essential for ensuring privacy and security.\n\nIn the past, it has been easy to follow their communication through a WIFI connection that you control. However, your devices are becoming more locked down and utilize mobile networks such as 4G and 5G for communication. As the devices communicate directly through mobile network base stations operated by Internet Service Providers (ISPs), tampering with or even monitoring their communication is outside your reach.\n\nWhile it is possible to set up a private base station, it requires expensive components and is hard to operate. However, many ISPs have begun offering private Access Point Names (APNs) to allow you to have a private network inside the ISP infrastructure.\n\nThis talk will show how you can affordably leverage ISP-operated mobile networks and their private APN services to control your mobile devices' network traffic. This technique lets you inspect, filter, and tamper with your mobile devices' IP traffic for offensive and defensive cyber security needs, such as penetration testing IoT devices or monitoring mobile device endpoints for malicious traffic.",
  "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
  "duration": 2486,
  "channel": "DEFCONConference",
  "uploader": "DEFCONConference",
  "upload_date": "20241016"
}

0.00s - 8.48s | This text was transcribed using whisper model: large-v2

 So my name is Aapo. I do a lot of things with IoT, but especially for my work
8.48s - 15.32s |  security research, consulting, and mainly look at IoT and other kinds of devices.
15.32s - 20.96s |  I work a lot on cryptography and network protocols. Outside of that,
20.96s - 25.88s |  bug bounty. So if you run a hardware bug bounty program, you know where to find
25.88s - 36.02s |  me, and I'm also a coach for a Finnish CTF team. But yeah, so here is some IoT
36.02s - 44.90s |  and my trusty EPC, and I want to kind of see how it works. I might want to hack it
44.90s - 53.22s |  or I might want to check out something about it. This particular device connects
53.22s - 58.98s |  to the Internet with an Ethernet port or Wi-Fi. So that should be quite
58.98s - 64.02s |  straightforward. If we want to know where it's connecting to, and that's a
64.02s - 71.98s |  big part of IoT, we can simply plug in an Ethernet cable to the device and the
71.98s - 78.08s |  other end to our laptop, and then on our laptop in the Ethernet interface, we will
78.08s - 85.62s |  see any traffic it's trying to send. We can do the same with Wi-Fi. So if we have
85.62s - 91.38s |  our access point, the device connected to it, and then again we will see the
91.38s - 99.97s |  traffic on the Ethernet interface. But there are some kind of devices, and
99.97s - 105.81s |  especially industrial IoT, that have a very specialized job and they have been
105.81s - 111.53s |  made to be very cheap. So they have only one way to connect to the Internet, and
111.53s - 117.61s |  nowadays a big part of it is some kind of something else than Ethernet or
117.61s - 125.05s |  Wi-Fi. So today we will be talking about these kind of devices. So some kind of
125.05s - 131.61s |  4G connection on this one, and how that works. There's a modem, you plug in a SIM
131.61s - 137.01s |  card and it connects to 4G, and it's connected to the Internet. But the
137.01s - 145.37s |  problem is, how do we actually see the data? So this is kind of the topic of
145.45s - 151.86s |  today. There are also different kind of devices, they might have multiple ways of
151.86s - 157.34s |  connecting to the Internet. So here we have a phone currently connected to my
157.34s - 161.18s |  Wi-Fi access point, and I can see everything it's sending to the Internet.
161.18s - 168.14s |  Or can I? So if you might not be able to see it, but there's a little 5G icon
168.14s - 173.62s |  there, so it's actually connected to 5G as well. So a question is, is there
173.62s - 181.30s |  traffic over 5G as well? So it might be something built-in, there might be
181.30s - 191.46s |  something malicious. So let's talk about that. So why would you want to look into
191.46s - 197.34s |  mobile network traffic? If you're just a tinkerer, if you're at DEF CON here, or
197.34s - 202.38s |  watching this video, then you might be curious. You might care about your
202.38s - 209.30s |  privacy. You might think, where do my devices talk to? Is there some kind of
209.30s - 215.18s |  unencrypted private information flying through the air? And if you are in
215.18s - 220.54s |  traveling in some countries, you might want to kind of know what exactly your
220.54s - 227.94s |  devices are sending. From offensive security point of view, we might think
227.94s - 232.02s |  that can we tamper with the traffic and find vulnerabilities? If you do
232.02s - 238.06s |  penetration testing, and you have these kind of devices, you need to hack them,
238.06s - 245.06s |  you need to kind of do some kind of security audit or something like that,
245.06s - 250.14s |  you might want to look at the traffic. Can you actually say we've kind of
250.14s - 258.49s |  penetration tested this, if you haven't looked at the mobile traffic? From
258.49s - 263.29s |  defensive point of view, you might want to detect malicious traffic, or just
263.29s - 269.53s |  filter some unwanted things. You can do this in most cases on the device, so if
269.53s - 274.37s |  you have a phone, you can install something and it will kind of filter
274.37s - 278.85s |  things, or endpoint detection, or whatever. But if you have some advanced
278.85s - 284.53s |  malware on your phone, it might not respect the on-device protections. So if
284.53s - 291.61s |  it's running on very low inside the operating system, it can easily just not
291.61s - 299.30s |  allow the endpoint protection to run. So we might not want to do the testing on
299.30s - 310.97s |  the device, we want to do it outside. So what are these mobile networks? Sometimes
310.97s - 316.53s |  called like cellular networks, and these come in different flavors. There is 2G,
316.53s - 325.53s |  3G, 4G, 5G, etc. Some older ones, but I think these are the currently used,
325.53s - 333.37s |  and in 4G and in 5G, there are these slightly different, like NB-IoT is a
333.37s - 342.53s |  special version of LTE, but the main thing is that you have different
342.53s - 352.54s |  frequency bands and special radios that can talk with these protocols. Usually
352.54s - 358.38s |  the mobile networks are operated by Internet Service Providers, so ISPs, and
358.38s - 364.30s |  they have massive amounts of base stations all over the world, so if you
364.30s - 371.02s |  have a civilization, you most likely have a mobile network there. And how
371.02s - 376.62s |  you use these mobile networks is you just buy a subscription from an ISP, they
376.62s - 381.86s |  will give you a SIM card or send it to you, and then you plug the SIM card into
381.86s - 388.10s |  your device, the SIM card authenticates you to the network, and then you connect.
388.10s - 396.02s |  You can kind of send text messages or phone calls, but nobody really does that
396.02s - 401.74s |  anymore, so it's mainly used to route IP traffic from your device to the
401.74s - 408.65s |  Internet or wherever you want to go. Usually this mobile network stuff, this
408.65s - 414.13s |  is very specialized, and it comes in this separate hardware module called a modem,
414.13s - 422.69s |  and that's a separate device on your device. These modems are very complex,
422.69s - 427.81s |  usually run their own operating systems, and are basically a small computer that
427.81s - 438.51s |  you just connect to your board, and then it handles a lot of things. Here you can
438.51s - 445.43s |  see a picture of this, so on the left we have a mobile device, it has a CPU and
445.43s - 452.35s |  the modem, and the CPU runs some kind of operating system, Linux or Windows or
452.35s - 458.07s |  whatever, and then talks with the modem and offloads the actual mobile network
458.07s - 463.83s |  thing to it, and then you just give it data, and the data somehow finds its way
463.83s - 473.47s |  into the Internet. On 4G we have these cell towers, so these are the base
473.47s - 478.59s |  stations that are all over the world, and they, your device usually connects to
478.59s - 484.51s |  the nearest one or the one with the least amount of traffic, so they handle
484.51s - 491.99s |  it themselves. The cell tower is connected to the ISP infrastructure, so
491.99s - 497.27s |  there's different kind of things, we're not going to go that in depth, because
498.07s - 503.99s |  this talk not only covers 4G, but all kinds of mobile networks. But usually
503.99s - 510.39s |  they have similar way of operating, so first the radio connects to a serving
510.39s - 515.99s |  gateway, and some kind of authentication mechanism, so the SIM card from the
515.99s - 523.31s |  modem talks with the off, and then your mobile device is then, the traffic is
523.31s - 533.13s |  routed with these packet gateways to the Internet. So how do we intercept this
533.13s - 537.57s |  mobile network traffic? So if we want to analyze it, we need to see it, to see it
537.57s - 546.43s |  we need to somehow intercept it at some point. So of course the easiest way to
546.43s - 553.71s |  intercept traffic is on the device, so just log in, run TCP dump, and you will
553.71s - 560.31s |  see the network connections. So if you have a Linux based device, the CPU is
560.31s - 564.83s |  sending the data to the modem, but you can look at the traffic already on the
564.83s - 570.47s |  device, so no need to any kind of fancy things. So if you can do this, it's very
570.47s - 575.35s |  powerful, and especially if you want to know if your device is accepting
575.35s - 579.31s |  connections, you might want to take a look at that from the device, so no need
579.31s - 584.47s |  to scan all UDP ports if you just like type in a command that shows you all
584.47s - 591.32s |  listening UDP ports. This however requires root access to the device, so if
591.32s - 597.24s |  you want to hack something, you might not have this, or if you have some kind
597.24s - 602.88s |  of secure device, like an iPhone, you might not be able to have enough
602.88s - 608.98s |  permissions to do this. And also I said that the modem has its own operating
608.98s - 613.66s |  system, might have some over there updates, some of the functionality can be
613.66s - 619.30s |  offloaded to it, so for example FTP, you can just say to the modem that hey
619.86s - 625.82s |  download these files from this FTP server and hand them to me. And that might not
625.82s - 630.78s |  be secure, and if you're just looking at the traffic on the device, you will most
630.78s - 635.26s |  likely you will not see that. And if you're doing penetration testing and you
635.26s - 638.50s |  have been relying on this, I have bad news for you, you haven't kind of done
638.50s - 647.53s |  your work really well. And we have found a lot of vulnerabilities in the modems.
647.53s - 652.93s |  So if we actually want to intercept the traffic, there is the CPU, there's a
652.93s - 657.89s |  modem, so how this works is they have some kind of serial interface, usually
657.89s - 663.93s |  not encrypted because you don't really need to, and just kind of look at the
663.93s - 670.09s |  board and find the traces and sniff them. This works with these AT commands,
670.09s - 674.21s |  sometimes a little bit different, but basically the CPU says to the modem that
674.25s - 678.85s |  hey please make a connection to Google and then send this data through that
678.85s - 686.59s |  connection and you will get the traffic. So you can just kind of sniff the
686.59s - 692.39s |  traffic and decode it and or even replace the modem with your own software
692.39s - 698.79s |  based simulator. So this is something that we've already done. In some cases
698.79s - 704.27s |  it's quite simple, if you have kind of clear traces and you just solder your
704.55s - 711.26s |  serial port adapter to it and pull the modem to reset, you can do this. But of
711.26s - 716.14s |  course it requires hardware reverse engineering and some tiny soldering and
716.14s - 725.74s |  again there's very complex setup sometimes, so it might not be as easy as
725.74s - 736.07s |  it's shown here. And we cannot see the modem's own traffic here as well. I
736.19s - 740.63s |  usually have written my own because it's quite simple to do, but I found this
740.63s - 746.67s |  GitHub project that seems to have made a similar thing. So if you want to play
746.67s - 756.04s |  around with this you can look at that and try it out. The next thing I tried to
756.04s - 761.08s |  run my own base station because I want to see the modem traffic. I figured out
761.08s - 767.84s |  just like I have my Wi-Fi access point, I'll do a 4G access point. You can buy
767.84s - 774.88s |  this, you can build your own and the software defined radios can do pretty
774.88s - 779.60s |  much anything nowadays. Just a couple hundred bucks and open source software
779.60s - 786.08s |  and you can do this. A basic setup, like a thousand bucks, not that much if you're
786.08s - 792.76s |  into kind of professional penetration testing, but it requires a week worth of
792.80s - 798.40s |  work to get running and if you want to buy the commercial thing then it's really
798.40s - 806.52s |  expensive. The problem with these, the radio bands are heavily licensed, they
806.52s - 812.96s |  are used quite a lot and that means that kind of all available bands have been
812.96s - 817.96s |  bought by someone, so it's illegal to run without a Faraday cage. So you need to
817.96s - 824.60s |  have a kind of pouch that you put your radio in and your device in and you
824.60s - 829.40s |  have to kind of be careful not to leak the thing because someone will find you
829.40s - 835.52s |  and come find you. Some of the frequency bands can be licensed, some countries
835.52s - 843.32s |  have unlicensed bands, but then you're limited into that one band or that one
843.32s - 850.12s |  physical location where you are running this. So if you're into this, there's the
850.12s - 855.08s |  LibraCellular, they have good instructions how to do this, but I have
855.08s - 863.88s |  something better for you. So let's go back to the 4G mobile network. We have
863.88s - 869.32s |  our mobile device, it talks to the cell tower and then there's the authentication
869.32s - 879.14s |  and some other stuff. But what exactly is that other stuff? Like I said, the devices
879.18s - 884.54s |  are authenticated with SIM cards and then there's some kind of a packet
884.54s - 891.22s |  filtering going on. There is this thing called lawful interception that kind of
891.22s - 898.38s |  means that your traffic can be inspected by the authorities and I think if it can
898.38s - 906.10s |  be inspected then it most likely is. But yeah, other things that it does is
906.34s - 913.06s |  policy enforcement. So basically it means if you buy a slow speed subscription, it
913.06s - 919.38s |  will make sure that you won't get good speeds. They calculate the billing, so if
919.38s - 923.54s |  you have a kind of per megabyte subscription then they calculate the
923.54s - 931.62s |  megabytes. Then they route the packets. So there might be some kind of ISP
931.66s - 936.74s |  internal things like DNS servers, so some of the things might not go to the
936.74s - 941.74s |  internet at all. So they might go to kind of internal DNS service or something
945.74s - 952.47s |  else. But also the packets can be routed internally in different ways. So we have
955.75s - 960.75s |  these things called APNs, which basically mean that how to get the packets to the
963.39s - 968.39s |  ISP. So this is something that if you're old, you know what these are because
973.03s - 977.63s |  you've kind of gotten a new SIM card, you put it in and it doesn't work and then
977.63s - 982.11s |  you need to go to these and then you need to type something. But nowadays you
982.11s - 986.51s |  might not even know that these exist. So and if you need to type something in
986.51s - 992.39s |  just type in internet and then you get to the internet. These are actually used
993.39s - 999.23s |  so you might have some kind of restricted internet or something. So some
999.23s - 1004.23s |  ISPs offer some kind of child safe things so they do more more things. Or
1008.37s - 1012.95s |  some kind of at some point you can kind of get more speed if you from some
1012.95s - 1016.95s |  certain ISP. So if you don't use the internet but you use something else then
1016.95s - 1020.99s |  it's faster because there's less people there or or something like that. But
1020.99s - 1025.99s |  usually just you use what comes with the subscription. So these a big change
1030.07s - 1034.63s |  nowadays that these are pushed usually through the network to your device. So
1034.63s - 1038.37s |  when you connect and you don't have an APN, the modem asks the network that
1038.37s - 1045.38s |  hey, do you have something for me and then it's automatically set up. You can
1045.38s - 1051.40s |  change these, configure them manually on the device, but it might not be
1051.42s - 1055.30s |  respected by the ISP. So it basically just tells the ISP hey this is what I
1055.30s - 1063.72s |  want but it can of course offer you whatever it wants. So there is this
1065.00s - 1070.00s |  thing called private APN. So something that you can kind of buy as a service.
1074.36s - 1079.38s |  So if you run big fleets of IOT devices you might want to buy this from an ISP.
1079.84s - 1084.84s |  So they will set you up with a internal network where your devices are
1088.12s - 1092.52s |  connected to and they might not go to the internet at all or there might be
1092.52s - 1098.16s |  some kind of you can run your own server there or something like that. It's a
1098.16s - 1103.16s |  private network and they can be configured however. So routing, filtering,
1104.16s - 1109.16s |  everything can be configured and direct connection between the devices are
1113.50s - 1118.54s |  kind of at least you can configure them to be allowed. So this is a big
1118.54s - 1123.54s |  difference to normal APNs that you cannot really do device to device
1125.14s - 1131.88s |  connections but you just have to go through the internet. These also by
1131.90s - 1136.90s |  default usually they don't filter the traffic so it's sometimes the ISPs
1139.58s - 1144.58s |  don't really like some kind of traffic or if you do like port scanning over
1145.68s - 1149.92s |  mobile network they might shut you down and you don't really know if it works
1149.92s - 1157.83s |  or not. So you can also set the private APN to connect you to the internet so
1159.33s - 1163.57s |  you have access to all these services and then you can get also access to the
1163.59s - 1168.59s |  internet maybe even with a static IP that then you can configure your servers
1169.53s - 1174.57s |  to only accept that. So then it's used quite heavily in IOT so that's where I
1174.57s - 1179.57s |  found out that this exists and a lot of the customers are using this so I kind
1181.61s - 1186.61s |  of figured that this sounds something that kind of if we want to do this
1187.33s - 1192.33s |  inspect the packets and they use the same things for the lawful interception
1194.07s - 1199.07s |  things then it might help me kind of look at my traffic. So I had this
1201.37s - 1206.37s |  question, could I get a private APN? Sounds expensive and I went on a journey
1208.35s - 1213.35s |  to send a bunch of emails to different ISPs and I found out that there's a huge
1214.33s - 1219.33s |  variance in price and usability so the first place I went they had this contact
1221.97s - 1226.97s |  us for pricing and they were like yeah we can do this no problem so you just pay
1228.21s - 1233.21s |  us 3k to get started and then 500 monthly. So I was like yeah this sounds a bit
1234.31s - 1239.31s |  too much for a proof of concept and I might as well just run my own base station
1239.45s - 1244.45s |  in a couple of offices. The next place they were also contact us for pricing but
1245.45s - 1250.45s |  I was pleasantly surprised so they also said that hey just go to our self service
1251.19s - 1256.19s |  portal and activate it and you're good to go and they had a 100 USD setup fee
1257.03s - 1262.03s |  and 150 bucks a month so this is completely doable. There was another place
1262.33s - 1267.33s |  so they have this $10 setup fee and 75 cents per hour so different pricing
1271.27s - 1276.27s |  models so if you look at the similar schemes here so the first one if you kind
1277.51s - 1282.51s |  of no matter how much you use it it's 250 bucks but then for a year it's just
1283.65s - 1288.65s |  2k so you can rent it out easily and the other one so 30 bucks a day, 100 bucks
1289.35s - 1294.35s |  a week and 6.5k for a year so if you want to kind of do it for longer times you
1296.85s - 1301.85s |  might want to have this local ISP and then if you just want it for a day then you
1304.59s - 1311.80s |  have options for that. There is a lot of difference in coverage so some of them
1312.88s - 1316.78s |  are really global they offer services in all the countries some are US specific,
1316.80s - 1321.80s |  some are EU specific and even some that are just inside your country. There's
1323.30s - 1327.42s |  also differences in the available data plans so you still have to pay for your
1327.42s - 1331.92s |  subscription even though you have the private APN because it's kind of a
1331.92s - 1336.92s |  different thing. Some of the like IOT orientated ISPs they offer a really good
1339.82s - 1345.76s |  pricing that you only pay like 2 bucks a year if you send 100 kilobytes of data a
1345.78s - 1350.78s |  year but if you want to look at your phone traffic I recommend not using these
1351.28s - 1356.28s |  IOT things because it might be costly but some of just regular ISPs offer this so
1361.20s - 1366.02s |  then you can use your own already kind of own subscriptions and you can have
1366.02s - 1371.04s |  like unlimited data for whatever you are paying for it in your country and then
1371.72s - 1379.12s |  you can push that to the private APN. So how to configure this? Like once you buy
1380.52s - 1385.52s |  it first you want to disable the traffic filtering if there is one and then disable
1386.96s - 1390.64s |  routing to the internet because we don't want the devices to connect to the
1390.64s - 1397.97s |  internet without us seeing it. Next just the side to side VPN between the private
1398.31s - 1403.31s |  APN and kind of our cloud server. We could do a bunch of these things inside the
1406.41s - 1410.99s |  private APN but I kind of like to be in control of what I do and have like a
1410.99s - 1415.99s |  command line access to the server where the data is. But once we have the tunnel we
1418.23s - 1423.73s |  can set our cloud server as the default gateway for the network and that basically
1423.73s - 1428.77s |  means that all the data from the SIM cards first go to the private APN and then it
1428.77s - 1433.77s |  just shoves through the VPN tunnel into our cloud server. And this basically should
1436.05s - 1443.13s |  mean that the SIM card data is now on our cloud server. And so we need to add some
1444.87s - 1449.87s |  mobile devices to the private APN so we have our SIM card we push it into the device.
1450.47s - 1455.47s |  Some ISPs kind of require you to set some APN settings on the device so they are not
1459.75s - 1464.75s |  pushing anything to the devices or they are roaming and you kind of need to set it up.
1467.09s - 1471.95s |  And kind of some of the ISPs that say that you need to set the APN and you need to set it to
1471.95s - 1476.97s |  this you might be able to just set anything and it works like on some carriers and then some
1477.47s - 1482.47s |  it doesn't and it's it's weird. So yeah. And then there are some ISPs like if you actually
1489.05s - 1494.05s |  run both the kind of ISP and you run the kind of infrastructure and you run the cell towers
1496.09s - 1501.59s |  and everything then you have more control over things. So you have this easy checkbox that
1501.59s - 1506.59s |  just redirect everything to the private APN no matter what settings or anything there's on
1507.63s - 1512.63s |  the device. So this is something that you want to look for when you are testing devices that
1515.47s - 1520.47s |  don't want to be tested because then you just it thinks it's connected to normal internet
1521.51s - 1528.18s |  but it's connected to you. And this should be it. So profit. Let's see. So this is a
1533.38s - 1539.62s |  proof of concept for this. So I talked about there's the IOT device connected to some of uh
1539.62s - 1544.62s |  some AS uh ISP and then there's a private APN configured uh inside the ISP. We have the IPsec
1547.74s - 1552.74s |  VPN tunnel to our server and then I'm SSH into the server. So uh yeah. Let's see a demo. So
1553.74s - 1560.69s |  um here we have the trusty EPC. So it's kind of it's doesn't require fancy things. You just SSH
1569.63s - 1574.63s |  into your uh server. My server is called private APN because it's connected to the private APN.
1577.27s - 1582.27s |  Here we are running a TCP dump so the interface is the IPsec VPN tunnel interface and we're looking
1582.29s - 1587.29s |  for traffic from host uh 10.2.2.2.0.1. And yeah. So we have the uh 2.2.2.0.1 SIM card so that
1599.88s - 1604.88s |  has like statically uh static IP that every time that uh SIM card is pushed to a device it will
1606.82s - 1611.82s |  get that IP address and here it is uh booting up and I'm really interested to see what it
1612.44s - 1617.44s |  actually does. So let's see. Of course takes some time to boot but uh we have uh yeah. And now
1628.04s - 1633.04s |  we see some traffic. So it wants to know the time. So not that interesting after all. But if
1635.98s - 1642.16s |  you leave the SIM card to the device then you will see some uh unencrypted uh location data
1642.20s - 1648.18s |  from that. So if you're running that tracker somewhere uh the ISP knows where you're kind of
1648.18s - 1653.18s |  uh exactly where you're running it. It's kind of uh one nice thing about this setup is also you
1655.22s - 1660.22s |  just kind of uh put the SIM card into something and then you see the traffic. So here is a modem.
1662.02s - 1667.02s |  It's used to connect uh laptops or ethernet based devices uh to the internet with 4G. So here is
1667.96s - 1672.96s |  booting up. And I like I said it runs its own operating system. This is kind of a device but it
1677.00s - 1682.50s |  might also be a chip uh inside your phone or whatever. But they run their own operating
1682.50s - 1687.52s |  systems. So once it's uh booted up uh we can see that it has so it actually says here zero
1688.52s - 1693.52s |  devices connected. But there's uh data there. So it's connecting to netgear.com and uh a lot
1700.06s - 1706.04s |  of doing a lot of things. So it's downloading or checking for software updates. But luckily we
1706.04s - 1712.38s |  can see that it's uh 443 so it's DLS traffic and I couldn't use my certmetem tool to hack that.
1712.38s - 1717.38s |  So it's it can be okay. Maybe. But now we at least know that it's it's not uh completely bad.
1722.96s - 1727.96s |  And also so here next up we have an iPhone. So we're booting it up and uh I have the SIM card
1732.40s - 1738.14s |  tray and you can see me struggle getting it in. So this is the kind of hardest part of the setup
1738.14s - 1743.92s |  is actually kind of boosting the SIM card inside. One of our project uh kind of penetration
1743.94s - 1747.88s |  tester push the SIM card and it we heard this kind of clunk clunk clunk clunk clunk clunk
1747.88s - 1752.38s |  because it was inside the device and it was a competition and we only had 30 minutes so we had
1752.38s - 1758.80s |  to shake it out of the device. But yeah that was one joke and uh that's all of them. So if we
1758.80s - 1763.66s |  push the SIM card into the device you can see it's actually locked. So this is the first boot
1763.66s - 1768.66s |  before unlock. We push a SIM card into an iPhone. So let's see what happens. Yeah. Okay. So
1774.74s - 1779.74s |  it's connecting to somewhere. So in here I think it's doing uh some uh push notifications or it's
1783.92s - 1788.92s |  uh getting that. And I think it uh won't do that many things before you actually unlock it but we
1791.06s - 1796.06s |  are actually seeing something. So if you are into hacking locked devices you might want to look
1796.52s - 1801.52s |  into this. So yeah. So this is the demo. That's uh how easy to actually run it uh once you've set
1808.56s - 1825.88s |  it up. Yeah. And let's get back to this question. So is there traffic over 5G as well? So of
1828.28s - 1834.06s |  course when I made this slide I was like hey I will need to kind of uh check it. So this is my
1834.08s - 1840.74s |  phone. I connected it to a Wi-Fi then I pushed the uh same SIM card into it and this is what I
1840.74s - 1847.58s |  see. So even though it's connected to Wi-Fi you will see traffic. So these are the I think
1847.58s - 1854.00s |  similar Apple push notifications. Uh first when I Googled these IP addresses I went kind of uh
1854.00s - 1859.00s |  the first match was some weird Russian forum and I got scared. Um but turned out it's uh maybe
1859.88s - 1864.88s |  not some uh malware on my phone. But yeah. So there is uh something that the operating system
1872.51s - 1877.51s |  does so it overrides the Wi-Fi and pushes directly uh through the 5G. And if there is some kind of
1879.89s - 1887.54s |  advanced malware then it might also pop up here. So proof of concept it's kind of uh kind of uh
1887.54s - 1897.28s |  yes. So um from kind of moving on from the proof of concept to an actual project that uh you
1903.86s - 1910.10s |  guys can use. Uh so I collaborated with some IOT security experts. Some of them are here in the
1910.10s - 1915.10s |  room and they have had the same exact problems. So how do you of course they have the problem
1916.06s - 1921.06s |  of how do you kind of uh look at your mobile traffic and a bunch of them were in the face of
1925.00s - 1930.00s |  using their own uh kind of mobile base stations and uh software defined radios and kind of
1932.54s - 1936.88s |  trying to use an IOT device that you need to kind of boot up and then you need to press a
1936.88s - 1942.76s |  button but once it's inside the bag it's kind of hard to best press the button and uh then I
1942.78s - 1947.86s |  told about this that hey I I kind of uh think this is the way to do it and and kind of I can
1947.86s - 1952.86s |  see them contemplating kind of uh years worth of uh doing all the previous steps. And yeah
1959.44s - 1966.10s |  some uh I also heard that they are running similar set up than I do. Like one person I know
1966.10s - 1972.73s |  that has been running this uh and a bunch of that haven't. So of course I'm from Finland so I
1974.41s - 1979.69s |  we have the kind of uh holy land of uh network connectivity so of course everything works
1979.69s - 1984.69s |  there. Uh it's cheap and uh stuff like that. So I wanted to look at the US where it's uh the
1986.79s - 1991.79s |  opposite. There's it's not cheap um and also EU and global stuff. So it turns out a lot of ISPs
1996.47s - 2001.47s |  and companies offer affordable private APNs and um they a little bit different but uh
2004.47s - 2009.47s |  I created this uh GitHub page so it's online now and you can see this uh to get uh uh some up
2014.21s - 2021.05s |  to date things. So they are there are global ISPs there so one of the 10 bucks uh to set up
2021.05s - 2026.05s |  and 20 bucks per day thing works everywhere. Uh if you've seen me uh before today I've kind of
2026.65s - 2031.65s |  given out bunch of SIM cards so you can try this out. But uh there would should be something
2035.29s - 2040.29s |  there that works in your country. Uh but of course the global IOT uh providers might be too
2043.43s - 2048.57s |  expensive for your use case if you want to use more data. Uh and if you want to use kind of
2048.57s - 2055.15s |  your existing uh mobile plans or you want to use your local ISP you really should kind of show
2055.15s - 2061.05s |  them this GitHub page and ask them hey do you want to go there on the GitHub page so people
2061.05s - 2066.05s |  will buy your services and uh figure out if you can make it work. Uh if they have to do it
2068.53s - 2073.53s |  manually it will be expensive so uh usually you want to look for something that has a the
2075.23s - 2080.23s |  self-service portal that works. But yeah. So this is my kind of uh full set up that I run.
2085.15s - 2090.15s |  So I have the multiple IOT devices connecting to an ISP and so a global ISP one of the
2092.99s - 2097.99s |  devices is in Helsinki in uh in my home. One is here in Las Vegas and they are kind of in the
2100.33s - 2105.33s |  opposite sides of the world but they still connect to the same ISP and they do the ISP stuff,
2107.73s - 2112.73s |  go into the private APN uh different uh ISPs have different kind of uh uh tunnels and
2115.51s - 2120.51s |  but uh they end up in my my server. There I run WireGuard but you can also set up uh another
2123.09s - 2128.09s |  type of VPN uh for your pentesting like uh workstations. So here I have on the right like a
2130.89s - 2137.03s |  Kali Linux machine in Helsinki. Uh some of my colleagues maybe be in Stockholm and they have
2137.03s - 2143.63s |  the same. And uh with this uh VPN server on the cloud server so there's like one to one mapping
2143.63s - 2148.63s |  so IOT like SIM card one goes to VPN profile one and the VPN profile one is opened uh in
2151.63s - 2158.63s |  Helsinki and then the traffic just magically goes there. So once you set up this uh if you are
2158.63s - 2163.71s |  working in a company your colleagues will love you. You just hand them out SIM cards, you
2163.71s - 2170.11s |  hand them out uh uh VPN profiles and then like they're hey what's this and you just hey push
2170.13s - 2174.97s |  that in there and then just double click that and then you see the data and everyone's amazed
2174.97s - 2182.01s |  and you're a hero. So uh it's that easy. But you can of course don't have to have this fancy
2182.01s - 2190.68s |  setup if you just want to do everything on the cloud server you can do it there. Um also like I
2190.68s - 2195.68s |  said on the GitHub you have um a list of uh private APNs. The list is quite small because I use
2195.90s - 2200.90s |  things that uh I found that work in Finland. When I flew here I tested uh London so in London
2207.14s - 2214.08s |  this works great. Uh here uh I run into the problem that the modem I was using to kind of uh
2214.08s - 2219.08s |  figure out if they uh work in different ways uh doesn't work in the US because I noticed that
2219.62s - 2224.62s |  the US has different mobile network stuff for some reason. But um I I tested that this work
2229.76s - 2234.74s |  well on my phone here so I just push it into my phone and I can see the traffic uh on the
2234.74s - 2239.74s |  server. So confirm to work well um everywhere in the world. But yeah so let's wrap up and uh
2243.50s - 2248.50s |  go back to the kind of why do would you want to do this. So curiosity and privacy. 50 bucks
2255.18s - 2261.76s |  couple days of tinkering on a device like that's a good deal. Um it's actually the 15 bucks
2261.76s - 2266.76s |  is like one evening of like six hours of of uh tinkering with this. So not not much. And
2267.76s - 2274.76s |  even if you use this uh if you want to use it for a longer time you might want to look into uh
2274.76s - 2281.70s |  other options that are kind of uh less expensive in long term. But hacker spaces or even
2281.70s - 2287.94s |  tiniest penetration testing companies like 2K a year should be a no brainer to uh if you
2287.94s - 2296.02s |  really want to uh look at the traffic. Offensive security. So if you do penetration testing
2296.04s - 2302.76s |  on IOT and you say to your customers that hey we kind of look at the device and look where
2302.76s - 2308.90s |  it's connects to. You really need some kind of setup to do this and 2,000 a year versus an
2308.90s - 2315.60s |  illegal setup that's kind of you need to have your best engineers work a week on. So that's
2315.60s - 2322.78s |  really expensive as well. These devices especially in industrial IOT um they're everywhere and
2323.10s - 2330.08s |  people don't really look at their traffic that much. So there's just a handful of companies
2330.08s - 2335.08s |  that actually I know that do this but now anyone can do this. And like I said maybe some
2339.64s - 2345.34s |  new attack vectors for high security devices that if you can push a SIM card in and it you get
2345.34s - 2350.34s |  the data then you can also uh send some uh malicious things uh to it. Defensive point of
2350.86s - 2357.24s |  view. So these private APNs uh like I connect it to my server but you can connect the IPsec
2361.98s - 2366.98s |  tunnel to your existing firewall. So if you are running some kind of uh firewall then uh you
2369.52s - 2376.06s |  can just connect this to it. Um kind of the first devices might kind of uh cost something and
2376.08s - 2381.58s |  then you it costs more to add devices but the cost goes down uh gradually. So if you have
2381.58s - 2386.58s |  thousands of devices it will be still uh quite uh inexpensive to do. And the mobile phone
2390.93s - 2395.93s |  malware infections they might not communicate at all over Wi-Fi or VPN. So there was this uh
2398.13s - 2404.37s |  hack some kind of some triangles or something and uh they said that when they noticed the
2404.37s - 2409.71s |  hack uh they were uh scanning their or like they were looking at their Wi-Fi traffic at their
2409.71s - 2416.19s |  office and they noticed that some of the phones were sending things to some uh C2 servers and
2416.19s - 2421.93s |  they also analyzed the malware and the malware had these properties that it can detect if it's
2421.93s - 2426.93s |  been uh run over Wi-Fi or VPN and not do it. So it might have been misconfigured to allow
2429.57s - 2434.27s |  traffic over Wi-Fi because why would you, people are looking at the Wi-Fi why would you do
2434.27s - 2439.55s |  your malicious stuff over that? So it might have just been a misconfigured malware that they
2439.55s - 2446.25s |  happened to find through their Wi-Fi. So I might think that the advanced malware that sometimes
2446.25s - 2452.35s |  are not that advanced uh might get better at this or they might already be doing this and that
2452.35s - 2457.37s |  was just one uh slip up. Uh so yeah, if you are targeted by uh advanced malware you might want
2457.47s - 2462.47s |  to do kind of this kind of thing. But you can find the project details, you can uh snap a uh photo
2470.36s - 2476.60s |  of this to remember it and you can find me here at Defcon or the internet and um thank you for
2476.60s - 2478.60s |  your uh coming here.


url: https://www.youtube.com/watch?v=0L6TlFYwy2U
title: DEF CON 32 - HookChain A new perspective for Bypassing EDR Solutions - Helvio Carvalho Junior
type: video
author: DEFCONConference
ingestion_date: 2024-11-10
transcription_model: large-v2
chunking_status: completed
content:
{
  "webpage_url": "https://www.youtube.com/watch?v=0L6TlFYwy2U",
  "title": "DEF CON 32 - HookChain A new perspective for Bypassing EDR Solutions -  Helvio Carvalho Junior",
  "description": "In the current digital security ecosystem, where threats evolve rapidly and with complexity, companies developing Endpoint Detection and Response (EDR) solutions are in constant search for innovations that not only keep up but also anticipate emerging attack vectors. In this context, this article introduces the HookChain, a look from another perspective at widely known techniques, which when combined, provide an additional layer of sophisticated evasion against traditional EDR systems.\n\nThrough a precise combination of IAT Hooking techniques, dynamic SSN resolution, and indirect system calls, HookChain redirects the execution flow of Windows subsystems in a way that remains invisible to the vigilant eyes of EDRs that only act on Ntdll.dll, without requiring changes to the source code of the applications and malwares involved.\n\nThis work not only challenges current conventions in cybersecurity but also sheds light on a promising path for future protection strategies, leveraging the understanding that continuous evolution is key to the effectiveness of digital security.\n\nBy developing and exploring the HookChain technique, this study significantly contributes to the body of knowledge in endpoint security, stimulating the development of more robust and adaptive solutions that can effectively address the ever-changing dynamics of digital threats. This work aspires to inspire deep reflection and advancement in the research and development of security technologies that are always several steps ahead of adversaries.",
  "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
  "duration": 2364,
  "channel": "DEFCONConference",
  "uploader": "DEFCONConference",
  "upload_date": "20241016"
}

0.00s - 6.50s | This text was transcribed using whisper model: large-v2

 Welcome Vegas! Woohoo! Let's go guys! Ladies and gentlemen, LVO. Thank you guys to be here.
15.68s - 20.68s |  It is amazing experience to be here for my first time. And let's talk about EDR, XDR bypassing.
20.68s - 25.68s |  First of all, our agenda. Uh, a little bit who I am. Uh, a little bit what is hookchain and what
36.76s - 41.76s |  is not hookchain is. And these research limitations. And after that, let's talk about the
42.76s - 47.76s |  basics to understand how does hookchain works. And after, look through hookchain itself.
57.64s - 62.64s |  Who I am? My name is Elvio Junior, aka Maverick. Uh, my undergraduation school, uh, I did
65.44s - 70.44s |  uh, airplane pilot. So yes, I am a pilot. And I was the pilot for the first time in my life.
71.76s - 76.76s |  The first in America Latina to learn OSCE 3. Uh, and I studying, or maybe studying to OSCE.
82.80s - 87.80s |  This research, uh, that I have no much time to study OSCE. And what I like? I like low
91.34s - 96.34s |  level exploitation, buffer overflow, shell coding, malware and so on. And also I am CEO
97.34s - 102.34s |  of SecForUs, a company, a Brazilian company that uh, provide training and researching,
107.84s - 112.84s |  consulting and so on. What is the motivation of this talk and this research? Many people
117.12s - 122.12s |  believes that I, I have solution X, that I top of the Gartner and I am a hundred percent
127.30s - 132.30s |  protected. No, there is no silver bullet. And when I doing, uh, pentesting or red team
138.58s - 143.58s |  exercise, I just need to bypass your environment. I don't need to bypass everything, just
148.28s - 153.28s |  your environment. So, and fool undetectable. Sometimes it happens. And what is hookchain?
156.60s - 161.60s |  Hookchain is a technique. It is not a tool. I'm not pretending to launch a tool. Yeah,
164.20s - 169.20s |  obviously I need to do some programming. I need to, to create a code to prove the concept,
172.04s - 177.04s |  but it is a technique. Okay? And what is the focus of hookchain? Do hook, do bypassing of
177.40s - 182.40s |  monitoring of EDRs and XDRs inside of NTDLL. In other words, a hundred percent user land.
195.94s - 200.94s |  Nothing about kernel land. Okay? And limitation. As I said, it is uh, 64 bits programming
201.50s - 206.50s |  and limited just to NTDLL. User land hooks. And I'm not pretending here to classify the EDR
220.58s - 225.58s |  vendors, XDR vendors as his effectivity. I'm just launching a new technique. Okay? I need
225.84s - 230.84s |  to do some tests, but I'm focused just in one point, not entire ecosystem. So, I'm not
240.08s - 245.08s |  pretending to classify EDRs. It must be clear for, for all guys. During this study, I, I
246.00s - 251.00s |  did a public call for vendors to work together with me to get early access to this code, to
260.14s - 265.14s |  this technique, and no one answered. When I published some bypass in LinkedIn, I received
265.68s - 270.68s |  some calls, not too happy calls sometimes, but I received just three years, three weeks ago,
279.06s - 284.06s |  some calls in order to test this. Okay? And let's go to the technical things. First of
287.26s - 292.26s |  all, we have almost two layers in order to protect and split the time block. Okay? So,
295.44s - 300.44s |  inside of our operational test. Ring zero, kernel land, and ring three, user land. To
304.94s - 309.94s |  transition between user land to kernel land, in a few words, we have just one way to do
314.48s - 319.48s |  this. Using a gate calling system call, or system dispatcher and so on, that I can
319.84s - 324.84s |  transit from user mode to the kernel mode. In general, we have this picture where we have
332.92s - 337.92s |  user process, system services, and so on. Doesn't matter what, uh, is the, his level is, it
340.68s - 345.70s |  is a system level, a certain level, or uh, a simple non-privileged user process. Every time
349.84s - 354.84s |  I need to do this transition. In general, this, this is a responsibility to NTDLL to do this
359.38s - 364.38s |  for me. So, this is a common stack call where I have one common application for, that must
369.72s - 375.06s |  be, that need to be, for example, need to execute first a virtual lock to locate a, a
375.08s - 380.08s |  memory, and the common stack call, the application will call this from kernel 32 DLL, and
383.16s - 388.16s |  the kernel 32 will call another, uh, API inside of NTDLL, and the NTDLL will prepare stack
394.20s - 399.20s |  call parameters, register parameters, and we will execute uh, assembly instruction
399.98s - 404.98s |  called sys call that will transit from user mode to the kernel mode. This is just another
413.86s - 418.86s |  image to explain that. So, I have the user mode application and the important here, system
421.60s - 426.60s |  service number. There is a table called system service descriptor table where I have an
427.54s - 432.54s |  indexes from zero to, I have no idea how much, more than four hundred, uh, numbers that match
441.82s - 446.82s |  each ID from a pointer to a kernel function. So, when I need to transit from the user mode
447.82s - 452.82s |  to kernel mode, I put the system service number inside of register RAL, sorry about the
460.80s - 465.80s |  names, but uh, I put this number in this register and I execute the system call
469.14s - 474.14s |  instruction. And system call will come to the kernel uh, address space and this ke- this
474.14s - 479.14s |  function will uh, match inside of this table and get the pointer to the real kernel
483.58s - 488.58s |  function. And the real kernel function will be executed and return to the user mode space
490.38s - 495.38s |  with the return of the the executed function. Doesn't matter uh, what this means. This is
497.76s - 504.06s |  another uh, uh, uh, image to illustrate that. System service call, system service
504.06s - 509.06s |  descriptor, the table with the indexes and uh, the pointers to the function and the real
511.66s - 516.66s |  code from, to the function. Okay? And when I execute some proxies uh, un-executed or uh,
521.70s - 526.70s |  when I load uh, DLL inside of another proxies, when I initiate uh, uh, a proxies, several
527.42s - 532.42s |  actions are, have been done behind the scenes. One of the most important that uh, uh, to the
537.70s - 542.70s |  stock is related to import address table. Import address table in a few words is a table
545.10s - 550.10s |  that uh, the name of the function that I want to execute, for example, uh, create proxies and
550.78s - 555.78s |  the name of the function and the uh, ID representing, ID representing inside of this table
563.18s - 568.72s |  that when I run this, when I run the the application, the operational system that the
568.72s - 573.72s |  loading of the operational system will look through uh, all loaded DLLs and fill this table
573.78s - 578.78s |  with the address, the real address of this function in the operational system. Virtual
582.26s - 588.36s |  address of course, but the virtual address. So when I try to execute this proxies, my
588.36s - 593.36s |  application will comes uh, to the import address table and request, import address table, what
596.00s - 601.00s |  is the real virtual address to this function? And put this for example in a history and
603.74s - 608.74s |  after that call the address of this hazards, history in order to call the uh, real uh,
611.52s - 616.52s |  function code. And another things that image loader do is uh, look through the, the loaded
620.62s - 625.62s |  DLLs if this DLLs uh, it uh, was already loaded, the recursive loading and so on and fill
628.20s - 633.44s |  this, this function pointer. There is another image to, to illustrate this, this same
633.46s - 638.46s |  thing that I have the name of the DLLs and the functions that I imported in my application by
640.46s - 647.00s |  the full and the, the, the name and the code of the application inside the, the uh, the
647.00s - 652.00s |  targeted DLL. Let's start talking about the techniques. Uh, when I, when I saw this, this
652.04s - 657.04s |  kind of hooking uh, we need to say, we must say that the techniques that we use to attack,
666.88s - 671.90s |  it's the same techniques that the EDR vendors are using to protect. The first time that uh,
674.94s - 679.94s |  someone published uh, uh, something related with this techniques in, in Windows, in
682.04s - 687.04s |  1995. So uh, a few, few years ago. Uh, and what is hook? Like a interception of the stack
694.52s - 699.52s |  call. When I try to execute some function, this function uh, I, I'm not executing the
702.10s - 707.10s |  real function. I, I will be uh, intercepted and redirected to another function. Like a
707.88s - 712.88s |  man in the middle. In this case it will be a function in the middle. But it's the same
715.68s - 720.68s |  concept. Okay? The, the most common techniques to do that is using instruction jump or call
724.36s - 729.36s |  or I can use import address table hooking. How can I do that? Let's see. Uh, talking
730.36s - 735.36s |  about jumping. This is a strategy that I, I'm uh, that we can see here. It's the most
740.94s - 745.94s |  common strategy uh, used by EDR and XDR vendors. That is, when I run a, an application,
750.38s - 755.38s |  uh, NTDLL is the first DLL uh, that is, is attached to my process. And after that, the
759.76s - 764.76s |  EDR DLL will be uh, attached and the DLL from EDR will patch or change in memory uh, the
771.54s - 776.54s |  instructions of the NTDLL. So when I try to execute for example this function, this
778.20s - 783.22s |  function is uh, uh, changed to jump to address space of EDR uh, code. In order that the EDR
783.58s - 788.58s |  can uh, check our parameters and response and so on. This is uh, an example of original
797.70s - 803.20s |  execution flow where when I get the address of the, the, the, the function, the target
803.20s - 808.20s |  function, we will receive the real address. But if after the EDR changes or after our
808.92s - 813.92s |  change that, because we will change this table, uh, after that, instead of receive the
818.10s - 823.10s |  address of the real function, the original function, we will receive an address to a
825.64s - 832.64s |  hooked function or to a uh, function in the middle in this case. Okay? So we will receive a
832.66s - 837.66s |  malicious function or uh, when I EDR doing that, the, the, the address of the EDR function.
840.10s - 845.10s |  So when I call this address, we will call the received function. In this case, uh, arbitrary
848.28s - 853.28s |  code from EDR or from our perspective and it is a decision if we will or not execute the
853.80s - 858.80s |  real function. In uh, talking about EDR infrastructure, the EDR will uh, analyze the
865.88s - 870.88s |  parameters and decide yes, uh, LV is a good guy, I can allow this. Oh no, LV is a bad
874.56s - 879.56s |  guy, let's drop this. And uh, uh, pop, pop up alert and uh, send this to, to SOC and
879.58s - 884.58s |  so on. Uh, this is a, that I am talking, this is a, uh, when I EDR load EDR, put an, another
893.76s - 898.76s |  DLL and it will be redirected and decide yes, no, execute or no, and do if, yeah, LV is a
903.04s - 908.04s |  good guy, let's uh, permit this execution, so do system call, include, uh, uh, uh, uh,
909.64s - 914.64s |  kernel land, it's everything as usual and after that come back to the user land process.
918.12s - 923.12s |  Ok, I can see here an unhooked API, ok, so I can see this typically uh, instructions where
928.46s - 933.46s |  uh, there is a, a, a moving from RLCX to R10 and after that there is a, an ID here in
934.00s - 939.00s |  hexadecimal, uh, where I put this ID in uh, EAX or RAX, doesn't matter, and after that there is a
948.62s - 953.62s |  system call. This is a typical uh, uh, code from NTDLL's uh, uh, functions that will
956.86s - 961.86s |  translate from user land to kernel land. And here we can see how can I, how can I, uh,
964.06s - 969.06s |  HTTP API where he is unhooked and he is unhooked API. As you can see, there is no this
972.40s - 977.40s |  typical code, there is a jump code where I'm doing uh, uh, jumping or interception to the
981.18s - 986.18s |  API or to the EDR uh, code. Ok, such that the most common bypasses is remapping NTDLL and
993.56s - 998.56s |  but to do that we need to load the NTDLL from the disk or from the another uh, memory area.
1003.10s - 1008.10s |  It's, seems to be malicious in most of case. Uh, another technique, it's direct syscall
1011.04s - 1016.04s |  when instead of goes through NTDLL, I get the system service number and do uh, and execute
1016.24s - 1021.24s |  the system call inside of my, my process. This sounds malicious again because the system
1029.18s - 1034.18s |  call was not coming back from NTDLL memory address space, it's coming from my application
1037.92s - 1042.92s |  address space. Ok, and EDR catch this. I direct syscall, it's a syscall, it's a syscall
1046.04s - 1051.04s |  similar to direct syscall but instead of uh, do uh, system call instruction inside of my
1053.74s - 1058.74s |  application, I will jump it to a address inside of NTDLL that exists just system call
1061.68s - 1066.68s |  instruction. So, who will execute NTDLL? So, sounds less malicious. But in this case, the
1066.68s - 1071.68s |  stack trace will be broken because I'm executing directly from my code inside of NTDLL. In
1080.02s - 1085.02s |  general, the application doesn't do that. The application in general calls system 32 DLL,
1088.16s - 1093.16s |  kernel base DLLs, GDI DLL and so on. And this DLL will call NTDLL and, and etcetera. So,
1097.40s - 1102.40s |  sounds malicious. Then, there are amazing guys uh, without this research, this talk today was
1107.28s - 1114.38s |  not possible. So, this is amazing guys uh, created he- uh, Heaven Gate technique, Hell's
1114.38s - 1119.38s |  Gate technique and the Hallows Gate technique. So, uh, for this research I used the part of
1120.36s - 1125.36s |  Hallows Gate technique in order to uh, resolve dynamically uh, the system service numbers.
1132.00s - 1137.00s |  Okay? So, let's understand a little bit better about Hallows Gate, this part of Hallows Gate.
1138.68s - 1143.68s |  In general, Hallows Gate locate the current address of the uh, target function, the function
1144.68s - 1149.68s |  that I want to execute. For example, uh, create a virtual memory or um, create processor and
1155.40s - 1160.40s |  look that uh, that code that we, we saw, the, the typically code and check if the typically
1164.04s - 1169.04s |  code uh, match with these uh, uh, these instructions. If yes, okay, this function was not
1169.04s - 1174.04s |  hooked. If no, the function is hooked. Okay? So, we can uh, see that and how can I uh, get
1186.56s - 1191.56s |  the system service number if, if the function is hooked. The real code uh, does not exist
1192.02s - 1197.02s |  more there. How, how can I see this? Let's, let, let's see uh, uh, another screen here. We can
1202.16s - 1207.16s |  see that the functions are linear. So, we can see here, for example, the system service
1211.08s - 1216.08s |  number six, system service number seven, this function is hooked. So, in general I have no
1217.08s - 1222.08s |  idea what's the, the, the system service ID of this function and the next function will be
1225.28s - 1230.28s |  nine and we, the next function will be ten. In other words, we can see the neighborhood uh,
1233.22s - 1238.22s |  functions before and after our uh, uh, uh, desired function in order to calculate. If this
1238.38s - 1243.38s |  function is six, that function is seven. My function will be eight. So, we can dynamically
1249.82s - 1254.82s |  check this. This is hollows gate technique, okay? And uh, before this, this research, this
1258.76s - 1263.76s |  study, uh, in a sunny day, uh, playing with my two uh, boys, my two boys, uh, playing with
1268.28s - 1273.28s |  two kids, I have an idea. Whoa! The major part of EDRs are hooking in an NTDLL. If I step
1279.82s - 1284.82s |  above and try to catch or try to do a, a, a function in the middle, uh, changing the flow
1288.16s - 1293.16s |  before NTDLL, okay? Because, because some uh, uh, several EDRs are checking if we are
1294.02s - 1299.02s |  changing NTDLL. So, because they are changing there, the EDR are checking if we also change
1305.26s - 1310.26s |  there and flag us as malicious. So, if EDR are checking there, and what if I did or I do
1315.70s - 1320.70s |  this in a step above? So, to check the uh, viability of this technique, I, I create a
1323.34s - 1328.34s |  code in order to check the EDRs hookings. So, nine percent of EDRs are hooking just in NTDLL.
1336.02s - 1341.02s |  So, the technique can be viable, okay? And I created the new technique called hook chain.
1341.02s - 1346.04s |  What is hook chain? Hook chain, the first step, I will uh, resolve system service number.
1355.70s - 1360.72s |  After that, create a table where I store their system service number in uh, on an address of
1361.02s - 1366.02s |  my custom stub function. In other words, the function that will be executed instead of
1373.90s - 1378.90s |  original function. EDR do this, whoa, I can do this also. And implant or change the import
1384.48s - 1389.48s |  address table from our proxies to our custom stub function. So, this is the hook chain
1390.72s - 1395.72s |  and after that, continue as usual. With this, I have in more details the hook chain implant
1402.06s - 1407.06s |  flow. First one, uh, I resolve dynamically, map some critical functions because in this,
1412.34s - 1417.34s |  in this phase, uh, I have no, uh, I had no bypass at EDR yet. So, I have no bypass at EDR
1420.72s - 1425.72s |  yet. So, I need to use some functions not uh, monitored by EDR in order to change some
1428.12s - 1433.12s |  memory address and change some uh, memory areas and so on. Read and et cetera. Uh, so I
1436.66s - 1441.66s |  map statically these main functions that I call it here, critical functions. And after that
1444.10s - 1449.10s |  I create a table. We will see the structure of this table uh, later on. Uh, uh, we create a
1450.78s - 1457.78s |  table that holds system service number that I already mapped. The function address, the real
1457.78s - 1462.78s |  function address inside of NTDLL and uh, the, the closest uh, address that holds a system
1468.86s - 1473.86s |  call instruction inside of NTDLL. And after that, I load it, I preloaded some functions.
1474.82s - 1479.82s |  For example, uh, if we will run uh, MetaExploit payload or Havoc payload uh, payload and uh, I
1485.56s - 1490.56s |  have no idea. Uh, there are another uh, DLLs that must be called before the execution of
1492.94s - 1497.94s |  this payload. Because a simple uh, a simple executable for example has no access to the uh,
1498.26s - 1503.26s |  URL uh, APIs or uh, crypto APIs. So we must preload this DLL in order to change the import
1510.88s - 1515.88s |  address table of this DLL. So here we preloaded these DLLs and after that do the magics.
1518.44s - 1523.44s |  Change the import address table of all needed DLLs. In this time I, I, I need to say that
1524.32s - 1529.32s |  there's not a good idea to change the import address table of all loaded DLLs. I tried that
1535.36s - 1540.36s |  and I lose my hair trying that and that not a good idea. Believe me, believe me. So I, I, I
1544.66s - 1549.66s |  created like a, a blacklist DLLs where I put inside of the application and uh, do this
1553.58s - 1558.58s |  preloading and after that changing that. This is the typical flow after hookchain planting,
1561.88s - 1566.88s |  the simplified version. So my application must call virtual unlock for example to create a
1570.06s - 1576.20s |  malicious error in order to put there a MetaExploit shellcode for example. And I need to
1576.20s - 1581.20s |  create this, I will call kernel to 32 DLL. Instead of call direct NT DLL, I will call
1583.44s - 1588.44s |  DLL this function. As I hook it, as I did the uh, function in the middle, I will, this,
1591.68s - 1596.68s |  this flow will be redirected to my application stub code. This application's stub code
1599.36s - 1604.36s |  will, will check several things and create a stack frame, a stack call, create uh, uh, put
1606.40s - 1612.00s |  the, the, the, the uh, the information inside the registers and after that do a
1612.10s - 1617.10s |  indirect syscall in order to execute. And kernel mode as usual. Here there is uh, I can
1622.54s - 1627.54s |  present you the, the first struct that I, that I uh, I will use. I will use here uh, a
1630.82s - 1635.82s |  list, okay, with this system call information and each entry of this list uh, has a system
1636.38s - 1641.38s |  service number, real address from the real function, okay, uh, the, the um, address of the
1652.36s - 1657.36s |  function or the address that holds the system call instruction and the address of my stub
1658.36s - 1663.36s |  function that will be redirected to. And uh, a hash use it to bypass uh, static analysis
1669.16s - 1674.16s |  instead of the name of the functions and the, and so on, okay. And this is the entire flow
1677.84s - 1682.84s |  where the same, my application want to execute some function, will call kernel 32 DLL and
1682.84s - 1687.84s |  that call kernel base DLL and after that I will check in a import address table. When I
1695.34s - 1700.34s |  check in import address table instead of receive the original address I will receive my stub
1701.82s - 1706.82s |  function. So the application will be, will redirect to my stub function. Inside of this
1707.10s - 1712.10s |  function uh, I can check the system call list, the list that I saw and the list there are
1716.78s - 1721.78s |  several items in order to check, get the parameters that were the real function, syscall uh, uh,
1723.36s - 1728.36s |  address and so on. Create the stack call frame and put everything that we, we need in the
1729.04s - 1734.04s |  registers and after that do indirect syscall. Okay? What are the advantages of that? How this
1741.72s - 1746.72s |  works? What, why? Uh, reduction of probability to identification for several reasons. The
1750.36s - 1755.36s |  first one, total time of execution. As I have several times similar to the, to the, to the
1758.44s - 1763.44s |  regular execution, the, the DDR cannot flag us because the, the total time of execution, the
1765.34s - 1771.64s |  telemetry of total time of execution will be okay. Will be uh, seems, seems to be uh,
1771.64s - 1776.66s |  valid. And stack, stack framing. Talking about stack frame, I'm flowing, I'm uh, uh, uh,
1780.92s - 1785.92s |  executing as usual. Coming for a kernel 32 for example, kernel base and so on. So, I'm
1789.16s - 1794.16s |  not stepping steps. Yeah, I'm not skipping steps. Okay? And the most important, I did not or I
1802.18s - 1807.18s |  uh, uh, uh, I, I didn't need to change the code of the original application. For example, we
1807.92s - 1812.92s |  will see in demo, when I want to execute mimikatz for example, I uh, uh, uh, I don't need to
1821.86s - 1826.86s |  touch the code. I just execute the hookchain in order to implant and execute mimikatz. That's
1827.00s - 1832.00s |  it. Because the hookchain implant and uh, uh, uh, uh, everything that we need. The
1838.44s - 1843.44s |  function in the middle. Let's see the first demo where we will execute and uh, command and
1847.78s - 1852.78s |  control using Havok. Okay, this is uh, flag as malicious as usual. And let's see this
1857.76s - 1864.14s |  first one. Here we can see uh, Bitdefender updated with uh, advanced threat protection
1871.32s - 1876.32s |  full enable and in the left side we can see our uh, console where uh, we will have, let me
1881.92s - 1886.92s |  see, uh, let's stop here. Just uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh,
1888.76s - 1893.76s |  uh, let me put here together in order to, okay. Okay, no problem. Uh, first time I will implant a
1900.54s - 1905.54s |  hookchain, after that download remotely from HTTP the malicious payload and after that
1906.26s - 1911.26s |  execute this malicious payload. So we are using Havok to do that. And here we go. Executed
1922.90s - 1927.90s |  our first, first command control. We can interact with that without any identification. Thank
1928.18s - 1937.46s |  you guys. You are amazing. It's just a starting one. Let's go. Let's go back to the
1945.36s - 1952.89s |  presentation. Whoa. Here it is. Looking in details we have. The first step is mapping everything
1953.69s - 1958.69s |  that we need. So in this operational system we have four, 472 NTDLL functions where we can
1970.77s - 1975.77s |  map and these functions are hooked or I need this as a critical function. And after that I
1975.99s - 1980.99s |  create this table, okay. This is the, the, the showing of this table and after that I do the
1987.77s - 1992.77s |  import address table hooking. So I can see here the output of this, this hookings in kernel
1995.47s - 2002.20s |  to 32, kernel base user 32 and so on. Let's talk about the real environment. The first of
2002.20s - 2007.22s |  all, thank you for all friends, all Brazilian community that there are most, most guys
2014.46s - 2019.46s |  here. Woo hoo! Uh that share with me uh your environment license and help me and support me in
2019.46s - 2024.48s |  order to make this research works. Uh without you guys nothing will be possible. Uh thank
2034.88s - 2039.88s |  you, thank, thank you so much. And because obviously some EDR vendors are not too happy
2040.72s - 2045.72s |  and not sharing their licenses or their environment in order that I test. So uh there are a
2052.30s - 2057.30s |  lot of friends uh sharing his environment with me in order to, to make this test, okay. So
2059.80s - 2064.80s |  we did this in a closed environment as the focus, the focus of this research is talking about
2070.74s - 2075.74s |  uh EDR protections uh and we will touch here else, else us uh I disabled the run as PPL and
2081.62s - 2087.82s |  coordination guard inside of the windows because the tar- the, the target the full focus, it
2087.82s - 2092.82s |  is not the windows, it is not the EDR and the XDR products. So, let's execute the second
2093.82s - 2098.82s |  DEM for you guys. The same as again, we have the antivirals, EDR working with advanced
2111.56s - 2116.56s |  threat protection and we will execute uh as usual, Mimikatz. I download it from GitHub
2124.64s - 2129.64s |  put on my server as you can see and execute this guy in a protected environment. And let's
2135.12s - 2144.73s |  see this working. Woo! It's not all. Execute and show a coffee, it, it's uh easiest part.
2150.71s - 2155.71s |  Let's dump the creds. Without creds, we have nothing. So, let's dump the creds. Let's
2159.01s - 2164.01s |  yeah, we can touch the else us without any identification with regular Mimikatz guys.
2171.09s - 2180.38s |  Woo! Coming to the end, uh we are able to test with 66 uh percent of the products uh
2187.52s - 2192.52s |  present in Gartner and in additional four other products. So, we are able to test with
2195.10s - 2200.10s |  and this is the result table. Okay? We tested remote process injection, meterpreter and
2206.04s - 2211.04s |  have walk as non-privileged user and meterpreter loading QE model, Mimikatz, procdump
2218.58s - 2223.58s |  and a custom a hundred percent developed by me, a hundred percent developed by me.
2225.02s - 2230.02s |  Assembly else us dump. Okay? I create this in order to do this test. And this is the result
2236.50s - 2241.50s |  table. Okay? This is the products that I, I was able to test. And as you can see uh we, we
2243.78s - 2248.78s |  have uh a lot of effectivity in order to, to doing this little things. Uh as I told you guys,
2255.88s - 2260.88s |  uh three weeks ago two vendors uh reached me in order to, to test uh again. The sentinel one
2266.06s - 2271.06s |  was the first and the second one was the trench. This results that I'm showing you uh the
2274.30s - 2279.30s |  sentinel one was able to uh stop hookchain. They did some patches in his product and uh
2285.10s - 2290.10s |  I was able to, to test this and stop it at all. Okay? Deadline, end line. Uh and the trench uh
2295.12s - 2300.12s |  still working on this, this uh uh identification. And they tried some behaviors but, but not
2302.56s - 2307.56s |  at all. Okay? The other vendors, non-contact. Okay? I love it this uh this phrase of
2317.89s - 2322.89s |  Sif Hawking. I'm just a child asking who, how and why. Occasionally I found the answer.
2328.67s - 2333.67s |  Thank you guys. This is my contact. Full white paper, source code, reference and feel free
2340.61s - 2345.61s |  to contact me, feel free to reach me by uh LinkedIn. My LinkedIn, it's my name and
2347.83s - 2352.95s |  just Elvio Junior and that's it. Now it's your time, time to ask questions. Thank you.


url: https://www.youtube.com/watch?v=EHdI6bdw-wo
title: DEF CON 32 - Feet Feud
type: video
author: DEFCONConference
ingestion_date: 2024-11-10
transcription_model: large-v2
chunking_status: completed
content:
{
  "webpage_url": "https://www.youtube.com/watch?v=EHdI6bdw-wo",
  "title": "DEF CON 32 - Feet Feud",
  "description": "Another fascinating DEF CON Game show.",
  "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
  "duration": 2380,
  "channel": "DEFCONConference",
  "uploader": "DEFCONConference",
  "upload_date": "20241016"
}

0.00s - 15.14s | This text was transcribed using whisper model: large-v2

 all right oh god there's lights okay thank you for coming I am Tiberius
15.14s - 21.02s |  however when I'm when I'm hacking with the only feet CTF team I take on a new
21.02s - 29.06s |  moniker of Tobias so that's who I will be presenting as this evening you also
29.06s - 32.70s |  have to forgive the nerves the we've technically done this unofficially at
32.70s - 39.74s |  Defcon the past two years the largest crowd we got was 40 there are and it's
39.74s - 42.18s |  great actually because the lights I can't really see many people but I
42.18s - 46.70s |  assume there are quite a lot more than 40 here we are gonna try and speed
46.70s - 52.42s |  through this because we really want hacker jeopardy to go on time so I'm
52.42s - 55.54s |  gonna introduce the team captains and then we are very very quickly choose
55.54s - 61.86s |  some teams from the audience the captain of team left foot tonight you may know
61.86s - 65.78s |  her from her software dev twitch streams you may know her as the writer
65.78s - 70.54s |  and host of hack fives threat wire or maybe as one of the minds behind the net
70.54s - 82.06s |  gala please give a very warm welcome to Ali diamond hello now for the captain
82.06s - 89.66s |  of team right foot some people call him orange flavored John others know him as
89.66s - 97.90s |  cyber security Ed Sheeran ladies and gentlemen we spared no expense it's John
97.90s - 107.04s |  Hammond you've got so many more cheers than me okay if you would like to play
107.04s - 115.76s |  please stand up that okay so basically team right for actually one last year so
115.76s - 121.20s |  let's forget to pick first I want him he's look at him who bright orange okay
121.20s - 129.32s |  bright orange come over here there are steps over there John pick your teammate
129.32s - 141.73s |  orange bowtie all right Ali okay I'm seeing that guy over there point at
141.73s - 151.09s |  someone okay yes there we go get over here over here pointing girl I want you
151.25s - 169.69s |  next let's go I like it can't see distances oh wait oh my god put your
169.69s - 178.02s |  hands over the lights it helps so much yeah quickly John always I'll choose for
178.02s - 181.74s |  you we have another group of people pointing at people I want you yeah all
181.74s - 193.83s |  right yes yeah come through come through come through the feralness Ali
193.83s - 201.97s |  right back there yeah not everyone needs a hype crew but I'm hyping you up I
201.97s - 213.85s |  don't know who you are yeah no I'm sorry I want that one are we done
214.01s - 225.58s |  hang on sorry lost count hell yeah women okay I love women I'm so sorry for
225.58s - 233.50s |  kicking you off bro yo I got a ginger what are you gonna do about it okay I
233.50s - 237.58s |  am now realizing I probably should have asked if any of these people have played
237.58s - 246.30s |  or watched family feud before oh well so round one for a good time Ali and
246.30s - 253.42s |  John please step up to these two mics hi everyone and grab a buzzer which I
253.42s - 266.01s |  hopefully will work really tall I'm really short am I taller than you okay
266.01s - 271.65s |  here's how it's gonna work I'm gonna read the question and if you think you
271.65s - 276.65s |  know an answer buzz in do not just blurt out the answer because I have to look
276.65s - 284.10s |  down and figure out who buzzed in first I will ask you okay all right let's see
284.10s - 289.38s |  the question oh I should probably preface we can these these were survey
289.38s - 293.06s |  questions that were generated by members of the community they're also answered
293.06s - 297.30s |  by members of the community I did remove all wrong answers and there were
297.30s - 303.90s |  quite a lot of them the data has been normalized so basically the max total is
303.90s - 307.46s |  a hundred for each one anyway we're gonna go all right show the question
307.46s - 312.94s |  please name a threat actor or a cybercrime group all right that is team
312.94s - 324.30s |  right foot John lock bit lock bit survey says okay lock bit is there but it is
324.30s - 329.38s |  not the top answer so Ali if you can get answers two or one you will win
329.38s - 333.14s |  control of the board okay I'm gonna say a funny one but then in a real one
333.14s - 342.41s |  Crowdstrike but actually scattered spider what do you say scattered spider
342.41s - 348.05s |  scattered spider survey says unfortunately the fourth answer sorry
348.05s - 350.89s |  the sick lancers so John you've been controlled the board that means do you
350.89s - 363.21s |  want to play or pass we will play okay thank you please sit down okay bowtie
364.53s - 370.01s |  rubber duck bowtie please speak into the mic as well thank you all right name a
370.01s - 384.66s |  threat actor or cybercrime group apt 40 survey says no okay glasses and beard
384.66s - 392.96s |  I'm just gonna call you that cult of the dead cow a survey says no okay other
392.96s - 404.06s |  glasses and beard sorry name a threat actor or cybercrime group no conferring
404.06s - 411.43s |  John there's no conferring no you can only confer when it's they're going over
411.43s - 426.89s |  to steal yes you say sorry napsis sorry lapses no all right so Ali's team you
426.89s - 431.09s |  can confer for a bit and then Ali I want an answer from you if you get any answer
431.09s - 448.47s |  on the board you will win the board US government is that your final answer the
448.47s - 465.98s |  US government no okay so team right foot win 25 points in round one okay so the
465.98s - 477.42s |  seventh answer was alpha the fifth alpha sorry the fifth answer sandworm for cozy
477.42s - 492.14s |  bear second Lazarus group and number one fancy bear okay all right bowtie and hard
492.14s - 500.42s |  hat please come up to the podium grab yourself a buzzer we have a construction worker because
500.42s - 509.22s |  we're getting to work okay you guys ready all right show the question please name a
509.22s - 519.18s |  security portmanteau do you need an example yes please a secure portmanteau is where you
519.18s - 525.34s |  take two or more words and combine them into one word e.g. you could call red teaming reaming
525.34s - 545.25s |  okay so that was left foot smishing smishing a survey says it was fishing etc smishing the
545.25s - 552.13s |  second answer that means right foot can steal the board I guess by that logic malware ransomware
552.13s - 558.05s |  like that what combination what answer do you want oh let's go malware malware a survey says
558.05s - 577.67s |  do you want to play or pass John we will pass oh okay left foot are gonna play yes yes yes
577.67s - 589.47s |  I don't know how this works all right go with white claw hi yeah I'm gonna steal our collective
589.47s - 602.14s |  one can you speak a little louder sorry oh yeah I'm gonna go with ransomware ransomware no discount
602.14s - 625.41s |  John Hammond yeah timeout okay I don't know I'm running out of clever nicknames now blackhead guy
625.41s - 634.89s |  I'm gonna go with Thunder what do you say sorry Thunder can you speak a little bit sorry Thunder
634.89s - 653.84s |  Thunder yes Thunder no so John's team you can steal this we need milk do we have to answer this
653.84s - 659.96s |  one correctly to get the points I mean ideally in a quiz show you would give correct answers John
659.96s - 665.60s |  I'm not sure it's one of the answers chairman so I consulted with my team and they told me if you put
665.60s - 679.16s |  together lock picking you get licking is that the final answer you're going with no all right
679.16s - 689.93s |  so team left foot technically win that one black hair you're gonna be disappointed with the answer
689.93s - 700.94s |  number seven you're so close it's grunting wait we should get that you don't get that you said
700.94s - 712.70s |  thunting I was very clear all right sick dancer Dev second thing me there just say Deborah pen
712.70s - 726.62s |  test that's a good for info sec and third activism this is my 13th reason and I just noticed the left
726.62s - 739.78s |  foot did score a very nice number of points that round so that's good okay okay glasses and beard
739.78s - 753.95s |  versus go with white claw okay round three we see the question please oh the buzzers yes you should
753.95s - 763.19s |  probably get the buzzers thank you we'll be better next year all right question please it's not
763.19s - 770.63s |  showing it could you go back to scores and try and just go start oh why start round two and why
770.63s - 779.07s |  is right for have zero okay hang on you need to update how are we winning there should be a way
779.07s - 792.26s |  to update the just the scores Helen oh hang on now they've got zero you just over it oh no they
792.26s - 804.15s |  don't okay sorry sorry it looked like they had zero all right round two and well we keep in
804.15s - 808.71s |  manual track of the school because honestly I don't really trust this system I so just show
808.71s - 816.38s |  the question because this might be round if this is round two we just have to go through again okay
816.38s - 827.14s |  technical difficulties is this running in PowerPoint it is running in PowerPoint yeah how many engineers
827.14s - 833.06s |  does it take to fix a PowerPoint presentation yeah we might to end the show and then okay hang
833.06s - 841.70s |  on well hang on well no because we need to reveal the answers anyway name no no no dude let's just
841.70s - 868.49s |  restart the presentation okay seriously we can start over all right all right Helen oh y'all
868.49s - 874.69s |  saw nothing we have backups we have backup questions don't get too excited we got backup
874.69s - 896.63s |  questions I mean time how's everyone doing John yes Ali how's your DEF CON been it's been really
896.63s - 912.99s |  good how many years yeah yeah correct yeah huge L um anyways hi everyone thank you so much for
912.99s - 920.71s |  coming in the meantime I figured we should be entertaining the you were doing what is it called
920.71s - 927.43s |  I did do that what's everyone's favorite village meant so far I want everyone to scream in three
927.43s - 936.19s |  two one go yo that's sick those are awesome answers I didn't understand a single one of them
936.19s - 944.03s |  but I have to say that's really exciting um who here I want to see a raise of hands who here is
944.03s - 951.47s |  this your first DEF CON raise your hand oh my god wait this is so awesome okay wait I'm gonna
951.47s - 957.99s |  pick what keep your hands up I'm gonna pick one person and I want you to be very loud and you're
957.99s - 966.56s |  gonna tell me why you're here I saw you put that hand on real slow you want it you want to stand
966.56s - 974.24s |  up and tell me why no um okay keep your hands up if you want to answer my question um I'm not
974.24s - 980.08s |  choosing someone on the stage oh we got to move sorry we'll finish this later love you the most
980.08s - 986.04s |  all right I don't know which question everyone saw so we're just gonna forget that happened
986.04s - 996.48s |  all right so round three fingers on buzzers all right show the question name an item considered
996.48s - 1004.72s |  be PII that is right foot that's me right yeah okay you know what team you're on you're on the
1004.72s - 1013.76s |  left side of the stage it should be obvious SSN SSN survey says it's the top answer what do you
1013.76s - 1024.60s |  want to do John you want to play or pass your play or pass John I assume you're playing John
1024.60s - 1035.84s |  are you playing excellent okay other beard and glasses guy mobile number sorry mobile number
1035.84s - 1050.21s |  mobile number survey says correct phone number yes nice nice okay all right name and items
1050.21s - 1061.84s |  considered be PII address address our survey says mailing address correct okay back to John
1062.28s - 1076.33s |  birth date date of birth birth date a survey says yes name of school or workplace name of school or
1076.33s - 1086.13s |  workplace survey says no we've lost sound but whatever like a health ID a health ID our survey
1086.13s - 1095.81s |  says no get ready to steal team left foot banking information like credit card number CVV credit
1095.93s - 1112.94s |  card number a survey says yes you're gonna sweep the board email address email address survey says
1112.94s - 1128.34s |  yeah John for the win driver's license number driver's license number no okay left foot you
1128.34s - 1141.31s |  have one chance for the second most popular answer to this survey question full name full
1141.31s - 1155.69s |  name the survey says they 100 points to team left foot coming in with the clutch at the end okay
1155.69s - 1167.09s |  round four can we have discount John Hammond and a other guy of beard and glasses please okay take a
1167.09s - 1176.74s |  buzzer all right please show the question name a prestigious cybersecurity company might have
1176.74s - 1186.98s |  been laid off from left foot buzzed in first speaking to the mic McAfee McAfee survey says
1186.98s - 1199.70s |  no right foot crowd straight crowd strike a survey says the number one answer John John a
1199.70s - 1211.06s |  play or pass we will play playing excellent okay pass the mic so go on the end please thank you
1211.26s - 1218.26s |  okay name a prestigious cybersecurity company you might have been laid off from Avast sorry what
1218.26s - 1232.38s |  Avast Avast survey says no back to John Microsoft Microsoft survey says no blame the community they
1232.38s - 1245.26s |  didn't list them enough um Norton Norton a survey says yeah please don't do that no alley
1245.26s - 1253.50s |  team left it left foot you can you can uh confer have you already yeah um I'll start with the
1253.50s - 1258.42s |  trauma dump and it's not a real answer I personally got laid off from a cybersecurity company but
1258.42s - 1262.22s |  we're gonna go with snick not that one but we're going with snick I forgot the name what are you
1262.22s - 1272.98s |  going with snick sneak sneak I think all right sneak a survey says no so right foot win that
1272.98s - 1286.91s |  round with 37 points hey this is an interesting one Oh what happened yeah but does mean you got
1286.91s - 1308.66s |  laid off from are we sure what round are we on all right so this all right well I'm just gonna
1308.66s - 1311.74s |  read the answers don't click anything because I think we might build the next round but anyway
1311.74s - 1319.26s |  all right in sixth place was Palo Alto fifth was Splunk okay we are on the good round fourth was
1319.26s - 1332.75s |  NCC third was Rapid 7 second was Mandiant and number one was CrowdStrike okay yeah next round
1332.75s - 1346.90s |  all right excellent take a buzzer please yours is on the floor for some reason okay show the
1346.90s - 1358.31s |  question please name a DEF CON party oh is it not okay that is team left foot buzzing in all right
1358.31s - 1363.59s |  sorry we'll figure that out in a minute team effort the blue team pool party the blue team
1363.59s - 1380.00s |  pool party no right foot red team pool parties the red team pool party no left foot what is
1380.00s - 1387.52s |  goth con excuse me goth con that's that's in about an hour but okay goth con correct you
1387.52s - 1396.16s |  win control the board Ali do a play or pass play play okay all right Ali it's your turn name a
1396.16s - 1407.78s |  DEF CON party queer con queer con oh yeah queer con oh survey says sorry yeah thank you queer
1407.78s - 1427.37s |  con second oh god no no conferring no no ninja party the ninja party no arcade con arcade con
1427.37s - 1443.47s |  yes for 15 masquerade party the masquerade party no I know don't blame me why I didn't come up
1443.47s - 1455.23s |  with these blanket con sorry you speak up blanket con blanket con no John's team can steal what are
1455.23s - 1471.20s |  you gonna go with John that con party the vet con party no so team left foot win those points
1471.20s - 1485.85s |  66 points to team left foot okay the sixth answer Florida man the fifth answer Illuminati
1485.85s - 1510.14s |  party the third answer DEF CON fuzz okay yeah it's you and John again yeah we should check
1510.14s - 1519.20s |  the buzzers hang on for the second round of feet con we will only press buzzers with our feet
1520.80s - 1534.91s |  that one works okay they both work good I just changed it all right you can have the buzzers again
1538.18s - 1548.98s |  Mimi okay all right show the question please name a popular cyber security cert that is left foot
1549.78s - 1558.26s |  CompTIA plus I'm sorry the the CompTIA plus one I only my brain only knows it as CompTIA plus
1558.26s - 1568.10s |  I'm a I'm a software engineer no John certified ethical hacker certified ethical hacker
1570.50s - 1574.66s |  is that you've been controlled you already answered do you want do you want to play
1574.66s - 1581.78s |  we will play all right John's gonna play is it by CompTIA yes it's a plus all right
1582.66s - 1595.58s |  sec plus sec plus survey says yes net plus that's not you didn't say anything close to sec plus
1596.30s - 1600.22s |  CompTIA plus it's by CompTIA they have they have multiple certs
1600.38s - 1606.22s |  I call it in my brain okay wait do you take plus net plus no good huh
1608.38s - 1619.68s |  what's your answer I'll go with net plus net plus no all right next
1621.44s - 1632.34s |  OSCP huh OSCP OSCP I'm definitely coming back next year all right OSCP a survey says
1632.66s - 1634.82s |  yeah number one answer
1639.95s - 1654.32s |  CESA you speak intimate yep CESA CESA a survey says no back to John CISSP CISSP survey says
1655.36s - 1666.53s |  yes bow tie it's all on you let's go with Cisco CCNA Cisco CCNA a survey says
1667.49s - 1669.97s |  no okay Ali's team you have a chance to steal
1672.05s - 1675.33s |  Ali I need an answer from you you're a team captain
1677.92s - 1687.34s |  sans I'm sorry don't make fun of me I'm a software engineer I don't know what it's called
1687.98s - 1693.55s |  what you you can confer with your team what you can confer with your team
1693.79s - 1706.18s |  no they said sans sans a survey says no right foot win the round it's okay those are pity points
1707.54s - 1715.46s |  all right and the fifth answer was PNPT the fuck is that that's like a transistor
1716.90s - 1722.18s |  yo they still haven't figured out if P equals NP this is not fair okay next round math jokes
1725.02s - 1738.11s |  okay hoping the buzzer's gonna work can you just hold them up so I can see which buzzer's
1738.11s - 1745.31s |  the first yeah okay all right ready to show the question all right name a programming language
1745.87s - 1752.83s |  that was team right foot Python Python you're kidding survey says I'm gonna number one answer
1753.79s - 1757.31s |  play this mofo that's John's decision but thank you
1759.71s - 1767.71s |  yeah I think we'll play this one we're playing excellent all right name a programming language
1767.71s - 1777.15s |  let me carry I think we're gonna win this one so I'm gonna go with APL APL a survey says
1778.03s - 1782.91s |  no John's dreams of being shattered in front of his eyes
1784.35s - 1794.35s |  see was that just where you speak up see see a survey says correct yes third
1795.87s - 1809.63s |  did they say lowercase c or uppercase c sharp a survey says yes John rust rust a survey says
1810.43s - 1819.98s |  number two answer I'm gonna get hate for this but java java a survey says no
1826.77s - 1830.53s |  all right one life left well crap um
1834.10s - 1843.60s |  I should have said c but I got the two yeah c plus plus c plus plus a survey says
1844.56s - 1849.76s |  yes you're literally kidding me you could tell a bunch of security people filled this out because
1849.76s - 1858.24s |  this is not right most popular survey question survey answer was come on down say I'm asking
1858.24s - 1870.32s |  you sorry uh uh javascript javascript our survey says no team left foot for the steal good answers
1871.04s - 1871.54s |  confer
1874.08s - 1880.96s |  it's all right it's all right go lang go lang survey says
1882.96s - 1891.04s |  no I ever bolted by this list y'all are wrong for this I'm just gonna say
1892.32s - 1898.08s |  there are a lot of funny people who answered the surveys because the sixth answer is of course
1898.80s - 1914.78s |  cobalt nasty for this all right the final round round eight
1916.30s - 1924.62s |  that was crazy we'll figure out the schools at the end let's
1925.42s - 1928.94s |  let's not let's not risk upsetting powerpoint again all right
1932.46s - 1935.18s |  we're gonna need that money back we don't win with what
1937.18s - 1939.66s |  he didn't he didn't bribe anything I don't remember anything all right
1939.74s - 1945.50s |  okay hang on we gotta wait for the I did it oh he did show the question I read the question I
1945.50s - 1952.46s |  think that's fair right no in my own head I read it did you see the question sim grip I did too
1952.46s - 1955.82s |  okay all right we got an honest person over here all right so right foot
1955.82s - 1960.94s |  neighbor's name a code scanning tool sim grip sem grip our survey says
1963.81s - 1967.57s |  yes but it's not the top answer so if you can get either one two or three
1970.43s - 1984.59s |  sink sorry um I'm gonna need an answer sink skink spank stinky
1989.55s - 1998.74s |  no it's actually it's a reg ex you should like fill it in okay john do you wanna play or pass
2001.47s - 2006.83s |  we will play okay all right name a code can you pass the mic down yeah I'm sorry
2006.83s - 2016.94s |  it name a code scanning tool link can you speak up link link lint
2018.14s - 2032.03s |  our survey says no sorry all right name a code scanning tool honestly don't have a clue
2032.03s - 2041.71s |  diff diff all right no john check marks check marks our survey says yes
2042.51s - 2052.34s |  uh I'm gonna answer sneak sneak our survey says yes the top answer
2053.78s - 2066.48s |  alley in no world was that sneak you can't just take four letters to begin with s n y and just
2067.52s - 2079.49s |  iterate through the other one okay name a code scanning tool um in solidarity with our
2079.89s - 2084.53s |  friends over there I don't have one because she should have got that we love honesty
2086.77s - 2093.17s |  okay well unfortunately that's incorrect so team left foot you do get a chance to steal
2104.05s - 2111.86s |  coverity do you have an answer though alley you have to say it
2112.66s - 2122.10s |  coverity coverity so they says yes wait wait hang on okay that should be four points
2123.62s - 2130.50s |  oh hell yeah that's a typo that I did not check earlier okay so just subtract 50 from the score
2130.50s - 2142.06s |  we're gonna add all right yeah so left foot does win that work repos right now okay so 305
2142.86s - 2150.06s |  all right that is it so go back to the scores who did win oh yeah I'm sorry I'm new at this
2150.06s - 2162.94s |  all right the third answer was sonar cube the second answer bearer code all right back to the
2162.94s - 2182.85s |  scores who won oh well left foot crushed it but yeah left foot win 305 to 255 what comtia plus
2186.67s - 2197.31s |  honestly okay honestly I accept it all right so we'll give out the prizes now so
2197.95s - 2199.07s |  wait can I get my money back
2204.03s - 2212.59s |  really okay good all right the money is mine okay so the grand prize for each of you hack the box
2212.59s - 2219.63s |  have kindly sponsored for one year vip voucher each also if you bring the card with this on to
2219.63s - 2223.71s |  the hack sorry the hack the box booth tomorrow they will give you a free piece of merch
2226.94s - 2232.86s |  can you distribute them so they they do have the codes printed on them so just keep that secret
2234.22s - 2244.66s |  um you also get their prizes you also get the uh the black badge raffle feet feud card
2254.06s - 2261.52s |  okay uh helen yep yeah so you also get a pair of coding socks
2262.48s - 2275.18s |  each okay but wait wait this has a list of languages on it of course team right
2275.18s - 2283.34s |  thought valiantly you get a weirdly little it's like a baby foot candle it's I don't know
2285.92s - 2292.08s |  it was a very weird night on amazon um so yeah it's like a candle in the shape of a baby foot
2292.64s - 2302.66s |  and also fruit by the foot that's the real prize we really won honestly damn that's crazy
2306.10s - 2311.14s |  we have extras excellent everyone gets fruit by the foot all right best day ever
2312.42s - 2318.74s |  there's starburst flavored that is our show uh I hope everybody enjoyed it um hopefully we'll be
2318.74s - 2325.30s |  back next year with bigger prizes wait I'd like a bigger audience but you've been fantastic so give
2325.30s - 2332.02s |  yourselves a round of applause thank you everybody please also a huge round of applause to helen and
2332.02s - 2343.23s |  wolfie for helping make this and run it and tips give it up for tips please also give a hand to
2343.23s - 2347.87s |  wolfie and helen my assistants who have corrected me numerous times as i've fucked everything up
2348.59s - 2355.79s |  no powerpoint next year maybe another round of applause for right foot sorry left foot
2358.03s - 2360.99s |  and right foot i'm selling my socks if anyone wants
2360.99s - 2364.67s |  i've been so furious this has been feet view thank you so much for watching
2364.67s - 2368.35s |  does anyone want to buy my left sock or what have a great rest of the evening and enjoy
2368.35s - 2377.09s |  hacker jeopardy thousand dollars thank you guys you did amazing i'm so proud of you


url: https://www.youtube.com/watch?v=70yyOMFylUA
title: DEF CON 32 - Gotta Cache ‘em all bending the rules of web cache exploitation - Martin Doyhenard
type: video
author: DEFCONConference
ingestion_date: 2024-11-10
transcription_model: large-v2
chunking_status: completed
content:
{
  "webpage_url": "https://www.youtube.com/watch?v=70yyOMFylUA",
  "title": "DEF CON 32 - Gotta Cache \u2018em all  bending the rules of web cache exploitation - Martin Doyhenard",
  "description": "In recent years, web cache attacks have become a popular way to steal sensitive data, deface websites, and deliver exploits. We've also seen parser inconsistencies causing critical vulnerabilities like HTTP Request Smuggling. This raises the question: what happens if we attack web caches' URL-parsers?\n\nIn this session, I'll introduce two powerful new techniques that exploit RFC ambiguities to bypass the limitations of web cache deception and poisoning attacks.\n\nFirst, I'll introduce Static Path Deception, a novel technique to completely compromise the confidentiality of an application. I\u2019ll illustrate this with a case study showing how such a breach can be replicated in environments like Nginx behind Cloudflare.\n\nNext, I'll present Cache Key Confusion, and show how to exploit URL parsing inconsistencies in major platforms, including Microsoft Azure Cloud. I\u2019ll then show how to achieve arbitrary cache poisoning and full denial of service.\n\nFinally, I'll reveal how to supercharge these vulnerabilities with a live demo that blends Cache Key Confusion with a \u201cnon-exploitable\u201d open redirect to execute arbitrary JS code for complete site takeover.\n\nAttendees will depart armed with a set of innovative techniques, along with a definitive methodology to find and exploit these and other URL or HTTP discrepancies.",
  "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
  "duration": 2638,
  "channel": "DEFCONConference",
  "uploader": "DEFCONConference",
  "upload_date": "20241016"
}

0.00s - 5.00s | This text was transcribed using whisper model: large-v2

 Uh welcome to track 2 talk number 2 on Saturday. Uh this morning we uh have Martin and he's
7.40s - 11.88s |  gonna talk about breaking the rules of webcache exploitation. So are you ready Martin?
11.88s - 21.66s |  I'm ready. Let's hear it for Martin. Thank you. Hello everyone. Welcome to Gara Cache Mall
21.66s - 28.14s |  bending the rules of webcache exploitation. Uh it's already not working. So this is the
29.14s - 34.14s |  agenda for today. First I'm going to briefly explain how a webcache works and then I'm
36.78s - 42.68s |  going to show how we can abuse um discrepancies between the URL parsers of HTTP servers and
42.68s - 48.28s |  cache proxies in order to steal secret information from victims using arbitrary webcache
48.28s - 54.32s |  deception. Next I'm going to show how use the same techniques or similar techniques to
54.32s - 60.68s |  obtain arbitrary webcache poisoning by modifying the keys of a store resource. Then I'm
60.68s - 66.16s |  going to combine both techniques in order to store whatever I want in the webcache with any
66.16s - 71.86s |  key so this will allow us to create a full defacement of a website. And finally I'm going to
71.86s - 76.00s |  present some protections that can be used to defend against those attacks and it takes hours
76.00s - 83.14s |  of the session. So before I start eh I want to clarify that most of the or all the examples
83.16s - 88.54s |  are going to be based on CDNs like CloudFront, CloudFront, Akamai but the idea is that you
88.54s - 94.20s |  can use all of these techniques with any HTTP with any webcache eh proxy or with any webcache
94.20s - 100.98s |  that you want even those that are included in the HTTP server. So whenever someone sends a
100.98s - 105.26s |  request to a website using a webcache basically the first thing that is going to happen is
105.26s - 109.56s |  that the cache proxy is going to handle the request and is going to try to generate the
109.58s - 114.28s |  response without having to communicate with the original server. And this is going to be
114.28s - 120.06s |  done by calculating a key and then this key is basically an identifier of the request that
120.06s - 125.26s |  states which is the resource that is being requested and it's not really important other
125.26s - 129.64s |  other things like the the headers like cookies or user agent because when we have two
129.64s - 134.94s |  requests that are looking for the same resource and that resource is static then we just need
134.96s - 140.10s |  to provide the resource and we don't care about processing anything else. So we will see
140.10s - 145.10s |  that the keys are just URLs and host names and in this talk we will just focus on the URL. So
147.74s - 153.44s |  when the cache proxy calculated this key and is looking for a re- a response that can be
153.44s - 159.08s |  served eh to this to this request it will he will look at the cache table and if there's
159.08s - 163.96s |  nothing there it will send the request to the original server. Now the original server is
163.98s - 169.84s |  going to parse the URL obtaining the path and then it's going to use an endpoint to resolve
169.84s - 174.66s |  the request. Sometimes it's just a static endpoint like a file and sometimes it's a dynamic
174.66s - 181.66s |  one that is generated by a program back then. So when this response is sent back to the cache
181.66s - 187.44s |  proxy the well the cache proxy is going to generate to evaluate some rules to see if this
187.44s - 193.44s |  response should be stored in the cache. So if it's static or not. These rules are usually
193.46s - 199.76s |  evaluated on the response using the cache control header but also in the request the cache
199.76s - 204.50s |  proxy is going to extract the path and it's going to evaluate it with some of the other rules.
204.50s - 210.70s |  For example the static extension one which is by default in most CDNs that looks for static
210.70s - 216.24s |  extensions. So basically this is the list that Cloudflare has. Uh it will look for the path
216.24s - 221.88s |  and the extension of that path and if it ends with any of these eh extension any of these
221.92s - 226.06s |  strings then it's going to decide that this is a static respo- resource and it's going to
226.06s - 232.20s |  store it in the in the web cache. So in this example we see that styles.css ends with a
232.20s - 236.94s |  static extension and therefore it's going to be cached using the key that was calculated in
236.94s - 243.70s |  the first step. Now whenever another client or even the same client sends a request to the
243.70s - 248.24s |  same resource the cache proxy is going to handle that, it's going to get the key of that
248.24s - 253.84s |  request and then it's going to retrieve the res- the store resource because it doesn't need to
253.84s - 259.26s |  interact with the origin server. And that's basically how a web cache works. Now we can
259.26s - 264.66s |  exploit this in many ways. The first one is usually is the first one I'm going to describe
264.66s - 269.66s |  actually is the one that was discovered by Omer Gil in 2017 which is a web cache deception
271.86s - 277.54s |  and basically this attack is based on the idea that some backend servers use this special
277.54s - 282.38s |  mapping where they are going to put some parameters in the path, this is called path
282.38s - 286.42s |  parameters, and they are going to just look for the beginning of the path and they will try to
286.42s - 293.08s |  match it with uh with any regex that is eh relate- related with the handler. So in this case
293.08s - 298.52s |  if we send a request to slash my account slash param1, param2 then the only the first part
298.52s - 305.04s |  is going to be used to map the endpoint and then this in this case is going to the origin
305.04s - 308.40s |  server is going to generate a response which contains a lot of information from the victim
308.40s - 315.28s |  because it is my account and it contains for instance the mail or the credit card details so
315.28s - 322.17s |  basically this is what we want to steal. And to do so the first thing is to send uh a link to
322.17s - 327.71s |  the victim, this requires user interaction and the link will start with the endpoint that we
327.71s - 332.65s |  want to to steal the response so in this case it's my account and then it will use as a
332.67s - 338.95s |  parameter a dot js. And the reason this is because first when it's handled by the origin
338.95s - 344.19s |  server this is going to be mapped to my account using the cookies of the victim and so the
344.19s - 349.39s |  response with the sensitive information is going to be sent back to the to the proxy and now
349.39s - 354.99s |  with the cache proxy it evaluates the the path, this in this case it doesn't know anything
354.99s - 359.13s |  about this mapping so it will think that the last part is just a parameter it's not a
359.15s - 363.63s |  parameter sorry it's a path it's part of a path and therefore it will think that it has a static
363.63s - 368.89s |  extension in this case dot js and so it will store this response containing all the secret
368.89s - 373.87s |  information. Now the attacker can send the same request and as the cookies are not important
373.87s - 379.07s |  as the only important thing is just getting the same path well the cache proxy is going to
379.07s - 383.85s |  match it with the key that already calculated in the first place and it will retrieve the
383.85s - 388.35s |  victim's response containing all the secret information. Now the problem with this
388.37s - 393.21s |  technique is that we need a special mapping for that basically we need this special my
393.21s - 397.75s |  account thing and then we also require that the parameters that we are sending in this case
397.75s - 403.31s |  a dot js doesn't have any effect on the backend because if not maybe the the response is not
403.31s - 409.74s |  going to contain that information that we want. So I started looking at different ways to
409.74s - 414.24s |  obtain arbitrary web cache deception which is basically obtaining the same thing but for any
414.26s - 420.72s |  kind of uh end point that we want. And to do this I started looking at how the the path is
420.72s - 426.76s |  obtained at each state. So basically the path is obtained from a URL by using these
426.76s - 431.90s |  delimiters which are basically characters that can then can be used to know where each part of
431.90s - 438.10s |  the URL or each part of the elements of the URL start and begin uh and end. There are a lot of
438.10s - 443.72s |  them defined by the RFC so we can see that the URL has things like the scheme, the
443.72s - 449.26s |  credentials, the host, the path query and fragment. So to split the each of them in a URL we
449.26s - 455.20s |  need these these delimiters. The problem is that the RFC says that any any of these
455.20s - 460.34s |  delimiters may or may not be or any of these characters may or may not be used as delimiters.
460.34s - 464.88s |  And so each implementation can choose new delimiters and this could create some
464.88s - 471.82s |  discrepancies. So let's say we find a server that is using a cache proxy and that is using
471.82s - 478.00s |  the dollar sign as a delimiter. Then if we send a request to my account dollar sign 8 dot JS
478.00s - 483.20s |  this is just going to be interpreted as my account because of the delimiter. However as I
483.20s - 488.20s |  said in the front end there's no delimiter being being said with eh the dollar sign and
488.20s - 492.18s |  therefore this is going to be interpreted the entire URL is going to be interpreted as the
492.18s - 497.92s |  path and the last part which is 8 dot JS is going to be used as the extension. In this case
497.92s - 502.52s |  actually it's dot JS the extension and so this is going to be stored in the cache just
502.52s - 507.70s |  because of that delimiter. And in this case we can use it with any kind of path we want
507.70s - 512.66s |  because we are not attacking a mapping we are attacking the entire way of obtaining the path
512.66s - 518.94s |  from the URL. Now which kind of delimiters we can find in the well well there are a lot of
518.94s - 525.12s |  them. Some I listed were in the frameworks is eh the first one is semi the semicolon in
525.12s - 531.72s |  spring so basically to define a matrix variable that is similar to a path eh parameter we can
531.72s - 537.46s |  use a semicolon. And the thing is that if we send something like my account semicolon 8 dot
537.46s - 543.84s |  JS anyone using Java probably Java in the well so they are going to consider this as my
543.84s - 548.48s |  account. This includes Google or big companies that are using Java because this is how
548.50s - 555.52s |  Spring works. And so we can use this for the attack. Now in Rails there's another really
555.52s - 561.96s |  interesting case which is they use the dot as a delimiter and that's because ehm the user is
561.96s - 567.46s |  able to send an extension with a valid endpoint and this will eh the effect that this will
567.46s - 573.26s |  have is that Rails will look for a specific view for that eh request. So let's say we define
573.32s - 578.76s |  an endpoint which is my account and then when we send a request to my account Rails will just
578.76s - 585.26s |  go and look for the HTML view. Now if we send something like my account dot CSS, Rails will
585.26s - 591.24s |  not just return a not a not found but it will look for the view of the the CSS view of my
591.24s - 597.78s |  account. Usually there's no CSS view for my account so we'll just receive a 406 error. But
597.78s - 602.62s |  the thing is that if the extension that we include is not recognized by Rails so let's say we
602.64s - 607.88s |  send something like my account dot AAA then Rails is just going to send us the default view
607.88s - 614.32s |  which is HTML. Remember all these eh all these extensions that are defined by Cloudflare.
614.32s - 620.32s |  Well in green you can see the ones that are recognized by Rails. So if we send my account
620.32s - 626.94s |  dot PNG or dot GPA or CSS then all of them are going to be recognized by Rails and it's going
626.94s - 631.98s |  to try to look for that response. And if there's no view for that extension then it's just
632.00s - 636.68s |  going to return us an error. But in red we see all the other extensions that are not
636.68s - 642.44s |  recognized by Rails. So if we send something like my account dot ICO then we are going to
642.44s - 649.74s |  get the default view but Cloudflare is going to think that this is a catchable resource. Now
649.74s - 653.88s |  there are other delimiters that can be used in HTTP servers instead of frameworks such as
653.88s - 659.28s |  the encoded null byte which is the case of open light speed. So basically this works like a
659.30s - 664.10s |  question mark if we send something like my account null encoded byte A dot JS then this is
664.10s - 670.00s |  just going to be used as my account. The problem for this is that we cannot send it through
670.00s - 675.44s |  many CDNs like Azure, Imperva, Cloudfront, Cloudflare, Google Cloud. That's because they
675.44s - 680.62s |  don't accept null encoded bytes. But fortunately Akamai and FastCDC yes so we can use them
680.62s - 687.36s |  for eh this explanation. And finally in Nginx we have a delimiter whenever whenever we
687.40s - 693.54s |  find a rewrite rule and the delimiter is the new line encoded and so basically it works just
693.54s - 698.58s |  as in open light with a null encoded byte and in this case we can send it through any CDN we
698.58s - 705.12s |  want. So if we send a request using any of these delimiters with the correct with the
705.12s - 710.42s |  corresponding eh backend then all of them are going to be interpreted as my account. The
710.42s - 715.26s |  response is going to be sent back but in this case the web caches are going to just store that
715.26s - 722.72s |  because of the static extension. But the problem with this attack is that many CDNs are
722.72s - 727.08s |  are already aware of this and they are implementing some detection some protection for for
727.08s - 731.86s |  this kind of attacks. In the case of Cloudflare for instance is web cache deception armor
731.86s - 737.50s |  which what it does is it looks for the extension of the request and the content type of the
737.50s - 742.30s |  response. And if they don't match for for instance for instance in this example we are using
742.30s - 748.68s |  dot CSS as the extension and the content type is text HTML so if they don't match basically
748.68s - 755.22s |  this is an attack and Cloudflare won't store this in the cache. So we can attack other rules
755.22s - 761.46s |  like the static directories one. That's a rule that we can create as a user that states that
761.46s - 766.16s |  anything starting with a specific path is going to be stored in the cache and that's really
766.16s - 771.20s |  common in many applications because we we usually store static files in the same directory
771.24s - 776.88s |  like share, public, assets, resources, anything that you can think. Uh and so we can use
776.88s - 782.24s |  this to attack this kind of eh this kind of web caches that are used in the rules. To create
782.24s - 788.18s |  an attack in this case we will need some payload that is interpreted in the front end with
788.18s - 793.60s |  the static directory at the beginning so basically in if the static directory is called static
793.60s - 799.36s |  then we want the path to start with static slash and anything we want. And then in the back
799.36s - 804.70s |  end we still want my account because we in this case want to steal my account we could steal
804.70s - 810.90s |  anything else we want. So if we have a delimiter in the back end we can try to do this but we
810.90s - 817.18s |  will need something else. And as I said to extract the path from a URL we just need the
817.18s - 822.82s |  delimiter and then that path is used in the keys rules and mapping. Well not really. There's
822.82s - 828.86s |  one extra step which is normalization. We need to normalize the path and most CDNs and
828.86s - 834.14s |  back end servers do normalize the path. This includes decoding the the characters. We can
834.14s - 840.80s |  send them with URL encoded eh notation that is percentage eh and the excite small value of a
840.80s - 846.54s |  character. And then in some cases the eh other other characters like segment delimiters like
846.54s - 852.88s |  the slash or the dot are also going to be decoded so this is going to be really useful. And
852.88s - 858.98s |  then the normalization also includes dot segments eh resolution which is just resolving
858.98s - 863.22s |  path traversals and whenever we find a dot segment then we just remove the previous
863.22s - 868.22s |  segment. And some parsers like the IIS one eh will also convert back slashes to slashes. So
872.20s - 876.94s |  now that we know this we can actually create a new payload that is going to use normalization.
876.94s - 881.48s |  Again we are using a delimiter so everything before the delimiter is going to be interpreted
881.48s - 888.12s |  in the back end server. But now we are also sending an encoded eh path traversal. And that is
888.12s - 894.46s |  because the front end is not using the delimiter so it's going to resolve this into this. And
894.46s - 900.72s |  this includes a path traversal which is going to convert it to static WCD. So now in the front
900.72s - 905.24s |  end we see that there are static characters there so this is going to be stored in the web
905.24s - 910.44s |  cache. And in this way we are able to bypass the web cache deception armor because there's no
910.46s - 918.36s |  extension so nothing to compare. Now let's see how how we can how we can apply this in a
918.36s - 923.36s |  real environment. And so first we need a a front end that is normalizing the path. Sorry. Eh
926.76s - 932.46s |  this is the case for CloudFront, Imperva and Azure. And so if we combine them with any back
932.46s - 937.64s |  end that is using a specific delimiter like for instance Tomcat then we can send the same
938.26s - 944.70s |  payload which is secret the semicolon with that represented delimiter. And this will cause
944.70s - 949.84s |  that in the back end we see secret. And in the front end again the semicolon is not a
949.84s - 954.88s |  delimiter for either CloudFront I Azure or Imperva. And therefore this is going to be
954.88s - 962.12s |  resolved to static slash X. And this way we will be able to steal any response we want in an
962.14s - 967.42s |  environment that is using basic basic and default configured eh Tomcat and any of the CDNs
967.42s - 972.42s |  that I showed. This can be used also with other back ends like eh Rails and Puma or even some
975.06s - 980.22s |  in the same case that we observed in the in the first example when we had a special mapping
980.22s - 985.80s |  in the back end. In this case there is no delimiter the the delimiter is just the the path
985.82s - 993.73s |  that we are attacking. Now we could also try another attack that is using the opposite
993.73s - 997.51s |  normalization so basically in the front end we don't have normalization that's the case of
997.51s - 1003.15s |  CloudFront eh Google Cloud Platform and Fastly. And in the back end we have normalization
1003.15s - 1008.15s |  in Nginx, IIS and open light speed. So if we use this we can create the attack by starting
1011.15s - 1016.89s |  with the static directory. In this case it's slash static. And this is because the path is
1016.91s - 1022.31s |  not normalized and so CloudFront for instance will think that this is the entire URL is the
1022.31s - 1029.21s |  path. But in the back end this is going to be normalized and therefore anyone using Nginx or
1029.21s - 1034.99s |  IIS or open light speed is going to see that the response is going to be for secret and this
1034.99s - 1041.43s |  will be storing the cache because of the static path. This worked on OpenAI and many other
1041.43s - 1046.45s |  bounties that I wasn't able to be eh I'm not able to talk about. And also this can be used
1050.45s - 1056.95s |  with IIS basically IIS is the only one that is decoding uh normalizing the backslash so if we
1056.95s - 1063.09s |  send something like using the in the slide this is going to be converted to static eh and
1063.09s - 1067.73s |  basically a dot segment with a backslash and it's going to be converted to account in IIS but
1067.73s - 1073.67s |  not in any other eh CDN because IIS decided that this is a cool idea. And so yeah they are
1073.67s - 1080.52s |  the only ones that I can be attacking with any other CDN. Now if we don't find one of these
1080.52s - 1086.84s |  rules well there's there's another rule that we can use which is the static files. Eh in many
1086.84s - 1093.90s |  in many CDNs we will see that there's a static file rule by default to store robots dot txt but
1093.90s - 1100.10s |  we can also find others for favicon or for index dot html or for any the any file that is
1100.10s - 1105.74s |  usually accessed by people and is static. And in this case we can use the same attack only
1105.74s - 1110.72s |  that we are not going to target a static directory we are just going to target something that
1110.72s - 1117.72s |  is being stored by the front end. So we can do something like secret then the delimiter
1117.74s - 1123.26s |  then coded eh dot segment and finally robot dot txt and this should work in many in many
1123.26s - 1128.92s |  applications. And also we can try to find other files. I found that there are many in in in the
1128.92s - 1136.20s |  wild so you should be able to find many others if you look for. So web cache deception is
1136.20s - 1140.94s |  pretty cool but the thing is you need user interaction and we don't really want user
1140.94s - 1147.38s |  interaction. So web cache poisoning is there for rescue us and the idea in web cache
1147.44s - 1153.28s |  poisoning is that we are going to try to store a malicious response in in a with a key that is
1153.28s - 1158.78s |  going to be accessed by other people. So let's say we want to eh hijack someone that is
1158.78s - 1163.72s |  using eh this this website this website and is accessing to the home page. Well if we are
1163.72s - 1167.96s |  able to find a payload that is stored in the home page then everyone navigating through the
1167.96s - 1172.76s |  web page is going to be affected. And usually you will look for something that is not
1172.78s - 1178.52s |  stored like that is not used in the key so in this case if the cookie value is reflected in the
1178.52s - 1184.62s |  in the body of the response then we are able to store this and anyone using the home page is
1184.62s - 1190.54s |  going to be affected and is going to re is going to retrieve the malicious response. The
1190.54s - 1195.64s |  problem with this attack is that in many cases we will find eh an example of a cache
1195.64s - 1200.38s |  poisoning but is with a is using a key or is using a resource that is not going to be
1200.40s - 1204.90s |  accessed by anyone. Like for instance if we are able to poison something that starts with
1204.90s - 1211.04s |  files then our username and then our unique ID no one is going to be accessing that eh that
1211.04s - 1216.04s |  path by themself so we will need user interaction. And that's not really cool. So to avoid
1218.88s - 1224.16s |  having to interact with any victim I designed an attack that is going to be able it's going
1224.18s - 1229.18s |  to allow us to change the key of the eh poison resource. Now in this case we will need key
1231.44s - 1237.12s |  normalization and this is by default in impaired but Azure is just normalizing the key eh and
1237.12s - 1243.16s |  it's partially by default also in Akamai and it's configured in all the other CDNs and what's
1243.16s - 1247.60s |  really interesting about this is that if you remember CloudFront for instance was
1247.60s - 1251.86s |  normalizing the rules but is not normalizing the key. So that means that they have different
1251.86s - 1257.14s |  parsers for each of these processes which is even in the same eh in the same HTTP parser
1257.14s - 1263.42s |  probably. So this this tell us that we can even find other desynchronization attacks or
1263.42s - 1268.42s |  other discrepancies even in the same eh in the same web cache. So as I said what we need is
1271.76s - 1277.06s |  a a front end that is normalizing the key and what we can do to detect this is to send a
1277.10s - 1281.94s |  request to a static resource like style.css and whenever we see evidence that this is of
1281.94s - 1288.30s |  being obtained from the web cache so for instance we have an X cache hit value eh hitter or we
1288.30s - 1293.98s |  can just compare the time of the response and whenever we find this then we need to modify the
1293.98s - 1300.48s |  path so that when normalized we still have style.css and if we send this request and the
1300.48s - 1304.66s |  response is still from the cache then it means that the key was normalized and so we
1304.66s - 1311.70s |  detected key normalization. And how we can exploit eh web cache poisoning with this? Well
1311.70s - 1316.88s |  basically we will try to modify as I said the key by sending first the exploit that we want
1316.88s - 1322.78s |  so this is the the the response that we will we will use to poison eh usually contains a
1322.78s - 1328.48s |  malicious payload then a delimiter so again we will need a delimiter in the back end but as I
1328.48s - 1334.22s |  said this is really common and we can find them almost anywhere and then we will need
1334.26s - 1339.86s |  some normalization in the front end so Imperva, Azure uh and sometimes Akamai and all the
1339.86s - 1345.10s |  others that are configured eh and this is going to be resolved with Poison so basically the
1345.10s - 1349.80s |  web cache is going to store the exploit response in the Poison endpoint which is something
1349.80s - 1360.13s |  we control so it's arbitrary we can just wire up a path we want usually the home page. So in
1360.13s - 1364.41s |  the in the first example I didn't talk w- about one of the delimiters because we weren't
1364.41s - 1369.55s |  able to send it with web cache deception and that's because the user needs to send it
1369.55s - 1374.95s |  through his browser and anything but uh and if the browser sees a hash then it's not going to
1374.95s - 1381.63s |  send the fragment but in web cache poisoning we can use the fragment. So I I started
1381.63s - 1386.19s |  looking at different at the behavior of different CDNs and HTTP servers and even
1386.19s - 1393.01s |  frameworks in regarding the the hash and I found that there were a lot of differences. The
1393.01s - 1397.41s |  most interesting one was again Imperva because they were no likes in the key. Eh Cloud
1397.41s - 1402.09s |  Front well sorry Cloud Front can be configured to to do the same thing so it's interesting to
1402.09s - 1407.09s |  see they are too. And basically if we have a hash and eh and key normalization then we can
1409.59s - 1414.37s |  use Imperva for for this exploit that I'm going to show. And in the back end we can use
1414.37s - 1419.63s |  pretty much any HTTP server like NGINX, Unicorn, Puma then we can use also frameworks like
1419.63s - 1425.73s |  Rails, Flash, Laravel they are not going to see the hash as a delimiter. Sorry they are
1425.73s - 1431.27s |  going to see the hash as a delimiter. Eh so basically we can send an attack that uses the
1431.27s - 1435.45s |  hash. First we are going to send what we want to be interpreted in the back end because it's
1435.45s - 1441.85s |  going to be using the hash and so if we send something like this then the XSS is going to be
1441.85s - 1446.09s |  interpreted as the path in the back end. It's going to retrieve retrieve the response the
1446.09s - 1451.29s |  malicious response and now this is going to be normalized the key so we will be able to
1451.29s - 1456.29s |  poison the home page with any any web cache poisoning eh payload that we have. This was
1458.63s - 1464.77s |  found in Vista, Mastercard and T-Mobile and again many others that I'm still unable to
1464.77s - 1469.77s |  talk about. And of course we can use the same attack for with eh Azure and Imperva and also
1470.17s - 1475.17s |  with Tomcat, with Rails, with NGINX, with a new line encoded and also again with any back end
1481.37s - 1486.71s |  that is using a special mapping. So it should be pretty easy and I already saw many people
1486.71s - 1491.75s |  exploiting this eh since I give this talk in black hat so it seems like there are a lot of
1491.75s - 1498.05s |  targets. Now we can do similar a similar thing with Azure only in this case Azure is
1498.07s - 1505.07s |  considering the front the hash as a delimiter and this is not the case for OpenLaySpeed,
1505.07s - 1511.69s |  Express and Django. So what we are going to do is we're going to send a payload that in this
1511.69s - 1517.39s |  case starts with what we want to poison so the keys the beginning and that's because the
1517.39s - 1522.59s |  front end is using the hash as a delimiter then the rest is going to be normalized and we
1522.61s - 1527.61s |  have the exploit. And again we can poison any path we want that is using Azure and OpenLight
1529.77s - 1534.65s |  or other frameworks there are other examples but as I in this research I wasn't able to look
1534.65s - 1539.65s |  for all all servers but yeah. And this was found again in OpenAI so yeah they they acted
1543.19s - 1547.29s |  really fast and they fixed the issue which I'm not sure really sure how they fixed because
1547.29s - 1555.77s |  it's still there. Eh and so what if we don't have a payload so let's say we don't have
1555.77s - 1561.37s |  anything that we can use to poison a malicious response we only we are only able to change
1561.37s - 1567.55s |  the keys well we still can do something we can start poisoning all the paths with a static
1567.55s - 1573.75s |  file. So let's say we use styles.css this will poison every single path and so when a user
1573.79s - 1579.23s |  sends a request to home my account or any other path he will just get CSS and therefore a
1579.23s - 1586.66s |  denial of service. Which many companies think that this is not a vulnerability. So sorry so
1588.60s - 1594.06s |  we already saw how we can poison anything we want and how we can change the keys so it's just
1594.06s - 1600.68s |  combining those ideas to create a very useful payload. And the first thing we need to do to
1600.68s - 1605.54s |  store everything anything we want in anywhere is to identify a malicious response that we
1605.54s - 1610.98s |  want to use. And this malicious response the the what's interesting about this attack is
1610.98s - 1616.04s |  that we are going to combine both so the response doesn't need to be eh cacheable. So we
1616.04s - 1620.22s |  can use anything we want in this case we are going to use an open redirect which is not
1620.22s - 1624.52s |  stored which is not cacheable and therefore it's it's not a vulnerability you can't report
1624.52s - 1630.20s |  this to anyone because it's really not a vulnerability. Then we need to choose what we are
1630.20s - 1635.80s |  going to poison and what I usually recommend is to poison some script that is being
1635.80s - 1640.44s |  included in uh in any web page usually the home page so in this case we have a home page
1640.44s - 1645.94s |  that is eh loading a main dot JS script. So we are going to poison that with a open
1645.94s - 1651.82s |  redirect. Why? Well because first as I said we are going to use this technique in this in this
1651.82s - 1657.16s |  example this demo I'm going to show it using Azure and Tomcat I finished coding this this
1657.16s - 1663.16s |  morning so it could work it could not work but eh the thing is if we send something that
1663.16s - 1667.80s |  starts with in this case logout which is the open redirect then the semicolon which is the
1667.80s - 1672.44s |  delimiter this is going to be interpreted as logout using the x forwarded host header which
1672.44s - 1677.44s |  is going to sh- to to show to send the the victim to the evil dot com. And then this is
1679.90s - 1684.92s |  going to be normalized so we are going to poison main dot JS and this is because it has a
1684.94s - 1690.12s |  static extension so even if the operator wasn't storable by default well we have the static
1690.12s - 1695.82s |  extension so we are able to poison main dot JS using this attack. Now when a user loads the
1695.82s - 1702.42s |  home page he will again load the main dot JS script and when this request is eh is received
1702.42s - 1708.80s |  by the by Azure he will just send the open redirect which will redirect the victim to the
1708.82s - 1715.56s |  evil dot com server will load the malicious JavaScript and we will be able to poison any any
1715.56s - 1721.54s |  victim with a malicious script that we completely control. And if we have the content security
1721.54s - 1726.84s |  policy enabled to provide to avoid this kind of attacks then we can use something like CSX
1726.84s - 1731.84s |  exfiltration to obtain anything we want you can ask Gareth about that he knows a lot. And so
1732.56s - 1742.49s |  yeah this is the demo. Eh so yeah so this is the web page. This is using Azure right now and
1749.17s - 1754.17s |  maybe this is yeah this is loading perfect. Internet. Eh so when we send a request to the
1756.51s - 1761.51s |  home page which is caches cache all Azure blah blah. Don't hack it right now please. Eh yeah
1762.23s - 1767.23s |  so we see that the main dot JS is being imported so that's what we want to attack. This is
1772.95s - 1780.32s |  main dot JS. Main dot JS is being obtained from the cache right now so it's going to last
1783.86s - 1787.80s |  twenty seconds. I said twenty seconds probably is going to in a real application is going to
1787.80s - 1792.90s |  be a day or maybe some days so you will need to wait for that to stop doing you can just use
1792.92s - 1799.72s |  uh an intruder session. Now this is the open redirect that we are going to use. This is the
1799.72s - 1808.84s |  evil dot com. And as you can see this is being this is an open redirect that is not being
1808.84s - 1813.84s |  cacheable. Config no cache means no cacheable. This is the evil dot com server that is going
1816.56s - 1823.56s |  to load our arbitrary JavaScript. And this is the exploit. Now I'm going to try to code it
1823.56s - 1829.14s |  by myself. So again the exploit is going to work like this. First we put the thing that we
1829.14s - 1834.14s |  want to to log so this is going to be used in the backend. Log out. Log out is the open
1836.18s - 1841.18s |  redirect. Then the semicolon to um to use as a delimiter. Now because the way that Java
1844.38s - 1850.26s |  use semicolons is eh is a semicolons only work for a segment so we cannot put a slash after
1850.26s - 1854.56s |  the semicolon because it's not going to be it's not going to work. So we need to send the
1854.56s - 1861.64s |  slashes and everything encoded in this case. So this is a slash encoded. This is a dot
1861.64s - 1871.08s |  encoded. Yeah the dot en- the dot segment encoded. And then we will put what we want to
1871.08s - 1879.60s |  poison which is main dot js. And it seems we are lucky enough. Yeah. So first as you if you
1880.60s - 1885.60s |  observe the first time we got a miss so nothing there. Now we got a hit so it seems it worked.
1890.24s - 1896.22s |  So if we load if we try to get the main dot js now instead of getting the js that we had we
1896.22s - 1901.22s |  have an open redirect. So now if we load let's try let's try it again. The problem is the
1907.25s - 1913.95s |  time because yeah so I need to wait the twenty seconds for this to stop being cached.
1919.03s - 1931.76s |  Unfortunately. See cached. So as you see I'm getting the main dot js that's because it's
1931.76s - 1937.66s |  being resolved the key and I'm getting the restore resource right now. But right now it
1937.66s - 1942.66s |  worked. So okay. So now if I go I do this. This is going to do everything that we saw. And
1949.64s - 1965.36s |  it's great. I'm running out of battery. Okay so I have time for this secret demo. Okay so we
1980.34s - 1984.98s |  already saw how we can use different discrepancies between URL parsers to exploit web
1984.98s - 1989.98s |  caches. But there are other kind of discrepancies where you can use and there are many other
1989.98s - 1995.02s |  discrepancies that I didn't research on but probably you can look for them. One of them is
1995.04s - 2000.04s |  the HTTP 0.9 smuggling. And basically I trust that most of you already know what HTTP is. The
2002.94s - 2008.82s |  basic protocol that we use everyday to communicate in the internet. But and probably you
2008.82s - 2013.82s |  already know the really new HTTP 3, the still new HTTP 2, the classic HTTP 1.1 and even
2017.00s - 2022.00s |  HTTP 1.0. But maybe you don't know that there's HTTP 0.9. 0.9 is basically a legacy
2022.44s - 2027.44s |  protocol that shouldn't be used by anyone unless you are NGINX, Apache, IIS or any other
2031.06s - 2037.42s |  HTTP server in the world. And the uh the thing with HTTP 0.9 is that we don't have any
2037.42s - 2042.56s |  headers. So no header for requests, no header for responses. And the con- the connections
2042.56s - 2049.00s |  are not persisted. Their add responses of course shouldn't be cacheable. So this is a
2049.00s - 2055.24s |  basically uh a basic 1.1 request, a HTTP 1.1 request. We send a request to home, we get a
2055.24s - 2060.26s |  response, nothing strange. Now if we remove the HTTP version and we just send get uh the
2062.82s - 2069.82s |  and the URL, the entire URL, then we will receive uh 0.9 response if we are targeting NGINX
2069.82s - 2075.40s |  as I said, Apache, IIS, Tomcat, any HTTP server basically. You will just get the response
2075.40s - 2081.98s |  which is the body. If you send something like this, then still HTTP 0.9. If you send
2081.98s - 2086.98s |  something like this, 0.9. And if you send this, 0.9. So why is this interesting at all? Well
2090.08s - 2095.42s |  browsers are a little bit smarter than others and they knew that it doesn't make sense to
2095.42s - 2101.86s |  keep talking this uh this old protocol so you cannot send a 0.9 request through browsers. So
2101.86s - 2107.40s |  most of the attacks are discarded just because of that. But let's see what happen if the
2107.40s - 2111.94s |  front end and the back end are not doing the same thing or they don't talk the same
2111.94s - 2116.94s |  protocol. So if we send a request like a HTTP 0.9 request and we send it through uh CDN,
2120.88s - 2125.88s |  most CDNs will take this request and they will upgrade it to HTTP 1.1. So then the back
2126.24s - 2131.24s |  end will just generate a 1.1 response. The body will be extracted and sorry, the response
2135.78s - 2142.40s |  will be actually downgraded and we will just be left with the body. And this works basically
2142.40s - 2149.30s |  even if no one should be using 0.9, this works because if the upgrade and downgrading is
2149.30s - 2155.40s |  done correctly then this works. And also connections are not persistent so a lot of
2155.42s - 2161.84s |  attacks are also discarded because of that. But the interesting thing is that some CDNs such
2161.84s - 2168.14s |  as Imperva, they don't verify the HTTP version before forwarding a request. So if we send a
2168.14s - 2173.14s |  request, a 0.9 request, Imperva will say well I will just forward it, I don't care. And the
2175.38s - 2180.06s |  problem is Imperva is not just going to forward it, it's going to think that this is a 1.1
2180.12s - 2185.12s |  request because Imperva doesn't know about 0.9. And the back end will send a body because it's
2188.02s - 2194.70s |  0.9 and so no headers and now Imperva is going to try to use this as a response and probably
2194.70s - 2199.50s |  it's going to fail. And so you are not going to see a lot of valid responses because of the
2199.50s - 2204.18s |  parsing. Because when Imperva tries to parse a body of a response that doesn't contain
2204.20s - 2210.78s |  valid HTTP headers then this is going to be interpreted as an error. But what if the body does
2210.78s - 2215.78s |  have these valid HTTP response headers? So let's say we have uh upload download uh file
2218.72s - 2225.02s |  feature or let's say we can control the body of a plain text or let's say we are even able to
2225.02s - 2230.86s |  modify the response using byte range or something like that. Then if by any means we are able
2230.88s - 2235.66s |  to control the response, then the body of the response no matter what the content type, no
2235.66s - 2245.25s |  matter anything, then we will be able to use this attack. Again this demo, I hope it works.
2245.25s - 2253.42s |  Okay, yeah. So in this super page, web page I created an upload file feature, okay. And in
2258.50s - 2263.52s |  this upload file feature we can upload any file we want, in this case an evil file that if you
2264.38s - 2269.38s |  want to see it, well it's really really small, but basically it's a response containing a
2272.48s - 2278.38s |  script and containing a cache control header, probably you can't see it, but yeah it
2278.38s - 2284.06s |  contains a cache control header saying that it should be stored for ten seconds I think.
2284.06s - 2289.06s |  Let's put one hundred seconds. And so when I upload this, I will also be able to retrieve it.
2289.08s - 2296.42s |  So, it seems like it was uploaded. So let's go back to burp. Okay, so this is the file that I
2305.68s - 2310.02s |  uploaded. Okay, let's say we have this upload download feature which is pretty common in
2310.02s - 2318.21s |  many applications. And this works, yeah. As you can see, I uploaded this file which in the
2318.21s - 2324.75s |  body will have a valid HTTP response. Containing headers, containing everything we want.
2324.77s - 2329.77s |  Now, if I send an, let's say I send a, an HTTP 0.9 request, which is basically this request
2333.73s - 2342.78s |  that doesn't contain any header here. As you can see, Imperva sends, says that it's an error
2342.78s - 2347.28s |  because it was trying to load the home page as a 0.9, it was trying to parse the response
2347.28s - 2352.28s |  body of the home page as 0.9, failed. Okay. Now, what if we send the same but for this
2352.36s - 2357.36s |  response? And as you can see, Imperva thinks that this is a valid response. It's actually
2366.64s - 2371.62s |  using the cache control header and it's contain, it's putting everything, it's also including
2371.62s - 2376.62s |  his own header so it seems like it's working. So, yeah, we can poison this file. Great. So
2379.22s - 2382.96s |  someone that is accessing this file is going to receive this response and is going to be
2383.00s - 2388.10s |  attacked. Now, not really cool. So we can use something that I just explained which is
2388.10s - 2393.74s |  modifying the old cache key. Again, we can use the request to the thing that we want to
2393.74s - 2399.22s |  store. Then the hash, in this case we are using Imperva. Imperva doesn't think that the hash
2399.22s - 2404.22s |  is a, is a delimiter. Ah, this, sorry, this time we are using Imperva and Nginx. And the,
2406.72s - 2412.56s |  then we have a path traversal. I added a few, eh, slashes because it's the way to avoid
2412.56s - 2417.56s |  being detected as an attack by, eh, many CDNs. And so, I have this path traversal that
2420.30s - 2425.30s |  removes the files, removes the abl file.txt and so basically is the path is going to be
2425.30s - 2431.64s |  converted to the home page which is index. And so we are going to poison the index. So if
2431.64s - 2438.08s |  we send this, I will have a little bit of luck. Yeah, it seems it's working because the
2438.08s - 2448.78s |  second time was obtained from the cache. So now, if I go back to the home page, it's not
2448.78s - 2453.78s |  working. Ah, sorry, I was using this. Ah, not good. This was, eh, this was the Azure one. This
2456.32s - 2463.21s |  is the Imperva one, fortunately. So if we go to Imperva, arbitrary cache. Okay, so how we
2480.24s - 2486.18s |  can defend against these kinds of attacks? Well, first we need to use the, eh, cache
2486.20s - 2491.68s |  control heater for web cache deception. So whenever we have a response that we don't want
2491.68s - 2495.92s |  to, anyone to store it, then just use private and no store for the web, for the cache
2495.92s - 2501.18s |  control heater. But also, eh, I wasn't able to show it in the example but there are many
2501.18s - 2506.26s |  CDNs that will override this cache control heater if there's a rule for the path. So
2506.26s - 2511.84s |  sometimes the rules for the request have priority and therefore it's important to check if
2511.86s - 2516.50s |  this is happening. If this is happening, try to modify this. And if it's not possible, then
2516.50s - 2523.40s |  try to consider using another CDN. And then, eh, for cache poisoning, well, we can try to
2523.40s - 2528.28s |  abort, to disable the key normalization. In some cases it's possible, in some cases not. So
2528.28s - 2533.12s |  in those cases, I suggest using the tool that I will provide that will let you know if
2533.12s - 2537.86s |  there's any discrepancy between normalization and, eh, delimiters and anything. So this
2537.88s - 2542.62s |  will allow you to know if you need to change either your HTTP server or your front end
2542.62s - 2549.62s |  server. These are all the references, eh, for the research. I, I use all this information.
2549.62s - 2556.20s |  All the other attacks that inspired me. And also in the slide you have the, eh, the links
2556.20s - 2561.00s |  for the cache poisoning and cache deception academy topics. We just released the academy
2561.00s - 2567.34s |  topic for deception. And also the extension for cache killer which is the, the extension I
2567.36s - 2573.96s |  created for, to detect this. It's still in progress. Eh, it should work but I will, I will
2573.96s - 2579.06s |  keep working on, on this, on this tool for the next probably month or two. And the
2579.06s - 2584.70s |  takeaways. Basically I showed that we can use any kind of discrepancy we, we find in the, in
2584.70s - 2589.78s |  the wild to exploit, eh, different servers using web cache. So web cache poisoning and
2589.78s - 2594.48s |  deception, eh, can be used in many, many cases wherever we find these kind of
2594.48s - 2601.02s |  discrepancies. Then we can use the exploitation things in many bounties because I found
2601.02s - 2605.76s |  that these kind of delimiters are being used in many systems. So I invite you to look for
2605.76s - 2610.56s |  them in different bounties and collect them and send me a message to just add you in the list
2610.56s - 2616.90s |  of people that are using this. And finally, eh, I found, I showed you a way of changing
2616.90s - 2621.28s |  these vulnerabilities but you probably can think of many other ways to combine it with
2621.28s - 2625.62s |  different vulnerabilities. Probably to upgrade vulnerabilities that are not really
2625.62s - 2631.06s |  useful and probably to be, to get better payments in bug bounties so I hope this helps
2631.06s - 2634.56s |  everyone. Uh, that's it. Thank you.


url: https://www.youtube.com/watch?v=zH6qbJuqhOM
title: DEF CON 32 - Mutual authentication is optional - Xavier Zhang
type: video
author: DEFCONConference
ingestion_date: 2024-11-10
transcription_model: large-v2
chunking_status: completed
content:
{
  "webpage_url": "https://www.youtube.com/watch?v=zH6qbJuqhOM",
  "title": "DEF CON 32 - Mutual authentication is optional - Xavier Zhang",
  "description": "Physical access control systems are often exploited in a number of ways. It could be weaknesses found within the credential itself, the antiquated communication protocol, the hardware itself, or the firmware it is running. But more often than not, it is a combination of factors that allow a variety of attacks from multiple dimensions. Some are extremely trivial and require little to no skill to perform, whereas some attacks require a bit more setup and knowledge of how the underlying technology works. We will go into detail on how these systems work, why verifying mutual authentication is important for physical access control systems and the exploits that can be accomplished, as well as ways to mitigate these exploits to make your facility more secure. This talk will include interactive demos involving official HID readers and hardware, proxmark3, and the flipper zero.",
  "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
  "duration": 1031,
  "channel": "DEFCONConference",
  "uploader": "DEFCONConference",
  "upload_date": "20241016"
}

0.14s - 5.14s | This text was transcribed using whisper model: large-v2

 All right. Are we ready? So today's talk is called Mutual Authentication is Optional. And
8.48s - 14.08s |  it's going to get into how we can uh get past HID I-class SE readers that are enforcing
14.08s - 19.22s |  secure credentials. We'll also go over a technical overview of all the other I-class
19.22s - 24.22s |  credentials. So who am I? So my name is uh Xavier. And I like all things physical security
25.12s - 30.12s |  and I am a RFID researcher. I'm a HID credential enthusiast. I like NFC. So yeah, that's me.
39.56s - 43.70s |  So today we're going to cover a few things in our agenda today. We're going to go over
43.70s - 49.14s |  cloning, downgrading, emulation, and we're going to get into buildings enforcing secure
49.14s - 54.14s |  credentials without one. So we're going to go over a few ways to get an exploit HID I-class
55.04s - 60.04s |  credentials to gain unauthorized access into a facility. So we're going to go over four
62.94s - 68.22s |  different attacks today in varied degrees of complexity. So the most simple one attack
68.22s - 74.62s |  we're going to go over is cloning, which is going to be the least complicated attack.
74.62s - 79.86s |  Most I-class legacy credentials use the standard mas- standard authentication key, which
79.86s - 84.86s |  has been a- a- which has been a- obtained in the Heart of Darkness paper in 2010. Now, if
87.56s - 93.40s |  you've not read this legendary paper yet, you should go, it's totally worth a read. If the
93.40s - 98.40s |  systems are not standard key, those keys can be extracted using a attack called low-class.
102.61s - 107.59s |  Overall, these credentials are fairly trivial to make a copy of. It's been, it's it's it's
107.59s - 112.99s |  been done for over a decade now, so nothing much to write home about. But we're going to have
112.99s - 119.13s |  to cover it anyways. Our next attack we're going to go over is a downgrade attack, where we're
119.13s - 123.17s |  going to take a secure credential, we're going to move the uh payload over to a insecure
123.17s - 131.37s |  credential. So for example, this would be turning an SE credential into a legacy credential.
131.37s - 137.99s |  We're also going to cover emulation today. So what if that downgrade didn't work? We're going
138.01s - 143.51s |  to do something by emulating a virtual credential. So this can be done with like ele- electronic
143.51s - 149.99s |  tools such as the Proxmark 3 or the Flipper 0. This exploits the HID tra- transitional
149.99s - 156.59s |  credentials, also known as SR. And finally, the fun part of today's talk is going to be exploiting
160.03s - 166.97s |  the bugs in reader firmware to gain access to facilities that only su- only enforce using a
166.99s - 174.93s |  secure credential. So we're going to go over some background tech today, so you guys sort of
174.93s - 179.37s |  know what we are talking about. So, you guys are all familiar with physical access control
179.37s - 186.15s |  systems, right? You probably interacted with one without thinking twice. There are a ton of
186.15s - 191.15s |  those HID Rev-E readers scattered around this convention floor today. Uh, WGAN protocol. That
199.85s - 204.81s |  is the signaling protocol between the reader and the controller. Simple signaling protocol
204.81s - 211.19s |  over two wires. Really old actually. Developed by John Wiegand in 1975. Still used today in the
211.19s - 216.73s |  majority of installations. It is basically just binary data that is being transmitted to the
216.73s - 221.93s |  reader, to the door controller. And it is up to the controller to determine access. The reader is
221.93s - 227.27s |  just there to read the credential. These are usually padded with a preamble to tell the
227.27s - 234.05s |  controller what bit length the uh, the credential is. And also has a uh, checksum to ve- verify
234.05s - 240.82s |  the data integrity. You know, access control systems, access control controllers are pretty
240.82s - 247.60s |  dumb. They can't tell what type of credential is presented for the most part. So they only read
247.60s - 251.84s |  credentials. So in the case of my class, it is going to read any credential that has the same
251.84s - 256.84s |  key. And we know that- well the cards have the same key for the most part. This just doesn't
256.84s - 263.98s |  cover like special functions such as readers in key rolling or any uh- and they- they're just
263.98s - 271.02s |  gonna- it's up to the controller to determine access. Whether to allow or deny the access. And
271.02s - 276.36s |  finally card cloning. That is just gonna be making a physical copy of a credential onto another
276.36s - 282.46s |  credential. Just a direct copy. Card emulation would be to make a digital copy of a physical
282.46s - 288.80s |  credential. With the- and emulating that with a device like a Proxmark 3 or a Flipper 0. A real
288.80s - 295.76s |  world use case for this would be the HID Transitional Credentials called SR, iClass SR.
295.76s - 302.84s |  And uh, those have a unique part where these uh, the secure part of the credential, the se-
302.84s - 307.88s |  secure identity object is actually s- is actually cryptographically bound to the card's serial
307.88s - 314.48s |  number. And since there are no uh, serial number changeable iClass cards, you're unable to make a
314.84s - 321.62s |  direct copy. Which means we're going to emulate those. Here are just an overview of all the
321.62s - 326.50s |  different uh, HID credential types. Starting with Prox. It's a bit out of scope for today's
326.50s - 333.64s |  topic but we have a reader here anyway. It's a- it's a low frequency credential. And then moving
333.64s - 338.88s |  on to the- around 2000 when we have iClass Legacy that first came out. And uh, those were
338.88s - 346.38s |  unique because those were actually encrypted with a 64-bit key. These were broken in 2010 by
346.50s - 350.56s |  Miloš Maryak during the Heart of Darkness attack. Where two readers were actually
350.56s - 354.64s |  destructively attacked in a way to dump the firmware to derive the master identification
354.64s - 363.99s |  keys. Going on to around the 2010s we have the uh, we have iClass SE. Also known as SIO
363.99s - 373.94s |  Enabled. SIO meaning secure identity object. And SEOS would still be the uh, flagship
373.94s - 382.00s |  credential from HID up to today. So we're going to go over a few possible attack vectors. So
382.00s - 388.02s |  we're going to go over card only attacks and card and reader attacks. Card only attacks are
388.02s - 392.92s |  very simple because it involves one thing. The physical credential itself and that's it. But
392.92s - 397.66s |  you can only do a limited things with um, with just the card. One attack that's possible is
397.66s - 403.86s |  just a downgrade to iClass Legacy because they would just send the same PAX payload on a
403.86s - 409.84s |  different type of media. You can think of cards as just an encrypted storage device like a USB
409.84s - 414.84s |  stick but crappier. Since you can just read the distance from quite a distance in the case of
414.84s - 422.96s |  iClass. The power efficiency, simple cryptography and the ISO 15693 protocol all contribute to
422.96s - 432.78s |  enhanced read ranges using long range readers. This is when card and reader attacks come in.
432.82s - 438.26s |  Card and reader attacks are very fun because you would think the reader on the building is there
438.26s - 445.67s |  to secure the, secure the building right? But we're going to use that exact reader that's
445.67s - 450.05s |  outside the building and use that to compromise the security of the building which we'll get
450.05s - 461.81s |  into a bit later. So how secure are each of these HID credentials? So starting with the least
461.81s - 467.69s |  secure one that is broken for over a decade now, we have iClass Legacy. The cryptography is
467.69s - 472.93s |  completely broken. Keys are leaked. Card's serial number is not relevant in the PAX payload
472.93s - 481.17s |  which means cloning is extremely trivial. And elite keys are recoverable by low class attack.
481.17s - 489.51s |  PAX payload is easily transferable by copying block six to nine to the, to another card. We
489.51s - 495.41s |  have iClass SR which is a transitional credential and the application instruction area on the
495.41s - 502.91s |  card is going to tell the reader that it has a secure identity object. This is good for
502.91s - 509.09s |  transitioning between uh buildings that have SE secure readers and legacy readers. And these
509.09s - 514.85s |  are easily downgradable to iClass Legacy or you can emulate this credential with the correct
514.85s - 523.04s |  card serial number for a complete clone. Going to iClass SE, standard key and the key
523.06s - 529.46s |  deprivation function are not leaked and the keys are not recoverable by low class. Currently
529.46s - 534.48s |  we still do not have the master authentication key for SE. These can be standard, elite keyed
534.48s - 539.48s |  or custom keyed and these can also be duplicated if it's an open format like uh HID 26 bit and
542.38s - 547.38s |  you'll be able to use a HID encoder to uh create another SE card, actually a legit copy because
548.00s - 553.00s |  they are freely encodable, it's an open format. These are also downgradable to Legacy or SR and
557.68s - 564.19s |  these can also be emulated which we'll get into and it will work on most readers. So we're
564.19s - 568.69s |  going to get pa- so how are we going to get past HID readers that are enforcing secure
568.69s - 573.99s |  credentials? The first would be a downgrade. So using official HID hardware against their
573.99s - 579.31s |  credentials to perform downgrade attacks. So so we're going to use a HID Omni key reader
579.31s - 583.85s |  which is the official HID hardware that plugs into a computer, acts like a keyboard and we're
583.85s - 588.85s |  just going to configure it to uh spit out the uh payload data along with the commands in the
590.95s - 596.49s |  Proxmark 3 to just make another copy without ever typing on the computer. We have a we have a
596.49s - 615.47s |  quick demo of this here. And one of the cards is SE and one of the cards is Legacy. We're just
615.47s - 620.75s |  going to put the Legacy card on the Proxmark and we're going to read the SE card on the uh HID
620.75s - 628.44s |  reader. Types out the command and there you go. The downgrade is done. Like this person said,
641.58s - 645.78s |  like the cheapest way isn't even to use the flipper with that add-on board to downgrade the SE
645.78s - 651.46s |  cards anymore. You can literally just buy that HID reader on Amazon for about $130, load up
651.46s - 660.15s |  that config and it's ready to downgrade any amount of standard keyed SE credentials. So now
660.15s - 665.13s |  we're going to get into I-Class SR. We have another way. So we're going to emulate an I-Class
665.13s - 670.67s |  SR with the correct card serial number and the secure identity object. And to accomplish this
670.67s - 675.13s |  we're going to actually need to do something special which is dump the SIO, the secure identity
675.13s - 683.06s |  object. So here in this example we're going to get the publicly accessible parts of the card
683.06s - 687.96s |  which is the uh the card serial number, the E-Purse and the application instruction area from
687.96s - 692.90s |  that I-Class SE card. And uh we're going to populate those values in a dump file and get them
692.90s - 697.90s |  ready to emulate. And that those bytes 0006 instructs the reader that hey there is an SIO.
700.14s - 703.74s |  We're going to use the reader on the building that's supposed to secure against uh secure the
703.74s - 712.82s |  building against uh like attackers to generate a successful challenge to dump the card. So once
712.82s - 717.00s |  you have emulated that partial card against that reader you're going to gather a trace. So that's
717.00s - 720.40s |  what the reader is uh that's what the reader is sending to the card and we're intercepting that
720.40s - 727.40s |  data. And we see here where we underline here as that there is a MAC that that we got. A
727.40s - 730.68s |  message authentication code and that is what we're going to use to generate a successful
730.68s - 737.94s |  challenge to dump that credential. So after we have got gotten that we're going to run that
737.94s - 743.24s |  standard uh dump command. And what what you got? The card just puked out all of its secure
743.24s - 748.84s |  contents that's supposed to be secure. So manual dumping of that SIO is possible on the
748.86s - 754.44s |  Proxmox 3 on the command line. But to build on that recently Eric Betts, Betsy, also
754.44s - 759.40s |  implemented NRMAC dumping on the Flipper Zero very recently with a very intuitive workflow.
759.40s - 763.68s |  And this allows reading the publicly accessible parts of the card without authentication.
763.68s - 769.96s |  Emulating that on the reader and and using that to generate a successful challenge to dump the
769.96s - 776.96s |  card. This as a side effect also allows users of the Flipper Zero who do not have a NARD
777.02s - 783.56s |  add-on board or the secure access module to dump the credential. Send it to someone who has
783.56s - 790.56s |  that reader and to use that to use that and use that to decrypt the copy and make a logical
790.56s - 796.90s |  copy off-site. So we're going to prepare our traditional transitional credential by just
796.90s - 803.08s |  transferring those over to the to another card. Like we see here those uh the the blue
803.10s - 808.10s |  blocks are transferred under the uh under the legacy payload and there we go we have the SR
810.80s - 822.62s |  credential already done. So now we're going to get into the readers that enforce secure
822.62s - 828.96s |  credentials. So now we do since we have a full dump of that SE card already we can emulate
828.96s - 833.60s |  the high security SIO only secure credential using a bug in reader firmware that was
833.62s - 838.52s |  originally discovered by NVX about a year and a half ago and also ported over to the uh
838.52s - 845.56s |  Flipper Zero by editing some code. So these actually work on signal readers revision uh on
845.56s - 852.40s |  firmware 10.0 5.6 or older and REV-E readers are still not patched at the moment. And you can
852.40s - 856.20s |  also use reader manager to inspect those signal readers without even being close to them
856.20s - 870.70s |  and you can verify what firmware they are running. So we can make we can make installations
870.74s - 877.28s |  more secure in a few ways. By using a high security key set such as standard two. Elite custom
877.28s - 883.92s |  keys for SE installations. Updating firmware. Disabling or disabling all legacy technologies
883.92s - 889.66s |  and issuing secure credentials such as SEOS, DESPIRE, mobile credentials. And now we're
889.66s - 894.94s |  going to get into some fun stuff. What not to do. Over the years I've seen a lot of terrible
894.94s - 901.64s |  installations of secure credentials being issued to clients. Such as this one here. You can
901.64s - 905.96s |  see this is a very popular vendor in Canada. They're located in B.C. and they have all the
905.96s - 910.72s |  latest signal readers on all their buildings. But um all their readers are pro- all their
910.72s - 914.22s |  credentials are programmed the same facility code as laid out in their official documentation
914.22s - 922.05s |  that's available publicly online. And what do you know? All the all the legacy technologies
922.05s - 927.05s |  are enabled on that reader too. So yeah. You can just they are easily downgradable. And that's
929.69s - 934.85s |  not even the worst part. Is that you don't even need the credential to make a copy because
934.85s - 940.13s |  since we know that the facility code is the same we can just uh if the numbers aren't rubbed
940.13s - 944.71s |  off on the back yet we can just uh take those numbers and just encode another one. And it's
944.71s - 952.08s |  gonna work. So given what we know we can literally make a copy of that fob in that picture
952.10s - 957.44s |  there. And we can make a logical copy that is gonna work on all of those readers from this
957.44s - 965.16s |  specific integrator. And that concludes my presentation for today. I want to say I have a
965.16s - 970.16s |  special thanks for the uh RFID hacking community. Uh Eric Betts also known as Betsy. Has the uh
972.56s - 977.56s |  pretty good documentation on all the uh I-class credentials and how the data is laid out. NBX
977.98s - 982.98s |  who discovered, originally discovered the authentication bypass bug which allows
984.06s - 989.06s |  working uh allows um working I-class SE emulation. As well as the people class emulation
991.00s - 996.00s |  code for the Flipper Zero. Kate or Miss Steel Dev. Genius on all things HID. And have to
999.16s - 1004.18s |  thank her for uh taking in my uh HID readers that I have pricked by bad firmware updates. And
1005.04s - 1010.04s |  Iceman. And I wouldn't be here without Iceman urging me to do this talk on uh I-class SE
1014.58s - 1021.60s |  today. So if you have any questions I'll be at the NFC table in the RF village. And thank you
1024.04s - 1025.04s |  for attending.


url: https://www.youtube.com/watch?v=oAriLYN-5HA
title: DEF CON 32 - Exploiting Cloud Provider Vulnerabilities for Initial Access - Nick Frichette
type: video
author: DEFCONConference
ingestion_date: 2024-11-10
transcription_model: large-v2
chunking_status: completed
content:
{
  "webpage_url": "https://www.youtube.com/watch?v=oAriLYN-5HA",
  "title": "DEF CON 32 -  Exploiting Cloud Provider Vulnerabilities for Initial Access - Nick Frichette",
  "description": "\"In this talk we will explore vulnerabilities in Amazon Web Services (AWS) products which allowed us to gain access to cloud environments.\n\n\nTraditionally, adversaries have abused misconfigurations and leaked credentials to gain access to AWS workloads. Things like exposed long-lived access keys and exploiting the privileges of virtual machines have allowed adversaries to breach cloud resources. However, these mistakes are on the customer side of the shared responsibility model. In this session, we will cover vulnerabilities in AWS services that have been fixed and that previously allowed us to access cloud resources.\n\n\nWe will start with an exploration of how Identity and Access Management (IAM) roles establish trust with AWS services and cover the mechanisms that prevent an adversary from assuming roles in other AWS accounts. We\u2019ll then demonstrate a vulnerability that bypassed those protections. We\u2019ll cover a real world example of a confused deputy vulnerability we found in AWS AppSync that allowed us to hijack IAM roles in other accounts.\n\n\nNext, we'll highlight potential misconfigurations involving IAM roles leveraging sts:AssumeRoleWithWebIdentity. These misconfigurations cloud permit unauthorized global access to these roles without the need for authentication, affecting services like Amazon Cognito, GitHub Actions, and more.\n\n\nFinally, we\u2019ll cover a vulnerability we found in AWS Amplify that exposed customer IAM roles associated with the service to takeover, allowing anyone the ability to gain a foothold in that victim account. We\u2019ll also discuss how security practitioners can secure their environments, even against a zero-day like one we\u2019ll demonstrate.\n\n\nJoin us to learn how attackers search for and exploit vulnerabilities in AWS services to gain access to cloud environments.\"",
  "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
  "duration": 2361,
  "channel": "DEFCONConference",
  "uploader": "DEFCONConference",
  "upload_date": "20241016"
}

0.40s - 5.90s | This text was transcribed using whisper model: large-v2

 Good afternoon DEF CON. Thank you so much for being here. Welcome to my talk on exploiting AWS
5.90s - 11.08s |  service vulnerabilities for initial access. My name's Nick Frechette. I'm a security
11.08s - 16.72s |  researcher over at Datadog where I specialize in AWS offensive security. So finding ways to
16.72s - 21.48s |  more effectively attack AWS environments and then hopefully how do we detect that behavior
21.48s - 26.22s |  or potentially prevent it entirely. As a result of that research sometimes I find
26.22s - 30.42s |  vulnerabilities in the underlying AWS services. Some of which we're going to be talking
30.42s - 35.72s |  about today. Uh in my free time I'm the creator and maintainer of Hacking the Cloud which
35.72s - 40.40s |  is an open source encyclopedia of offensive security techniques that you can use in cloud
40.40s - 44.34s |  hacking adventures. Uh if you'd like a Hacking the Cloud sticker come find me after. I have
44.34s - 51.18s |  quite a few to give away. Now breaking into AWS accounts is a topic that's near and dear
51.18s - 56.82s |  to my heart. And I'm sure it is for many of you as well. But I think if we're being honest
56.82s - 61.82s |  with ourselves the traditional methods for gaining access to AWS environments are boring.
64.32s - 69.52s |  If we were to poll the audience for real world AWS breaches, compromises and other security
69.52s - 75.96s |  incidents I guarantee 95 percent of scenarios would fall into one of three categories. They
75.98s - 81.82s |  were either caused by a leaked access key, an exposed S3 bucket or an exploited EC2
81.82s - 86.82s |  instance. These are the stereotypical ways that adversaries gain access to AWS accounts. And
89.30s - 94.90s |  to be clear these are very real risks that we do have to contend with but it's nothing you
94.90s - 100.98s |  haven't seen before and again these methods are boring. What we're going to be doing instead
100.98s - 105.64s |  is kicking it in the door to the cloud and exploiting vulnerabilities in AWS services to get
105.66s - 112.60s |  initial access. Particularly by abusing pre-existing trust. What I mean by that is a
112.60s - 118.98s |  common design pattern in AWS is to have IAM roles in your account that have a trust
118.98s - 123.98s |  relationship back to some AWS service. This enables that service to perform actions and uh do
127.12s - 131.68s |  functionality in your account. Today we're going to look at some vulnerabilities that would
131.70s - 136.80s |  allow us to hijack that trust relationship and gain access to victim accounts so that from
136.80s - 142.74s |  there we can escalate privileges, move laterally, access resources and more. In this session
142.74s - 147.92s |  we're going to talk a little bit about how trust is established between IAM roles and AWS
147.92s - 152.86s |  services. Then we're going to dive deep on two vulnerabilities as well as a more general
152.86s - 157.52s |  misconfiguration that would allow us to take advantage of this. And finally we're going to
157.54s - 163.04s |  discuss prevention options. In the absolute worst case scenario of an adversary having a
163.04s - 168.04s |  cross-tenant zero day in AWS, what options do we have to try and mitigate some of these
168.04s - 173.18s |  attacks? Now it's worth noting that the majority of the vulnerabilities we're going to be
173.18s - 178.48s |  talking about today have been remediated by AWS. So think of this more from the perspective
178.48s - 182.96s |  of what can we learn from these vulnerabilities that we can apply to better attack or defend
183.00s - 188.80s |  cloud environments. To get us started on the topic of abusing pre-existing trust, we should
188.80s - 193.82s |  probably talk about how that trust is established. In AWS, every IAM role is going to have a
196.82s - 201.82s |  role trust policy. And this policy determines who or what is permitted to assume a role.
204.96s - 211.56s |  Trust policies can trust a variety of identities. Users, roles, AWS organizations, uh identity
211.56s - 217.40s |  providers and many others. But where I think it's most interesting is when they trust an AWS
217.40s - 223.70s |  service itself. What you see here on the slide is the default trust policy that is associated
223.70s - 228.94s |  with roles for the Lambda service. If you were to create a Lambda function today, the role
228.94s - 233.66s |  that would be created for you would have this very same trust policy. And what it says is
233.66s - 239.32s |  that we're going to allow the Lambda service to assume role with web or sorry to assume
239.38s - 246.26s |  role. Now the way this works in practice is every time your Lambda function runs, the Lambda
246.26s - 252.46s |  service assumes that role in your account and does whatever it is that you want it to do. This
252.46s - 257.90s |  behavior introduces some really interesting questions. Since I already said that this is the
257.90s - 264.90s |  default policy, theoretically the majority of roles using Lambda would have it. And since the
264.90s - 270.56s |  trust policy just says that it trusts Lambda, could I do something a little bit sneaky? Could I
270.56s - 276.10s |  create a Lambda function in my account but point to or reference a role in a different AWS
276.10s - 282.14s |  account? Preferably one that I don't have legitimate access to. Uh could we cause that Lambda
282.14s - 287.78s |  service to assume that role in the victim account and carry out whatever operations would be
287.78s - 292.92s |  beneficial to us as adversaries? Well, the good news or bad news depending on your
292.94s - 298.14s |  perspective is there is something that stops this. And that's called pass role. Anytime you are
298.14s - 304.68s |  providing a role to an AWS service for it to assume it, you're interacting with pass role. One
304.68s - 311.16s |  of the core rules of which is that you can only pass a role within your same AWS account. If
311.16s - 315.76s |  you try to pass a role outside of it, you get the error message, access denied, cross account
315.76s - 321.80s |  pass role is not allowed. Now this is a very important security boundary in AWS because
321.82s - 327.22s |  without it, theoretically we could just arbitrarily assume role into pretty much any account and
327.22s - 331.66s |  access any of the resources associated with it. So this is going to be the goal for the first
331.66s - 336.56s |  part of our session today. Finding a vulnerability in the pass role implementation to allow us
336.56s - 343.00s |  to weaponize a service against roles that trust it and access resources in victim accounts.
343.00s - 348.50s |  So the first vulnerability is a confused deputy vuln in AWS AppSync. If you're not familiar
348.54s - 354.32s |  with AppSync, AppSync is a managed GraphQL service offered by AWS. And it allows you to
354.32s - 360.62s |  quickly and easily build GraphQL APIs. The way that it works is pretty similar to what I
360.62s - 365.76s |  described with Lambda. Where you first create an AppSync API in your account, configure it, set
365.76s - 370.60s |  your schema, uh and so on. And then as a part of setting your data source, a role will be
370.60s - 376.94s |  created for you that by default has a trust policy back to the AppSync service. Then when
376.94s - 382.42s |  your AppSync API is run, the AppSync service assumes that role and does whatever that API is
382.42s - 387.48s |  supposed to do. And again, if you try to pass a role outside of your account, cross account
387.48s - 393.22s |  pass role is not allowed. Now, I recognize this is DefCon and I'm sure many of you are already
393.22s - 397.90s |  trying to come up with ways that we can bypass this control. Things like unique encoding
397.90s - 403.54s |  schemes, maybe something like HTTP request smuggling or maybe even something more exotic.
403.54s - 408.42s |  These are all great ideas and I strongly encourage you to pursue them. But we actually found a
408.42s - 413.42s |  much simpler approach to get around this. Through the power of Spongebob memes. See, under
415.92s - 421.52s |  normal circumstances, the AWS API is typically case sensitive. And what I mean by that is, if
421.52s - 428.20s |  you pass a parameter to the AWS API, it expects it in a very particular casing format. If you
428.20s - 434.00s |  don't meet that expectation, you'll get an error. So to demonstrate this with a completely
434.00s - 440.18s |  unrelated API, here we're performing secrets manager create secret. And that API expects a
440.18s - 446.54s |  name parameter with a capital N. So long as you pass it in that format, 200 OK, everything
446.54s - 453.76s |  worked, you created a secret. However, if we were to deviate from that format in any way, in
453.78s - 459.26s |  this case we've put it to all uppercase, we try to pass that to the API, it complains. 400 bad
459.26s - 465.30s |  requests, member must not be null. The reason for this is, well, we didn't pass the parameter
465.30s - 470.64s |  it was looking for. It might be the right alphabetical characters, but it's not the specific
470.64s - 476.64s |  one and so it rightfully so throws an error. While this may sound like a fairly obvious thing
476.64s - 481.44s |  to point out, the reason it matters is that by comparison, the AppSync API was a bit of an
481.46s - 488.14s |  oddity. It was not case sensitive. We could pass arbitrary parameters of mixed casing and
488.14s - 493.28s |  the API would take it totally fine. Presumably somewhere on the back end it was normalizing
493.28s - 498.28s |  these values in some way. Now, the parameter you would use to pass a role ARN to the AppSync
501.18s - 507.28s |  service was called service role ARN. Capital R, capital A. If you try to pass a role outside
507.30s - 513.40s |  of your account, once again, cross account pass role is not allowed. But, if we change the
513.40s - 519.04s |  casing on that parameter in any way, even on just a single character, in this case we made it
519.04s - 524.72s |  all lowercase, and we try to pass a role outside of our account, for some unknown reason,
524.72s - 528.92s |  whatever filtering or allow listing or checking that was supposed to occur, either didn't
528.92s - 534.06s |  happen at all or completely failed. And so by changing the casing on a parameter, we were
534.08s - 540.84s |  able to now arbitrarily specify roles for the AppSync service to try and assume. And so we
540.84s - 547.72s |  were able to break a major security boundary in AWS with Spongebob memes. To put this in
547.72s - 553.82s |  another, in another context, what this allowed us to do was to target any role that
553.82s - 560.20s |  legitimately used the AppSync service in any AWS account. We could now provide that role to
560.20s - 566.24s |  the service and arbitrarily assume it and access whatever resources were associated with it.
566.24s - 571.68s |  This is an example of a cross service confused epi attack, where a lower privilege entity, in
571.68s - 576.88s |  this case us, was able to trick or coerce a higher privilege entity, in this case the
576.88s - 582.46s |  AppSync service, to do something on our behalf. In this scenario, that was assuming a role in
582.46s - 587.64s |  a victim account. It's really important to stress, the thing that makes this possible, the
587.64s - 593.68s |  thing that enables this attack, is that pre-existing trust relationship between those IEM
593.68s - 599.22s |  roles and the AppSync service. Because the roles trust AppSync and because we found a way to
599.22s - 604.12s |  weaponize that service against them, this is the path that allows us to assume those roles in
604.12s - 610.36s |  victim accounts. Now if you've ever done cross tenant attacks in AWS, you know that assuming a
610.36s - 615.94s |  role is only half the battle. From here, we have to actually do something useful. If we were
615.94s - 620.98s |  in a scenario where, say we had a vulnerability similar to this, but we were limited to
620.98s - 627.22s |  something like S3 list buckets as the only API call we could make, well yeah it's not great, um
627.22s - 631.52s |  I wouldn't be very happy of it if I was a defender, but at the end of the day there's very
631.52s - 636.26s |  little risk that could be demonstrated there. The good news for us is that in this scenario,
636.26s - 642.14s |  the AppSync service is more than happy to provide. In particular, AppSync allows you to run
642.14s - 647.14s |  arbitrary AWS API calls. So you now have the full spectrum of API calls that you can invoke at
650.54s - 656.82s |  your choosing. Now for a step by step practical example of how we'd exploit this, as an
656.82s - 662.32s |  attacker we'd create our own AppSync API. Configure it, set the schema, create our data
662.32s - 667.36s |  sources and so on. Then we would perform that pass role bypass to specify what role in the
667.36s - 672.86s |  victim account we want to assume. And then under normal circumstances, the AppSync service
672.86s - 677.46s |  would assume the role in our account. But because of this vulnerability, it now assumes the
677.46s - 683.40s |  role in the victim account and carries out whatever API operation we asked it to perform. The
683.40s - 688.04s |  neat thing about this, from a attacker's perspective, is if you've ever done pen testing or
688.04s - 692.42s |  red teaming in AWS, you know that one of the challenges you have is that you're trying to find
692.42s - 696.18s |  credentials with access to things that you care about. And if you can't find them
696.20s - 700.88s |  immediately, how can you move laterally, escalate privileges, find other credentials and
700.88s - 706.44s |  things like that? Becomes a bit of a hassle. By comparison, with confused deputy attacks, you
706.44s - 712.12s |  can just land as the identity of a production application. If there's a production AppSync
712.12s - 718.12s |  API, it has privileges to do whatever that service needs to do. Whether that is upload S3,
718.12s - 724.70s |  write to DynamoDB, read from RDS or anything else. Because we land as that role, we have
724.72s - 729.46s |  access to whatever privileges are associated with it and these tend to be fairly beneficial.
729.46s - 735.10s |  So no need for weird bloodhound-esque lateral movement paths, you just pass that role and
735.10s - 739.64s |  yow you're in. That's how we can do things like this in this screenshot where we dump the
739.64s - 745.64s |  contents of a DynamoDB table, cross tenant in AWS, by asking it to perform the DynamoDB
745.64s - 750.88s |  scan operation. If you're interested in this vulnerability and you'd like to learn more,
750.90s - 756.74s |  AWS did release a security bulletin for it and in addition we have a blog post with a ton of
756.74s - 761.48s |  additional information on conducting confused deputy attacks in AWS and some things to look
761.48s - 766.48s |  out for. Um the respective QR codes with the arrows will take you to each of those. Now
768.48s - 772.96s |  before we get to our next vulnerability, it's important that we learn a few things about a
772.96s - 779.06s |  assume role with web identity. Up until this point, we've been talking about plain assume
779.14s - 785.92s |  role, which is typically used for AWS to AWS role assumption. As an example, if you have an
785.92s - 790.98s |  AWS account and you would like me to assume a role in that account to do something, we'd
790.98s - 796.86s |  probably end up using assume role. Now while this is great, it does have sort of a limitation
796.86s - 803.66s |  and that it requires the entity doing the role assumption to originate from AWS. And
803.66s - 807.70s |  unfortunately there are a variety of use cases in which you might want to have an identity
807.74s - 813.44s |  from outside Amazon have the ability to assume role in your account. What do you do? Well
813.44s - 818.08s |  that's where assume role with web identity comes in. The core idea here is that you
818.08s - 823.72s |  essentially offload the responsibility of authorizing that role assumption onto an identity
823.72s - 828.80s |  provider of your choosing. This can be a private identity provider, one that you or your
828.80s - 833.70s |  company own and maintain, or it can be a public identity provider. Think things like
833.70s - 840.34s |  Facebook, Amazon, Google, GitHub and many others. The sort of core idea here is that your
840.34s - 845.02s |  external users, whether they be human or machine, interact with the identity provider to
845.02s - 850.36s |  generate a JWT. They pass that into the assume role with web identity call and this enables
850.36s - 855.70s |  them to assume a role in your account. Now before we go any further, I really want to stress
855.70s - 861.10s |  that assume role with web identity is a general design pattern. A variety of technologies
861.10s - 867.38s |  and services, both native to AWS and outside of it, will use this general design pattern. In
867.38s - 872.12s |  the next couple slides we're going to use GitHub Actions as an example, just to sort of
872.12s - 875.78s |  explain how this works and talk a little bit more about the underpinnings, but I really want
875.78s - 884.80s |  to stress, this is not limited to just GitHub Actions. To one common problem with CIC pipelines,
884.80s - 888.50s |  particularly if they're outside of your cloud provider, if you're using something like GitHub
888.50s - 894.14s |  Actions, well, how do you authenticate to an S3 bucket if you wanted to upload, say, some
894.14s - 898.48s |  sort of build artifact or logs or anything else? Well, we could use assume role with web
898.48s - 904.68s |  identity. To set this up, we initially create a trust relationship between our AWS account
904.68s - 909.26s |  and an identity provider. Since we're already using GitHub Actions, we might as well use
909.26s - 915.96s |  GitHub's provided identity provider. Once that trust is established, we then create an IEM
915.96s - 921.92s |  role. And like I said before, all roles have a role trust policy, and in that policy we
921.92s - 928.10s |  specify the identity provider as the entity that's allowed to assume the role. Then every
928.10s - 934.40s |  time our Git GitHub Action job runs, it will retrieve a JWT that has been signed by that
934.40s - 939.74s |  identity provider, that JWT is passed to an assume role with web identity call, and thus
939.74s - 943.98s |  that job at that pipeline is able to assume the role in our account and do whatever it
944.00s - 950.28s |  wants to do, whether that is upload something to S3 or call any other API calls. Now,
950.28s - 953.64s |  throughout that explanation, you might have noticed something a little bit scary and
953.64s - 958.28s |  thought, perhaps I misspoke or maybe I said something wrong. And I assure you, it's just as
958.28s - 964.66s |  scary as it sounds. What you see here is the most basic, bare bones, simple trust policy
964.66s - 970.30s |  you could possibly write when using assume role with web identity. And what it says is that
970.32s - 975.12s |  we're going to allow the GitHub identity provider the ability to assume role with web
975.12s - 979.80s |  identity. And that's a really scary thought, because if we think about it, GitHub's
979.80s - 985.96s |  identity provider is public. It's global. Anybody has access to it. There's nothing
985.96s - 989.88s |  stopping someone from creating their own GitHub account, creating their own GitHub
989.88s - 996.48s |  Action to generate a JWT signed by that very same provider and passing that into the
996.50s - 1001.90s |  assume role with web identity call, allowing them to access those roles in your account. If
1001.90s - 1007.34s |  you had this trust policy on a role, anybody with access to a GitHub account could have
1007.34s - 1013.24s |  just hijacked it and accessed anything associated with it. That's why anytime you use assume
1013.24s - 1019.48s |  role with web identity with a identity provider that is public or global, you need to also
1019.48s - 1025.92s |  have a condition which restricts specifically who is allowed to assume that role. Again,
1025.96s - 1030.50s |  assume role with web identity is a general design pattern. The specific condition you'll need
1030.50s - 1035.70s |  will differ depending on the service and what specific scenario you're in. Um, to continue
1035.70s - 1041.54s |  using GitHub Actions as an example, uh, you need to specify what GitHub repo or what GitHub
1041.54s - 1047.54s |  organization is allowed to assume that role. Uh, if you're a heavy AWS user, strongly
1047.54s - 1051.62s |  encourage you to audit your roles that use assume role with web identity for this
1051.72s - 1056.56s |  misconfiguration, because again, if you make this mistake, theoretically anybody could just
1056.56s - 1063.20s |  assume that role in your account, which is not optimal. Um, now the good news to all this is
1063.20s - 1067.50s |  this attack that I just described was very much in vogue last year. Number of security
1067.50s - 1071.48s |  researchers, including my own colleague Christoph, identified that this was a potential
1071.48s - 1076.08s |  issue and raised awareness about it. If you'd like to learn a little bit more about the GitHub
1076.08s - 1080.08s |  specific situation, uh, we have a blog post with a ton of details and that QR code will take
1080.08s - 1085.82s |  you there. Uh, now, to AWS's credit, they actually did something about this. In
1085.82s - 1092.12s |  particular, they made what was once an optional condition now mandatory. You can no longer
1092.12s - 1097.06s |  create a role trust policy for the GitHub identity provider that doesn't include a
1097.06s - 1103.44s |  condition. And so, that's good. Kudos to AWS, that's a great change. But, if we were to
1103.44s - 1107.90s |  nitpick it, remember, I said assume role with web identity is used all over the place in a
1107.92s - 1113.16s |  variety of contexts and they only fixed it specifically to GitHub actions. Any of the other
1113.16s - 1119.00s |  technologies or services that have this very same potential misconfiguration were slash are
1119.00s - 1124.34s |  still vulnerable. So again, definitely encourage you to audit your roles. There is one such
1124.34s - 1129.18s |  service that I'd like to dive a little bit more deeply on, uh, and that is Cognito. Uh,
1129.18s - 1135.52s |  Cognito is a sign in as a service offering from AWS. If you want to store user credentials
1135.54s - 1140.14s |  but you don't want to deal with the hassle of, uh, hashing, salting, password reset flows
1140.14s - 1144.68s |  and things like that, you can pay Amazon a fee and they'll take care of that for you. One of
1144.68s - 1150.02s |  the core features of Cognito is something called an identity pool. And an identity pool is a
1150.02s - 1155.26s |  resource you can create in your account that allows you to distribute short lived IEM
1155.26s - 1162.40s |  credentials to your users. So that they can assume roles in your account. As an example, say
1162.42s - 1167.42s |  you're building a web application and you want your users to be able to upload content to S3
1167.42s - 1172.22s |  or invoke an API gateway using IEM auth, the way you could achieve this is creating a
1172.22s - 1176.56s |  Cognito identity pool, allowing your users to interact with it, to generate tokens. They
1176.56s - 1180.40s |  would then, uh, through a library in the client, pass that to the assume role with web
1180.40s - 1186.70s |  identity call and assume the role in your account. The good news about Cognito is that they
1186.70s - 1191.40s |  seemingly foresaw the very same attack that occurred with GitHub actions. They do by
1191.40s - 1198.48s |  default include a condition to defend against it. Specifically, the condition will restrict
1198.48s - 1204.56s |  which identity pool is allowed to assume a role. And this is very important because if the
1204.56s - 1209.56s |  trust policy looked like this, anyone in the world would be able to assume that role. And
1209.56s - 1212.90s |  we're gonna talk about some reasons why that trust policy might look like this in just a
1212.90s - 1219.20s |  little bit. Uh, the way that we can take advantage of it is slightly different than what we
1219.24s - 1224.24s |  described with GitHub actions. This is mostly an implementation detail specific to Cognito
1224.24s - 1229.24s |  identity pools. Specifically, what we're gonna do is use a, uh, Cognito offload. Offloads
1230.98s - 1235.98s |  are essentially like the, uh, essentially the step by step operations, uh, that a user
1238.66s - 1243.42s |  would use to generate those credentials. There are two offloads currently supported. An
1243.42s - 1249.06s |  enhanced offload and a basic offload, sometimes referred to as classic. We're gonna
1249.06s - 1254.06s |  focus on classic because that allows us to actually specify the role we want to assume
1254.06s - 1259.24s |  because the final operation in this chain is assume a role with web identity. Um, the
1259.24s - 1263.20s |  specific step by steps are as an attacker, we're gonna create our own Cognito identity
1263.20s - 1268.64s |  pool. We're gonna, going to then perform get ID and get open ID tokens to generate that
1268.64s - 1273.68s |  JWT. And once we have that JWT, we pass it in an assume role with web identity call and
1273.68s - 1278.32s |  this allows us to assume the role in the victim account. So for our purposes, for the
1278.32s - 1283.82s |  remainder of this talk, we'll refer to this as the variant one vulnerable trust policy. A
1283.82s - 1288.72s |  trust policy tied back to, uh, the Cognito service but not including a condition to
1288.72s - 1294.00s |  restrict which identity pool is allowed to assume it. The reason we call this variant one
1294.00s - 1297.46s |  is cause there's a variant two as well. You might have noticed there's actually two
1297.46s - 1303.30s |  conditions on the default trust policy associated with roles using Cognito. And this
1303.32s - 1309.12s |  condition is the AMR condition or the authentication methods references. And this value can
1309.12s - 1313.24s |  be either authenticated or unauthenticated. You might wonder, could we assume a role if
1313.24s - 1317.94s |  this condition was set? And it turns out, yeah, absolutely, we totally could. If the value
1317.94s - 1322.28s |  was set to unauthenticated, then literally everything I just said works completely the same,
1322.28s - 1327.28s |  no changes required. If the value is authenticated, then we do have to jump through some
1327.28s - 1331.52s |  extra hoops. In particular, in our attacker controlled account, we have to create a
1331.54s - 1336.74s |  Cognito user pool, configure it alongside the identity pool, and then from there, basically
1336.74s - 1342.58s |  all the same steps apply with some minor changes. All this condition is doing is checking
1342.58s - 1347.36s |  that you are authenticated to an identity pool, not that you are authenticated to a
1347.36s - 1351.76s |  specific identity pool. And since we can just create our own identity pool, we can
1351.76s - 1357.04s |  configure it however we like, so we can add authentication if we need to. Now, earlier
1357.04s - 1361.10s |  you might have noticed that I called this a misconfiguration, and that's because it is.
1361.12s - 1366.06s |  The GitHub action stuff, this Cognito stuff, any of the other myriad of services which have
1366.06s - 1371.80s |  this very same issue, all of these are misconfigurations, not a vulnerability. Had you
1371.80s - 1376.54s |  suffered a breach, compromise, or other security incident as a result of it, well,
1376.54s - 1380.34s |  unfortunately that would be your responsibility as a part of the AWS shared responsibility
1380.34s - 1385.08s |  model. And so you're probably wondering, well Nick, I was promised a second vulnerability,
1385.08s - 1390.22s |  you're calling this a misconfiguration, what gives? Well, uh, this is where things get weird,
1390.24s - 1395.68s |  and we dive into our second vulnerability of the day, the AWS Amplify service exposing
1395.68s - 1401.02s |  IEM roles to take over. Uh, see, earlier this year, when we noticed this capability in
1401.02s - 1405.86s |  Cognito, one of the first questions we had was, okay, cool, I wonder how common this
1405.86s - 1411.00s |  misconfiguration is? And because it could be used for initial access, literally just a
1411.00s - 1415.24s |  Zoom rolling into victim accounts, we wanted to know if we could use publicly available
1415.26s - 1421.26s |  data to find vulnerable roles and notify their owners so they could be fixed. To do this, we
1421.26s - 1425.94s |  used a tool called Sourcegraph, one of the free features of which is essentially like a
1425.94s - 1432.08s |  Google search, but for public GitHub repos. Uh, we, we wrote a regular expression to find
1432.08s - 1437.78s |  IEM role ARNs, threw that into Sourcegraph, got back results, deduplicated them, removed
1437.78s - 1442.22s |  like clearly placeholder ARNs, like account IDs with all one digit or incrementing digits,
1442.24s - 1447.90s |  things like that. Uh, and that left us with a little over 8,000 results. We then tried to
1447.90s - 1452.88s |  assume each role twice, once authenticated and once unauthenticated, and we let our
1452.88s - 1458.52s |  automation run, and pretty much immediately it was clear, something was terribly wrong.
1458.52s - 1463.56s |  Remember, this was across all of public GitHub, across teams, organi- organizations,
1463.56s - 1469.10s |  individuals, and companies, so why was it then that the overwhelming majority of roles
1469.10s - 1475.02s |  that we found to be vulnerable all had this weirdly specific naming convention? What you
1475.02s - 1479.82s |  see here on the slide are real, real, real roles that we found in the wild that were
1479.82s - 1484.60s |  vulnerable, that we have slightly modified the names of so as to respect the anonymity of
1484.60s - 1489.80s |  their owners. Now, what made these roles so weird was that they all ended in either auth
1489.80s - 1494.96s |  role or unauth role. And additionally, they had what appeared to be either a time stamp
1494.96s - 1501.04s |  dating back to 2018 or 2019, or they had a six digit integer. Presumably, these were the
1501.04s - 1505.98s |  newer ones. To make things even more weird, as we were going through the results, we actually
1505.98s - 1511.88s |  found three roles that belonged to AWS themselves. And so at this point, it wasn't clear if
1511.88s - 1517.02s |  this was some sort of uh, tutorial gone wrong, where somebody was demoing how to set this
1517.02s - 1522.02s |  up with OIDC, not realizing that it was vulnerable and letting anybody just assume these
1522.02s - 1528.30s |  roles in anybody's account. Uh, using source graph, we were able to see the name of the file
1528.30s - 1533.34s |  where these roles were being found. And again, the overwhelming majority were found in
1533.34s - 1540.14s |  files called team provider info dot Jason. And through this, this led us ultimately to the
1540.14s - 1546.08s |  culprit. The thing that was causing all these roles to let anybody assume them. Turns out it
1546.08s - 1552.98s |  was AWS Amplify. Normally, when we're talking about a misconfiguration, we're talking
1552.98s - 1557.26s |  about a mistake, misunderstanding, developers moving a little bit too quickly and breaking
1557.26s - 1562.66s |  things or not realizing that something is as vulnerable as it actually was. This was a rare
1562.66s - 1568.60s |  circumstance in which an AWS service itself was exposing IEM roles such that anyone could
1568.60s - 1573.64s |  assume them. The vulnerability was not that you could misconfigure roles, it's that AWS
1573.64s - 1578.44s |  was doing it to you. And fortunately not just once, they actually did it twice. If you're not
1578.44s - 1583.44s |  familiar with Amplify, Amplify is a uh, web and mobile application framework that lets you
1583.44s - 1588.92s |  build these technologies. Uh, one of the core selling points to Amplify is essentially that
1588.92s - 1595.22s |  uh, by using Amplify, it will abstract away a lot of the underlying infrastructure. So, if
1595.22s - 1599.96s |  you build a web application and you want your users to be able to create accounts, sign up,
1599.98s - 1605.12s |  change the password and so on, your developers don't even need to know what Cognito is. They
1605.12s - 1611.20s |  can simply add authentication through Amplify and Amplify goes in, creates Cognito resources,
1611.20s - 1616.96s |  configures them, sets them up and so on. Key emphasis, Amplify configures those Cognito
1616.96s - 1622.60s |  resources. Now, there's two ways you can build an Amplify app or configure them. Uh, they're
1622.60s - 1627.48s |  not mutually exclusive, you can use either or. Uh, there is the studio which is the web
1627.50s - 1632.40s |  console portal equivalent and then there is the CLI which you can run from your local machine
1632.40s - 1637.84s |  or as a part of your CI city pipeline. Now, typically when we do cloud security research, one
1637.84s - 1642.24s |  of the challenges we often have is it's not really clear why this vulnerable, vulnerable
1642.24s - 1646.68s |  behavior is occurring. We might be able to see that something has security impacts but it's
1646.68s - 1650.76s |  not clear what exactly happened or why it's doing this. For example, with the first
1650.76s - 1654.96s |  vulnerability, for some remarkably strange reason, changing the casing on a JSON
1654.98s - 1660.18s |  parameter let us break a major security boundary in AWS. So, we run into these types of
1660.18s - 1664.62s |  scenarios fairly frequently. What's neat about this Amplify vulnerability and why it's
1664.62s - 1669.56s |  personally one of my favorites is that this is actually a rare instance where we are able to
1669.56s - 1675.56s |  know not only how did this become vulnerable but when. That's because the Amplify CLI
1675.56s - 1681.40s |  written by AWS is actually open source and by digging through the commit history we can
1681.44s - 1686.88s |  see how and when these vulnerabilities were introduced. And we didn't know it at the time
1686.88s - 1692.38s |  but later on in speaking to AWS employees we came to find out that the Amplify studio, the
1692.38s - 1698.46s |  web console equivalent actually uses parts of the CLI as libraries. And that's how I can
1698.46s - 1704.20s |  tell you with confidence that both the Amplify studio and CLI became vulnerable on or around
1704.20s - 1710.64s |  July 3rd 2018 to the variant 2 vulnerable trust policy. Allowing anyone in the world the
1710.64s - 1716.32s |  ability to assume these roles. What happened here is that any time you create an Amplify
1716.32s - 1721.76s |  project 2 roles are created for your account. An auth role and an unauth role like we saw
1721.76s - 1728.76s |  previously. Unfortunately as a part of this commit the default trust policy associated with
1728.76s - 1733.74s |  those roles was set to this which would allow anybody in the world to assume them. So, if
1733.74s - 1737.70s |  you thought you were building a private internal only application that required
1737.70s - 1741.18s |  authentication and you were gatekeeping who could create accounts so that you could
1741.18s - 1746.50s |  protect your stuff, well, um, sorry, no, no you weren't. Anybody could just assume those
1746.50s - 1751.58s |  roles and access whatever was associated with them. Like I said that was introduced on July
1751.58s - 1756.58s |  3rd 2018. A little over a year later on August 8th 2019 AWS sort of fixed this issue. Uh,
1760.22s - 1765.12s |  seemingly unintentionally. What they did was they set the default effect to a deny and
1765.12s - 1769.76s |  they removed the condition entirely. This is actually the exact same behaviour we see
1769.76s - 1775.30s |  today when uh creating a new Amplify project. It's really important to stress though that
1775.30s - 1780.56s |  what this change did in August of 2019 was make it so that no future roles would become
1780.56s - 1786.24s |  misconfigured. Any that had already been created were still vulnerable and that's how we
1786.24s - 1791.24s |  were able to assume them in 2024. Later on in July of 2020 variant 1 was introduced to the
1791.60s - 1796.60s |  Amplify CLI in studio. And this one was a bit more complicated than the previous one. What
1798.98s - 1804.08s |  happened here was that, like I said previously, you can add authentication to an Amplify
1804.08s - 1809.52s |  project. If at any time you chose to remove authentication for any reason, that would
1809.52s - 1814.56s |  introduce the variant 1 policy. Uh, in speaking to developers who were affected by this, it
1814.56s - 1820.06s |  seems the most common reason that this occurred was situations in which uh, uh, projects
1820.08s - 1825.26s |  started with the built-in Amplify controlled Cognito resources and then at a later time
1825.26s - 1830.46s |  moved to Cognito outside of Amplify or to just a different identity provider entirely.
1830.46s - 1835.50s |  Regardless of the reason why, what occurred was that a cloud formation template would run
1835.50s - 1840.00s |  that would start setting the default trust policy to what you see on the slide, which is
1840.00s - 1845.78s |  totally fine. This is the secure trust policy. However, under the line, uh under the
1845.80s - 1850.38s |  condition for the deletion of the authentication configuration, there was a line to delete
1850.38s - 1855.64s |  the condition off of the trust policy, which left you with this. The variant 1 vulnerable
1855.64s - 1861.72s |  policy. Whatever identity permissions were associated with this rule, when this exposure
1861.72s - 1867.26s |  occurred, it still had access to. So if your rule could do something like uh, read from an
1867.26s - 1872.70s |  S3 bucket, write to a demo deep table, it still had that access. Additionally, at the end of
1872.72s - 1878.02s |  the day, these are just plain IEM rules. And unfortunately, developers love to over
1878.02s - 1883.36s |  privilege their identities. When we were reacting to this vulnerability and, and notifying
1883.36s - 1887.80s |  customers so they could mitigate it, we found a number of customers who had highly privileged
1887.80s - 1893.58s |  roles, including some with S3 full access, Kinesis full access, and more. Unfortunately, as a
1893.58s - 1897.58s |  part of this vulnerability, now anyone could have just accessed them and then accessed
1897.60s - 1904.00s |  whatever resources were associated with those accounts. Uh, like I said, that was introduced
1904.00s - 1908.84s |  in July of 2020, and then between then and when we eventually disclosed in 24, more and more
1908.84s - 1914.02s |  roles were becoming misconfigured. One thing I'd really like to stress, uh, that makes this
1914.02s - 1920.42s |  cognito issue even more extreme, is that cognito role RNs are easily discoverable. If we
1920.42s - 1924.22s |  compare this to the first vulnerability we talked about today, a confused deputy volunteer in
1924.24s - 1930.04s |  some AWS service, an adversary still has to contend with the challenge of finding role RNs. To
1930.04s - 1935.08s |  be clear, an RN is not a secret, not in the same way a password or API key is, but it does
1935.08s - 1940.06s |  represent some level of challenge. They would have to brute force them, enumerate them, drive
1940.06s - 1944.36s |  them from IEC, pull them out of build logs, rip them out of GitHub like we did, or a myriad of
1944.36s - 1949.36s |  other ways. Suffice to say, it's something you could do across, say, a small number of
1949.40s - 1954.74s |  organizations, but not something that you could easily scale to the size of the internet. By
1954.74s - 1959.34s |  comparison, the whole point of cognito is that these roles are supposed to be available to your
1959.34s - 1964.58s |  users. They're basically public. And in addition, they're also deterministic in the sense that
1964.58s - 1969.28s |  if you know the unauth role, you now also know the auth role by default. And so what an
1969.28s - 1974.22s |  adversary could have done is simply taken advantage of the default domain associated with
1974.22s - 1980.20s |  Amplify apps to suddenly crawl tens of thousands of apps looking for vulnerable roles. And to
1980.20s - 1984.40s |  be clear, the majority should be fine, but undoubtedly some number of them were created
1984.40s - 1989.94s |  between July, er sorry, uh 2018 and 2019, or who have since changed their authentication
1989.94s - 1994.94s |  configuration since July of 2020. Like I said before, we did disclose this to AWS in January
1997.04s - 2002.38s |  of this year, and AWS released comprehensive fixes in February and April. Uh they did the
2002.38s - 2006.32s |  same thing they did with GitHub Actions, where you can no longer create these vulnerable trust
2006.32s - 2011.16s |  policies anymore, but in addition they sort of went above and beyond. Specific to this
2011.16s - 2016.86s |  Amplify issue, they actually made changes to STS so that you can no longer assume a role
2016.86s - 2022.90s |  cross-account in AWS. So even if you had a variant one or variant two role in your account,
2022.90s - 2027.64s |  somebody from a different account can no longer assume it. If you'd like to learn more about
2027.64s - 2032.72s |  this vulnerability, AWS did release a security bulletin for it, and we have a blog post here
2032.72s - 2036.72s |  with a ton of additional content and information on carrying out these types of attacks, how we
2036.72s - 2042.16s |  found this vulnerability, and more advice on dealing with assume role with web identity.
2042.16s - 2047.60s |  Really quickly though, I did want to mention, um throughout this talk we've been focused on
2047.60s - 2052.94s |  creating resources in an attacker account so that we could then victimize a victim account.
2052.94s - 2057.94s |  The reason for doing this is simplicity. If you own the resource, you can configure it, you can
2057.94s - 2062.48s |  set it up however you like, you know it'll always be available, you can troubleshoot it, but I
2062.48s - 2068.12s |  wanted to mention it's not strictly required, it's just easier. Specific to the Amplify
2068.12s - 2073.72s |  vulnerability, they fixed cross-account role assumption using identity pools. It's no longer
2073.72s - 2079.24s |  possible for me to use one of my identity pools to assume a role in your account. But they
2079.26s - 2083.14s |  didn't do anything about the same account scenario. And to be clear, I don't really think
2083.14s - 2088.40s |  there's much they could do. Uh the reason they were able to block cross-account assume role
2088.40s - 2093.42s |  using this is, well that's, it's a very weird thing to do. It's very likely zero if any real
2095.62s - 2100.12s |  organizations were doing this in the wild. And so they said hey, we can just block this off, we
2100.12s - 2105.10s |  can fix this attack vector and we'll mitigate a lot of the risk. But you can still absolutely
2105.12s - 2110.46s |  do this in the same account and this is something that you can still take advantage of today.
2110.46s - 2115.16s |  Um to be clear, this is more difficult, things are much nicer if you can just bring your own
2115.16s - 2119.44s |  identity pool, but it's definitely something to be aware of if you are penetration duster, red
2119.44s - 2124.94s |  teamer, or you simply use Amplify a lot. The way this would work is that if there is a variant
2124.94s - 2131.48s |  one or variant two role in that account, if there is also a identity pool that is configured
2131.50s - 2137.44s |  with basic authentication or the basic offload, not necessarily associated with Amplify, just
2137.44s - 2143.48s |  any Cognito identity pool with that enabled, you as an external adversary can still interact
2143.48s - 2148.58s |  with that identity pool, generate a JWT, pass it into Sumo with WebIdentity, and still assume
2148.58s - 2153.78s |  one of those variant one or variant two vulnerable roles. Um in the bottom left hand side
2153.78s - 2157.52s |  there's a QR code that'll take you to a Hacking the Cloud article that will walk you through
2157.54s - 2164.24s |  step by step how you can do this in both authenticated and unauthenticated environments.
2164.24s - 2168.34s |  With the time we have left, I wanted to talk a little bit about what we can do to prevent
2168.34s - 2174.34s |  cross-tenant attacks in AWS. To be clear, uh your time is probably much better spent doing
2174.34s - 2178.92s |  things like deleting old access keys, uh cleaning up old IAM users, following least
2178.92s - 2183.72s |  privilege and so on, but if you're already doing all those things, or you have the technology
2183.72s - 2187.70s |  and processes in place, it might be helpful to know that we do have some controls that can
2187.70s - 2193.50s |  potentially prevent some of these attacks. In particular, AWS offers us a number of
2193.50s - 2199.70s |  condition keys that allow us to restrict essentially where an adversary is coming from. Sort
2199.70s - 2205.04s |  of similar to what we saw with a Sumo with WebIdentity previously. These conditions allow us
2205.04s - 2210.42s |  to restrict either the ARN, the account, or the organization that is permitted to assume a
2210.42s - 2215.22s |  role. It's really important to stress, these are not included by default, uh if you were to
2215.22s - 2219.06s |  try to create an instance profile today, it wouldn't automatically add one of these, so you
2219.06s - 2223.36s |  would have to manually attach these conditions to each role you want to assume, or want to
2223.36s - 2228.16s |  protect, and more importantly, make sure that those services are compatible with these
2228.16s - 2234.04s |  condition keys. From some nominal testing, EC2, Lambda, AppSync, all work totally fine, but
2234.04s - 2238.44s |  again, really encourage you to test before you try pushing this into production. To
2238.46s - 2243.64s |  demonstrate how this would work in practice, uh using source account as an example and
2243.64s - 2248.88s |  tying us back to the first uh confused deputy attack we had in AWS, uh by exploiting this
2248.88s - 2253.52s |  vulnerability, an adversary would be able to assume a role which trusted the AppSync service.
2253.52s - 2259.32s |  But if we had set a condition key on that trust policy, limiting what AWS account is
2259.32s - 2264.56s |  permitted to assume that role, it would successfully block that adversary. The way we
2264.58s - 2268.42s |  know this is back in the day when this vulnerability was present, we were able to actually
2268.42s - 2273.52s |  test setting that condition key, and it did successfully block an external adversary. So
2273.52s - 2278.66s |  even in that worst case scenario of someone having a cross-tenant zero day in AWS, by
2278.66s - 2284.60s |  setting this condition key, we can block that access. The other reason I like this is from a
2284.60s - 2290.04s |  defense in depth perspective. If pass rule says, under absolutely no circumstance should
2290.06s - 2294.60s |  someone be able to assume a role cross-tenant, well great, we broke that with SpongeBob
2294.60s - 2299.36s |  memes. So the next time that happens, or if somebody finds a more inventive way around it,
2299.36s - 2304.80s |  let's have an additional layer of protection to block these types of attacks. You're also
2304.80s - 2310.64s |  unlikely to run into many issues, if it works, it's very unlikely to cause uh problems, it's
2310.64s - 2315.88s |  this is more of a static thing that stays the same over time. So in conclusion, we took a
2315.92s - 2320.86s |  look today at how confused epi attacks can weaponize cloud services against us. We took a
2320.86s - 2325.90s |  look at a very dangerous misconfiguration when using assume role with web identity. And
2325.90s - 2330.16s |  again, I strongly encourage you to audit your roles that use it, because if you make this
2330.16s - 2334.28s |  mistake, theoretically anybody can assume roles in your account. It's an easy mistake to
2334.28s - 2339.88s |  make. AWS actually made it twice. Um and then we also took a look today at how we can use
2339.88s - 2345.38s |  uh condition keys to block confused epi attacks in AWS. If you enjoyed this talk, you can
2345.38s - 2349.98s |  find more of our research at securitylabs.daydoghq.com. If you have any questions, feel free
2349.98s - 2354.32s |  to find me after or reach out to me on social media. Again, my name's Nick Frechette. Thank
2354.32s - 2355.00s |  you all so much.


url: https://www.youtube.com/watch?v=n8w85LpRplY
title: DEF CON 32 - Measuring the Tor Network - Silvia Puglisi, Roger Dingledine
type: video
author: DEFCONConference
ingestion_date: 2024-11-10
transcription_model: large-v2
chunking_status: completed
content:
{
  "webpage_url": "https://www.youtube.com/watch?v=n8w85LpRplY",
  "title": "DEF CON 32 - Measuring the Tor Network - Silvia Puglisi, Roger Dingledine",
  "description": "Millions of people around the world use Tor every day to protect themselves from surveillance and censorship. While the Tor Browser and its protocol are widely known, the backbone of the Tor ecosystem, its extensive network of volunteer relays, is often subject to speculation and misinformation. The Tor Project is dedicated to supporting this network and fostering a vibrant, diverse community of relay operators.\n\nThis talk will focus on our efforts to maintain a healthy network and community, and detect and mitigate attacks -- all with the help of metrics and analysis of usage patterns. By illustrating how we collect safe-enough metrics for an anonymity network, we will offer insights into how we identify unusual activity and other noteworthy events on the network. We will also discuss our ongoing strategies for addressing current and future network health challenges.\n\nIf you are interested in understanding the inner workings of the Tor network and its relay community and how we keep this vital ecosystem running, this talk is for you.",
  "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
  "duration": 2495,
  "channel": "DEFCONConference",
  "uploader": "DEFCONConference",
  "upload_date": "20241016"
}

0.88s - 10.22s | This text was transcribed using whisper model: large-v2

 Okay, wow, this microphone is really loud.
10.22s - 11.06s |  Awesome.
11.06s - 13.66s |  Welcome everybody to another Tor Talk.
13.66s - 16.10s |  I'm Roger, this is Sylvia,
16.10s - 19.14s |  and we're gonna tell you about the Tor metrics,
19.14s - 21.24s |  how to understand what's going on in the network,
21.24s - 24.92s |  and in general, how to grow the community that we're going.
24.92s - 27.58s |  So part of what I'm trying to do here,
27.58s - 30.74s |  many of you have seen only me as a Tor speaker.
30.74s - 32.82s |  I'd like to show you that there are other Tor people
32.82s - 34.74s |  in the world who actually do things,
34.74s - 37.46s |  and eventually there will be more voices of Tor
37.46s - 43.23s |  than just me, and maybe eventually, not even me.
43.23s - 47.19s |  Okay, so some motivation, some things to think about.
47.19s - 51.57s |  Here is a fellow who was elected in Iran back in 2009,
51.57s - 53.23s |  but he ended up not in charge,
53.23s - 56.97s |  and many of you might remember the Green Party movement.
56.97s - 59.07s |  So long ago, I was talking to a friend
59.07s - 62.83s |  who was doing trainings of how to be safe in Iran,
62.83s - 64.23s |  how to get around censorship,
64.23s - 67.51s |  and he was teaching people open proxies, VPNs,
67.51s - 72.35s |  Tor, Syphon, other tools, and then he realized,
72.35s - 75.43s |  everybody that I've taught anything other than Tor to
75.43s - 79.35s |  is now dead or in jail, so I'm only gonna teach Tor,
79.35s - 82.11s |  and that was an eye-opening story when I heard it,
82.11s - 84.31s |  and there are two lessons there that we take here.
84.31s - 87.27s |  One of them is we really need to understand
87.27s - 88.79s |  what's going on in the Tor network,
88.79s - 91.55s |  so when a lot of users show up from certain countries,
91.55s - 93.07s |  we know that it's happening,
93.07s - 95.19s |  but also we need to do it safely.
95.19s - 98.19s |  We need to be aware of how we can measure things
98.19s - 103.37s |  and understand things without putting users at risk.
103.37s - 105.25s |  So we've got four main topics
105.25s - 106.27s |  we're gonna talk about today.
106.27s - 108.45s |  The first one is introduction to Tor
108.45s - 110.73s |  for the people who need a refresher,
110.73s - 113.83s |  and then we're gonna talk more about the relay community
113.83s - 117.37s |  and who runs the relays and how we grow that as a community,
117.37s - 119.17s |  and then we're gonna talk about
119.17s - 121.37s |  understanding what's going on with metrics
121.37s - 123.29s |  and measurements and datasets,
123.29s - 126.79s |  and at the end, we'll ask you to help us save the world.
128.65s - 130.17s |  Okay, did I?
130.17s - 131.05s |  Yes.
131.05s - 133.85s |  Okay, so Tor is a lot of different things.
133.85s - 135.61s |  Tor is a nonprofit.
135.61s - 137.37s |  It's a free software project.
137.37s - 139.09s |  It's a network of volunteers.
139.09s - 142.49s |  So raise your hand if any of the following apply to you.
143.39s - 147.37s |  Anybody here run a Tor relay or a bridge or a snowflake?
147.37s - 149.25s |  Anybody here help with research
149.25s - 152.69s |  or helping in writing research papers?
152.69s - 155.89s |  Anybody here use Tor, use Tor browser?
155.89s - 158.65s |  Anybody here find themselves explaining to their relatives
158.65s - 162.45s |  why I have nothing to hide makes no sense?
162.45s - 164.37s |  Excellent, you're all part of Tor.
164.37s - 167.73s |  Tor is an ecosystem of people trying to fix the fact
167.73s - 172.97s |  that people need privacy and not enough of them have it.
172.97s - 174.93s |  Okay, so I use the word anonymity
174.93s - 176.21s |  when I'm talking to researchers,
176.21s - 179.73s |  but really it comes down to communications metadata,
179.73s - 183.25s |  and here's creepy NSA guy that we learned a lot more about
183.25s - 184.53s |  from this note in documents,
184.53s - 187.85s |  and he actually said we kill people based on metadata.
187.85s - 190.21s |  So that's where we're starting from,
190.21s - 194.17s |  and the goal of Tor is to protect that metadata
194.17s - 198.53s |  so that it's hard for people to tell who's doing what.
198.53s - 201.77s |  So I only use the word anonymity talking to researchers.
201.77s - 204.13s |  When I'm talking to my parents and grandparents,
204.13s - 205.95s |  I work on privacy systems,
205.95s - 208.81s |  because anonymity, I don't know, that's kind of scary,
208.81s - 210.85s |  but privacy is an important value.
210.85s - 212.61s |  Everybody should have privacy.
212.61s - 216.21s |  And then when I'm talking to corporations,
216.21s - 217.89s |  Google, Walmart, and so on,
217.89s - 220.45s |  I work on communication security,
220.45s - 222.61s |  because privacy is dead, anonymity,
222.61s - 223.75s |  why would they want that?
223.75s - 225.97s |  But of course they want security.
225.97s - 227.73s |  And then when I'm talking to governments,
227.73s - 229.21s |  I work on traffic analysis
229.21s - 231.41s |  resistant communication networks.
231.41s - 233.01s |  And again, it's the same system,
233.01s - 234.49s |  the same security properties.
234.49s - 236.59s |  The goal is to get all these users together
236.59s - 238.97s |  so you have your cancer survivors
238.97s - 241.27s |  and your Egyptian activists and other groups
241.27s - 243.43s |  all inside the same network.
243.43s - 245.47s |  And then there's the fourth category
245.47s - 247.57s |  that we've been thinking a lot about recently,
247.57s - 249.67s |  which is the reachability side,
249.67s - 252.61s |  the people in Vietnam who can't get to BBC,
252.61s - 255.31s |  or people trying to reach the Tor network
255.31s - 257.35s |  even though censors are trying to block it.
257.35s - 260.39s |  So the goal here is to get all of these different groups
260.39s - 262.19s |  into the same anonymity system
262.19s - 263.67s |  so that they can blend together.
263.67s - 266.73s |  Otherwise, if you just had an anonymity system
266.73s - 268.63s |  for cancer survivors,
268.63s - 271.39s |  then people would know why you installed it,
271.39s - 274.64s |  why you're using it.
274.64s - 276.40s |  Okay, so how do you actually build one of these?
276.40s - 279.00s |  How do you build a metadata,
279.00s - 281.80s |  communications metadata privacy system?
281.80s - 284.80s |  The easy answer is you have some centralized system
284.80s - 287.60s |  like a VPN, and all the users connect to it
287.60s - 290.12s |  and request some webpage or something.
290.12s - 292.14s |  And a bunch of things can go wrong here.
292.14s - 294.20s |  The first thing that can go wrong is
295.20s - 298.66s |  what if that central point of failure fails?
298.66s - 300.42s |  So we hear this all the time.
300.42s - 302.46s |  There's some new VPN company that says,
302.46s - 305.98s |  we promise we won't look at any of your traffic.
305.98s - 308.18s |  Okay, actually, we do look at all of your traffic.
308.18s - 310.50s |  We promise we won't log anything.
310.50s - 312.36s |  Okay, actually, we do write all of it down.
312.36s - 314.26s |  We promise we won't tell anybody.
314.26s - 316.46s |  Oh, well, of course we answer subpoenas.
316.46s - 319.66s |  And so the problem there is you don't know.
319.66s - 321.14s |  It's a centralized system.
321.14s - 323.90s |  They promise they won't screw you, but they have the data.
323.90s - 326.34s |  They know you, they know what you do.
326.34s - 327.94s |  It's privacy by promise.
328.92s - 330.18s |  And it's worse than that.
331.78s - 334.10s |  Even if that centralized provider
334.10s - 336.78s |  is somehow magically honest in all ways,
336.78s - 338.50s |  it's still one point.
338.50s - 340.94s |  And the cable goes in, the same cable comes out.
340.94s - 342.54s |  And if you can get access to that,
342.54s - 345.08s |  then you can start doing timing and volume analysis
345.08s - 348.42s |  to say that user wanted to get that webpage.
348.42s - 351.46s |  So even if the central point is totally honest,
351.46s - 352.64s |  it's still bad news.
353.86s - 356.62s |  So the goal of Tor is to distribute the trust
356.62s - 359.60s |  over multiple relays so there's no single point
359.60s - 362.06s |  that knows both you and what you're doing.
362.06s - 365.34s |  So it's privacy by design, privacy by architecture,
365.34s - 367.42s |  rather than privacy by promise.
367.42s - 370.66s |  And part of the goal is we've got these,
370.66s - 372.34s |  you're able to see the slides, right,
372.34s - 373.28s |  even though they're not here?
373.28s - 374.46s |  Okay, good.
374.46s - 377.30s |  So part of the goal is that we've got these three relays
377.30s - 380.00s |  and you build your path through the three relays
380.00s - 382.52s |  and no single point in the network
382.52s - 384.40s |  is able to learn both who you are
384.40s - 386.34s |  and what destinations you're going to.
387.84s - 389.40s |  And another key point here,
389.40s - 391.52s |  transparency for Tor is really important.
391.52s - 393.80s |  Yes, it's open source, it's free software.
393.80s - 395.60s |  Yes, we give you the design documents
395.60s - 398.04s |  and the specifications and the research papers.
398.04s - 401.96s |  And yes, we say, hi, I'm Roger, she's Sylvia.
401.96s - 403.76s |  We're people, we're actually developers
403.76s - 407.02s |  who are happy to publicly identify ourselves.
407.02s - 410.64s |  And the key to that is transparency is really important.
410.64s - 412.96s |  Every time I talk to somebody, they say,
412.96s - 414.64s |  oh, ha ha, the privacy people
414.64s - 415.96s |  are talking about transparency.
415.96s - 417.38s |  That doesn't make any sense.
417.38s - 421.16s |  And the reason why it makes sense is privacy is about choice.
421.16s - 423.04s |  Transparency is about choice.
423.04s - 426.52s |  Our goal here is we choose to be transparent
426.52s - 428.56s |  in order to build a stronger community,
428.56s - 431.80s |  in order for people to be able to trust what we're doing
431.80s - 436.60s |  and to become part of our community.
436.60s - 438.36s |  Okay, so here's our first graph.
438.36s - 440.82s |  This is a graph of the capacity and the load
440.82s - 442.96s |  on the Tor network over the past 10 years.
443.00s - 446.28s |  The green line is what sort of capacity we have,
446.28s - 448.68s |  and the purple line is the load on the network.
448.68s - 451.80s |  And we're up at 250 gigabits per second
451.80s - 454.28s |  of actual load on the network,
454.28s - 457.52s |  which is a pretty big global network at this point.
457.52s - 460.40s |  On the other hand, the capacity has still been growing,
460.40s - 462.60s |  and the purple line, the load line,
462.60s - 465.08s |  has sort of flattened out over the past couple of years.
465.08s - 468.42s |  So we can definitely talk about how to scale
468.42s - 477.38s |  and how the performance should work.
477.38s - 479.58s |  No, keep going.
480.42s - 481.26s |  That's fine.
482.50s - 483.78s |  Now it works?
483.78s - 484.62s |  Oh, it doesn't.
485.58s - 487.10s |  Hello, that's fine.
487.10s - 492.10s |  Okay, so this is the number of relays per flags that we have.
492.50s - 496.58s |  Flags are like roles that relay can have on the network.
496.58s - 499.42s |  And for example, if we have a stable relay,
499.42s - 501.82s |  that usually is gonna be a guard as well,
501.82s - 504.70s |  meaning that that's the relay that the Tor browser picks
504.70s - 506.66s |  when entering the network.
506.66s - 508.62s |  And some of you might notice
508.62s - 510.70s |  there is an interesting curve here.
510.70s - 514.66s |  It's the HSDR graph here.
514.66s - 516.88s |  In the middle, we have 4,000 relays,
516.88s - 519.26s |  and then sometimes around June, something happens,
519.26s - 522.06s |  and we lose 2,000 of those relays.
522.06s - 524.30s |  And why does that happen?
524.30s - 527.30s |  So there was a Debian update.
527.30s - 529.46s |  There was a library that had a few dependencies
529.46s - 530.74s |  in common with Tor.
530.74s - 534.02s |  So that updated, and the Tor daemon restarted,
534.02s - 536.42s |  and the relays lose the flag.
536.42s - 538.70s |  And this introduced us to a topic
538.70s - 542.02s |  that is important for the Tor network, which is diversity.
542.02s - 544.94s |  And actually shows that we have a little bit
544.94s - 548.62s |  of a monoculture issue when it comes to Tor relays.
548.62s - 551.70s |  A lot of them don't run Debian.
551.70s - 554.46s |  And that is because the easiest way to run a Tor relay
554.46s - 558.18s |  is you spin up a Debian machine,
558.18s - 560.90s |  you install the package, and then you have the relay.
562.02s - 566.02s |  And so it shows these monoculture issues,
566.66s - 568.30s |  but I think we're not the only one
568.30s - 569.58s |  to have monoculture issues.
569.58s - 574.14s |  I think Microsoft said that they had 8.5 million devices
574.14s - 578.86s |  affected by the CrowdStrike incident, so anyway.
580.30s - 582.74s |  And this is why we talk about diversity a lot
582.74s - 585.70s |  when we talk about the Tor network.
585.70s - 588.46s |  The Tor network is driven by a diverse community
588.46s - 593.46s |  of relay operators, and we have developers,
593.62s - 597.06s |  we have privacy researchers, we have security professional,
597.06s - 598.46s |  we have academics.
599.58s - 602.54s |  And diversity is not just a desirable attribute,
602.54s - 603.98s |  but it's a critical component
603.98s - 605.94s |  when we talk about the network.
605.94s - 610.10s |  And you might wonder, what do we mean by diversity?
610.10s - 612.38s |  Why we place so much emphasis on it?
613.70s - 618.18s |  And what mechanism, and we do have to foster
618.18s - 619.46s |  this diverse community.
619.46s - 623.34s |  So the first thing is when we talk about the network,
624.22s - 625.86s |  we mean diversity of hardware,
625.86s - 629.18s |  software configuration, location, but also of users.
630.26s - 631.98s |  Hardware and software configuration,
631.98s - 634.38s |  we saw a little bit when a lot of relays
634.38s - 636.02s |  run the same operating system,
636.02s - 637.82s |  they are affected maybe by a bug.
637.82s - 639.78s |  That means a big part of the network
639.78s - 641.90s |  is affected by the same bug.
641.90s - 645.70s |  So we strive, our objective is to have
645.70s - 648.94s |  a uniform distribution of operating systems.
648.94s - 651.70s |  Of course, we're not there, but that's what we aim for.
652.66s - 655.74s |  We place a lot of emphasis on location as well
655.74s - 657.34s |  for a variety of reasons.
657.34s - 660.38s |  We want third circuit to spread
660.38s - 662.02s |  different geographical location
662.02s - 665.58s |  because it would make harder for an observer
665.58s - 670.58s |  to place the origin and the destination of a communication.
671.22s - 674.58s |  We want also to our relays to be spread globally
674.58s - 676.46s |  to avoid single point of failure.
677.78s - 679.98s |  Because if there is an issues locally
680.02s - 682.78s |  that is affecting a number of relays,
682.78s - 684.74s |  we know that the rest of the network,
684.74s - 686.86s |  we maintain the stability.
686.86s - 690.26s |  And we also think that by having relay operators
690.26s - 691.46s |  spread around the world,
691.46s - 694.98s |  we have a more diverse community, diverse location,
694.98s - 699.10s |  which in the end will help maintain the network growth
699.10s - 701.90s |  and sustainability in the long run.
701.90s - 705.34s |  And we also want diversity of users.
705.34s - 708.94s |  So people use Tor for a variety of reasons.
708.94s - 712.58s |  We estimate that we have 400,000 users
712.58s - 714.82s |  daily in the United States.
714.82s - 718.74s |  It means that we have diversity of usage patterns,
718.74s - 720.86s |  profiles, interest.
720.86s - 725.02s |  We have minority groups using Tor, social right activists.
725.02s - 728.26s |  We have privacy enthusiasts who would want to be followed
728.26s - 731.58s |  by surveillance company, why they surf the web.
731.58s - 736.10s |  And for many people like activists,
737.10s - 740.70s |  people belonging to different kind,
740.70s - 742.50s |  for example, people researching,
746.07s - 747.11s |  people researching,
748.99s - 751.79s |  people researching like information
751.79s - 753.51s |  on reproductive rights online.
753.51s - 757.19s |  People needed looking for information
757.19s - 758.23s |  on domestic violence.
758.23s - 762.47s |  For example, Tor provides a platform to access information
762.47s - 764.83s |  and to express themselves in environments
764.87s - 767.55s |  where they could otherwise face persecution,
767.55s - 769.99s |  discrimination, and retaliation.
771.63s - 776.63s |  So how do we actually foster this diverse community
777.15s - 777.99s |  in a healthy way?
777.99s - 781.55s |  So we place a lot of emphasis on direct communication.
781.55s - 785.55s |  We use IRC, we use mailing lists, we use forum.
785.55s - 788.27s |  We organize online and in-person meetups,
788.27s - 791.67s |  online meetups for our community up and every month.
791.67s - 794.99s |  We get feedback from relay operators, for example,
794.99s - 799.47s |  and we tell them how development is going,
799.47s - 801.27s |  if there is any update,
801.27s - 804.23s |  and we get this communication going on.
804.23s - 808.15s |  We have in-person meetups, we are here at DEF CON.
808.15s - 810.15s |  In Europe, we know that there are other events
810.15s - 812.27s |  where people in our community like to gather.
812.27s - 815.91s |  So for example, we have FOSDEM, we have CCC.
815.91s - 818.67s |  And we do develop transparent policies.
819.67s - 823.43s |  We have open process to create and maintain policies
823.43s - 824.67s |  for our community,
824.67s - 827.11s |  which are central to how we run the community.
828.31s - 832.19s |  This discussion of how to propose and maintain this policy
832.19s - 835.39s |  happen on our backtracker on GitLab.
835.39s - 839.15s |  And everybody's welcome to give feedback.
839.15s - 842.39s |  And it's a way for us to give new tools
842.39s - 843.51s |  to the relay community,
843.51s - 844.55s |  because in the past,
844.55s - 847.67s |  we noticed that they were organizing informally.
847.67s - 849.67s |  And this is a tool to help them
849.67s - 854.67s |  with managing expectation and with governance.
860.56s - 862.00s |  Testing, testing, great.
862.00s - 864.40s |  So where do these relays come from?
864.40s - 867.24s |  And originally, it was people that I had met
867.24s - 869.16s |  at hacker conferences like this.
869.16s - 871.00s |  And I said, tour is really important.
871.00s - 872.72s |  And you said, how can I help?
872.72s - 874.64s |  And I convinced you to run a relay.
874.64s - 877.76s |  And that worked great for the first 50 relays
877.76s - 879.00s |  or 100 relays.
879.04s - 880.92s |  But if we need to scale the network
880.92s - 882.48s |  up to thousands of relays,
882.48s - 885.28s |  and they need to be in diverse locations around the world,
885.28s - 887.40s |  I can't go meet all of the people.
887.40s - 890.44s |  In order to have an actually global safe network,
890.44s - 892.76s |  we need to have them everywhere
892.76s - 895.12s |  and run by people that we don't know as well.
895.12s - 896.28s |  And the challenge there,
896.28s - 898.32s |  especially in the past few years,
898.32s - 901.40s |  the set of reasons why bad people
901.40s - 903.64s |  might want to set up relays has grown.
903.64s - 907.04s |  We've ended up with people in Russia setting up relays
907.04s - 911.40s |  in order to do a man-in-the-middle attacks on exit traffic.
911.40s - 914.60s |  We've ended up, we've seen people running relays,
914.60s - 916.00s |  a lot of them for reasons
916.00s - 918.40s |  that we still don't quite understand what's going on.
918.40s - 921.96s |  So this is an ongoing and growing challenge.
921.96s - 923.76s |  How do we make sure that the tour network
923.76s - 926.68s |  has the relays that are doing it for the right reasons?
926.68s - 928.52s |  And part of that, so how do we do that?
928.52s - 930.32s |  How do we actually accomplish that?
930.32s - 932.64s |  The first answer is we need to get rid
932.64s - 936.08s |  of application-level vulnerabilities.
936.08s - 939.04s |  For example, the reason why those Russian people
939.04s - 941.48s |  could run man-in-the-middle attacks
941.48s - 946.48s |  is because some tour users were doing HTTP, not HTTPS.
946.64s - 949.08s |  So we managed to fix the browser
949.08s - 952.32s |  to do HTTPS by default always,
952.32s - 956.96s |  and that is a structural way to resolve that sort of attack.
956.96s - 959.64s |  Another angle to look at this,
959.64s - 962.88s |  one of the few tools that we have in this community,
962.88s - 965.28s |  in this social group of relays,
965.28s - 966.96s |  is the social graph.
966.96s - 968.44s |  It's who knows who.
968.44s - 971.76s |  So if we, as a community of relay operators,
971.76s - 973.72s |  get together and know each other,
973.72s - 977.58s |  then we can be stronger against outsider or FBI people
977.58s - 979.52s |  or whatever trying to come in and join.
980.64s - 984.12s |  And speaking of that, one of the really successful models
984.12s - 986.32s |  over the past 10 years or so
986.32s - 988.76s |  is relay operator associations.
988.76s - 991.16s |  These are non-profits around the world who,
991.16s - 994.88s |  these are individuals who banded together in their country.
994.88s - 998.92s |  One of them exists in France, in Switzerland, in Germany.
998.92s - 1000.16s |  There's one in Sweden.
1000.16s - 1001.36s |  There was one in Luxembourg.
1001.36s - 1002.40s |  There was one in Iceland.
1002.40s - 1003.64s |  There are two in Canada.
1003.64s - 1008.28s |  It's a global phenomenon where non-profit hacker spaces
1008.28s - 1010.72s |  get together to run tour exit relays.
1010.72s - 1012.52s |  And one of the great things about this
1012.52s - 1014.14s |  is it's sustainable.
1014.14s - 1016.00s |  You can run really big relays.
1016.00s - 1018.92s |  Another one is they can establish relationships
1018.92s - 1022.32s |  with the local cops so they know how tour works.
1022.32s - 1023.80s |  And also, we know who they are.
1023.80s - 1025.92s |  We meet them at conferences like DEF CON
1025.92s - 1027.32s |  and grow that community.
1027.32s - 1029.44s |  So having them be part of our community
1029.44s - 1032.48s |  is one of the ways we can handle that asymmetry
1032.48s - 1035.40s |  where we need to have a lot of relays around the world,
1035.40s - 1037.76s |  but also we need to know who they are
1037.76s - 1040.08s |  and have them work together.
1040.08s - 1044.20s |  And speaking of that, EFF a year ago launched a campaign
1044.20s - 1047.12s |  to get universities in particular running relays.
1047.12s - 1049.52s |  And part of the reason for that is universities,
1049.52s - 1051.96s |  that's where freedom of speech,
1051.96s - 1054.20s |  freedom of thought is supposed to be.
1054.20s - 1058.32s |  But it's not just the freedom side.
1058.32s - 1060.36s |  Universities are about education.
1060.36s - 1062.44s |  You can run a relay at your university
1062.44s - 1066.40s |  and learn how to put a computer on the actual internet,
1066.40s - 1069.76s |  which is a thing that we all used to do.
1069.76s - 1072.44s |  And now the cloud exists, so not as many people do that.
1072.44s - 1074.12s |  So there's an education side.
1074.12s - 1075.24s |  There's a community side.
1075.24s - 1076.52s |  There's a research side.
1076.52s - 1078.04s |  If you want to help do that,
1078.04s - 1080.40s |  they have these amazing coins
1080.40s - 1082.12s |  that they created a little while ago
1082.12s - 1084.08s |  that I'll show you afterwards,
1084.08s - 1086.42s |  that if you run a relay at your university,
1086.42s - 1088.72s |  they will send you one of these amazing coins.
1089.72s - 1093.46s |  And relay incentives in general is a broad topic
1093.46s - 1095.32s |  that we really need to work harder on.
1095.32s - 1100.32s |  So gamification, like Reddit style, Stack Exchange style.
1100.96s - 1104.24s |  I got a gold badge for my five-year anniversary.
1104.24s - 1106.84s |  I got a silver badge because I was an exit relay.
1106.84s - 1110.24s |  I got a silver badge because I'm in an ISP
1111.08s - 1112.32s |  that there aren't very many other relays in.
1112.32s - 1116.60s |  So coming up with ways to highlight what we need most
1116.60s - 1119.00s |  and make people feel happy and part of a community
1119.00s - 1120.48s |  and check out my gold badge.
1120.48s - 1122.08s |  I can put this on Twitter.
1122.08s - 1124.76s |  Leaderboards, I'm running the 17th fastest
1124.76s - 1126.56s |  tour relay in the world right now.
1126.56s - 1127.80s |  So that's one piece.
1127.80s - 1130.56s |  Another piece is just giving people more attention.
1130.56s - 1134.60s |  Meetups, having a time to answer their questions
1134.60s - 1137.00s |  and walk people through how to run relays.
1137.00s - 1139.28s |  And then there are the structural incentives.
1139.64s - 1143.28s |  Imagine we had a tour design where if you run a relay,
1143.28s - 1145.62s |  you get better performance on tour.
1145.62s - 1149.02s |  So there are a bunch of exciting security problems
1149.02s - 1150.84s |  that come with a lot of these designs,
1150.84s - 1153.14s |  but maybe there's a version of this design
1153.14s - 1156.08s |  where we can get rid of the security flaws
1156.08s - 1158.02s |  while still keeping the incentives.
1158.02s - 1161.68s |  And then the last topic is a lot of cryptocurrency people
1161.68s - 1164.32s |  these days keep saying, you know what you need to do?
1164.32s - 1167.04s |  You need an incentive coin thing
1167.04s - 1169.00s |  so that people will want to run relays
1169.72s - 1170.56s |  because they'll make money.
1170.56s - 1173.08s |  They'll make Bitcoins, dollars, and so on.
1173.08s - 1176.88s |  And part of the challenge there is the cryptocurrency world,
1176.88s - 1179.80s |  their fundamental building block is capitalism,
1179.80s - 1181.42s |  is maximizing profit.
1181.42s - 1182.60s |  And the tour world,
1182.60s - 1185.24s |  our fundamental building block is altruism.
1185.24s - 1187.36s |  It's I want to make the world a better place.
1187.36s - 1188.96s |  I have extra resources.
1188.96s - 1190.32s |  I can provide them.
1190.32s - 1195.32s |  So these are very different worlds in terms of organic.
1196.32s - 1199.08s |  Do you build an organic community of people
1199.08s - 1200.48s |  who want to help each other?
1200.48s - 1203.24s |  Or are you just trying to everybody for themselves
1203.24s - 1204.56s |  maximize your profit?
1204.56s - 1207.12s |  And in that case, if you want to do it for profit,
1207.12s - 1209.68s |  maybe you'll all go to the cheapest hosting provider
1209.68s - 1212.06s |  so you can spend as little money as possible.
1212.06s - 1214.00s |  Or maybe you'll sell your data on the side
1214.00s - 1215.24s |  to make a little bit more money
1215.24s - 1218.28s |  because capitalism means maximizing profit.
1218.28s - 1223.28s |  So these are fundamentally incompatible approaches
1223.68s - 1227.48s |  to how to build a community of volunteers around the world
1227.48s - 1232.16s |  who are trying to keep people safe.
1232.16s - 1236.00s |  Okay, so I want to tell you a bit about network health
1236.00s - 1238.12s |  and metrics and what we do.
1238.12s - 1240.00s |  But first I want to tell you about a story
1240.00s - 1243.56s |  because I want to put what we do in context.
1243.56s - 1246.44s |  So between May and June in 2020,
1246.44s - 1250.48s |  we removed a group of Torexit relays
1250.48s - 1253.16s |  that were doing an SSL strip attack.
1253.16s - 1254.84s |  And we didn't spot this at first
1254.84s - 1258.32s |  because they were leaving all the exit traffic alone,
1258.32s - 1260.68s |  but they were intercepting just a little bit of the traffic
1260.68s - 1265.08s |  that was going to some cryptocurrency exchange websites.
1265.08s - 1267.92s |  So what this were doing was that
1269.18s - 1272.56s |  if the user was visiting the HTTP website,
1272.56s - 1275.96s |  not the ghttps one, they wouldn't redirect the user.
1275.96s - 1278.84s |  And if they wouldn't notice that there was not a lock,
1278.84s - 1281.64s |  for example, in the browser, in the tab,
1281.64s - 1285.44s |  they would just continue to send their login information,
1285.44s - 1288.76s |  for example, and this information would be intercepted.
1288.76s - 1292.28s |  So we noticed this, we kicked the relays out.
1292.28s - 1294.20s |  And then a few weeks later in June,
1294.20s - 1297.28s |  another group of relays does the same attack.
1297.28s - 1300.88s |  And we noticed again, and we kicked them out.
1300.88s - 1303.24s |  And then we realized that network health
1303.24s - 1305.86s |  is not just about kicking relays out,
1305.86s - 1306.96s |  although that's important
1306.96s - 1310.04s |  because you don't want malicious relay on the network.
1310.28s - 1312.36s |  It's a little bit more.
1312.36s - 1314.80s |  So we got together with the browser team
1314.80s - 1319.12s |  and because of projects like HTTPS Everywhere
1319.12s - 1321.32s |  and a newer version of Firefox
1321.32s - 1325.80s |  that allowed the browser to default to HTTPS
1325.80s - 1330.52s |  and then fall back to HTTP just when needed,
1330.52s - 1334.20s |  we were able to enable HTTPS by default into our browser
1334.20s - 1337.08s |  and protect our user, not just the network level,
1337.08s - 1339.60s |  but also the application level.
1339.60s - 1343.48s |  And so out of incident like this,
1343.48s - 1347.52s |  that we understood the value of having a network health team
1347.52s - 1350.56s |  which is basically joining together a group of people
1350.56s - 1352.94s |  that keep track of the network.
1352.94s - 1357.08s |  And part of what we do is kicking relays out of the network.
1357.08s - 1362.00s |  So, but also part of what we do is community engagement,
1362.00s - 1363.68s |  is working with the relay community,
1363.68s - 1367.84s |  working with activists sometimes in the ground.
1367.84s - 1371.20s |  And the foundation of what we do also
1371.20s - 1373.68s |  is collecting the metrics about the Tor network
1373.68s - 1377.04s |  that gives us an idea of how the network is performing,
1377.04s - 1378.36s |  how the network is doing
1378.36s - 1381.50s |  and to understand if something is not right.
1382.60s - 1387.24s |  And so when there is a geopolitical event, for example,
1387.24s - 1388.70s |  we are able to spot it
1388.70s - 1392.56s |  because we are able to notice some patterns on the network.
1392.56s - 1397.32s |  And in this case, this is a picture of,
1397.32s - 1400.12s |  it's a graph of directly connecting user from Russia
1400.12s - 1404.06s |  in between 2021 and 2022.
1404.06s - 1406.68s |  And we start noticing that there was censorship
1406.68s - 1411.04s |  being applied in Russia between November and December,
1411.04s - 1413.64s |  which a few months before the Ukraine invasion.
1414.56s - 1417.72s |  And we saw directly connect user decline.
1417.72s - 1421.40s |  And as we saw this happening,
1421.40s - 1425.48s |  we also were able to work with the anti-censorship team,
1425.48s - 1426.84s |  with activists on the ground
1427.36s - 1430.16s |  and we were able to tell the user to use bridges.
1430.16s - 1433.72s |  And so we saw a spike in bridge users in Russia.
1433.72s - 1435.84s |  And that meant that we are able to help these people
1435.84s - 1437.84s |  reach the news and know what is happening
1437.84s - 1439.08s |  outside the country.
1440.74s - 1443.22s |  Yeah, so let me jump in and explain what a bridge is
1443.22s - 1444.76s |  for those who don't know.
1444.76s - 1446.76s |  We've been talking about relays so far.
1446.76s - 1449.82s |  These are public Tor relays that are in the list.
1449.82s - 1451.92s |  And that means that the censors can get that list
1451.92s - 1453.12s |  and block all of them.
1453.12s - 1456.68s |  So bridges are relays that aren't in the list.
1457.50s - 1459.52s |  And so the goal is to transform the traffic in some way
1459.52s - 1460.82s |  so that it's doing a protocol
1460.82s - 1462.40s |  the censors don't want to block
1462.40s - 1464.32s |  and it's talking to an IP address
1464.32s - 1466.14s |  that the censors don't think is Tor.
1466.14s - 1469.08s |  So that's the arms race that we're in there.
1469.08s - 1471.12s |  And there are a bunch of different approaches
1471.12s - 1472.48s |  we have for bridges.
1472.48s - 1476.18s |  One of which is?
1476.18s - 1481.18s |  Yeah, and similar situation in Iran in 2022,
1481.40s - 1483.42s |  the Massimini protests.
1483.42s - 1488.42s |  So people, we started noticing that there was aggressive
1488.84s - 1490.64s |  censorship happening in the country.
1490.64s - 1492.56s |  People were in the streets protesting.
1492.56s - 1494.80s |  They couldn't access the internet.
1494.80s - 1496.24s |  They couldn't access the news.
1496.24s - 1500.20s |  They couldn't use communication messaging systems.
1500.20s - 1503.90s |  So by working again with people on the ground,
1503.90s - 1505.90s |  the activists, the local people
1505.90s - 1507.28s |  and the anti-censorship team,
1507.28s - 1511.70s |  we were able to teach these users how to use Snowflake.
1511.70s - 1516.70s |  And we saw how the users from Iran spiked
1516.90s - 1518.66s |  during the protest.
1519.42s - 1520.26s |  And for these people,
1520.26s - 1522.14s |  reaching Tor was a lifeline in a way
1522.14s - 1524.14s |  because they were able to communicate
1524.14s - 1527.34s |  between one another in ways that it was not possible
1527.34s - 1529.70s |  at the moment with any other tool.
1529.70s - 1531.62s |  And if you're wondering what Snowflake is,
1531.62s - 1533.50s |  so in the Tor ecosystem,
1533.50s - 1536.70s |  Snowflake is what we call a pluggable transport.
1536.70s - 1539.74s |  And you can think of it like as a tool
1539.74s - 1541.84s |  that Tor used to disguise the traffic.
1541.84s - 1543.84s |  And in the case of Snowflake,
1543.84s - 1546.94s |  it used temporary run volunteer proxies.
1546.94s - 1550.38s |  So it's by using the proxies,
1550.38s - 1553.82s |  more difficult for the sensors to actually block them.
1553.82s - 1557.90s |  And it helps people in very difficult situation
1557.90s - 1559.20s |  to reach the internet.
1560.94s - 1562.74s |  Yeah, so two things to point out
1562.74s - 1564.10s |  for this graph in particular.
1564.10s - 1566.24s |  One of them is that y-axis,
1566.24s - 1568.10s |  we're talking a hundred thousand
1568.10s - 1570.06s |  or maybe hundreds of thousands
1570.06s - 1573.10s |  of people using Snowflake in Iran during that event.
1573.10s - 1576.06s |  So as we're showing more recent graphs,
1576.06s - 1578.22s |  we end up with larger and larger numbers.
1578.22s - 1580.86s |  We're talking about many tens of thousands
1580.86s - 1582.42s |  or hundreds of thousands of people.
1582.42s - 1585.84s |  And the other answer is snowflake.torproject.org.
1585.84s - 1588.06s |  You too can volunteer as a Snowflake.
1588.06s - 1590.26s |  You can either install it as a browser extension
1590.26s - 1591.50s |  or there's a headless version
1591.50s - 1593.74s |  for your Linux system based on Go.
1593.74s - 1596.92s |  Please help.
1596.92s - 1599.32s |  And so more recent case,
1599.32s - 1603.68s |  we saw just a few months ago between May and June,
1603.68s - 1607.64s |  we started to see user spikes in Myanmar.
1607.64s - 1610.52s |  And then we got news that the military junta
1610.52s - 1613.48s |  and started to apply internet shutdown.
1613.48s - 1616.80s |  So again, what we did was get together with the browser,
1616.80s - 1618.38s |  the anti-censorship team,
1618.38s - 1622.04s |  and help users connecting to Tor
1622.04s - 1625.50s |  by distributing bridges directly for them in the country.
1625.50s - 1629.04s |  And we saw how from not having
1629.04s - 1631.14s |  almost zero bridge users in the country,
1631.14s - 1638.96s |  we got a spike in a few days to just half a thousand.
1639.16s - 1641.24s |  So one of the fascinating things
1641.24s - 1642.90s |  about the graph like this,
1642.90s - 1645.92s |  periodically we see spikes in our user graphs.
1645.92s - 1648.16s |  And here we're like, wow, a whole lot of people
1648.16s - 1651.68s |  from Myanmar just started using Tor over bridges.
1651.68s - 1652.60s |  This is great.
1652.60s - 1654.08s |  But sometimes we see the spikes
1654.08s - 1656.44s |  and we don't know if it actually happened that way.
1656.44s - 1658.44s |  We don't know what happened in the country.
1658.44s - 1660.08s |  So every time we see a spike like this,
1660.08s - 1662.04s |  we're wondering, is there a bug?
1662.04s - 1664.72s |  Is there like a GOIP database mess up
1664.72s - 1666.90s |  where actually these are people in Germany
1666.90s - 1670.34s |  who got misclassified as being in Myanmar for that week?
1670.34s - 1671.74s |  So part of the challenge we have
1671.74s - 1674.06s |  is understanding what actually happened.
1674.06s - 1676.58s |  And one of the principles that we have for that situation
1676.58s - 1679.46s |  is trying to learn the same data points
1679.46s - 1681.14s |  from a bunch of different dimensions,
1681.14s - 1682.86s |  from a bunch of different angles,
1682.86s - 1685.36s |  so that we can confirm that what we're seeing
1685.36s - 1686.86s |  actually really did happen.
1686.86s - 1688.42s |  And part of that is angles,
1688.42s - 1691.10s |  including talking to people on the ground and saying,
1691.10s - 1692.94s |  what just happened in your country?
1692.94s - 1695.34s |  Why is it that we're seeing these graphs,
1695.34s - 1699.78s |  these changes in our graphs?
1699.78s - 1703.70s |  Yeah, so we've been talking about user data
1703.70s - 1710.06s |  and you might have wonder...
1710.06s - 1710.90s |  That's loud.
1711.94s - 1712.78s |  Okay.
1714.66s - 1715.50s |  All right, go on.
1715.50s - 1716.32s |  Okay.
1716.32s - 1717.70s |  I think they can hear us better than we can.
1717.70s - 1718.54s |  Okay.
1718.54s - 1722.54s |  So you might wonder, Tor is the anonymity network.
1722.54s - 1724.38s |  How come you have user data?
1724.38s - 1727.18s |  And so the simple answer to that
1727.18s - 1729.82s |  is that we live by a principle
1729.82s - 1731.42s |  that is we do not collect data
1731.94s - 1733.18s |  that cannot be public.
1733.18s - 1736.90s |  So all our data that we have in metrics
1736.90s - 1739.14s |  is actually public data that we publish.
1740.58s - 1743.34s |  So we do not collect in-client data.
1743.34s - 1745.38s |  Our user data that we've been talking about
1745.38s - 1746.58s |  are actual estimations.
1746.58s - 1748.90s |  And I'm going to say how we do this.
1748.90s - 1751.34s |  But other products like VPN products,
1751.34s - 1755.06s |  they actually count the users and we don't.
1755.06s - 1758.62s |  So what we have is aggregated data from relays.
1758.62s - 1762.50s |  So the Tor browser or Tor clients in general,
1762.50s - 1763.74s |  when they connect to the network,
1763.74s - 1766.38s |  they request a list of relays information
1766.38s - 1768.50s |  and they need this to build the circuits.
1768.50s - 1772.14s |  So we estimate that if...
1773.30s - 1775.82s |  First of all, we count all these requests globally,
1775.82s - 1777.90s |  how they are collected from relays.
1777.90s - 1780.06s |  And then we estimate that if a Tor client
1780.06s - 1784.70s |  was connected 24 hours, it will make 10 requests.
1784.70s - 1788.10s |  So we count all the requests that we get from the relays
1789.10s - 1791.74s |  when they upload their statistics.
1791.74s - 1796.30s |  And we divide them by 10 and we have the user estimation.
1796.30s - 1799.54s |  So in reality, we might be underestimating our user count
1799.54s - 1801.66s |  by an order of magnitude.
1801.66s - 1803.38s |  But that's fine for us.
1803.38s - 1805.02s |  It's the trade-off that we want to make
1805.02s - 1808.42s |  in order not to put our user at risk.
1808.42s - 1812.98s |  So we do have, though, historical data.
1812.98s - 1816.86s |  And we actually serve this data
1816.86s - 1818.74s |  on collector.torproject.org.
1818.74s - 1821.18s |  You find all the documents that have ever, I think,
1821.18s - 1823.30s |  been created on the network.
1823.30s - 1827.62s |  And out of this data, we have network-wide statistics.
1827.62s - 1830.82s |  And we have also the status of individual nodes
1830.82s - 1834.74s |  that will give us an idea of how the network is doing.
1834.74s - 1836.26s |  And then something that we do
1836.26s - 1839.74s |  is making the data accessible to the public,
1839.74s - 1842.42s |  but also presenting the documentation
1842.42s - 1845.34s |  of how we estimate the metrics that we have.
1845.34s - 1848.86s |  And we believe in algorithm transparency
1848.86s - 1850.90s |  so that if you download these tarballs
1850.90s - 1852.54s |  and you process this data,
1852.54s - 1854.90s |  you can come up with the same results as we do.
1854.90s - 1856.66s |  And if you don't, you explain to us
1856.66s - 1858.14s |  why you don't have the same results
1858.14s - 1862.22s |  and maybe we have to change something on our side.
1862.22s - 1865.70s |  So this is an overview of how this process works.
1865.70s - 1868.74s |  So the relays have this little piece of information
1868.74s - 1871.02s |  that they upload to directory authorities,
1871.02s - 1874.34s |  which are special nodes in the network.
1874.34s - 1876.42s |  And there is a service that's called collector
1877.38s - 1882.30s |  which downloads these documents and archives them.
1882.30s - 1886.30s |  And then we have a service that takes all these documents,
1886.30s - 1888.02s |  parses them, puts them in a database
1888.02s - 1891.62s |  and extracts time series and network statistics.
1891.62s - 1894.66s |  And I said that all the data is public
1894.66s - 1897.54s |  and this is true, but there are a few exceptions.
1897.54s - 1902.14s |  The first exception is that we are in a race with sensors.
1902.14s - 1904.46s |  And these sensors have a lot of resources
1904.46s - 1907.54s |  and sometimes we need to protect some of the data
1907.54s - 1909.74s |  in order to keep up with this race.
1910.82s - 1913.46s |  And the other exception that we do
1913.46s - 1915.66s |  is that sometimes we don't publish data
1915.66s - 1918.14s |  when it can put user at risk.
1918.14s - 1920.26s |  For example, if we have a specific country
1920.26s - 1922.90s |  which has very few Tor users,
1922.90s - 1925.46s |  and we believe that publishing those numbers
1925.46s - 1928.30s |  can help somebody else that has other data
1928.30s - 1929.58s |  to anonymize those users,
1929.58s - 1931.66s |  so we don't publish that numbers.
1931.66s - 1937.28s |  And I think the threshold is about 100 users.
1937.28s - 1941.28s |  So we've been talking about metrics
1941.28s - 1943.32s |  for understanding the status of the network,
1943.32s - 1945.04s |  but also we have other metrics
1945.04s - 1948.80s |  that helps us understand if a node is misbehaving
1948.80s - 1952.72s |  and also give us an idea of how the network is performing
1952.72s - 1955.08s |  from a point of view of performances
1955.08s - 1958.28s |  and bandwidth and so on.
1958.28s - 1961.12s |  So we look at traffic users patterns,
1961.12s - 1963.96s |  we have actual bandwidth measurements
1964.64s - 1966.44s |  and we have performance measurements.
1967.72s - 1970.96s |  So one thing that we do is bandwidth probing
1970.96s - 1972.72s |  and we have a service,
1972.72s - 1974.08s |  we actually have a bunch of service
1974.08s - 1976.08s |  that are called bandwidth authorities.
1976.08s - 1979.60s |  And what they do is that they create a two hop circuit
1979.60s - 1981.12s |  with the measured relay,
1981.12s - 1983.76s |  connect to a web server, download the files
1983.76s - 1986.40s |  and gives us an idea of the time it takes
1986.40s - 1988.44s |  to download that file.
1988.44s - 1991.80s |  And we can compare this with the bandwidth
1991.80s - 1993.96s |  that actually the relays declares
1993.96s - 1997.32s |  and when they send their descriptors file
1997.32s - 1998.64s |  to the directory authorities.
1998.64s - 2000.64s |  And in certain case we can find out
2000.64s - 2001.84s |  if some relays, for example,
2001.84s - 2004.52s |  is trying to game the system and tells us
2004.52s - 2009.19s |  that they have more bandwidth than they actually do.
2009.19s - 2012.39s |  And another thing that we do is performance measurements
2012.39s - 2014.79s |  and we do them for two reasons.
2014.79s - 2017.63s |  One of them is we want to know how the network performs
2017.63s - 2019.59s |  from different points in the world.
2019.59s - 2023.47s |  For example, we want to know how the location
2024.35s - 2026.91s |  of a user compared to the location of relays distribution
2026.91s - 2031.31s |  globally affects the bandwidth,
2031.31s - 2034.03s |  affect other measurements.
2034.03s - 2035.99s |  And the other thing is that we want to know
2035.99s - 2040.39s |  for censored regions how the network performs
2040.39s - 2043.71s |  for those users in those extreme situation.
2043.71s - 2047.55s |  So the way this works is that we have a Tor client
2047.55s - 2049.99s |  and a web server and then we build a server,
2049.99s - 2052.03s |  a circuit directly to the web server,
2052.03s - 2054.99s |  download again a file and we measure the bandwidth,
2054.99s - 2057.23s |  we measure the time it takes to build the circuit
2057.23s - 2063.46s |  and if there are errors, for example.
2063.46s - 2068.14s |  And so with all this data, we know it's open,
2068.14s - 2069.22s |  we know it's accessible,
2069.22s - 2072.70s |  we know everyone can start learning about Tor
2072.70s - 2077.66s |  and basically it's documented, it's data,
2077.66s - 2079.94s |  but we also know there is a learning curve.
2079.94s - 2083.02s |  We also know that you need resources sometimes
2083.02s - 2086.62s |  to store this data, you need time to parse it.
2086.62s - 2091.62s |  And so we decided to open a little bit more
2092.38s - 2095.06s |  the way people can access metrics data.
2095.06s - 2098.02s |  We are working on an API which is already embedded,
2098.02s - 2099.78s |  it's already accessible.
2099.78s - 2103.50s |  And we hope that this API can lower the barrier
2103.50s - 2107.02s |  for researchers to start working with metrics data.
2107.02s - 2108.90s |  We have few endpoints available,
2108.90s - 2111.86s |  more coming up soon as we are developing.
2111.86s - 2115.34s |  So you can access metrics-api.torproject.org,
2115.34s - 2117.54s |  the two endpoints that are available are details
2117.54s - 2121.74s |  and summary and give you some information of relays.
2121.74s - 2123.78s |  I forgot to mention the API is called
2123.78s - 2127.54s |  the Network Status API or NSA for short.
2127.54s - 2132.54s |  So one nice thing is that besides that you're querying
2135.14s - 2138.02s |  the NSA to get the information,
2138.02s - 2141.10s |  is that it has a little bit more flexibility
2141.10s - 2142.54s |  compared to parsing the data
2142.54s - 2144.78s |  or what we are offering right now.
2144.78s - 2148.86s |  And is that you can do, for example, historical queries.
2148.86s - 2151.26s |  So there is a published variable
2151.26s - 2153.78s |  that you can parse in the URL.
2153.78s - 2158.78s |  And you parse, you send a date, just year, month, day,
2158.94s - 2161.50s |  and then you get all the relays and bridges
2161.50s - 2164.38s |  that were in the network within a week of the parcel date.
2164.38s - 2168.94s |  So this is something new and that I believe
2168.94s - 2172.18s |  can give a lot of flexibility to researchers
2172.18s - 2173.86s |  to start experimenting.
2173.86s - 2175.82s |  I hope it helps relay operators
2175.82s - 2178.22s |  checking the status of their nodes.
2178.22s - 2180.06s |  And you can check out the documentation.
2180.06s - 2184.30s |  We welcome feedback and bugs reports, of course,
2184.30s - 2186.72s |  because expect it to be broken.
2189.34s - 2193.22s |  And yeah, I think it's part of the Tor Project mission
2193.22s - 2196.32s |  to create and develop anonymity and privacy technology,
2196.36s - 2199.28s |  but also to further its understanding,
2199.28s - 2202.16s |  scientific understanding, popular understanding.
2202.16s - 2208.43s |  And this is one step forward in that direction.
2208.43s - 2211.87s |  Okay, so we've learned a bit about the relay community side
2211.87s - 2214.35s |  and we've learned a bit about the metrics
2214.35s - 2216.31s |  and network measurement side.
2216.31s - 2218.23s |  What can you do to help us?
2218.23s - 2222.15s |  So one big piece is get involved in the relay community
2222.15s - 2224.99s |  and run a relay around the world at your university,
2224.99s - 2229.75s |  at your work, at your VPS, wherever you happen to be.
2229.75s - 2231.83s |  Maybe there are sysadmins in here right now
2231.83s - 2234.07s |  who would love to add an extra relay
2234.07s - 2236.59s |  on their volunteer rack.
2236.59s - 2239.35s |  And so part of that is the more relays we have,
2239.35s - 2241.03s |  the more performance we can handle,
2241.03s - 2243.99s |  the more users we can give good performance to.
2243.99s - 2246.77s |  And the other piece of it is the more relays we have
2246.77s - 2250.35s |  run by good people, the smaller the impact is
2250.35s - 2253.35s |  when some bad person shows up to try to run a relay.
2253.39s - 2256.75s |  So the way that we dilute the people trying
2256.75s - 2260.51s |  to do the attacks is we get more and more hacker spaces
2260.51s - 2263.87s |  and individuals and hackers and volunteers
2263.87s - 2266.03s |  to be participating in this community.
2266.03s - 2267.87s |  And yes, location matters.
2267.87s - 2270.39s |  If you can run a relay that's not in Germany
2270.39s - 2273.55s |  or the Netherlands, then you'll be showing up
2273.55s - 2275.11s |  in a part of the Tor network
2275.11s - 2277.67s |  that doesn't have as many relays right now.
2277.67s - 2280.97s |  So for example, the U.S. is nowhere near the top
2280.97s - 2283.01s |  in terms of countries that are contributing
2283.67s - 2284.51s |  to the Tor network.
2284.51s - 2285.77s |  I hear the U.S. has lots of hackers
2285.77s - 2289.47s |  and lots of internet connections and lots of computers.
2289.47s - 2292.55s |  So there's certainly an opportunity for you
2292.55s - 2295.63s |  to help us make the U.S. more impactful,
2295.63s - 2297.83s |  which in turn makes performance better
2297.83s - 2299.83s |  for users in this hemisphere.
2299.83s - 2302.61s |  So think about diversity of location,
2302.61s - 2304.67s |  but wherever you can run it is great.
2304.67s - 2306.19s |  Please add more relays.
2306.19s - 2307.87s |  And also I'm happy to chat with you more
2307.87s - 2312.15s |  about exit policies and whether you'll end up
2312.17s - 2315.37s |  in IP block lists and so on after this.
2316.41s - 2319.97s |  And the other side of that is community.torproject.org
2319.97s - 2323.63s |  has a lot of how-tos and guides on,
2323.63s - 2325.97s |  I want to set up a bridge on Debian.
2325.97s - 2327.01s |  How do I do it?
2327.01s - 2328.13s |  Here are the steps.
2328.13s - 2329.57s |  I want to do a snowflake.
2329.57s - 2330.65s |  Here's the steps.
2330.65s - 2333.19s |  I want to go to a relay operator meetup
2333.19s - 2335.65s |  or learn about the relay associations.
2335.65s - 2338.03s |  Here are the lists and documentation.
2338.03s - 2339.39s |  So there's a lot there.
2339.39s - 2341.09s |  And join the community in general.
2341.09s - 2342.83s |  Come to the relay operator meetups.
2342.83s - 2346.43s |  We don't have a formal official relay operator meetup
2346.43s - 2347.99s |  here at DEF CON this year
2347.99s - 2351.13s |  because this is one enormous room
2351.13s - 2353.43s |  and we didn't figure out how to do the spaces.
2353.43s - 2356.59s |  Maybe next year we'll join Privacy Village or something.
2356.59s - 2359.15s |  But you can come talk to us afterwards
2359.15s - 2361.79s |  and we'd love to chat with you more about relays.
2361.79s - 2363.35s |  And if you can't run a relay,
2363.35s - 2370.55s |  run a bridge or a snowflake or contribute in other ways.
2370.57s - 2372.21s |  Yeah, so you can also get involved
2372.21s - 2374.37s |  with the network health team.
2374.37s - 2376.65s |  If you're interested in network data analysis,
2376.65s - 2379.57s |  hunting for bad relays, there's certainly space for that.
2379.57s - 2382.31s |  We hope the API can help you.
2382.31s - 2384.21s |  There is also one thing that we do
2384.21s - 2387.33s |  which is creating tools for relay operators.
2387.33s - 2388.89s |  For example, Tor Weather,
2388.89s - 2391.09s |  which gives a notification to operators
2391.09s - 2393.61s |  when something is not right with the relay.
2393.61s - 2395.81s |  But also you can get involved with
2395.81s - 2400.59s |  for different projects, the network status API,
2400.59s - 2405.59s |  the parser that we use to parse the network documents in Java
2405.59s - 2408.07s |  and also we are trying to work on
2408.07s - 2412.91s |  a better way to visualize network data.
2412.91s - 2417.31s |  We are trying to move the parser from Java to Rust.
2417.31s - 2420.69s |  There is a link to our wiki, please get involved.
2420.69s - 2423.35s |  And if you're thinking to contribute to Tor
2423.35s - 2425.19s |  in any possible way,
2425.19s - 2427.05s |  you're helping me, you're helping Roger,
2427.05s - 2429.35s |  you're helping our colleagues,
2429.35s - 2431.49s |  you're helping us fighting the good fight
2431.49s - 2445.07s |  for privacy, anonymity and internet freedom.
2445.07s - 2445.95s |  Yeah, excellent.
2445.95s - 2449.05s |  So we've told you a bit about how you can help us,
2449.05s - 2450.97s |  but it is about helping us
2450.97s - 2453.73s |  and it's also about helping all of the activists
2453.73s - 2455.07s |  and people around the world,
2455.07s - 2457.73s |  especially the people who don't have the power
2457.73s - 2460.93s |  to have a military or bodyguards or whatever,
2460.93s - 2463.05s |  yet they want to change their society.
2463.05s - 2466.79s |  So we will be doing an Ask Us Anything You Like
2466.79s - 2470.27s |  at the Tor booth in the vendor area right after this.
2470.27s - 2473.03s |  We've got these cool badges and other things available
2473.03s - 2475.83s |  or you can just ask us every possible Tor question.
2475.83s - 2477.77s |  We will be there until you have run out
2477.77s - 2479.27s |  of all of your Tor questions
2479.27s - 2481.25s |  and Argoon has kindly volunteered
2481.25s - 2483.79s |  to escort all of us over there
2483.79s - 2486.03s |  and we'll hang out as long as you would like
2486.03s - 2487.55s |  to answer all your questions.
2487.55s - 2488.39s |  Thank you.


url: https://www.youtube.com/watch?v=dldX9UFhNTs
title: DEF CON 32 - Laundering Money - Michael Orlitzky
type: video
author: DEFCONConference
ingestion_date: 2024-11-10
transcription_model: large-v2
chunking_status: completed
content:
{
  "webpage_url": "https://www.youtube.com/watch?v=dldX9UFhNTs",
  "title": "DEF CON 32 - Laundering Money - Michael Orlitzky",
  "description": "CSC ServiceWorks is a large vendor of pay-to-play laundry machines in apartments and condomiums. Most are Speed Queens, but newer CSC-branded machines use an app for payment and have custom circuitry inside. Many however accept quarters as well. We show that, when all else fails, you can always physically bypass the coin slot to run the machines for free.",
  "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
  "duration": 1173,
  "channel": "DEFCONConference",
  "uploader": "DEFCONConference",
  "upload_date": "20241016"
}

0.24s - 5.88s | This text was transcribed using whisper model: large-v2

 Alright. So, uh, my name is Michael, and this is the title of my talk. It is Laundering
5.88s - 10.08s |  Money. If you're here to learn about laundering money, you're in the wrong place. Because
10.08s - 15.08s |  this talk is laundering money. Um, what we're going to do today is do some laundry for free
18.02s - 23.02s |  on machines that would really rather that you pay for them. Uh, in other words, what we're
23.02s - 28.02s |  going to do is save, that's right, laundering money. And from whom are we going to save?
31.16s - 37.60s |  A company called CSC ServiceWorks, who is the villain of this talk. Uh, and that is despite
37.60s - 42.62s |  the fact that, uh, to the untrained eye it's going to look like I am the one doing all of the
44.32s - 49.32s |  bad things. Uh, but I promise you, uh, I know we are the protagonists. Um, this is our
51.52s - 56.52s |  target. It is, uh, these are the washers in my building. Uh, they are CSC branded. Um, and
57.48s - 63.94s |  these are the dryers on the other side of the hallway. Uh, they are speed queens. Speed
63.94s - 69.42s |  queens are the most popular model of commercial washer and dryer in the United States. Uh, and
69.42s - 74.56s |  I made that up, but it's probably true. Uh, the manuals for these can all be found online.
74.56s - 79.10s |  They're extensively documented. And the manuals tell you how to hack the machine. But the
79.10s - 85.14s |  problem is that CSC, because they suck, have modified the base speed queens so that nothing
85.16s - 91.52s |  works. Right? So, here's a comprehensive list of things that do not work. And I have to do
91.52s - 96.26s |  this because if I don't, inevitably someone is going to come up to me at the end of the talk
96.26s - 101.44s |  and tell me that I'm an idiot and that I've wasted my time and that he saw a TikTok video and
101.44s - 105.88s |  all you have to do is press two buttons and the machine will do whatever you want. It's not
105.88s - 110.58s |  true. Uh, so we're going to start with the stupidest tricks. Who played video games in the
110.58s - 116.10s |  80s or 90s? Uh, you used to be able to go to an arcade, uh, and then go to the hardware store,
116.10s - 119.90s |  buy a washer that was just quarter sized and put it in the machine and it would give you
119.90s - 125.30s |  credit for a quarter. Uh, these days the washers cost more than the quarters do, but it's the
125.30s - 129.70s |  principle. There's a type of person who is willing to spend 40 cents to make sure that the
129.70s - 135.84s |  other guy gets zero cents and it's nice to meet you too. Uh, the second bullet point here is
135.84s - 140.78s |  if the first one fails, uh, if it knows that the washer is not a quarter, you can drill a tiny
140.78s - 145.38s |  hole in a real quarter, tie a string around it and go fishing like Wile E. Coyote and just pull
145.38s - 149.28s |  the quarter back out. Doesn't work. These machines have trap doors in the coin slot
149.28s - 155.46s |  mechanism. So, manual mode. The two button trick from TikTok puts the machine into
155.46s - 160.34s |  something called manual mode which believe it or not is documented in the manual. Uh, there
160.34s - 165.10s |  are several ways to get into manual mode which gives you admin privileges and none of them
165.10s - 171.06s |  work because CSC has changed the base model, uh, and all the circuitry. Uh, there's also a
171.06s - 176.88s |  dip switch to turn off the entire payment system. Doesn't work. And, uh, I'm from Baltimore
176.88s - 182.14s |  and there's a guy up the street, uh, in Philadelphia who has a video on YouTube, uh, where he
182.14s - 186.48s |  just takes a pocket knife basically and shorts out these two wires. This doesn't work on my
186.48s - 193.06s |  model because my model doesn't have those wires. Alright, but who is CSC ServiceWorks?
193.08s - 197.46s |  Well, what do they do? They take these laundry machines, they outfit them with a pay module
197.46s - 202.42s |  and they put them into condo buildings and apartments and the residents pay to use them and
202.42s - 207.44s |  in exchange CSC is supposed to maintain the machines. There are three ways to pay. Uh, the
207.44s - 212.00s |  first is a refillable chip card, the second are these two stupid apps, and the third is with
212.00s - 217.48s |  quarters, the old fashioned way, right? But, a picture is worth a thousand words. And so
217.50s - 222.64s |  here's a picture of what CSC actually does. Uh, I hope you can see that it's quite
222.64s - 227.84s |  impressive because it's not possible to give a lower rating than one star and there are
227.84s - 234.66s |  343 reviews up there. Uh, but right, the better business bureau sucks, so what if you trust
234.66s - 240.22s |  Yelp? Ooh, we're, we're getting better at least. Uh, but Yelp has been known to manipulate
240.22s - 246.56s |  ratings, so what if we ask Google? Yikes. Okay, uh, well this is unfair because what is a
246.58s - 253.36s |  one star rating even mean? We need to be able to put this into context somehow, uh, to really,
253.36s - 258.56s |  you know, be able to judge what one star means. I'm from Baltimore, uh, so to put this in
258.56s - 264.30s |  context for me, I decided to find out what former residents think of the Baltimore jail after
264.30s - 270.54s |  they've been released. You heard it here first, folks. Uh, having any sort of business
270.54s - 276.14s |  relationship with CSC service works is less fun than jail. And not just any jail, Baltimore
276.20s - 283.20s |  jail. Alright. Now, yes, a picture is worth a thousand words, but words are also worth words.
283.20s - 287.70s |  Uh, specifically each word is worth one word. So I'm going to use some words to tell you
287.70s - 292.74s |  about the dark patterns that CSC uses to steal money from its customers. And for two
292.74s - 296.74s |  reasons. Because, right, in a minute I'm going to rob these guys and I don't want anyone to
296.74s - 301.78s |  feel guilty. Uh, that's the first reason. Uh, but the second reason is that this is a
301.82s - 306.60s |  security-minded audience and some of these are actually kind of good and would be funny if it
306.60s - 311.76s |  wasn't my money they're stealing. The first dark pattern is with the, um, refillable chip
311.76s - 316.64s |  cards, which have the same downsides that any, like, gift card have, right? Uh, if the
316.64s - 321.04s |  company stops supporting them or if they change the machines, if you lose it, if it's, you
321.04s - 326.04s |  know, you're fucked. Uh, these specific card machines, they have, uh, a separate refill
326.42s - 333.28s |  machine and some fine print in the lower right hand corner of the refill machine that says
333.28s - 337.96s |  that they do not accept one dollar bills. Oh, they accept one dollar bills. What they won't
337.96s - 343.00s |  do is put one dollar on your card and you have to call customer service and spend an hour on
343.00s - 347.76s |  the phone to get your one dollar back. Uh, the third bullet point is one of my favorites. So
347.76s - 352.60s |  CSC has to coordinate with the people who own the buildings to do any sort of maintenance
352.60s - 356.98s |  on the machines and they're upgrading from the card machines now to the app machines. So
356.98s - 360.84s |  they coordinated with my building and with several other people's buildings and you know
360.84s - 365.68s |  they picked a date, put up flyers, sent out emails, uh, let everyone know when the machines
365.68s - 369.92s |  are going to be changed over and then two weeks ahead of schedule at like 630 in the morning
369.92s - 374.92s |  they sent in a, a covert laundry ops team to steal the old card reader machines and replace
374.92s - 379.00s |  them with app machines. Now why is this a dark pattern? Because it strands like a hundred
379.00s - 383.14s |  people with a stupid card with money on it that they can't spend that they've already paid
383.14s - 388.44s |  for. Uh, and the only way to get it back now is to just snail mail the card to them and pray
388.44s - 391.94s |  because you don't even know how much money is on the card because they stole the fucking card
391.94s - 398.12s |  reader. Uh, the machine settings also don't work in your favor. When I originally did this
398.12s - 402.46s |  work the machines in my building were a dollar eighty five a piece and you could only put ten
402.46s - 406.96s |  dollars at a time on your card and if you ever tried to spend ten dollars one eighty five at a
406.96s - 412.00s |  time you can't. So when you lose that card they get some of your money. Now they've since
412.00s - 416.64s |  remedied this by raising the price to two dollars but also fucked them for raising the price.
416.64s - 422.74s |  Um, if you put quarters into the uh coin operated machines you lose the ability to change any
422.74s - 428.04s |  of the cycle options so if you try to change from say uh cold water to hot water and you
428.04s - 432.22s |  forget after you've put a quarter in you can't. You have to take your laundry, move it next
432.22s - 436.92s |  door, pay again and then run with the hot water. Alright. I don't know too much about the
436.92s - 441.88s |  apps because I would rather die than install an app on my phone. Uh, but I do know that the
441.88s - 446.70s |  apps will take your money at any time over cellular data but you can only redeem the credit if
446.70s - 452.26s |  the building's wifi is up and if CSC servers are working. Finally, the oldest dark pattern in
452.26s - 456.46s |  the world is to just have shitty customer service. Once they have your money they don't want
456.46s - 461.10s |  to answer the phone and they especially don't want to give it back. Uh, they also do not fix
461.10s - 464.78s |  the machines which you may recall if you were paying attention is the one thing that they're
464.80s - 468.84s |  supposed to do in exchange for your money. And the last one doesn't fit anywhere but I threw
468.84s - 475.52s |  it in there. They lost a re- a Rico lawsuit for doing Rico shit. So, I've concluded the CSC's
475.52s - 479.28s |  business model is just stealing quarters from old ladies because what are you going to do
479.28s - 485.30s |  about it? Well, what am I going to do about it? This is me. Um, why do I care so much? So,
485.30s - 490.26s |  I've been in the industry for like 20 years doing programming, system administration,
490.26s - 494.20s |  hacking. I was one of the teenagers who got in trouble for taking down MySpace. These
494.22s - 499.92s |  days I do more benign stuff. I, uh, file CVEs against open source projects and contribute
499.92s - 505.96s |  fixes. Uh, I'm a Gen 2 Linux developer. I have a PhD in math. Doesn't matter. Nobody cares.
505.96s - 511.00s |  All you need to know about me is that I really hate being taken advantage of. Especially when
511.00s - 516.20s |  it's for a small amount of money. And especially when the implication is that I just won't do
516.20s - 520.44s |  anything about it. Now, this is DEF CON. So, there are a lot of people here who are way
520.44s - 525.46s |  better than me with electronic thing it, uh, what's it's and thingamabobs. However, I do
528.32s - 534.32s |  have a very particular set of skills. So, I'm going to use those skills to give myself a
534.32s - 541.29s |  refund. Why? So, first we gotta get into these things, right? These are the dryers. That's the
545.95s - 551.13s |  front view. That's the top view. Those are Phillips screws. And this is a Phillips
551.13s - 556.15s |  screwdriver. This is a 20 minute talk. I'm not sure what you expected. The washers are a
558.61s - 562.59s |  little bit harder, right? There's a tubular lock on the front of that thing and so we've got
562.59s - 568.09s |  some options. The first option, hardcore mode. If you know how to pick tubular locks, that'll
568.09s - 572.69s |  get you right in. But there's a problem with this. Now, there's an entire village of people
572.69s - 576.63s |  here who would love to discuss the intricacies of lock pick along with you and then they
576.63s - 580.73s |  would tell you not to pick any lock that doesn't belong to you. I'm not one of those
580.73s - 586.75s |  people. Follow your heart. Do what the fuck you want to. Now, we can sidestep that issue
590.35s - 594.65s |  though. Right? Inside the door of the washers there's a sticker that says Alliance Laundry
594.65s - 600.05s |  Systems. Now I guess they supplied these machines to CSC before the retrofit. Uh and I might
600.05s - 604.53s |  have got the idea to just type in Alliance Laundry Systems into Google. I'm not sure if you
604.53s - 609.83s |  can read the fine print. Uh but that says Alliance Washer Dryer Key that I've typed into the
609.83s - 616.47s |  search box and you only get one result and it's a key that costs $5.80 which I am more than
616.47s - 621.49s |  willing to risk to to look cool up here on stage. And what do you know? Not only does it
625.49s - 631.75s |  open this one, it opens them all. Now, once we're inside of course we have to actually give
631.75s - 638.69s |  ourselves a refund somehow uh via subterfuge. And so there are some caveats here. Right?
638.69s - 643.69s |  Chronologically this is where all the things that don't work would go in my story. Um and
643.69s - 647.57s |  once you have physical access to a machine, hacking it really shouldn't be that hard. But
647.57s - 651.31s |  there are some extra constraints here that make it difficult. First is that I can't break the
651.31s - 655.41s |  machine. Right? My neighbors use these machines to do laundry. I like my neighbors and you
655.41s - 659.01s |  really don't want to be on the bad side of someone who wakes up at 5 in the morning on a
659.01s - 664.15s |  Sunday to do laundry. Really though, I can't break the machine because if I break the
664.15s - 667.19s |  machine the property manager is going to get involved. They're going to call the board. The
667.19s - 671.39s |  board is made up of my neighbors. I like my neighbors. I don't want to waste their time. But
671.39s - 676.83s |  seriously, I can't break the machines. If I break the machines the $2 joke doesn't go so
676.83s - 684.66s |  well. What are they going to do? Sue me for $1,000? Sticker to the rescue. Inside the
684.66s - 688.64s |  dryers we see uh they have another Alliance laundry system sticker. So all these things come
688.64s - 692.70s |  from the same place. And you can't find too much information about the circuitry from
692.70s - 697.20s |  Alliance's website. They don't have pictures of like the coin drop mechanism or anything. But
697.20s - 703.64s |  Google does. Oh, sorry. There's only three wires coming out of this thing. How hard could it
703.64s - 708.58s |  be? And we see inside the washers that we've got the same three wires going into a bullet
708.58s - 714.58s |  connector. This is a paper clip with some tape around it. Just shove it in there. Ding ding
714.58s - 719.58s |  ding. Each time you short it out you get credit for putting one quarter in. In seconds. It
720.18s - 726.43s |  doesn't take weeks in snail mail to get a refund. Ha. The dryers are a little bit more
729.13s - 734.37s |  difficult because uh you see there's no bullet connector here. But again uh when faced with
734.37s - 739.21s |  adversity I find that it's always best to follow your heart and just do what the fuck you want
739.21s - 744.21s |  to. And strip a little shielding off those fuckers. And just touch them together. It works.
744.73s - 749.73s |  It's fine. Ding ding ding. And you get a refund from the dryers too. Now the next slide says the
754.61s - 759.55s |  end but it's not really the end. Because of course there's a live demo that's pre-recorded
759.55s - 769.53s |  uh but it's live. I just, it was live when I pre-recorded it. But it's live. What's up Defcon?
769.53s - 778.02s |  We're doing full screen. This is important. And this is laundering money. I'm here in my
778.18s - 782.18s |  building's laundry room at three in the morning on a Monday because I'm about to do absolutely
782.18s - 787.18s |  nothing wrong. Before we get started I've got to attach my phone to my Yopro so that you can
790.52s - 799.07s |  see what I can see. So bear with me for a second. That should just about do it. And since I've
801.55s - 807.55s |  already told you how this is gonna go uh we can just jump right into the demo. So over here on my
807.57s - 813.93s |  right we see a bunch of stupid washing machines and the first thing that we need to do to get some
813.93s - 820.55s |  free money out of these guys is to get into this service panel. Now obviously the best way to do
820.55s - 825.55s |  that in terms of how cool it makes you look is with this Covert Instruments tubular lockpick.
827.95s - 833.35s |  This is much more than a lockpick. It is actually an impressioning tool which means that you can
833.35s - 838.39s |  save your game once you've picked the lock. And that is why I do not have to stand here for 30
838.39s - 845.03s |  seconds and jiggle this thing back and forth uh to get the lock open. Uh however you'll notice I've
845.03s - 849.77s |  closed the panel back up and the reason for that is because as it turns out in the state of
849.77s - 854.77s |  Maryland at least it is illegal to use a lockpick to commit a crime. And so instead I have here
856.57s - 861.59s |  with me my great hundred key that I have purchased from Alliance Laundry Systems and what I'm
861.59s - 866.59s |  going to do is open the service panel with this guy. Once it's open we can immediately identify
871.91s - 876.91s |  the trio of red black and white wires running in from the coin drop mechanism and to perform the
878.77s - 884.25s |  cha-ching operation I'm going to be using this custom hacking tool here that you can find at any
884.25s - 889.25s |  Radio Shack. And all I'm going to do is insert the ends of it into the holes occupied by the
889.25s - 899.53s |  red and black wires. Eventually the machine will go apeshit to indicate success and you can close
904.41s - 913.22s |  up the service panel once more. And that's it for the washers. Now we move on to the dryers. The
917.46s - 922.46s |  dryers are protected by phillips screws and you will need a phillips screwdriver to remove them.
922.46s - 927.46s |  If you do not have a phillips screwdriver I think you can find these on ebay but once we've got
933.67s - 938.67s |  them removed the service panel comes right off and we once again identify the trio of red black and
943.01s - 948.01s |  white wires. On this machine it looks like someone was kind enough to pre-strip the red and black
949.01s - 954.01s |  wires and cover them with electrical tape. So all we have to do is remove this electrical tape and
960.45s - 965.45s |  then to perform the cha-ching operation on the dryer I'm just going to smoosh the exposed portion
967.09s - 972.09s |  together with my hands. And you'll notice I'm doing this with my bare hands and the reason for that is
973.05s - 978.05s |  because it's actually safe to touch any exposed wires with your bare hands. Uh it's called grounding
980.53s - 1008.57s |  look it up. Once again the loud screeching noise from the siren indicates that everything is fine. We
1011.61s - 1016.61s |  can replace the electric tape and close the panel back up. Wait there's more. I'm so glad this
1016.73s - 1045.79s |  happened. Defcon it's your boy back in the LR with an update. It is August 5th a few days before the
1058.75s - 1063.55s |  conference and with all the publicity surrounding this talk it looks like CSC has upgraded the
1063.55s - 1067.39s |  security on the machines in my building. Now it's not clear if this is a company wide
1067.39s - 1072.29s |  initiative or just a personal fuck you to Michael Arlitsky and so to be on the safe side uh we're
1072.31s - 1077.31s |  going to bypass it. The first thing that they've done is to replace the old shitty tubular locks on
1078.75s - 1084.15s |  the washers with the new shitty tubular locks. So unfortunately the great hundred key will no
1084.15s - 1089.23s |  longer open that up uh but if you have the covered instruments tubular lock pick of course uh it
1089.23s - 1094.23s |  comes right open. So fuck out of my face with that shit. On the dryers what they've done is to
1095.21s - 1100.21s |  replace the Phillips screws with these security Torx screws. Now the nice thing about these is that
1103.55s - 1109.03s |  if you don't have the bit you can buy a set on Amazon for about $6.99 and the nice thing about
1109.03s - 1114.29s |  $6.99 is that if you don't have it what you can do is completely ignore everything that I just
1114.29s - 1119.97s |  said and get a standard flat head screwdriver and just shove it in that bitch. Now what I want you
1119.97s - 1124.71s |  to do next is close your eyes and acknowledge that through the power of Christ anything is
1124.71s - 1141.88s |  possible and then slowly begin to turn the screwdriver. Inside the only changes that have been
1141.88s - 1146.86s |  made are that someone has gone just fucking bonkers on this thing with a roll of electrical
1146.86s - 1151.86s |  tape but that should not stop anyone who is not an idiot. Happy DEF CON. This is all completely
1164.26s - 1165.16s |  legal thank you.


url: https://www.youtube.com/watch?v=5S_1uHLn4SI
type: video
author: DEFCONConference
ingestion_date: 2024-11-10
transcription_model: large-v2
chunking_status: completed
title:
DEF CON 32 - Finding & exploiting local attacks on 1Password Mac desktop app - J. Hoffman, C. Morgan

content:
{
  "webpage_url": "https://www.youtube.com/watch?v=5S_1uHLn4SI",
  "title": "DEF CON 32 - Finding & exploiting local attacks on 1Password Mac desktop app - J. Hoffman, C. Morgan",
  "description": "Password managers are routinely granted a massive level of trust from users, by nature of managing some of their most sensitive credentials. For any noteworthy password manager, the encryption standards for user data are well understood and highly scrutinized. What is less understood is the attack surface of the software itself. This presentation explores the local security of the 1Password MacOS desktop application and answers the question of \u201chow safe are my passwords if my computer is infected or otherwise compromised?\u201d.\n\nThis talk will cover the outcome of our research into 1Password, presenting several different attacks to dump local 1Password vaults. This includes describing multiple application vulnerabilities and security weaknesses we identified in the 1Password MacOS desktop application, as well as discussing the inherent limitations in its usage of IPC mechanisms and open source software. Additionally, we will discuss novel vulnerabilities found in Google Chrome that aided our exploitation of the 1Password browser extension.",
  "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
  "duration": 2330,
  "channel": "DEFCONConference",
  "uploader": "DEFCONConference",
  "upload_date": "20241016"
}

3.25s - 8.25s | This text was transcribed using whisper model: large-v2

 Hello. Hi everyone. Uh, I'm Colby Morgan. I'm here with Jeffrey Hoffman. We're both
9.59s - 14.77s |  offensive security engineers at Robinhood. And earlier this year we did some research into
14.77s - 20.67s |  1Password for Mac. Uh, with the goal of identifying local attacks to dump vaults. And
20.67s - 23.91s |  today we're gonna be sharing with you the outcome of our research and some of the uh,
23.91s - 31.03s |  vulnerabilities we discovered. So, why look at 1Password for Mac? Um, well if you use
31.07s - 35.17s |  1Password, it's probably a place where you have some of your most sensitive credentials.
35.17s - 38.71s |  And we know that APTs, red teamers, one of the first things they're gonna do when they
38.71s - 44.21s |  land on an endpoint is look for credentials. Things like uh, cloud credentials, browser
44.21s - 48.25s |  cookies, um, passwords, API keys, things they might be able to use even if they were
48.25s - 54.39s |  evicted from the laptop uh, to retain um, some sort of access or even more access. Um, if
54.39s - 58.79s |  they were to be grepping around uh, looking for files that have the word password in the
58.79s - 64.29s |  file name, one thing they would find is this 1Password dot SQLite file. And this is a
64.29s - 69.69s |  SQLite database uh, it's encrypted but it contains your uh, 1Password vault contents. So
69.69s - 74.53s |  that file itself wouldn't get you much. Um, so we're gonna talk about some attacks later
74.53s - 79.97s |  that might be able to get you some of that content. Um, so to talk a little bit about the
79.97s - 84.55s |  local security model of 1Password on Mac. Um, when you log in with your password to
84.55s - 89.21s |  1Password, uh, it doesn't actually store your password. It's not even sending your password to
89.21s - 95.35s |  the server. Instead what's going on is um, your password along with a locally stored secret
95.35s - 99.93s |  key are combined cryptographically to generate a master unlock key uh, which is now
99.93s - 103.97s |  referred to as an account unlock key. You'll hear us throughout this talk refer to it as a
103.97s - 110.97s |  muck because that's how we saw it referenced in the code. Um, so we use SSO meaning uh, we
110.99s - 116.83s |  don't have a password for 1Password and how it works in this case is uh, you authenticate and
116.83s - 122.67s |  then you get a device key um, and you receive a credential bundle from 1Password's API and
122.67s - 127.17s |  the device key is stored in your Mac keychain and that is used to decrypt the credential
127.17s - 132.65s |  bundle that contains that muck that I referenced. So the muck is basically the keys to the
132.65s - 138.65s |  kingdom. It allows you to decrypt that SQLite database where your vault content is stored.
138.67s - 144.11s |  Malware on your machine uh, without a SIP bypass on Mac or a way to inject into processes
144.11s - 149.31s |  arbitrarily shouldn't be able to get your vault contents uh, decrypted um, without in the
149.31s - 154.85s |  SSO case unrestricted keychain access and in the non-SSO case um, access to your 1Password
154.85s - 160.59s |  password meaning something you know um, and your locally stored secret key. Once you have
160.59s - 165.33s |  the muck uh, the full decryption process um, isn't super straightforward but it's fully
165.35s - 171.05s |  explained in this blog post from David Shoots aka Darthnal uh, where he also had some code
171.05s - 176.90s |  that we referenced for the decryption and so this was really helpful in our research.
176.90s - 180.94s |  Yeah, so it's generally understood that if somebody has full control over your machine
180.94s - 185.04s |  they're gonna be able to access whatever data it protects in some way shape or form. That
185.04s - 189.08s |  being said, 1Password still cares about creating meaningful security boundaries wherever
189.08s - 194.82s |  possible and on Mac OS specifically with things like SIP and code signing uh, there is a
194.84s - 199.20s |  possibility to create these significant boundaries. Uh, these are just some CVEs that are Mac
199.20s - 203.74s |  specific uh, we took a look at these CVEs when we first started our work and they indicated
203.74s - 208.52s |  to us that we were thinking about the right things and looking in the right places. So when
208.52s - 212.18s |  we started out looking at 1Password we didn't really look at the app, we looked at the
212.18s - 217.92s |  framework that it was built on. Uh, we looked at the framework that it was built on uh, which
217.92s - 222.92s |  is Electron and so Electron is a chromium and node based framework. It has a pretty well
222.92s - 227.90s |  defined attack surface, the standard attack you'll see is somebody trying to get XSS and
227.90s - 232.64s |  convert that into RCE or something else but 1Password is an app, doesn't have a whole lot of
232.64s - 238.50s |  opportunities to inject user controlled content. Uh, the XSS to RCE route is usually
238.50s - 243.22s |  considered uh, a remote attack but because we're local attackers we have the ability to
243.22s - 247.72s |  consider local things and one of the local security controls in Electron is something
247.72s - 253.00s |  called fuses. Uh, these fuses control the behavior of 1Password in certain situations. I'm
253.00s - 256.96s |  not going to read all of these off to you but I'll tell you that 1Password correctly set all
256.96s - 261.80s |  of these. Uh, but some of the interesting ones that, that you might take a look at are things
261.80s - 267.64s |  like only load app from ASAR. Uh, if this had been enabled or rather disabled, uh, you might
267.64s - 272.78s |  be able to proxy the app and inject JavaScript that way. And the enable node CLI inspect
272.78s - 277.76s |  arguments fuse is also pretty interesting. It controls whether or not this Chromium flag
277.76s - 283.12s |  called the remote debugging port works. If that had been enabled in 1Password, we would
283.12s - 288.16s |  have, uh, started it with this flag, uh, connected a Chromium debugger to it and from there
288.16s - 292.34s |  you can see all the pages that the app has. You would have been able to inject, uh,
292.34s - 298.54s |  JavaScript or change the currently running JavaScript in any of those pages. So, once we
298.54s - 303.24s |  saw that all the electron, uh, attack surface was, was considered and wasn't going to be
303.24s - 308.24s |  viable, we started looking at how the app functioned and there were 3 main ways that we saw
308.24s - 314.98s |  to get passwords out of the desktop app and those were the SSH agent, the CLI, and the
314.98s - 320.12s |  browser extension. Now the SSH agent and the CLI were disabled by default, but the browser
320.12s - 325.64s |  extension is enabled by default. And so once we saw this, we installed the browser extension
325.64s - 329.44s |  and as long as the app is unlocked, if you click the browser extension, it's going to
329.44s - 332.60s |  seamlessly sync all of your passwords and they're going to show up without any kind of
332.60s - 336.34s |  re-authentication. And so as soon as we saw that behavior, we started trying to figure out
336.34s - 342.24s |  how we could abuse that. And so we briefly considered the idea of maybe if we had some
342.24s - 347.08s |  cool RCE and sandbox escape, we could control Chrome and try to do that remotely. Those are
347.08s - 351.56s |  prohibitively expensive and again, because we're local attackers, we have much less
351.58s - 355.88s |  expensive options. And uh, that fuse that I mentioned before that controlled the remote
355.88s - 361.32s |  debugging port, that's a Chromium flag. And so while Electron can turn that off, Chrome and
361.32s - 366.50s |  other Chromium based browsers can't. It's just how they work. So as soon as we realized
366.50s - 371.40s |  that, we found the JavaScript responsible for doing the syncing. It's that green URL that's
371.40s - 376.40s |  on screen now. And we opened it up, looked at the console and we saw some log messages
376.40s - 381.14s |  related to native core. This is a native messaging host and we'll re- we'll revisit that in a
381.16s - 387.76s |  little bit. Uh, but essentially what we did is we read the JavaScript from that page, uh,
387.76s - 392.80s |  beautified it, modified it to do that syncing behavior immediately and with the remote
392.80s - 397.74s |  debugging port, we just injected our new extension into that page and uh, everything was
397.74s - 404.04s |  synced and we were able to extract passwords and decrypt them. And so this is our mental
404.04s - 408.62s |  model of 1Password. We'll revisit this image a couple of times during the presentation. Uh,
408.66s - 413.30s |  but what's important to note here is that we didn't really have a vulnerability in 1Password
413.30s - 422.15s |  exactly. We're kind of just abusing how Chrome and extensions function. Yeah, so sticking
422.15s - 426.53s |  along the path of attacking the browser extension, um, one question we were asking
426.53s - 430.27s |  ourselves is would we be able to spoof the legitimate 1Password extension with our own
430.27s - 436.23s |  extension or make changes to the 1Password extension and still uh, still have it do that
436.27s - 442.27s |  automatic syncing process Jeff, uh, referred to before. Um, so if you're not familiar, uh, you
442.27s - 447.01s |  can start Chrome from the command line, uh, along with this dash dash load extension flag to
447.01s - 451.75s |  load extensions from disk, um, which are referred to as unpacked extensions. So what we
451.75s - 456.59s |  were wondering is if we could load a modified version of 1Password from disk, uh, and have it
456.59s - 460.79s |  do that syncing process, obviously the modified code we would add would be JavaScript that
460.79s - 464.27s |  makes it do what we want, such as like taking those passwords after they sync and sending
464.27s - 469.11s |  them to a remote server. Um, there were a couple of kind of challenges that, uh, made this
469.11s - 475.41s |  not straightforward. So, first, uh, the, when the extension talks to the Mac OS 1Password
475.41s - 479.05s |  app, uh, the 1Password app is going to verify that the connection is coming from the
479.05s - 484.49s |  legitimate extension's extension identifier. Um, and then further, we have a managed Chrome
484.49s - 489.09s |  policy that prevents, uh, loading unpacked extensions altogether, meaning we couldn't
489.09s - 494.79s |  launch Chrome with that CLI flag. Um, and also, all extensions are disallowed by default, with
494.79s - 500.77s |  only some being allow listed. So we started trying to tackle these challenges one by one. Uh,
500.77s - 506.03s |  first, uh, spoofing the extension identifier. Thankfully this was extremely straightforward
506.03s - 511.31s |  because Google tells us exactly how to do it on their browser extension, uh, security FAQ
511.31s - 517.59s |  page, where they basically explain that they don't consider this a bug. And, um, basically,
517.61s - 523.19s |  um, every browser extension has a manifest.json file with a bunch of key entries. Um, one of
523.19s - 528.93s |  those is called key, and, um, basically, if you take that key entry and put it in your own
528.93s - 533.69s |  extension's manifest.json file, the extension IDs will be the same because the extension
533.69s - 539.07s |  identifier is derived from that key entry. So we can just grab the legitimate extension's, um,
539.07s - 544.01s |  key identifier, put that on our own extension, and then it'll match. So now we're left with
544.03s - 548.53s |  the challenges from the managed Chrome policy. Uh, we basically started just trying to
548.53s - 554.11s |  figure out how it worked and how it is enforced, and we came across this policy directory in
554.11s - 559.37s |  Google Chrome's, um, application support directory. And we decided to just remove it and see
559.37s - 565.51s |  what happens. Um, so, if you delete that policy directory and you start Chrome, Chrome is
565.51s - 569.29s |  actually smart enough to realize it should be there and it'll recreate it and it'll start
569.29s - 574.47s |  reinforcing itself. However, um, as I mentioned before, we were prevented from starting
574.47s - 578.67s |  Chrome with the dash dash load extension flag, but what you can do is delete the policy
578.67s - 583.05s |  directory, start Chrome with the dash dash load extension flag, which will load your extension,
583.05s - 588.25s |  um, and it, that works because the policy directory doesn't exist at the time Chrome starts,
588.25s - 592.15s |  but then Chrome will recreate that directory and start reinforcing itself, um, but your
592.15s - 596.33s |  extension is already loaded, um, but Chrome will start going through and saying which
596.33s - 601.13s |  extension should be allowed, if it's not allowed, remove it. Um, but thankfully, it's only
601.13s - 605.91s |  doing a shallow check of comparing the extension identifier to the allow list, and we're
605.91s - 610.41s |  spoofing the legitimate extension's extension identifier, so it doesn't remove our modified
610.41s - 617.41s |  1Password. Um, so that overcomes all of the challenges we had, and here is us just, uh,
617.41s - 622.95s |  adding a POC.html file to the extension. We have the legitimate extension identifier, so
622.97s - 627.45s |  our passwords do sync when 1Password is unlocked, um, and it pulls in all our passwords as
627.45s - 631.95s |  expected, and obviously, we could have also added, um, edited the JavaScript or added
631.95s - 638.40s |  JavaScript to, uh, make it grab those credentials and send them elsewhere.
638.40s - 643.04s |  So earlier, I brought up the term native messaging host, and what that is is a binary that
643.04s - 647.70s |  acts as a middleman between, uh, Chrome or other browsers that support this protocol and
647.70s - 653.92s |  the desktop app, and, uh, how it normally works is Chrome will launch it, it will, uh, hold
653.92s - 658.08s |  pipes to the standard in and standard out of that process, and then send JSON messages back
658.08s - 664.28s |  and forth. And if you're familiar with IPC on macOS specifically, that's a big deviation
664.28s - 669.02s |  from the norm. Normally, things use XPC, and part of the reason they do that is because when
669.02s - 673.60s |  you use XPC, you have something called an audit token available to you, and you can use
673.60s - 677.54s |  that audit token to securely look up the code signing information of the process sending
677.54s - 682.34s |  the message. Because there's no audit token, and there's no way to attach it to this method
682.36s - 688.84s |  of IPC, uh, there's not really a great way for 1Password to go and verify the code signing
688.84s - 693.04s |  information of whatever is sending that JSON. And they seem to know about this and, and
693.04s - 698.48s |  still try to implement, uh, a parent PID check, and essentially what they did was they would
698.48s - 702.78s |  call getPPID, go look up the code signing information of the parent, and check it a bunch
702.78s - 708.82s |  of, uh, check it across the list of accepted and trusted browsers. But that's a generally
708.84s - 714.18s |  understood, you know, anti-pattern on XPC. The reason they tried it here is, again, there's
714.18s - 720.82s |  no audit token, so they can't really do much better. Uh, so, going forward, we're going to
720.82s - 724.42s |  talk about dumping, and instrumenting, and hooking, and our tool of choice for that was
724.42s - 728.26s |  Frida. We're going to release some Frida scripts for people to look at other native
728.26s - 735.30s |  messaging hosts. Um, what you can see on screen here is me hooking the read and write
735.32s - 739.76s |  methods for the native messaging host, and this is how we figured out what JSON to send
739.76s - 745.36s |  when we actually wrote our exploit. And so, like I said, it's a well understood problem
745.36s - 750.04s |  that if you're doing XPC, you should be using the audit token because if you try to look up
750.04s - 754.90s |  the code signing information by PID, what can happen is someone will send a message, exec
754.90s - 760.54s |  into the actual process that was supposed to send that message, and by the time the PID is
760.56s - 765.76s |  used, the co-signing information will be that of the trusted process and not the malicious
765.76s - 770.26s |  one that sent that initial message. And so we considered doing that, but we actually don't
770.26s - 775.04s |  have to, there's a, a bunch better way. What you can do is create this process tree of
775.04s - 780.04s |  three processes, where the parent holds the standard in and standard out write pipes, the
780.04s - 785.24s |  child execs into a trusted browser, and the grandchild is actually the browser support
785.28s - 792.02s |  process. And when it goes to do that, uh, get PPID check, we've satisfied that condition. So
792.02s - 796.16s |  this is what the exploit looks like in process, and once you've set up these pipes, you can
796.16s - 802.66s |  just send the JSON that we showed, uh, in the Frida slide, and the browser support process
802.66s - 809.00s |  will happily return back to you the muck. And so here's our mental model again. We've
809.00s - 812.88s |  totally cut Chrome and the need for the actual extension out of the equation, this time
812.88s - 817.16s |  we're attacking browser support. And we were satisfied with this, this is definitely a
817.16s - 821.96s |  valid way to get the muck and, and dump all your vaults, but we thought, well what if we
821.96s - 825.80s |  could also cut out browser support? So we started looking at the browser helper XPC
825.80s - 832.78s |  server. And I've talked a lot about XPC already, I'll spare you, but they do everything
832.78s - 836.52s |  correctly here that you would expect. They take the audit token, they get the co-signing
836.52s - 841.60s |  information in a secure way, and they check the team identifier and bundle ID.
841.60s - 846.54s |  Unfortunately, what they forgot to do was consider, uh, the hardened runtime, or the
846.54s - 850.58s |  co-signing flags of the client. And so, if you're not familiar with hardened runtime,
850.58s - 855.58s |  essentially what it is, is a co-signing flag that says to macOS to, uh, block all different
857.72s - 863.62s |  manners of injecting, uh, code dynamically, and that makes sense, right? If your security
863.62s - 869.16s |  model is going to be built around co-signing, you don't want ways to have untrusted code
869.16s - 874.10s |  running in a process without invalidating that signature. And so that was introduced in
874.10s - 878.76s |  September of 2018, and it would have been literally impossible for them to include that
878.76s - 883.78s |  control, and they were a company before 2018. And so we thought, maybe if we could find
883.78s - 892.14s |  some old clients, we could do this injection and satisfy their checks. Um, yeah, so like
894.20s - 898.24s |  Jeff mentioned, what they were doing is checking the code signature information of the XPC
898.24s - 901.84s |  client to see if it had a matching team identifier. They weren't checking for the
901.86s - 906.30s |  hardened runtime, so we knew what we needed to find. An old version of 1Password that
906.30s - 911.78s |  matched these conditions. Uh, we couldn't find any official source from 1Password that was
911.78s - 916.32s |  offering old versions of clients, so we tried to get a little bit creative, and what we did
916.32s - 920.82s |  is tried to enumerate subdomains of 1Password dot com by looking at certificate
920.82s - 926.62s |  transparency logs. And one of the subdomains we found was c dot 1Password dot com, and it
926.62s - 931.30s |  appeared to be some sort of web cache, we didn't really know what it was used for. Um, so we
931.32s - 937.00s |  used Google dorks to see what files Google had indexed, um, on the URL, and we found a bunch
937.00s - 941.46s |  of old versions of 1Password for all different operating systems. Um, one of the oldest ones
941.46s - 946.58s |  we could find for Mac was version 6.5.2, so we pulled that down, and it looked like a
946.58s - 950.74s |  perfect candidate because it had the correct team identifier, even though it was very old, um,
950.74s - 956.04s |  and it didn't have the hardened runtime. So we figured we likely could inject into it. If
956.06s - 961.60s |  you have a binary that is not signed with the hardened runtime, um, the easiest way to
961.60s - 967.20s |  inject into it is to write your own dynamic library, uh, that runs whatever code you want in
967.20s - 972.34s |  its constructor, and then you load that dynamic library, uh, as you launch the program using
972.34s - 977.34s |  the dial D insert libraries environment variable. This is very similar to LD preload on
977.34s - 982.22s |  Linux, uh, for loading shared objects. Um, basically you can, it'll be the first dynamic
982.24s - 987.04s |  library that the program loads, and it'll run your code pretty much immediately. So we knew
987.04s - 991.82s |  that the XPC server exposed one method called echo, which just echoed a string back to you.
991.82s - 995.66s |  So as a proof of concept, we just wrote our dynamic library to send the string hello world to
995.66s - 1000.76s |  the server. Um, as you can see here, uh, once our code runs, it does send that string, we get a
1000.76s - 1005.76s |  successful response, and this is great because it proves that, um, the XPC server validation
1005.76s - 1011.74s |  was passed by our client. So obviously we didn't want to just echo strings back to
1011.76s - 1016.16s |  ourselves, um, so we looked at what other methods were available, and the one that performed
1016.16s - 1020.54s |  actually interesting functionality was called send to main, uh, where basically a message
1020.54s - 1025.50s |  gets sent through browser helper and then to the main application, um, and gives you a
1025.50s - 1030.84s |  response. Um, so what we tried doing was just sending the same messages that the browser
1030.84s - 1035.58s |  extension was triggering, um, but unfortunately we were just getting no responses
1035.58s - 1040.52s |  whatsoever when we would do that. It would just totally hang. Um, so through some debugging,
1040.54s - 1044.44s |  we found that when our connection was created, there was a connection object and there was a
1044.44s - 1050.48s |  boolean flag on it of is browser equals false, and we figured this might be problematic. Um,
1050.48s - 1055.18s |  and it seemed to be getting set to false because our client's package ID, one password six in
1055.18s - 1060.36s |  this case, uh, didn't match what was expected being the browser support process. Um, so we
1060.36s - 1065.42s |  didn't give up there because we knew that the one password CLI tool, um, called OP, uh, we
1065.46s - 1070.54s |  knew that that also communicated with the same XPC server using XPC messages, so we wanted to
1070.54s - 1076.94s |  understand how that worked. To do so, we pulled down this XPC spy, uh, open source program for
1076.94s - 1082.38s |  inspecting XPC traffic that a process is sending, and we started using, uh, the OP tool, uh,
1082.38s - 1087.28s |  while looking at the XPC traffic, and we saw that it was sending, uh, similar XPC traffic, but,
1087.28s - 1091.72s |  um, it was including this client info JSON blob. Um, and I'll explain kind of the
1091.72s - 1096.40s |  significance of some of the values you see, uh, highlighted in red here in the coming slides,
1096.40s - 1101.46s |  but, um, the point is, we were able to send messages in the same way the one password CLI
1101.46s - 1106.80s |  was and mimic the functionality of the one password CLI. And this was nice, um, but there's
1106.80s - 1112.44s |  quite a few limitations that we weren't satisfied with. First, CLI integration is disabled by
1112.44s - 1117.48s |  default. So, uh, if you're sending these messages and the CLI integration is disabled, nothing's
1117.48s - 1123.12s |  gonna happen. Um, and, you know, most people probably haven't turned it on. So, we were
1123.12s - 1128.46s |  wondering, could we just find where the settings are stored and enable it ourselves? And we
1128.46s - 1133.60s |  found this settings dot JSON file. Um, we opened it, we saw a bunch of Boolean flags for various
1133.60s - 1138.36s |  settings, one of them being CLI integration. Um, but we also saw that every setting had a
1138.36s - 1142.34s |  corresponding auth tag, which looked like a big random string that we assumed was
1142.34s - 1146.34s |  protecting the integrity of the settings through some authenticated hash or something like
1146.34s - 1151.40s |  that. So, we assumed this wasn't gonna work, but we tried it anyway. Switched to CLI
1151.40s - 1155.34s |  integration from false to true and it just worked. So, it seemed like these auth tags were
1155.34s - 1160.58s |  here, probably secure, but they just weren't being enforced yet. So, we can enable CLI
1160.58s - 1166.62s |  integration, great. Uh, but there's still, uh, some serious limitations here. Um, the first
1166.62s - 1171.00s |  being that when you use the one password CLI, you have to approve the session. So, if you've
1171.00s - 1174.30s |  used it before, you're probably familiar with this image on the left, where it's gonna say
1174.36s - 1180.06s |  your terminal is requesting access to one password, do you wanna authorize it or not? Um, and
1180.06s - 1186.64s |  so what we saw was that your session basically gets created based on the XPC messages that
1186.64s - 1191.88s |  get sent when you run the, uh, OP tool. And the session identifier is basically based on
1191.88s - 1196.88s |  the PID of your shell, being the parent process that calls the tool, and the start time of
1196.88s - 1202.06s |  that parent process. And so, it'd be pretty easy for malware on the machine to basically
1202.08s - 1207.66s |  look for any shell that spawns the OP CLI, figure out the PID of that shell, and then grab the
1207.66s - 1211.26s |  start time of the process, and then you'd know a valid session identifier and you could
1211.26s - 1216.26s |  ride along with it, sending whatever messages you want with an approved session. However,
1216.26s - 1220.30s |  um, this, you know, obviously isn't optimal either because it requires that there's a
1220.30s - 1224.64s |  legitimate user sitting there, really using the one password CLI, having an active session
1224.66s - 1229.66s |  that you could spoof alongside. So we kept digging here, um, we, uh, saw that in the
1232.24s - 1236.90s |  client info JSON blob, there was a session type of CLI being sent when the CLI was being
1236.90s - 1242.18s |  used, and, uh, we were wondering if any other session types might be supported. We
1242.18s - 1247.08s |  disassembled the browser helper binary to take a look, and we saw that it seemed like the
1247.08s - 1251.98s |  string of browser could also be a valid session type. And this was interesting to us because
1252.00s - 1258.00s |  we hadn't seen this string being sent by, uh, the extension. But, uh, we tried it anyway to
1258.00s - 1261.58s |  see what would happen. We removed all of those other session identifiers that I was
1261.58s - 1266.78s |  mentioning before, and it just totally worked. If one password was unlocked, we were able to
1266.78s - 1272.02s |  send a session type browser, request the muck, and we were able to grab it. So that kind of
1272.02s - 1276.96s |  completed the chain of, um, injecting into an old client, sending XPC messages, and grabbing
1276.98s - 1281.98s |  the muck. Um, so you may have noticed that every issue we've discussed so far requires that
1284.08s - 1289.18s |  one password is unlocked. Um, this next and final issue we're gonna discuss does not require
1289.18s - 1293.46s |  that one password is unlocked. And it focuses on how biometric authentication is
1293.46s - 1298.26s |  implemented. If you, uh, use one password on a MacBook, you're probably familiar with this.
1298.26s - 1303.80s |  You launch one password, you provide your fingerprint, it unlocks. Um, so a very common way
1303.84s - 1308.28s |  that macOS apps are gonna implement fingerprint checks is by using this evaluate policy
1308.28s - 1313.52s |  method. And it's pretty simple. You call it, prompts user for their fingerprint. If they
1313.52s - 1318.18s |  provide it and it's valid, it returns true, otherwise it returns false. So what we did to make
1318.18s - 1322.92s |  sure this method was in use, we hooked it with Frida, uh, used the code on the right, uh,
1322.92s - 1328.10s |  screenshot here, and just basically made it always return true no matter what. And so after
1328.10s - 1332.54s |  doing that, we launch one password, prompt it for our fingerprint. We could either hit cancel
1332.54s - 1337.54s |  or provide someone else's fingerprint, and the vault unlocks. So, um, you might be
1339.72s - 1342.98s |  thinking, well, you know, if you're a red team or you land on someone's laptop, you can't just
1342.98s - 1347.16s |  disable SIP and hook methods with Frida. And that's true, but this taught us something
1347.16s - 1351.76s |  really interesting, which is that even with an invalid fingerprint, somehow one password is
1351.76s - 1355.70s |  grabbing the muck and it is decrypting the vaults and displaying your passwords. So we wanted
1355.70s - 1360.96s |  to understand how that worked. Um, and also it's not obvious that this was gonna work just
1360.96s - 1366.50s |  hooking, um, methods with Frida, cause it is possible to protect items at the keychain level
1366.50s - 1370.14s |  with flags that indicate a fingerprint is required. And you wouldn't be able to just hook
1370.14s - 1373.74s |  something with Frida and make that work, cause the keychain would protect you with your
1373.74s - 1379.30s |  fingerprint. So that was not what was going on here. So how does it work? We figured the
1379.30s - 1383.38s |  keychain was involved in some way, so we opened the keychain access application and started
1383.38s - 1388.28s |  looking around, and we saw that when using biometric authentication, there were, uh, these
1388.30s - 1392.44s |  items, or there was a item in the keychain called biometric unlock, and it would be named
1392.44s - 1396.38s |  differently depending on if you were using SSO or not. And we saw that it could only be
1396.38s - 1400.48s |  accessed by one password, so if it wasn't obvious from the name, we knew it was, uh, being
1400.48s - 1406.76s |  used by one password for biometric unlock. Now, along with, um, the item in the keychain,
1406.76s - 1410.60s |  there were these comments that were added, which tell us that it's, the item here is
1410.60s - 1415.36s |  encrypted by the secure enclave. And this tracked because when we looked at what was stored
1415.38s - 1420.56s |  in the item, it was just a bunch of random ciphertext that we couldn't do anything with. So,
1420.56s - 1424.12s |  we were wondering where the secure enclave key entry is, because we couldn't see it in the
1424.12s - 1428.80s |  keychain access application, we couldn't query it with the security CLI tool, which is the
1428.80s - 1432.94s |  built-in, uh, macOS CLI tool for interacting with the keychain. Uh, so we wanted to
1432.94s - 1438.14s |  understand where it was, what was going on and how it was being used. What we did for this
1438.14s - 1443.44s |  was hooked the sec item copy matching, uh, method with Frida, and logged the parameters
1443.46s - 1448.46s |  passed to it and the return values from it. This method is used to query the keychain. And so
1448.46s - 1452.54s |  we wanted to trigger the fingerprint unlock and see what was, uh, being accessed from the
1452.54s - 1457.70s |  keychain by one password when going through that flow. And amongst other things, one of the
1457.70s - 1464.38s |  things we saw accessed was a sec key ref or a secret key reference. And, um, that's what was
1464.38s - 1469.46s |  returned. And in the query was an A tag or an application tag, uh, with those hex bytes you
1469.48s - 1474.98s |  see in the top red box. And if you decode those, you get the string of com agile bits one
1474.98s - 1479.58s |  password memory enclave key. So this seemed likely what we were, uh, looking for based on
1479.58s - 1485.92s |  the name, but we wanted to be sure. So, uh, we wanted to see if it was being used to decrypt
1485.92s - 1491.06s |  anything or decrypt that biometric unlock entry. Um, now secure enclave keys, it's not the
1491.06s - 1495.36s |  case that you just access them and then do what you wish with the bytes. Instead, you ask the
1495.36s - 1500.14s |  secure enclave to encrypt something, decrypt something, or sign something. In this case, it, we
1500.14s - 1504.40s |  thought it would be decrypting something. So we did the same thing with hooking sec key create
1504.40s - 1509.42s |  decrypted data, which is how you would ask the secure enclave to decrypt something. And we log
1509.42s - 1514.28s |  the parameters and the return value. And we do see that that's during the biometric unlock, that
1514.28s - 1518.72s |  sec key ref is being used. It's decrypting the ciphertext that was stored in the biometric
1518.72s - 1524.16s |  unlock entry. And in return, we get 32 bytes. Uh, so we thought 32 bytes, that's great. That
1524.16s - 1529.94s |  sounds like an AES key. Maybe that's the muck. But it turned out it was not the muck. And we
1529.94s - 1533.84s |  didn't really know what to do. So we just took these 32 bytes and started trial and error
1533.84s - 1538.84s |  trying to decrypt things. And, um, finally, we found an item in the encrypted SQLite database
1541.32s - 1548.02s |  called enc unlock key. And we were able to successfully decrypt it with AES. And, uh, in
1548.02s - 1553.36s |  return, we got the muck. So, now we fully understood the chain here of the memory
1553.36s - 1557.76s |  enclave key in the secure enclave being used to decrypt the biometric unlock entry in the
1557.76s - 1562.06s |  keychain. From there, you have a key that can be used to decrypt this entry in the SQLite
1562.06s - 1569.08s |  database. Then you have the muck, which can decrypt the rest. Um, so now we fully understand
1569.08s - 1574.38s |  this process, but obviously we can't just access the keychain, um, on behalf of 1Password
1574.38s - 1578.88s |  unless we have a way to inject our own code into 1Password and query the keychain. Because
1578.88s - 1582.48s |  those items can only be queried by 1Password. So now we're gonna talk about how we
1582.50s - 1590.85s |  accomplished getting that code injection. Yeah, so we needed to, uh, have the 1Password
1590.85s - 1597.09s |  bundle ID and we needed to not invalidate the code signature. And, uh, there's a relatively
1597.09s - 1602.75s |  old technique on Electron where you can inject, uh, essentially the main.js file isn't
1602.75s - 1607.47s |  included in the signature check and so you can just overwrite it. And to protect against
1607.47s - 1612.93s |  that, they added this enable embedded ASAR integrity validation fuse. And this was enabled
1612.95s - 1617.45s |  in the modern version of 1Password. Uh, but we implied the same thinking that we did to the
1617.45s - 1621.23s |  hardened runtime and went through all of our scraped old versions. And we found one that
1621.23s - 1625.63s |  was created either before or around the time when this flag was introduced and it had it
1625.63s - 1630.31s |  disabled. Even if we couldn't find that though, there was this additional GitHub security
1630.31s - 1634.67s |  advisory that was a bypass to this fuse. And, uh, you know, if we couldn't find either of
1634.67s - 1640.79s |  those, we probably would have gone and hunted for some similar bug ourselves. So we can
1640.81s - 1645.95s |  overwrite the main.js file, but that isn't good enough to do what we need to do. Uh,
1645.95s - 1649.51s |  essentially there are no functions in Node.js that we can call directly that will let us
1649.51s - 1653.93s |  read the keychain and use the secure enclave. And so we need to call these native functions
1653.93s - 1658.59s |  from native code. And because of the hardened runtime, which is applied to this version of
1658.59s - 1662.87s |  1Password, we can't actually do the typical thing where you'd write your own library and
1662.87s - 1668.37s |  have the Node.js code hand off to it. And so we took a step back and saw that in the
1668.39s - 1673.79s |  entitlements, uh, there was this Apple security allow JIT entitlement, which essentially
1673.79s - 1677.83s |  means that somewhere in the process it would be okay to have a read write execute page of
1677.83s - 1683.57s |  memory. Once we saw that, we realized a browser exploit would probably work. And so we
1683.57s - 1688.27s |  found an applicable CVE that was exploited in the wild. We did the standard browser exploit
1688.27s - 1693.21s |  things and got our JavaScript heap and read write primitives and used that to run shellcode
1693.21s - 1697.79s |  by corrupting a WASM object. I'll show you a video of that in a second. But I also wanted
1697.81s - 1701.57s |  to call out that because we're not in a renderer process, we can kind of cheat a little bit
1701.57s - 1706.17s |  with ASLR by just running VM map outside of the process and then directly writing all
1706.17s - 1722.49s |  the addresses we need to a file and then reading that file into our exploit. Alright, so
1722.49s - 1728.17s |  hopefully you see that, it's on the monitor. Yeah, so in this video we're just adding a
1728.17s - 1732.91s |  demo password that we're gonna extract. Uh, we're adding it right now, then we'll show you
1732.91s - 1737.65s |  that SIP is enabled. And then what we're gonna do is run a makefile. And that makefile is
1737.65s - 1741.83s |  gonna compile a C program that prints out some function addresses in the core foundation
1741.83s - 1746.27s |  library. The reason this is important is because we need those functions, uh, but we didn't
1746.27s - 1750.63s |  want to do the work of adjusting the ASLR ourselves. And these functions are in the shared
1750.63s - 1755.37s |  cache, so we can just print them out and the addresses are the same between processes. That
1755.37s - 1761.51s |  print is gonna be used to generate our final ASLR adjusted, uh, shellcode in JavaScript.
1761.51s - 1767.15s |  And we're gonna start the old version of 1Password. The reason we do that is because on the
1767.15s - 1771.85s |  first run, a deep signature check is gonna be performed. Uh, but we can just start it, kill
1771.85s - 1777.93s |  it, and then run it again with our modified main.js. And because this fuse is disabled and
1777.93s - 1785.57s |  only a shallow check runs, we can run our browser exploit. Yeah, and so there it is. We'll
1785.57s - 1823.73s |  show you that the password matches. I think it's up. No, I'm not good. Oh man, this is the
1823.73s - 1833.28s |  hardest part of the talk. There we are. Okay, so this is the doc- oh man. Okay, so this is
1837.18s - 1840.88s |  the diagram of what we essentially did. Uh, and I think it's important to call out here that
1840.88s - 1846.02s |  although obviously 1Password could have used that enforcement flag, uh, but for everything else
1846.02s - 1850.22s |  in order to get that native code execution and actually exploit it, we essentially just
1850.22s - 1854.46s |  waited. And you know, we scraped the old versions, but the vulnerabilities were in Electron
1854.46s - 1858.26s |  and- and Chromium and it can be really hard to secure that when you're releasing new versions,
1858.26s - 1865.32s |  uh, every day. And so here's a vulnerability summary of what we reported. Uh, this is a
1865.34s - 1869.72s |  little bit, uh, there are a few more vulnerabilities here than what's going to be on the next
1869.72s - 1874.88s |  slide. The reason that is, is because the initial XPC entry vulnerability is really what
1874.88s - 1879.50s |  needed to be fixed, not the other stuff that we talked about with the session writing. And we
1879.50s - 1886.61s |  also reported the, uh, Chrome managed policy bypasses to them, or to Google rather. And so
1886.61s - 1890.81s |  here's an exploitability summary. Uh, I think it's important to call out that SIP doesn't
1890.81s - 1896.05s |  prevent this and root is not required. Low privilege malware can perform these attacks. Uh,
1896.07s - 1902.51s |  but, uh, the main issues that 1Password was able to fix are- are highlighted in green. And
1902.51s - 1908.05s |  so the XPC bypass was fixed, the biometric enforcement flag was fixed, and most
1908.05s - 1912.63s |  importantly, the settings modification is now being enforced. And you might say, well, why
1912.63s - 1917.83s |  is that important? It's important because the first two currently exploitable issues are not
1917.83s - 1921.79s |  exploitable if you just disable that setting. And so while it might degrade the user
1921.81s - 1927.51s |  experience a little bit, you are fully protected with the newest version of 8.10.38 which was
1927.51s - 1932.25s |  released in August. Now the- the browser extension injection and impersonation, like I
1932.25s - 1937.75s |  mentioned earlier, that's not really a- a bug per se, it's kind of just how browsers work. And
1937.75s - 1941.93s |  the browser support get PBID bypass is a little bit interesting because there's not really a
1941.93s - 1947.87s |  clear line of ownership to the issue. Uh, so 1Password can't really securely look up the code
1947.91s - 1954.35s |  signing information of their parent process. It is true that all browsers could maybe switch to
1954.35s - 1959.39s |  XPC to provide that audit token. Uh, it's one available method to perform inter-process
1959.39s - 1964.79s |  communication with, uh, or on Mac OS rather. But it's also true that Apple might be able to
1964.79s - 1970.63s |  provide some way to look up your parent processes code signing information securely. It's
1970.63s - 1978.49s |  just not super clear who, uh, who needs to go and fix this. And so, because you can't really
1978.49s - 1982.43s |  fix those things, we're going to talk about how to detect them. So, I pulled all of these
1982.43s - 1987.37s |  bundle IDs from the browser support process. These are all the chromium based browsers that
1987.37s - 1992.51s |  it will accept as a parent process. And these are the non-chromium based browsers that it will
1992.51s - 1998.25s |  accept as a parent process. And so, for the, uh, extension, injection, and impersonation,
1998.25s - 2002.11s |  those are relatively straightforward to fix. Essentially, what you have to do is go
2002.11s - 2006.55s |  enumerate the dangerous flags, which I've done for you. They're highlighted in orange. And
2006.55s - 2010.79s |  you just need to alert any time those flags are used. Uh, we also, out of an abundance of
2010.79s - 2014.23s |  caution, are just blocking the developer builds of certain browsers because they have a
2014.23s - 2019.93s |  slightly different threat model. And for the browser support issue, again, if you just disable
2019.93s - 2025.01s |  that setting, uh, you will be fine. Uh, I can tell you personally, I'm not going to disable
2025.01s - 2029.41s |  the setting. I like using the browser extension. And I think having malware on your computer
2029.41s - 2034.81s |  is already a pretty significant bar. Now, uh, if you don't want to disable the extension like
2034.81s - 2040.09s |  me, here's a, you can alert on it. Uh, basically, during normal use, when you click a browser
2040.09s - 2045.65s |  icon, what's going to happen is launch D actually spawns that process. And so, if you just set
2045.65s - 2050.67s |  an alert for any time a browser, uh, is spawned, and the parent is not launch D, it's likely
2052.79s - 2057.57s |  malicious, or it's going to be some kind of headless browsing. And you can usually limit
2057.57s - 2063.91s |  that to the people that need to do that at your company. Probably QA teams. And a malicious
2063.91s - 2068.65s |  actor might look at that alert and, and try to get around it, right? By getting launch D to
2068.65s - 2073.45s |  spawn their parent, uh, to spawn their exploit process and then execing in directly. And so
2073.45s - 2078.33s |  what you can look for there is the exec. Uh, the, the point of looking for the exec is that
2078.33s - 2082.27s |  there are a couple different ways to obscure or, or, you know, move around those read write
2082.27s - 2088.37s |  pipes. You don't have to use the process tree that we showed you here. But at every, um, for
2088.37s - 2093.57s |  every actual exploit, at some point, they're going to have to exec into a trusted browser. And
2093.57s - 2098.51s |  so, that's what you will, uh, would alert on and it looks like PID reuse in launch D child
2098.51s - 2104.21s |  processes. So here's our full disclosure timeline. We reported all the stuff that we found
2104.21s - 2110.19s |  initially in January of 2024. And then we followed up on March 8th and reported, reported
2110.19s - 2116.13s |  that the biometrics flag was missing. And we actually weren't convinced at the time that it
2116.13s - 2120.33s |  was going to be exploitable in practice. And so we put together a proof of concept and gave
2120.35s - 2124.47s |  it to them on March 22nd and that confirmed for both us and 1Password that this was going to
2124.47s - 2129.47s |  work. Here's a remediation timeline. Again, the, the browser support fix is unclear, but you
2131.63s - 2137.67s |  can toggle that setting. The XPC bypass was fixed in April. Uh, the settings file
2137.67s - 2141.77s |  modifications, this says it was scheduled to be fixed in August. That version is out now. We
2141.77s - 2148.19s |  just submitted the slides before. And the biometrics flag issue was fixed in 8.10.36 and
2148.19s - 2153.99s |  they followed up in 8.10.38 with just a reliability change. It was actually fixed in 3.6,
2153.99s - 2162.65s |  but I would just recommend upgrading to the, to the latest version. So here is where you can
2162.65s - 2167.65s |  find the code we've released at, uh, Morgan C3 slash 1Password exploits. Um, as of now we
2169.89s - 2174.69s |  released, uh, the Frida scripts we used while doing some of this research. Um, none of it is
2174.69s - 2179.47s |  really specific to 1Password at all. Um, it could definitely be extended and useful for
2179.49s - 2184.63s |  researching other applications. In terms of exploits, the only thing we've released is a proof
2184.63s - 2189.87s |  of concept for the XPC validation bypass issue. It's not going to decrypt your vaults, but
2189.87s - 2195.91s |  rather just demonstrate the issue. Uh, in the future, we, uh, will add more code here once ample
2195.91s - 2200.29s |  time has been given for people to be upgrading their 1Password to safe versions and for the
2200.29s - 2207.12s |  other issues to be fixed. Um, and yeah, we may have some time for a few questions if people
2207.20s - 2239.27s |  have them. Sorry, what was that? Yeah, we, we took a look at them. Um, I would just say any,
2239.27s - 2253.36s |  okay, sounds good. Yeah. Any other password managers? Yes. It was just us over the course of
2253.36s - 2258.32s |  about a month. And the, the guy to your left helped us a bit with the last coolest
2258.32s - 2272.92s |  vulnerability right there. I'm, I'm so sorry. Can you say that again? Oh, just, we wanted to
2272.94s - 2277.40s |  know how safe our passwords were. You know, really the detections, I would say are the most
2277.40s - 2282.58s |  important part for us. And I would say, practically, we expected this as, we expected this
2282.58s - 2287.08s |  as likely possible with, you know, most major password managers or similar attacks. We
2287.08s - 2290.78s |  wanted to do it on 1Password because we use 1Password and we wanted to use it on red team
2290.78s - 2307.20s |  operations. Um, and then of course get the issues fixed as well. So, uh, we, we took a brief
2307.20s - 2312.10s |  look at other password managers, but this was a significant amount of effort for us. And so you
2312.12s - 2317.78s |  can imagine that we wouldn't want to do the work, uh, especially on, you know, work time if,
2317.78s - 2328.10s |  uh, if it wasn't going to pay out on a red team. All right. Thank you.


url: https://www.youtube.com/watch?v=5P7KatZBr_I
title: DEF CON 32 - Grand Theft Actions Abusing Self Hosted GitHub Runners - Adnan Khan, John Stawinski
type: video
author: DEFCONConference
ingestion_date: 2024-11-10
transcription_model: large-v2
chunking_status: completed
content:
{
  "webpage_url": "https://www.youtube.com/watch?v=5P7KatZBr_I",
  "title": "DEF CON 32 - Grand Theft Actions Abusing Self Hosted GitHub Runners - Adnan Khan, John Stawinski",
  "description": "GitHub Actions is quickly becoming the de facto CI/CD provider for open-source projects, startups, and enterprises. At the same time, GitHub\u2019s security model is full of insecure defaults. This makes it easy for their customers to expose themselves to critical attacks from the public internet. The end result? A systemic vulnerability class that won\u2019t go away.\n\nDuring our research, we identified GitHub Actions misconfigurations at scale that would allow threat actors to backdoor major open-source projects. An example of this is our attack on PyTorch, a prominent ML framework used by companies and researchers around the world.\n\nThrough this attack, we could contribute code directly to the main branch of the PyTorch repository, upload malicious releases, backdoor other PyTorch projects, and more. These attacks began by compromising self-hosted runners, which are machines that execute jobs in a GitHub Actions workflow. From there, we leveraged misconfigurations and GitHub \u201cfeatures\u201d to elevate our privileges within GitHub Actions workflows.\n\nOur research campaign included dozens of reports, over $250,000 in bug bounties, and endless war stories. Tune in for a deep dive into the TTPs that allow turning a trivial runner compromise into a full supply chain attack.",
  "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw",
  "duration": 2663,
  "channel": "DEFCONConference",
  "uploader": "DEFCONConference",
  "upload_date": "20241016"
}

1.94s - 4.76s | This text was transcribed using whisper model: large-v2

 All right, hello everyone.
4.76s - 7.60s |  All right, let's address the burning question in the room.
7.60s - 10.08s |  No, we did not get access to the source code
10.08s - 11.48s |  for Grand Theft Auto 6,
11.48s - 14.00s |  nor do we know if the game will ever be released.
14.00s - 16.96s |  Maybe in the following presentation, but not today.
16.96s - 18.60s |  This is Grand Theft Actions
18.60s - 21.12s |  abusing self-hosted GitHub runners at scale.
22.20s - 23.94s |  Hello, everybody.
23.94s - 25.60s |  Before we get started, we read online
25.60s - 26.80s |  that as long as we have a disclaimer,
26.80s - 29.68s |  nobody's allowed to sue us, so here we go.
29.68s - 31.40s |  All vulnerabilities mentioned during this talk
31.40s - 32.80s |  have been remediated.
32.80s - 35.02s |  The views and opinions expressed are solely our own
35.02s - 37.12s |  and the content is not endorsed by,
37.12s - 40.20s |  nor does it represent the views of our employers.
40.20s - 42.36s |  All right, whoa, there we go.
42.36s - 44.00s |  Adnan, I think we're good to go.
47.48s - 48.92s |  All right, if you're wondering about us,
48.92s - 50.68s |  so my name is Adnan Khan.
50.68s - 52.96s |  For my day job, I work as a security engineer.
52.96s - 54.72s |  I'm also a security researcher
54.72s - 56.10s |  and bug bounty hunter on the side,
56.10s - 58.56s |  and you can find some of my socials below.
58.56s - 59.80s |  My name is John Stawinski.
59.80s - 62.24s |  I'm a red team security engineer at Praetorian.
62.24s - 65.36s |  I also do CICD security research on the side.
65.36s - 67.08s |  In the last year, I've watched Avatar,
67.08s - 69.92s |  The Last Airbender, three times, the animated version,
69.92s - 75.32s |  and in a past life, I was a collegiate wrestler.
75.32s - 76.96s |  All right, so we're going to be talking a lot
76.96s - 78.32s |  about self-hosted runners today,
78.32s - 80.08s |  so I want to make sure everyone has a background
80.08s - 82.28s |  and understanding of how GitHub runners work.
82.28s - 84.48s |  On one side, you have GitHub-hosted runners.
84.48s - 87.24s |  These are built by GitHub, updated frequently,
87.24s - 89.04s |  and as of writing, cover a wide variety
89.04s - 91.06s |  of operating systems and architectures.
91.06s - 92.98s |  The key thing with GitHub-hosted runners
92.98s - 96.34s |  is that they are ephemeral, meaning they are torn down
96.34s - 99.18s |  at the conclusion of each workflow job.
99.18s - 101.74s |  On the other hand, you have self-hosted runners.
101.74s - 105.34s |  These are managed and secured entirely by the end user,
105.34s - 107.22s |  and it's not surprising here
107.22s - 109.26s |  that the path of least resistance
109.26s - 112.30s |  when configuring one of these self-hosted runners
112.30s - 114.50s |  is a non-ephemeral self-hosted runner
114.50s - 116.58s |  that is actually the least secure.
116.58s - 119.06s |  Both GitHub-hosted and self-hosted runners
119.06s - 121.34s |  work to service GitHub Actions workflows
121.34s - 123.94s |  that are triggered from repositories on GitHub.
124.86s - 126.42s |  So why are we here today?
126.42s - 129.18s |  Well, all the organizations and projects
129.18s - 130.54s |  that you see on the screen here
130.54s - 133.10s |  use or at one point use self-hosted runners
133.10s - 135.34s |  on their public GitHub repositories.
135.34s - 138.04s |  Not only that, but they use them in an insecure manner,
138.04s - 139.64s |  and over the last 12 months,
139.64s - 141.62s |  we were able to identify vulnerabilities
141.62s - 143.66s |  in these organizations and many more
143.66s - 144.94s |  that we can't talk about.
144.94s - 147.46s |  Some of these cases could have led to widespread,
147.46s - 149.24s |  critical supply chain attacks.
150.82s - 153.22s |  You may be wondering if we just showed that last slide
153.22s - 154.82s |  to flex our cool bones at DEF CON,
154.82s - 157.58s |  and I just want to go on record saying that,
157.58s - 159.86s |  yeah, pretty much.
159.86s - 162.40s |  But it also raises a bigger question.
162.40s - 165.22s |  Why were we in a position to execute supply chain attacks
165.22s - 166.66s |  on all of these companies?
166.66s - 167.86s |  And the reason behind that
167.86s - 171.02s |  is that GitHub Actions provides this broad attack surface
171.02s - 173.58s |  that exposes these organizations to compromise,
173.58s - 176.46s |  especially when they use self-hosted runners.
177.70s - 179.78s |  All right, so how do we get started
179.78s - 181.42s |  with our vulnerability research campaign?
181.42s - 185.14s |  Well, it all started all the way back in August of 2022
185.14s - 188.06s |  when I was working on a red team with my former employer.
188.06s - 189.66s |  I was fairly new to red teaming at the time,
189.66s - 190.50s |  and I tripped a canary
190.50s - 191.94s |  and got us kicked out of the client network.
191.94s - 193.76s |  Maybe that's happened to some of y'all.
193.76s - 195.10s |  But while trying to get back in,
195.10s - 196.24s |  we did some social engineering,
196.24s - 198.74s |  and I found that we could use a GitHub access token
198.74s - 200.66s |  that we obtained to execute a workflow
200.66s - 203.02s |  and persist on a non-ephemeral self-hosted runner
203.02s - 204.22s |  inside that client's network,
204.22s - 206.94s |  and that got us back in and we reached objective.
206.94s - 209.02s |  That led to a talk at ShmooCon
209.02s - 210.30s |  called Phantom of the Pipeline
210.30s - 212.10s |  that was given in January of 2023
212.10s - 214.70s |  and the release of the original Gato tool.
215.90s - 218.26s |  Around the time that Anand, Mason Davis,
218.26s - 220.42s |  and Matt Chkosky were developing Gato,
220.42s - 221.50s |  I joined them on the red team,
221.50s - 223.36s |  and Gato was my entry to CICD Security.
223.36s - 225.02s |  So I started using Gato
225.02s - 227.02s |  to identify GitHub Actions vulnerabilities
227.02s - 228.46s |  during red team engagements,
228.46s - 231.86s |  and then we started diving deeper into GitHub Actions abuse,
231.86s - 234.06s |  especially around post-exploitation.
234.06s - 235.32s |  And as we were doing our research,
235.32s - 236.92s |  we were trying to brainstorm ways
237.76s - 238.60s |  to blow this up and bring it
238.60s - 241.20s |  from internal red team engagements to everybody,
241.20s - 242.52s |  and Anand had an idea
242.52s - 244.64s |  that really took it to the next level.
244.64s - 246.12s |  So one of the little asterisks
246.12s - 247.84s |  at the end of the ShmooCon presentation
247.84s - 250.72s |  was that if you fixed a typo on a public repository
250.72s - 253.16s |  that was using a non-ephemeral self-hosted runner,
253.16s - 254.48s |  you could just make a pull request
254.48s - 257.12s |  and modify the workflow file and get persistence.
257.12s - 258.92s |  I demonstrated that vulnerability
258.92s - 261.82s |  against GitHub Actions runner images repository,
261.82s - 264.36s |  which was accepted as a critical vulnerability.
265.36s - 268.40s |  After Anand hacked GitHub Actions runner images,
268.40s - 271.08s |  we started looking at all public repositories
271.08s - 273.28s |  and realized that these misconfigurations were everywhere.
273.28s - 274.80s |  So we decided to team up.
274.80s - 276.00s |  Our first joint operation
276.00s - 277.88s |  was breaching Microsoft's perimeter
277.88s - 280.24s |  by getting code execution on a domain-joined machine
280.24s - 282.04s |  through Microsoft DeepSpeed,
282.04s - 285.52s |  and then we launched a series of attacks,
285.52s - 288.44s |  the next one of which we will cover today.
288.44s - 290.16s |  All right, so at a high level,
290.16s - 291.60s |  there's three steps to discovering
291.60s - 293.56s |  this vulnerability at scale.
293.56s - 296.36s |  The first step is to search for candidate repositories
296.36s - 298.32s |  that might be using a self-hosted runner.
298.32s - 301.56s |  To do this, we used a combination of GitHub CodeSearch
301.56s - 304.28s |  and Sourcegraph CodeSearch dorks.
304.28s - 305.56s |  Both have pros and cons,
305.56s - 309.00s |  so we used both and deduplicated the results
309.00s - 311.10s |  to get a nice list of candidate repositories
311.10s - 313.12s |  for further analysis.
313.12s - 316.24s |  To do this, use the tool that's open source right now
316.24s - 317.50s |  called GatoX.
317.50s - 320.32s |  It uses automated workflow run log analysis
320.32s - 321.88s |  to determine if there's a non-ephemeral
321.88s - 323.48s |  self-hosted runner in use.
324.28s - 326.58s |  Once you have the results, you can plan your next step,
326.58s - 328.76s |  which is essentially, can you hack it,
328.76s - 330.82s |  and can you hack it for some good impact?
332.24s - 335.36s |  Self-hosted runner takeover is a special case
335.36s - 337.70s |  of public Poison Pipeline execution,
337.70s - 340.24s |  so Poison Pipeline execution at a high level
340.24s - 342.52s |  is any time you can get your own code
342.52s - 345.04s |  executed by a GitHub Actions workflow,
345.04s - 347.24s |  you can conduct a Poison Pipeline attack.
347.24s - 348.68s |  In the case of self-hosted runner,
348.68s - 350.24s |  we'll use this to deploy persistence
350.24s - 352.56s |  on the self-hosted runner via a pull request,
352.56s - 354.96s |  and that opens up a ton of lateral movement
354.96s - 356.28s |  and privilege escalation paths.
356.28s - 358.80s |  So the first two aspects of this diagram
358.80s - 360.12s |  are pretty straightforward.
360.12s - 363.28s |  This last section, the GitHub Actions post-exploitation,
363.28s - 365.24s |  is where we focus most of our research.
366.16s - 368.80s |  All right, so it's a tale as old as Unix time itself.
368.80s - 372.24s |  Misconfigurations are amplified by insecure defaults.
372.24s - 375.44s |  So the default setting for a pull request approval
375.44s - 377.36s |  on the pull request trigger is that
377.36s - 380.56s |  it only requires approval for first time contributors.
380.56s - 382.28s |  So if someone has a pull request,
383.00s - 385.04s |  no matter how trivial, merged into the default branch,
385.04s - 387.92s |  they won't need approval for workload executions
387.92s - 389.76s |  on their pull requests.
389.76s - 394.84s |  This is that bad insecure default.
394.84s - 397.56s |  Today, we're gonna walk through playing with fire,
397.56s - 399.92s |  which is what we've called our supply chain attack
399.92s - 401.64s |  on the PyTorch repository,
401.64s - 404.64s |  which began by compromising self-hosted runners.
404.64s - 406.36s |  This was one of our more complex
406.36s - 408.72s |  and more intricate attacks,
408.72s - 410.32s |  so we're gonna use it as a case study
410.32s - 412.76s |  to teach technical TTPs of GitHub Actions
412.76s - 415.08s |  exploitation and post-exploitation as we go.
416.00s - 417.88s |  All right, so what is PyTorch?
417.88s - 419.32s |  I'm sure a lot of you already know what it is,
419.32s - 420.76s |  but it's a machine learning framer
420.76s - 423.12s |  originally developed by Meta that's been open-sourced,
423.12s - 424.88s |  and it's used by companies around the world
424.88s - 427.64s |  like Google, Lockheed Martin, OpenAI, and more.
427.64s - 429.44s |  If someone were to backdoor PyTorch,
429.44s - 431.32s |  this could allow compromising developers
431.32s - 433.08s |  and organizations in that very much
433.08s - 435.06s |  red-hot AIML space right now.
436.56s - 439.68s |  In all seriousness, probably the hardest part
439.68s - 442.40s |  of compromising PyTorch was searching
442.92s - 443.74s |  through their workflows.
443.74s - 446.16s |  They had something like over 90 workflows,
446.16s - 447.64s |  over 15 GitHub secrets,
447.64s - 449.52s |  and more than five self-hosted runners
449.52s - 451.12s |  on their public repository,
451.12s - 453.56s |  and I vividly remember sitting at my desk.
453.56s - 455.72s |  I had like three different notebooks out,
455.72s - 457.76s |  and I was manually, pen and paper,
457.76s - 460.56s |  mapping out the data flow through all these workflows,
460.56s - 463.44s |  a lot of which were nested in preparing for this attack
463.44s - 465.52s |  so that we could try to figure out the potential impact
465.52s - 466.88s |  before we actually started.
468.08s - 469.88s |  When we confirm that a repository
469.88s - 471.74s |  has a self-hosted runner,
471.74s - 473.02s |  we look at the workflow logs
473.02s - 474.50s |  like Adnan talked about earlier.
474.50s - 477.06s |  So here's an example of the PyTorch repository,
477.06s - 478.42s |  and in this screenshot,
478.42s - 479.66s |  you can see in the workflow log,
479.66s - 481.02s |  it has the runner name,
481.02s - 481.88s |  the runner group name,
481.88s - 482.72s |  and the machine name.
482.72s - 484.78s |  So we see that this is a Jenkins runner.
484.78s - 485.82s |  It's a self-hosted runner
485.82s - 489.20s |  attached to their public repository in the default group,
489.20s - 490.46s |  and this is one of the runners
490.46s - 491.96s |  that we ended up compromising.
492.94s - 495.46s |  All right, remember that default approval requirement?
495.46s - 498.02s |  Well, for certain repositories that are very busy,
498.02s - 499.62s |  you can often determine that
499.62s - 501.10s |  just using contextual analysis,
501.10s - 501.94s |  and let me tell you,
501.94s - 503.54s |  PyTorch was very busy.
503.54s - 505.54s |  So what you look for is a pull request
505.54s - 507.02s |  submitted by a previous contributor
507.02s - 508.70s |  that's from a fork pull request.
508.70s - 510.50s |  You check that the workflow run
510.50s - 512.30s |  associated with it was not approved,
512.30s - 515.88s |  and that it ran on the pull request trigger.
515.88s - 517.10s |  When all these things come together,
517.10s - 519.50s |  it's very likely that the default approval requirements
519.50s - 520.34s |  are in place.
521.66s - 524.62s |  GitHub warns against attaching self-hosted runners
524.62s - 526.70s |  to public repositories for obvious reasons,
526.70s - 528.98s |  but they don't make their documentation
528.98s - 529.82s |  and their warnings obvious,
529.82s - 530.98s |  and you kind of have to go out of your way.
531.66s - 533.58s |  Part of the reason we think that so many repos
533.58s - 535.14s |  publicly use self-hosted runners
535.14s - 537.82s |  is because they don't end up seeing this documentation.
537.82s - 538.78s |  In this next slide,
538.78s - 540.34s |  we'll just walk through what it looks like
540.34s - 542.06s |  to register a self-hosted runner
542.06s - 545.46s |  and see what documentation you come across as a developer.
546.42s - 548.06s |  So in my GitHub repository,
548.06s - 549.48s |  I'm going to actions panel.
549.48s - 551.14s |  I'm trying to register a self-hosted runner,
551.14s - 553.34s |  but first I'm gonna look through their documentation
553.34s - 554.58s |  because I'm a good developer,
554.58s - 556.08s |  and I click on these links.
556.08s - 557.58s |  I don't see anything security related,
557.58s - 558.78s |  so I click on this other link,
558.78s - 560.90s |  and we get to a lot of GitHub
561.70s - 562.86s |  self-hosted runner documentation.
562.86s - 566.46s |  We learn about auto-scaling, usage limits.
566.46s - 568.22s |  I think it lists architecture stuff,
568.22s - 570.70s |  and then every API that the runner's going to use
570.70s - 571.98s |  in its lifetime,
571.98s - 574.62s |  and then way down here at the bottom,
574.62s - 576.42s |  it talks about self-hosted runner security,
576.42s - 579.20s |  and so GitHub does clearly say
579.20s - 581.46s |  they don't recommend using self-hosted GitHub runners
581.46s - 582.86s |  on public repositories,
582.86s - 584.90s |  and they have other articles out there too
584.90s - 586.98s |  elaborating on this,
586.98s - 589.74s |  but the point we want to drive is
589.74s - 590.86s |  it's not obvious.
590.86s - 591.70s |  So chances are,
591.70s - 593.30s |  if you're just registering a self-hosted runner
593.30s - 594.50s |  to a public repo,
594.50s - 597.58s |  you're not going to come across this documentation.
597.58s - 598.74s |  All right, the first step
598.74s - 600.90s |  was to infiltrate the contributor list.
600.90s - 602.78s |  This is what's important to take advantage
602.78s - 604.54s |  of those default approval settings.
605.46s - 606.98s |  To infiltrate the contributor list,
606.98s - 608.24s |  if you've been following our blogs at all,
608.24s - 609.82s |  we always call the grammar police.
609.82s - 612.36s |  So the grammar police is me and Adnan.
612.36s - 613.32s |  We show up,
613.32s - 615.18s |  and we run all of their markdown files
615.18s - 616.34s |  through Grammarly.
616.34s - 618.22s |  Again, fixing a typo here
618.22s - 619.70s |  and finding a grammatical error
619.70s - 620.70s |  doesn't matter at all.
620.70s - 622.72s |  To become a contributor,
622.72s - 624.30s |  we also could make a legitimate,
624.30s - 625.82s |  substantial code contribution
625.82s - 627.62s |  to the PyTorch repository,
627.62s - 630.00s |  but that's a lot of work.
630.00s - 632.10s |  So what we do instead is we notice
632.10s - 634.26s |  resolve was in the past tense,
634.26s - 635.66s |  and it should have been in the present tense.
635.66s - 637.46s |  So we fixed that for them,
637.46s - 639.58s |  and we submitted a pull request,
639.58s - 640.46s |  and a few days later,
640.46s - 642.42s |  it got merged by the maintainers,
642.42s - 644.52s |  and then we were now officially contributors,
644.52s - 646.26s |  which is not a security boundary, right?
646.26s - 647.80s |  And this is the same process we did
647.80s - 649.40s |  with all of our exploits.
649.40s - 651.32s |  All right, so this is where the hacking starts.
651.32s - 653.16s |  So the step, phase two,
653.16s - 654.92s |  was to install command and control
654.92s - 657.08s |  on those non-ephemeral self-hosted runners
657.08s - 658.40s |  attached to PyTorch.
658.40s - 659.92s |  To do this, we leverage something
659.92s - 662.04s |  we like to call runner-on-runner.
662.04s - 663.10s |  In a nutshell, what that is,
663.10s - 665.40s |  is we install another self-hosted runner
665.40s - 667.00s |  on their self-hosted runner.
667.00s - 669.52s |  This has the benefit of sending all traffic
669.52s - 672.00s |  to the same domains and IP addresses
672.00s - 674.04s |  that the original self-hosted runner uses.
674.04s - 676.58s |  So if there is any EDR or monitoring software
676.58s - 677.82s |  on that runner, it's very likely
677.82s - 679.72s |  that we'll fly right under the radar.
679.72s - 681.12s |  Oh yeah, so remember that default?
681.12s - 683.70s |  This is where those defaults lead to an actual compromise.
683.70s - 687.02s |  To do this, we just modified the workflow in our fork
687.02s - 689.40s |  and had it just install our C2.
690.54s - 693.70s |  The second runner is attached to our private C2 repository,
693.70s - 695.80s |  so we can task it to execute commands.
695.80s - 697.86s |  It's essentially a web shell.
697.86s - 699.10s |  Using this web shell,
699.10s - 701.54s |  we're able to look around the file system
701.54s - 703.16s |  and learn about the host,
703.16s - 705.98s |  and all of this is in preparation for phase three.
707.10s - 709.18s |  Phase three is the most important phase
709.18s - 711.42s |  of all of our CICD compromise.
711.42s - 714.26s |  So the great secret heist,
714.26s - 716.06s |  self-hosted runner post-exploitation,
716.06s - 718.66s |  is how you go from trivial RCE
718.66s - 720.02s |  to complete supply chain attack.
720.02s - 721.42s |  And when we started our research,
721.42s - 723.38s |  we submitted some reports and we saw some reports
723.38s - 725.54s |  where researchers would get code execution
725.54s - 727.66s |  on self-hosted runners, and then they'd say,
727.66s - 729.46s |  hey, and by the way, you can probably do
729.46s - 732.02s |  all these theoretical post-exploitation stuff.
732.02s - 734.86s |  And orgs just didn't care at all.
734.86s - 737.06s |  They should care if you're executing code on the runners,
737.06s - 739.86s |  but it's kind of what they're supposed to do anyway,
739.86s - 741.98s |  and so they didn't realize the actual impact
741.98s - 744.58s |  of these attacks unless you go and demonstrate it.
744.58s - 747.22s |  So that is what we did.
747.22s - 749.34s |  All right, so to understand how we were able
749.34s - 750.58s |  to achieve some of this impact,
750.58s - 752.02s |  I want to make sure everyone understands
752.02s - 753.46s |  the magical GitHub token.
753.46s - 755.50s |  So this is a token that is used
755.50s - 757.38s |  by all GitHub Actions workflows
757.38s - 760.74s |  to authenticate to GitHub for API or Git operations,
760.74s - 763.82s |  and it's an OAuth bearer token that has multiple scopes.
763.82s - 765.42s |  Some of these scopes can be read or write,
765.42s - 767.82s |  and they're configured at the repository,
767.82s - 769.54s |  organization level, or specified
769.54s - 772.10s |  within each specific workflow file.
772.10s - 775.26s |  So it's very important to know that these tokens
775.26s - 777.74s |  are only valid for the duration of each job
777.74s - 778.66s |  within a workflow.
778.66s - 781.62s |  As soon as that job concludes, the token is no longer valid.
781.62s - 784.78s |  So to conduct some of the post-exploitation attacks here,
784.78s - 788.18s |  we need to work around this limitation.
788.18s - 790.98s |  So let's see how PyTorch, at the time,
790.98s - 794.54s |  used or privileged the GitHub token.
794.54s - 797.62s |  Okay, so this GitHub token has all the right permissions.
797.62s - 801.58s |  This means we have a lot of options to play with
801.58s - 803.26s |  for some of our post-exploitation,
803.26s - 805.74s |  and also, this runner right here is the same one
805.74s - 807.06s |  that we now have C2 on.
808.90s - 811.66s |  So when a workflow uses the Actions checkout step
811.66s - 814.18s |  or some other steps, the GitHub token is stored
814.18s - 816.48s |  on the self-hosted runner's file system,
816.48s - 817.82s |  which we have compromised.
817.82s - 819.74s |  So the problem is that GitHub tokens
819.78s - 822.74s |  from fork PRs only have read permissions.
822.74s - 825.34s |  So if we tried to take the GitHub token
825.34s - 828.58s |  from the fork PR we used to actually implant the runners,
828.58s - 829.90s |  it's pretty much useless to us.
829.90s - 832.18s |  So the solution is we persist on the runner,
832.18s - 835.38s |  and then we capture a token from a future workflow.
835.38s - 836.90s |  This is the setup we typically see.
836.90s - 838.78s |  We have the workflow from our fork PR,
838.78s - 840.26s |  no access to secrets,
840.26s - 843.58s |  and a GitHub token with only read permissions.
843.58s - 847.18s |  Then we have a workflow from the base repository.
847.18s - 849.90s |  This workflow does have access to secrets,
849.90s - 851.62s |  and the GitHub token has write permissions.
851.62s - 855.46s |  Notice how they both execute on the same self-hosted runner.
855.46s - 857.42s |  So this is the reason that persistent,
857.42s - 859.66s |  non-ephemeral self-hosted runners are so bad, right?
859.66s - 862.54s |  If the runner's ephemeral and we implant it,
862.54s - 865.34s |  doesn't give us much access beyond secrets
865.34s - 867.02s |  on the file system or something like that.
867.02s - 868.98s |  If it's persistent and long-lived,
868.98s - 871.06s |  then it will execute future workflows.
871.06s - 873.14s |  So we implant the runner,
873.14s - 874.94s |  then we wait for future workflows
874.94s - 877.02s |  from the base repo to execute on the runner.
877.86s - 879.18s |  This is why we need to install persistence
879.18s - 881.38s |  rather than just showing code execution.
881.38s - 883.14s |  And then we compromise the GitHub token
883.14s - 886.46s |  and any GitHub secrets used by subsequent workflows.
886.46s - 889.14s |  So in the PyTorch attack, you can see here,
889.14s - 891.42s |  this was our C2 repository.
891.42s - 894.14s |  So we were executing code on their self-hosted runners,
894.14s - 895.70s |  and we waited for a future workflow
895.70s - 897.82s |  to execute from the base repo.
897.82s - 900.70s |  And then we just print out the git config file at runtime.
900.70s - 903.90s |  So this base64 encoded token is that GitHub token
903.90s - 904.94s |  that we were just talking about.
904.94s - 906.74s |  So we compromise that GitHub token.
907.42s - 908.26s |  And again, we could only use it
908.26s - 910.22s |  for the duration of the build,
910.22s - 912.82s |  but some of these jobs lasted a long time,
912.82s - 914.62s |  so we were able to use it for a while.
914.62s - 916.06s |  All right, one of the first things we did
916.06s - 917.22s |  after obtaining the GitHub token,
917.22s - 919.06s |  which had actions write access,
919.06s - 921.78s |  is we used it to delete the workflow run logs.
921.78s - 923.34s |  Because if we got caught,
923.34s - 925.58s |  then we likely wouldn't be able to show impact
925.58s - 926.94s |  in order for Meta's triage team
926.94s - 929.18s |  to understand the severity of the submission.
930.54s - 931.86s |  One of the most direct ways
931.86s - 933.50s |  to demonstrate impact with a GitHub token
933.50s - 935.50s |  is by modifying GitHub releases.
935.54s - 937.54s |  So GitHub releases is just one way
937.54s - 940.22s |  for a user to download a release
940.22s - 942.26s |  of an application from a repository.
942.26s - 944.42s |  What typically will happen is
944.42s - 945.82s |  you can upload GitHub releases,
945.82s - 948.70s |  so PyTorch uploads releases to their GitHub releases page.
948.70s - 950.22s |  And then if you wanna download PyTorch,
950.22s - 951.66s |  one of the ways you can do that
951.66s - 954.74s |  is by going to GitHub releases and downloading it there.
954.74s - 956.66s |  Depending on the permissions of the GitHub token,
956.66s - 960.82s |  it can allow you to modify releases and release assets.
960.82s - 963.96s |  So you could potentially backdoor a release asset,
963.96s - 965.92s |  upload it, and then everyone who downloads
965.92s - 968.76s |  and uses that asset will be running your malicious code.
970.00s - 973.24s |  We did not want to upload any backdoored assets.
973.24s - 975.32s |  We're trying to identify supply chain attacks,
975.32s - 976.52s |  not actually execute them.
976.52s - 978.84s |  So instead of tampering with assets,
978.84s - 980.52s |  we just tampered with the release name.
980.52s - 983.72s |  So we sent this curl request using the GitHub API,
983.72s - 985.32s |  and it uses the compromised GitHub token
985.32s - 988.72s |  just to add my name to the end of the release.
988.72s - 990.52s |  So after we sent that request,
990.52s - 993.32s |  you can see the latest PyTorch release at the time
993.32s - 995.20s |  was now signed by yours truly.
995.20s - 997.52s |  And all we did here was just take a screenshot
997.52s - 998.72s |  and then revert it right away.
998.72s - 1001.24s |  Obviously we didn't want this being seen by a lot of people,
1001.24s - 1002.84s |  but this was our POC to show
1002.84s - 1006.32s |  that we could tamper with GitHub releases if we wanted to.
1006.32s - 1007.44s |  A problem that we've encountered
1007.44s - 1008.56s |  with GitHub release tampering
1008.56s - 1010.70s |  is organizations have came back to us
1010.70s - 1011.88s |  and legitimately said,
1011.88s - 1015.02s |  well, no one really uses our GitHub releases.
1015.02s - 1018.10s |  And obviously that shouldn't be the case,
1018.10s - 1020.28s |  but to demonstrate further impact,
1020.28s - 1022.56s |  we wanted to look at GitHub secrets.
1022.56s - 1025.84s |  GitHub secrets are the crown jewels of any repo
1025.84s - 1027.12s |  that uses GitHub actions.
1027.12s - 1029.56s |  A lot of the times they're overprivileged,
1029.56s - 1032.04s |  and they can provide lateral movement opportunities
1032.04s - 1033.76s |  beyond the GitHub repository.
1033.76s - 1037.88s |  So we were back to being Charlie from It's Always Sunny,
1037.88s - 1039.08s |  and we were searching through
1039.08s - 1041.24s |  all of those PyTorch workflows again,
1041.24s - 1043.00s |  trying to see where GitHub secrets were used
1043.00s - 1044.48s |  and see if we could compromise them.
1044.48s - 1046.24s |  Here, you can see that they're using
1046.24s - 1047.60s |  some personal access tokens,
1047.60s - 1048.80s |  which are always super interesting.
1048.80s - 1050.38s |  They're using AWS keys,
1050.38s - 1051.96s |  and there were a bunch of other secrets
1052.48s - 1053.76s |  used by the PyTorch repository.
1053.76s - 1055.64s |  And before we go any further,
1055.64s - 1057.60s |  I want to quickly recap how we got here.
1057.60s - 1060.78s |  So remembering back to the grammar police,
1060.78s - 1062.50s |  we came, we fixed the typo,
1062.50s - 1064.02s |  that typo got merged,
1064.02s - 1066.40s |  that made us a contributor to the repository,
1066.40s - 1067.80s |  which was using the default setting
1067.80s - 1069.96s |  so we could execute arbitrary workflows.
1069.96s - 1072.20s |  We used that access to execute a workflow
1072.20s - 1074.84s |  that implanted three of their self-hosted runners.
1074.84s - 1076.60s |  Then we stole this GitHub token
1076.60s - 1080.24s |  from a future workflow executing on that runner,
1080.24s - 1082.24s |  and now we're going to try to use this GitHub token
1082.24s - 1084.60s |  to exfiltrate GitHub secrets.
1084.60s - 1086.96s |  All right, one problem that we had to work around
1086.96s - 1089.58s |  is that the workflows that had the GitHub secrets
1089.58s - 1090.92s |  that we wanted to take
1090.92s - 1092.92s |  didn't actually run on the self-hosted runners
1092.92s - 1094.12s |  that we were now sitting on.
1094.12s - 1097.04s |  So the solution here was to use the GitHub token
1097.04s - 1099.40s |  to conduct pipeline privilege escalation
1099.40s - 1102.76s |  and get arbitrary code execution within those workflows
1102.76s - 1105.24s |  so we could get those very nice secrets.
1106.36s - 1108.60s |  Another problem is GitHub tokens
1108.60s - 1110.20s |  aren't allowed to modify files
1110.20s - 1112.80s |  in the .github-workflows directory.
1112.80s - 1115.20s |  That seems like a kind of random restriction by GitHub,
1115.20s - 1117.48s |  but it's actually implemented a few years ago
1117.48s - 1119.68s |  to prevent these exact attacks.
1119.68s - 1121.10s |  So they want to stop attackers
1121.10s - 1123.08s |  from doing what we're trying to do right now,
1123.08s - 1124.92s |  and so you can't just make a new branch,
1124.92s - 1126.72s |  modify a workflow to exfiltrate secrets,
1126.72s - 1128.70s |  and then you're done.
1128.70s - 1130.96s |  The solution here is you need to find a workflow
1130.96s - 1133.38s |  that uses the GitHub secrets you're interested in
1133.38s - 1135.34s |  that executes code from outside
1135.34s - 1137.48s |  of this .github-workflows directory.
1137.48s - 1139.80s |  So this was our path here.
1139.80s - 1142.48s |  If you look back at PyTorch's weekly.yaml workflow
1142.48s - 1146.72s |  at the time, you can see it does two interesting things.
1146.72s - 1148.48s |  One of them is that it calls this
1148.48s - 1151.32s |  underscore update commit hash .yaml workflow.
1151.32s - 1154.76s |  The other is that it uses these update bot tokens
1154.76s - 1157.08s |  and PyTorch bot tokens, which based on context,
1157.08s - 1159.80s |  we were pretty sure were GitHub personal access tokens.
1159.80s - 1162.56s |  So let's look at update commit hash .yaml.
1163.76s - 1166.88s |  This workflow is still in the restricted workflows directory
1166.88s - 1170.44s |  so we can't modify this workflow with the GitHub token,
1170.44s - 1174.24s |  but it calls update commit hashes .py,
1174.24s - 1176.64s |  which is in the .github-scripts directory,
1176.64s - 1179.40s |  so it's not in the restricted workflow directory.
1179.40s - 1180.92s |  All right, so all we need to do now
1180.92s - 1184.08s |  is use the GitHub token to make a new feature branch
1184.08s - 1187.84s |  and modify that script, and we know that the workflow
1187.84s - 1189.68s |  that calls it runs on a certain event
1189.68s - 1191.24s |  so we can take advantage of it.
1192.72s - 1194.76s |  The modification that we introduced was quite simple.
1194.76s - 1197.04s |  We saved off the tokens that it received
1197.04s - 1198.80s |  via environment variables to files,
1198.80s - 1201.48s |  and then we used our public key to encrypt them
1201.48s - 1203.36s |  and print them to the build logs.
1203.36s - 1205.24s |  This would prevent any other attackers
1205.24s - 1208.52s |  from getting those secrets while we were conducting our POC.
1209.72s - 1210.80s |  Okay, so what did we actually do?
1210.80s - 1213.76s |  We used the GitHub token to create a new branch.
1213.76s - 1215.48s |  Then on the new branch, we added our payload
1215.48s - 1218.12s |  to update commit hashes .py script.
1218.12s - 1220.24s |  Then we used the GitHub token to trigger our payload
1220.24s - 1222.12s |  via a workflow dispatch event.
1222.12s - 1224.96s |  This is because the token had actions write privileges,
1224.96s - 1226.96s |  and then we could retrieve the encrypted secrets
1226.96s - 1229.16s |  from the build logs, cancel the workflow,
1229.16s - 1231.76s |  delete the workflow logs, and actually decrypt the secrets.
1231.76s - 1234.96s |  So at the time when we were doing this operation,
1234.96s - 1237.44s |  I was absolutely terrified that I was gonna forget
1237.44s - 1240.24s |  to take a screenshot, so instead of taking screenshots,
1240.24s - 1241.68s |  I just screen-recorded the whole thing,
1241.68s - 1243.52s |  so I have like an eight-hour screen recording
1243.52s - 1245.40s |  of all this stuff from PyTorch,
1245.40s - 1248.20s |  and I condensed some of it into a video on the next slide
1248.20s - 1251.32s |  just showing the GitHub personal access token exfiltration,
1251.32s - 1253.12s |  so we'll walk you all through that.
1254.24s - 1256.56s |  Here we have our run-on-runner C2,
1256.56s - 1259.44s |  and we're gonna navigate to our command output,
1259.44s - 1263.32s |  and we're just gonna get the base64-encoded GitHub token.
1263.32s - 1265.60s |  This is from a future workflow that was executing
1265.60s - 1267.08s |  on one of the compromised runners.
1267.08s - 1270.04s |  So we're gonna base64 decode that token,
1270.04s - 1271.92s |  and you can see it's just an authorization header
1271.92s - 1273.52s |  within the GitHub token.
1273.52s - 1275.60s |  Use the token to clone the repository,
1275.60s - 1278.64s |  and then we made a new branch in the PyTorch repository.
1278.64s - 1281.96s |  So within our new branch, we're going to modify
1281.96s - 1283.72s |  that Python script that we identified
1283.72s - 1285.20s |  as an injection point.
1285.20s - 1288.12s |  Here with my really poor Vim skills,
1288.12s - 1289.48s |  I'm gonna paste our new payload
1289.48s - 1291.20s |  to exfiltrate those secrets,
1291.20s - 1293.76s |  and then once the payload is set,
1293.76s - 1295.68s |  we're gonna actually commit this to the repository
1295.68s - 1297.68s |  because the token has contents right.
1297.68s - 1299.36s |  So we committed to the repo.
1299.36s - 1301.32s |  Now we're gonna use a curl request
1301.32s - 1304.04s |  just to trigger this workflow from our branch
1304.04s - 1305.88s |  which has the injected code in it,
1305.88s - 1307.16s |  and then we're looking at
1307.16s - 1308.72s |  the actual PyTorch repository now
1308.72s - 1310.48s |  where we just triggered a workflow,
1310.48s - 1315.16s |  and we're gonna see two base64-encoded encrypted blobs.
1315.16s - 1316.64s |  So we're gonna scroll down,
1316.64s - 1317.96s |  and we're gonna get these blobs
1317.96s - 1319.72s |  which is the output of our commands,
1319.72s - 1323.60s |  and then we're gonna go to this other terminal
1323.60s - 1324.92s |  and decrypt them.
1324.92s - 1327.04s |  So first, we're just gonna base64 decode them,
1327.04s - 1328.68s |  and then we're gonna save them to a file,
1328.68s - 1330.76s |  and we're gonna use our private key locally
1330.76s - 1332.52s |  to actually decrypt them and get the token.
1332.52s - 1335.80s |  So once you see some output that starts
1335.84s - 1337.64s |  with G-H-P underscore,
1337.64s - 1339.24s |  those are the personal access tokens,
1339.24s - 1342.96s |  and we're gonna take them and run them through Gato
1342.96s - 1345.20s |  which will do a bunch of automated enumeration
1345.20s - 1346.88s |  using the GitHub APIs,
1346.88s - 1349.72s |  and then Gato will tell us who the owners are,
1349.72s - 1350.80s |  what they have access to,
1350.80s - 1352.88s |  and what the potential impact is
1352.88s - 1356.00s |  because if they just had read access to the repository,
1356.00s - 1357.96s |  there's not a real risk here to PyTorch.
1357.96s - 1361.16s |  If they have other more sensitive privileges,
1361.16s - 1362.40s |  then it could be bad.
1362.40s - 1364.20s |  So here we're running them through Gato.
1364.24s - 1365.40s |  You can see one of the users
1365.40s - 1367.00s |  is the PyTorch update bot user,
1367.00s - 1369.08s |  the other is the PyTorch bot user,
1369.08s - 1371.24s |  and they're both PyTorch organization members.
1371.24s - 1373.20s |  The tokens have repo scope,
1373.20s - 1375.12s |  so we can use this now to interact
1375.12s - 1376.88s |  with the PyTorch repository directly.
1376.88s - 1378.52s |  These are long-lived credentials,
1378.52s - 1380.68s |  and they had access to 93 repos
1380.68s - 1382.40s |  within the PyTorch organization,
1382.40s - 1383.60s |  a lot of which were private
1383.60s - 1385.08s |  and did not have branch protection,
1385.08s - 1387.36s |  so you could have done whatever you wanted to them,
1387.36s - 1391.48s |  and there's a lot of paths here to supply chain compromise.
1391.48s - 1392.96s |  All right, that was a lot, right?
1392.96s - 1395.04s |  Okay, so what can we do?
1395.04s - 1396.80s |  So we have those two tokens,
1396.80s - 1398.52s |  so we can use them by themselves
1398.52s - 1400.64s |  to simply introduce code into the main branch.
1400.64s - 1401.96s |  One creates a pull request
1401.96s - 1403.68s |  and have the other one approve and merge it.
1403.68s - 1405.60s |  Boom, supply chain compromise.
1405.60s - 1408.20s |  Another form would be to backdoor a dependency
1408.20s - 1409.60s |  within the PyTorch organization
1409.60s - 1412.16s |  that's used by the primary PyTorch repo,
1412.16s - 1413.84s |  and that could be a more subtle way
1413.84s - 1417.04s |  to obtain that same final impact.
1417.04s - 1419.04s |  And finally, there's even more clever ways,
1419.04s - 1422.56s |  like smuggling malicious code into an in-progress feature
1423.00s - 1424.20s |  that a developer is working on
1424.20s - 1425.36s |  and then have that merged in.
1425.36s - 1427.96s |  So really, there's too many paths to count
1427.96s - 1430.88s |  for how this could impact end users of PyTorch.
1432.60s - 1434.92s |  If you want to see even more impact
1434.92s - 1436.88s |  and you're not really into GitHub,
1436.88s - 1438.24s |  this section is for you.
1438.24s - 1440.36s |  So if you noticed a few slides ago
1440.36s - 1441.52s |  when we were looking at their secrets,
1441.52s - 1444.04s |  we saw that they also had AWS keys.
1444.04s - 1446.72s |  We conducted a very similar attack
1446.72s - 1448.12s |  to the one we just showed, right,
1448.12s - 1449.84s |  where we do that kind of workflow hopping
1449.84s - 1452.04s |  and injection and triggering with the GitHub token.
1452.04s - 1454.80s |  We repeated that process to grab their AWS keys.
1454.80s - 1456.24s |  And the reason we wanted to do this
1456.24s - 1458.48s |  was if you don't understand GitHub,
1458.48s - 1460.84s |  it can sometimes be hard to even see
1460.84s - 1464.76s |  why code contributions to the main branch of a repository
1464.76s - 1465.76s |  could be impactful.
1465.76s - 1469.04s |  But everyone understands AWS and AWS releases.
1469.04s - 1470.52s |  So we grabbed these keys,
1470.52s - 1472.86s |  then we used the AWS CLI to authenticate,
1472.86s - 1475.64s |  and we confirmed that we were the PyTorch bot user.
1475.64s - 1477.48s |  So we didn't want to poke around too much,
1477.48s - 1480.68s |  but we did some basic AWS S3LS commands
1480.68s - 1483.36s |  and saw that this user had a lot of privileges.
1483.36s - 1485.16s |  We confirmed through the workflow logs
1485.16s - 1488.08s |  that they could actually write to these S3 buckets.
1488.08s - 1491.08s |  And one weird thing that we weren't really expecting to see
1491.08s - 1494.56s |  was there were PyTorch releases in these S3 buckets.
1494.56s - 1498.16s |  So at the time, honestly, I didn't think too much of it
1498.16s - 1500.12s |  because I was kind of freaked out with the access we had
1500.12s - 1502.28s |  and wanted to submit the report right away.
1502.28s - 1503.12s |  I think, like we said,
1503.12s - 1505.72s |  this was hour eight or nine at this point,
1505.72s - 1507.20s |  and I just wanted to get the report in.
1507.20s - 1509.36s |  So I didn't really pay attention to these releases.
1509.36s - 1511.56s |  And then we were rehearsing like a week ago
1511.56s - 1513.40s |  and we were talking and I was like,
1513.40s - 1516.12s |  why were these releases in these S3 buckets?
1516.12s - 1517.96s |  Like what system was pulling releases from here?
1517.96s - 1521.24s |  So I went to the PyTorch website
1521.24s - 1525.32s |  and just going to PyTorch.org or PyTorch.com,
1525.32s - 1526.68s |  whatever it is, you can scroll down
1526.68s - 1528.60s |  and you can see when they instruct users
1528.60s - 1531.00s |  to install, download and install PyTorch,
1531.00s - 1532.76s |  if you're using PIP or whatever,
1532.76s - 1536.96s |  you're using this download dot, what is it?
1536.96s - 1538.92s |  Download.pytorch.org URL.
1539.44s - 1541.36s |  So a lot of the releases were coming from this URL.
1541.36s - 1544.84s |  So we went to this URL in a new tab
1544.84s - 1548.20s |  and we're just looking at the request and response headers
1548.20s - 1551.16s |  and sure enough, in the headers,
1551.16s - 1553.60s |  they confirmed that these were actually pulling releases
1553.60s - 1555.80s |  from these S3 buckets
1555.80s - 1560.68s |  and the layout of the release assets on that website
1560.68s - 1563.04s |  were identical to the layout in the S3 buckets.
1563.04s - 1565.52s |  So basically with those AWS keys,
1565.52s - 1567.92s |  we could have uploaded our own PyTorch releases
1567.92s - 1572.92s |  to these S3 buckets and then anyone who downloads PyTorch
1573.16s - 1575.24s |  by following the instructions from the website
1575.24s - 1576.40s |  is going to be downloading
1576.40s - 1578.48s |  and potentially executing our malicious code.
1578.48s - 1581.56s |  So this is that kind of SolarWinds style supply chain attack
1581.56s - 1585.64s |  for anyone that uses PyTorch in the future.
1585.64s - 1587.64s |  Okay, that was a lot.
1587.64s - 1588.96s |  We just did a lot.
1588.96s - 1590.10s |  This is what it looks like
1590.10s - 1592.30s |  put into a nice little red team diagram
1592.30s - 1595.72s |  and if anyone got here late or fell asleep,
1595.72s - 1596.68s |  we'll do a quick recap.
1596.72s - 1599.20s |  So we started by opening a typo fix PR
1599.20s - 1601.20s |  in the PyTorch repository.
1601.20s - 1604.20s |  That PR got merged and then we were a contributor
1604.20s - 1606.38s |  so we submitted a malicious pull request
1606.38s - 1609.04s |  that installed our runner on runner C2
1609.04s - 1612.08s |  on three different self-hosted GitHub runners.
1612.08s - 1613.12s |  From one of these runners,
1613.12s - 1615.88s |  we stole a GitHub token from a future build
1615.88s - 1617.28s |  and we used that to do a bunch of stuff.
1617.28s - 1619.16s |  First, we just modified that release
1619.16s - 1622.40s |  by updating the release title with my name and reverting.
1622.40s - 1624.40s |  Then we created a feature branch
1624.40s - 1626.08s |  in the PyTorch repository
1626.08s - 1628.20s |  and we triggered additional workflows
1628.20s - 1630.26s |  using that workflow dispatch event
1630.26s - 1631.64s |  injecting through the scripts
1631.64s - 1635.12s |  that were outside of the restricted workflows directory.
1635.12s - 1637.24s |  Through this process, we stole some secrets.
1637.24s - 1639.64s |  We stole the PyTorch bot AWS keys
1639.64s - 1642.08s |  and then we stole two GitHub personal access tokens
1642.08s - 1644.00s |  used by PyTorch bot users.
1644.00s - 1645.24s |  There were a bunch more secrets in there
1645.24s - 1648.08s |  but we didn't think there was any need
1648.08s - 1650.52s |  for additional demonstration of impact.
1650.52s - 1651.64s |  Using the AWS keys,
1651.64s - 1655.02s |  we could upload new PyTorch releases to this S3 bucket.
1655.02s - 1656.66s |  Obviously, we did not do that.
1656.66s - 1659.14s |  Then using the GitHub personal access tokens,
1659.14s - 1661.34s |  we could have modified PyTorch dependencies.
1661.34s - 1662.68s |  We could have used them in conjunction
1662.68s - 1665.70s |  to commit and then approve merges to main
1665.70s - 1668.00s |  and done a lot of other bad stuff.
1668.00s - 1669.66s |  All of this goes to say, at the end of the day,
1669.66s - 1672.48s |  there were a lot of supply chain compromised paths here
1672.48s - 1677.22s |  which would have affected most people using PyTorch
1677.22s - 1679.02s |  both currently and in the future.
1680.42s - 1683.52s |  You would think that this amount of demonstration of impact
1683.52s - 1685.40s |  would be enough to convince PyTorch
1685.40s - 1689.08s |  to immediately and urgently apply fixes
1689.08s - 1692.64s |  and well, we'll let the disclosure timeline
1692.64s - 1693.52s |  speak for itself.
1693.52s - 1697.40s |  So we submitted this issue in August 2023
1697.40s - 1698.68s |  to Metabug Bounty.
1700.04s - 1704.38s |  A month later, Meta said there is no update to provide.
1704.38s - 1705.48s |  Two months later,
1705.48s - 1708.16s |  Meta said they consider the issue mitigated.
1708.16s - 1710.54s |  I was suspicious so I went back in
1710.54s - 1713.22s |  and just executed some code,
1713.22s - 1715.10s |  some shell commands on the runners.
1715.10s - 1716.22s |  I didn't do the full attack
1716.22s - 1718.58s |  but anyone could still compromise their runners,
1718.58s - 1721.34s |  any contributor and based on the workflow logs and stuff,
1721.34s - 1723.34s |  I don't think their configurations were still secure
1723.34s - 1724.78s |  so we responded saying,
1724.78s - 1727.44s |  hey, the issue is not fully mitigated, here's some proof.
1727.44s - 1731.78s |  Then another two months went by
1731.78s - 1733.24s |  with really no word from Meta
1733.24s - 1736.22s |  and we sent a strongly worded email
1736.22s - 1738.54s |  expressing concerns on the remediation
1738.54s - 1740.06s |  that has been implemented
1740.06s - 1741.18s |  which led to some back and forth
1741.18s - 1743.26s |  and then December 15th,
1743.26s - 1745.34s |  Meta applied some other fixes
1745.34s - 1747.10s |  which seemed to be sufficient
1747.10s - 1748.06s |  and then they offered a call
1748.06s - 1749.78s |  to discuss the remediation in depth.
1749.78s - 1752.14s |  And so we actually ended up meeting
1752.14s - 1753.42s |  two of the PyTorch maintainers
1753.42s - 1755.80s |  and they were very concerned about these issues,
1755.80s - 1757.58s |  they were on top of it
1757.58s - 1759.54s |  and they also expressed concern
1759.54s - 1761.66s |  at the potential gaps in remediation.
1761.66s - 1764.58s |  So I'm not totally sure where the disconnect was
1764.58s - 1767.18s |  but it worried us that PyTorch
1767.18s - 1771.49s |  was potentially vulnerable for this long.
1771.49s - 1773.93s |  This attack took a lot of hands-on keyboard,
1773.93s - 1776.49s |  workflow modification, command and control,
1776.49s - 1777.33s |  all that stuff
1777.33s - 1779.97s |  and so it kind of prompted us to say,
1779.97s - 1783.85s |  hmm, is there an easier way to automate a lot of this?
1783.85s - 1785.45s |  All right, well, now there is.
1785.45s - 1787.85s |  So with a tool that I've open sourced this week
1787.85s - 1788.89s |  called GatoX,
1788.89s - 1791.09s |  it actually automates that self-holstered
1791.09s - 1793.15s |  runner takeover process.
1793.15s - 1794.81s |  So now instead of all that work
1794.81s - 1796.49s |  that you saw John have to do,
1796.49s - 1797.49s |  which all you have to do
1797.49s - 1799.21s |  is you fix a typo, okay,
1799.21s - 1800.37s |  and you become a contributor,
1800.37s - 1801.65s |  then you run GatoX
1801.65s - 1803.85s |  and then one of two things happens, okay?
1803.85s - 1804.85s |  You either get a shell
1804.85s - 1806.93s |  and you're on top of the world as hacker man
1806.93s - 1808.83s |  or you learn that approval is required
1808.83s - 1811.71s |  and then you're sad because you didn't get to hack anything.
1811.71s - 1814.41s |  So let's walk through a quick video demo
1814.41s - 1815.81s |  of how GatoX works
1815.81s - 1818.77s |  to automate this self-holstered runner takeover process.
1818.77s - 1821.45s |  So first I use it to enumerate a runner, okay?
1821.45s - 1822.35s |  It's non-ephemeral.
1822.35s - 1824.37s |  So now I'm gonna go ahead and fix a typo
1824.37s - 1826.65s |  in this test repository here.
1826.65s - 1828.81s |  So I go ahead and fix a typo
1828.81s - 1830.09s |  and make that commit.
1831.17s - 1832.45s |  So after I make the commit,
1832.45s - 1835.41s |  I'm going to go ahead and create a pull request.
1835.41s - 1836.97s |  So I check that I fixed the typo
1836.97s - 1839.25s |  and I make a quick typo fix PR
1839.25s - 1841.01s |  and now that's merged in.
1841.01s - 1843.53s |  So after that's merged in with my attacker account,
1843.53s - 1845.51s |  I'm gonna go ahead and delete the fork.
1845.51s - 1847.49s |  So because GatoX will create a new fork
1847.49s - 1850.21s |  to throw the payload at the repository.
1850.21s - 1851.89s |  So I'm gonna quickly go ahead
1851.89s - 1853.93s |  and do the web flow to delete the fork
1853.93s - 1855.49s |  and I hit confirm.
1855.49s - 1857.37s |  All right, now I'm gonna run GatoX.
1857.37s - 1859.21s |  So here I've just configured it
1859.21s - 1860.53s |  to deploy a runner on runner
1860.53s - 1863.65s |  on a Linux x86 64-bit runner.
1863.65s - 1866.11s |  So it's gonna go ahead and create the payload
1866.11s - 1868.25s |  within a gist and ask for confirmation
1868.25s - 1870.25s |  because the next steps are overt.
1870.25s - 1872.57s |  So I'm gonna type confirm and hit enter.
1872.57s - 1874.69s |  So as soon as I do that,
1874.69s - 1876.43s |  it's gonna create a draft pull request
1876.43s - 1878.85s |  which is going to deploy that runner on runner payload
1878.85s - 1882.07s |  onto that self-holstered runner via fork PR.
1882.07s - 1883.91s |  So it's gonna take a second
1883.91s - 1885.45s |  because it's now pulling for that runner
1885.45s - 1886.29s |  to connect to GitHub
1886.29s - 1888.13s |  and then check back into that repository.
1888.13s - 1889.17s |  So as soon as it does that,
1889.17s - 1892.93s |  it will drop into a nice and very convenient shell.
1892.93s - 1896.25s |  Okay, now I'm just gonna test this with uname-a
1896.25s - 1897.37s |  and when I hit enter,
1897.37s - 1899.13s |  what's happening under the hood
1899.13s - 1901.97s |  is that GatoX is issuing a workflow dispatch event
1901.97s - 1904.45s |  to that workflow in the C2 repository
1904.45s - 1905.89s |  which is going to run code
1905.89s - 1908.85s |  on that malicious self-holstered runner
1908.85s - 1910.79s |  and you can see that output there.
1910.79s - 1913.33s |  All right, so now you know the techniques
1913.33s - 1915.81s |  and now since I've open sourced it, you have a tool.
1915.83s - 1917.05s |  And another thing I wanna point out
1917.05s - 1920.13s |  is that finding CICD misconfigurations
1920.13s - 1923.01s |  in open source repositories can be a very thankless job
1923.01s - 1925.29s |  because you submit a report, they fix a bug
1925.29s - 1926.49s |  and you don't even get a CVE.
1926.49s - 1928.25s |  So there's not really a lot of motivation for people.
1928.25s - 1931.69s |  So what I've done with GatoX is that on its wiki,
1931.69s - 1933.01s |  I've created a little hall of fame.
1933.01s - 1935.73s |  So if you use GatoX to find a pwn request,
1935.73s - 1937.85s |  injection or self-holstered runner vulnerability
1937.85s - 1940.71s |  in open source project and the maintainers fix it,
1940.71s - 1941.55s |  then just make an issue
1941.55s - 1944.17s |  and I'll add your name to the wiki to get some credit.
1944.17s - 1945.69s |  Just make sure it's actually fixed
1946.53s - 1947.37s |  because we don't want the hall of fame
1947.37s - 1948.19s |  to be the hall of O days
1948.19s - 1951.64s |  as much as some people might like that.
1951.64s - 1954.20s |  Okay, so we showed all of the technical TTPs
1954.20s - 1956.48s |  we use during our PyTorch attack,
1956.48s - 1957.92s |  but there are a bunch of other stuff
1957.92s - 1959.68s |  we've used on other targets.
1959.68s - 1962.44s |  And so we wanna take these next few minutes
1962.44s - 1964.84s |  to honor the diehard CICD nerds
1964.84s - 1967.28s |  who are here to learn all of the TTPs possible.
1967.28s - 1969.48s |  So we're gonna run through this pretty quickly,
1969.48s - 1971.08s |  but all the slides will be up there online
1971.08s - 1972.72s |  so you can check it out after.
1972.72s - 1974.32s |  All right, so one thing you can do
1974.32s - 1975.68s |  after getting persistence on a runner
1975.68s - 1977.92s |  is a solo run style build compromise.
1977.92s - 1980.52s |  So you get the persistence
1980.52s - 1983.66s |  and then you just have to modify scripts
1983.66s - 1986.22s |  for the build or source code after it's checked out.
1986.22s - 1987.28s |  And once you do that,
1987.28s - 1990.16s |  the final build artifacts from that run
1990.16s - 1991.30s |  are going to be poisoned,
1991.30s - 1993.08s |  but it's not gonna be linked back
1993.08s - 1994.04s |  to the original source code.
1994.04s - 1995.36s |  So it's a very stealthy way
1995.36s - 1997.76s |  to conduct a supply chain attack.
1997.76s - 2000.18s |  And here's another way that you could take
2000.18s - 2002.48s |  GitHub release asset tampering a little further.
2002.48s - 2004.72s |  So if you have that context write token,
2004.72s - 2007.68s |  then all you have to do to tamper with releases on assets
2007.68s - 2010.12s |  is use the API to delete the old asset
2010.12s - 2011.92s |  and then upload a new one with a post request
2011.92s - 2013.16s |  right on top of it.
2013.16s - 2015.76s |  And the indicator of compromise here
2015.76s - 2017.68s |  is just going to be the timestamp
2017.68s - 2020.80s |  when a user is looking at it through the web interface.
2020.80s - 2022.34s |  So it's not very obvious.
2022.34s - 2025.24s |  And if you wanna see a tale of a bug bounty gone wrong
2025.24s - 2027.36s |  and a program acting in some pretty bad faith,
2027.36s - 2030.36s |  go check out my blog post on my report to the ASTAR network,
2030.36s - 2040.17s |  which is a tiny polka dot parachain.
2040.17s - 2042.93s |  All right, another technique you can add to your arsenal
2042.93s - 2045.17s |  is something called a post checkout hook.
2045.17s - 2049.01s |  So what happens when a subsequent workflow
2049.01s - 2051.25s |  that you need to get secrets from or token from
2051.25s - 2053.81s |  runs only very infrequently?
2053.81s - 2056.69s |  Well, you need something that can have these qualities.
2056.69s - 2059.11s |  It needs, you need to be able to extend the build time.
2059.11s - 2061.51s |  You also don't wanna break how the workflow works
2061.51s - 2063.79s |  to inform someone that something's going wrong.
2063.79s - 2065.13s |  And you also wanna get notified
2065.13s - 2067.65s |  because you don't wanna be pressing F5 for two weeks.
2067.69s - 2071.37s |  Well, a quick solution here is to place a script like this
2071.37s - 2073.93s |  into the git hooks directory of the repository
2073.93s - 2075.73s |  in the original self-hosted runner.
2075.73s - 2079.29s |  This example here will just take that git config file
2079.29s - 2081.89s |  and code it and then send it off to your XFIL URL
2081.89s - 2083.25s |  and then sleep for 15 minutes.
2083.25s - 2084.93s |  So this is gonna give you 15 minutes
2084.93s - 2086.57s |  to take advantage of that GitHub token
2086.57s - 2089.57s |  to perform post exploitation actions.
2089.57s - 2092.61s |  All right, before we dive into other TTPs,
2092.61s - 2095.17s |  it's important to remember how self-hosted runner takeover
2095.17s - 2096.61s |  falls in within the landscape
2096.61s - 2099.23s |  of broader GitHub actions attack techniques.
2099.23s - 2101.97s |  So there's another type of attack called bone requests
2101.97s - 2105.41s |  where you basically get execution
2105.41s - 2107.01s |  within a privileged workflow.
2107.01s - 2108.93s |  That's the same thing that happens with runner takeover.
2108.93s - 2111.05s |  From there, there's different
2111.05s - 2112.61s |  post exploitation techniques you can use.
2112.61s - 2114.25s |  So you can steal secrets,
2114.25s - 2116.25s |  you could get the actions runtime token,
2116.25s - 2118.81s |  which has some interesting attacks with cache poisoning,
2118.81s - 2120.57s |  as well as OIDC abuse,
2120.57s - 2123.65s |  which I think the talk right after us in this track
2123.65s - 2125.09s |  actually goes into a lot more detail there.
2125.93s - 2126.77s |  So kinda cool.
2126.77s - 2129.09s |  And then there's just too many different attacks
2129.09s - 2129.93s |  to talk about.
2129.93s - 2132.35s |  There's so many ways to achieve great impact.
2133.25s - 2134.73s |  We talked about some of the permissions
2134.73s - 2135.77s |  of the GitHub token today,
2135.77s - 2138.09s |  but there's a lot of other dangerous permissions out there.
2138.09s - 2139.81s |  So some of them don't pose a risk,
2139.81s - 2141.93s |  but others like contents right and actions right
2141.93s - 2143.93s |  obviously pose a significant risk.
2143.93s - 2146.13s |  So other permissions we have identified
2146.13s - 2148.37s |  as dangerous and actually abused during our attacks
2148.37s - 2151.45s |  are pages right, pull requests right, and packages right.
2151.45s - 2152.89s |  And there's a lot of good documentation online
2152.89s - 2156.17s |  if you wanna learn more about what these actually do.
2156.17s - 2158.13s |  If you have just actions right and contents right,
2158.13s - 2159.81s |  you can create feature branches,
2159.81s - 2161.69s |  modify files, execute in workflows,
2161.69s - 2164.33s |  and then issue dispatches events to steal secrets.
2164.33s - 2165.25s |  If that looks familiar,
2165.25s - 2168.33s |  it's because that's exactly what we did during PyTorch.
2168.33s - 2170.65s |  All right, what if you have a workflow
2170.65s - 2171.49s |  that looks like this?
2171.49s - 2172.79s |  It runs on workflow dispatch,
2172.79s - 2174.33s |  but then uses some of that input
2174.33s - 2176.01s |  in a GitHub script or run step.
2176.01s - 2179.73s |  Well, this lets you do good old 90s style code injection.
2179.73s - 2181.45s |  And I'm gonna walk you through how this process works.
2181.45s - 2183.77s |  And you only need a GitHub token actions right
2183.77s - 2185.53s |  to make this hop.
2185.53s - 2189.33s |  So here you can see that the input from the dispatch event
2189.33s - 2190.61s |  has org and repository.
2190.61s - 2192.13s |  And within that script step,
2192.13s - 2195.13s |  repository is used by GitHub context expression.
2195.13s - 2197.65s |  So that's going to be our injection point there.
2197.65s - 2198.77s |  See that arrow.
2198.77s - 2201.01s |  So here's an example payload thrown
2201.01s - 2203.65s |  with a simple request script in Python.
2203.65s - 2206.41s |  The first thing here is to close out the script
2206.41s - 2208.49s |  because we don't want a syntax error or have it to crash
2208.49s - 2210.17s |  because we still need the code to finish
2210.17s - 2213.29s |  in order for us to get our payload execution.
2213.29s - 2215.97s |  The next thing, and this is where the actual payload is,
2215.97s - 2219.33s |  is a JavaScript injection payload
2219.33s - 2224.33s |  that essentially will pull code from a C2 URL
2224.45s - 2225.73s |  and pipe it to bash.
2225.73s - 2228.01s |  So then you can just have arbitrary code there
2228.01s - 2230.07s |  and get execution in that workflow,
2230.07s - 2231.45s |  get the token, get the secrets.
2231.45s - 2237.89s |  And then from there, do more post exploitation.
2237.89s - 2239.63s |  If you only have contents right,
2239.63s - 2241.63s |  there's still a lot of bad stuff you can do.
2241.63s - 2244.39s |  You can modify non-protected branches a lot of times,
2244.39s - 2246.87s |  which actually could end up in releases.
2246.87s - 2248.39s |  You can modify releases directly
2248.39s - 2250.33s |  like we did when we signed my name.
2250.33s - 2251.47s |  You can modify tags,
2251.47s - 2253.95s |  which a lot of times reusable actions are referenced by tags
2253.95s - 2256.75s |  so that could open up some fun possibilities.
2256.75s - 2259.11s |  And then you can also issue repository dispatch events
2259.11s - 2260.45s |  to execute workflows.
2261.71s - 2263.35s |  All right, here's a really unique way
2263.35s - 2266.07s |  to take advantage of contents right.
2266.07s - 2269.43s |  All right, so one way is if you want to get execution
2269.43s - 2271.91s |  on a workflow that only runs on the push trigger,
2271.91s - 2274.75s |  you can add code to that,
2274.75s - 2277.35s |  a feature branch that modifies a script
2277.35s - 2279.15s |  that's called by that workflow.
2279.15s - 2281.21s |  And the next time the developer will push changes,
2281.21s - 2283.47s |  now you have arbitrary code execution
2283.47s - 2284.99s |  within the context of that workflow.
2284.99s - 2287.47s |  And then you can jump further from there.
2287.47s - 2289.81s |  Another way is instead of targeting the workflows,
2289.81s - 2291.11s |  why not target the developer?
2291.11s - 2293.23s |  So you could modify a unit test
2293.23s - 2296.95s |  within a feature branch that a developer is working on
2296.95s - 2298.91s |  and add an info stealer payload.
2299.59s - 2301.15s |  Then when the workflow runs,
2301.15s - 2302.43s |  the dev accounts compromised
2302.43s - 2304.21s |  and often in organizations,
2304.21s - 2306.51s |  dev compromise can be game over.
2306.51s - 2309.35s |  Another way is if you have these two permissions
2309.35s - 2310.19s |  for the token,
2310.19s - 2311.99s |  you can get code modification
2311.99s - 2313.99s |  in the protected default branch.
2313.99s - 2315.61s |  So there's some preconditions here.
2315.61s - 2317.45s |  The repository has to have the setting
2317.45s - 2320.63s |  to allow get of actions to approve and merge pull requests.
2320.63s - 2323.51s |  There can be at most one reviewer required
2323.51s - 2326.33s |  for that pull request approval rule.
2326.33s - 2329.85s |  And there cannot be any code owner protection rule sets
2329.85s - 2331.73s |  enabled and enforced.
2331.73s - 2333.25s |  When all of these are together,
2333.25s - 2335.17s |  you can have attack like this.
2335.17s - 2337.61s |  Attacker gets the GitHub token.
2337.61s - 2339.77s |  They use their account to create a fork pull request
2339.77s - 2341.13s |  with malicious changes.
2341.13s - 2343.17s |  And then they approve that pull request
2343.17s - 2346.45s |  with that capture token and merge that pull request.
2346.45s - 2350.99s |  That leads to a good old supply chain compromise.
2350.99s - 2352.47s |  Now, if you're on a runner,
2352.47s - 2355.19s |  it's possible to steal the GitHub actions runtime token
2355.19s - 2356.43s |  from a future workflow.
2356.43s - 2358.51s |  This is where you'll have some attack bats
2358.51s - 2360.27s |  for GitHub actions cache poisoning.
2360.27s - 2362.43s |  And I have a blog post on my website
2362.43s - 2364.43s |  that goes into the details of that.
2364.43s - 2367.77s |  And if there's a organization level runner group
2367.77s - 2371.39s |  that shares that also private repositories are using,
2371.39s - 2373.71s |  then there's bats to jump to private repositories
2373.71s - 2374.81s |  and other runner groups.
2374.81s - 2377.65s |  And it just kind of blows those attack bats wide open.
2378.87s - 2381.83s |  Okay, that's the end of some of the more advanced TTPs.
2381.83s - 2384.31s |  If you're the type of person who likes big mind maps
2384.31s - 2385.99s |  to tell you exactly what to do,
2386.51s - 2388.71s |  I made one and I put it on my GitHub a few days ago.
2388.71s - 2389.91s |  So you can go check that out.
2389.91s - 2391.71s |  Obviously this is not comprehensive,
2391.71s - 2394.63s |  but if you're trying to get into GitHub actions exploitation
2394.63s - 2396.99s |  and you have read or write access to a repo,
2396.99s - 2399.71s |  but don't really know what to look for or what to do next,
2399.71s - 2400.55s |  check this out.
2400.55s - 2403.19s |  It should be able to give you some good guidance.
2403.19s - 2405.21s |  In our opinion, GitHub can do a few things better.
2405.21s - 2406.77s |  They can increase warnings and awareness.
2406.77s - 2410.15s |  So if you try to attach a selfless runner to a public repo,
2410.15s - 2412.15s |  it'd be great if there were big red letters saying,
2412.15s - 2413.43s |  be careful, this is dangerous.
2413.43s - 2415.47s |  They can also improve their secure defaults.
2415.47s - 2417.99s |  Like we talked about with the workflow approval requirements
2417.99s - 2421.03s |  and then they could implement granular approval requirements
2421.03s - 2423.63s |  so that maintainers don't need to select
2423.63s - 2426.99s |  between three options, all of which have broad privileges
2426.99s - 2429.39s |  and probably will hurt developer workflows
2429.39s - 2430.43s |  one way or the other.
2431.51s - 2432.99s |  If you're trying to defend your organization
2432.99s - 2435.55s |  from these attacks, start with the easy stuff.
2435.55s - 2438.55s |  Please change this radio button so that you require approval
2438.55s - 2440.75s |  for all outside collaborators.
2440.75s - 2442.61s |  We use the GitHub token extensively
2442.61s - 2444.05s |  in almost all of our attacks.
2444.05s - 2445.85s |  Set it to read only.
2445.85s - 2448.45s |  That will stop a lot of post-exploitation opportunities.
2448.45s - 2450.89s |  And then force your devs to use
2450.89s - 2452.37s |  fine-grain personal access tokens.
2452.37s - 2455.55s |  We hate discovering fine-grain tokens
2455.55s - 2458.49s |  on red team engagements because they only give you access
2458.49s - 2459.45s |  to one repository.
2459.45s - 2462.01s |  If we find a personal access token, that's classic.
2462.01s - 2463.09s |  We can do a lot of stuff.
2463.09s - 2466.01s |  If it's fine-grain, sometimes that's the end of the road.
2466.95s - 2470.07s |  All right, there's also ways to secure yourself more
2470.07s - 2472.79s |  by making sure that your self-hosted runners are ephemeral.
2472.79s - 2474.39s |  So GitHub maintains something
2474.39s - 2475.65s |  called Actions Runner Controller,
2475.65s - 2478.51s |  which allows you to automatically spin up and spin down
2478.51s - 2480.63s |  self-hosted runners using Kubernetes.
2480.63s - 2483.23s |  Also, a lot of cloud providers have auto-scaling groups
2483.23s - 2485.43s |  that can be linked with self-hosted runners
2485.43s - 2487.99s |  and have that also be done automatically.
2487.99s - 2490.11s |  And finally, there's a lot of third parties
2490.11s - 2493.53s |  that offer turnkey drop-in replacements
2493.53s - 2496.23s |  for GitHub runners that are also ephemeral
2496.23s - 2499.11s |  if you don't want to use just the GitHub-hosted runners.
2499.11s - 2501.95s |  And one thing that's important to remember here
2501.95s - 2504.63s |  is that ephemeral means the environment
2504.63s - 2506.65s |  that the runner is in also has to be ephemeral.
2506.65s - 2508.39s |  So if there's an ephemeral runner,
2508.39s - 2511.23s |  but it's using a non-ephemeral file share,
2511.23s - 2514.35s |  then attacker can simply jump to other workflows
2514.35s - 2516.43s |  by saving certain files off.
2516.43s - 2519.07s |  So it's important to really consider the entirety
2519.07s - 2522.91s |  of that environment when it comes to build agent security.
2524.07s - 2526.83s |  All right, so another way is by using GitHub's feature
2526.83s - 2528.41s |  for runner group pinning.
2528.45s - 2531.53s |  So GitHub has a feature where if you have a runner group,
2531.53s - 2534.09s |  you can actually pin it to a specific workflow
2534.09s - 2537.01s |  or even a specific SHA or reference of a workflow.
2537.01s - 2539.93s |  So with this, you have a lot of different options
2539.93s - 2542.41s |  to restrict which workflows can even use the runner.
2542.41s - 2544.37s |  And all of this has the ultimate effect
2544.37s - 2546.69s |  of protecting your most privileged runners.
2547.63s - 2550.29s |  We want to end our talk today by emphasizing
2550.29s - 2553.37s |  that this talk and this research is not about PyTorch.
2553.37s - 2555.55s |  So we have some key takeaways here.
2555.55s - 2558.17s |  If you've been zoning out, lock in for this last slide.
2558.81s - 2561.13s |  What we did to PyTorch, we've been able to do
2561.13s - 2563.93s |  to a lot of advanced, mature organizations
2563.93s - 2565.37s |  because of this lack of awareness
2565.37s - 2566.81s |  around CICD agent security.
2566.81s - 2569.69s |  So I think we submitted over 20 high and critical
2569.69s - 2570.97s |  bug bounty submissions.
2570.97s - 2572.45s |  Among them were critical vulnerabilities
2572.45s - 2575.89s |  in GitHub Actions, Microsoft, PyTorch, TensorFlow,
2575.89s - 2579.59s |  ASTAR, HydroGX, ByteDance, and a lot more
2579.59s - 2582.13s |  that we're not allowed to talk about.
2582.13s - 2585.21s |  Additionally, if you think that public CICD security is bad,
2585.21s - 2588.33s |  internal CICD security is a nightmare.
2588.33s - 2592.27s |  Almost not every red team engagement I've been on,
2592.27s - 2594.25s |  but a lot of red team engagements were able
2594.25s - 2596.37s |  to get access to a CICD platform
2596.37s - 2598.93s |  and then escalate privileges and use it to reach objective.
2598.93s - 2601.05s |  It's almost become the new ADCS for us
2601.05s - 2603.11s |  where internally everyone's vulnerable
2603.11s - 2605.47s |  and compromising these CICD platforms
2605.47s - 2607.65s |  can give you extensive privileges.
2608.57s - 2609.77s |  Finally, what we need everyone to do
2609.77s - 2611.27s |  is go learn about these attacks
2611.27s - 2612.81s |  to protect your org from compromise.
2612.81s - 2615.09s |  The main issue we see is just this lack of awareness.
2615.61s - 2617.63s |  If devs, architects, execs, and everyone
2617.63s - 2619.09s |  is learning about these attacks,
2619.09s - 2620.59s |  they'll be able to implement controls
2620.59s - 2622.57s |  that hopefully protect their organization
2622.57s - 2625.41s |  from the next critical supply chain attack.
2625.41s - 2627.09s |  All right, the moral of the story here
2627.09s - 2629.69s |  is learn about this stuff and don't let this be you.
2633.06s - 2634.48s |  All right, so we want to start off
2634.48s - 2636.94s |  by thanking all of the bug bounty triage teams
2636.94s - 2638.52s |  that handled some of the reports
2638.52s - 2640.94s |  and handled them well and applied solid mitigations
2640.94s - 2642.26s |  to their products.
2642.26s - 2643.90s |  We'd like to thank everyone who showed out today.
2643.90s - 2645.14s |  Thank you so much for watching us
2645.14s - 2646.34s |  and helping us raise awareness
2647.06s - 2648.26s |  and for everyone watching online.
2648.26s - 2649.34s |  Thank you.
2649.34s - 2651.38s |  We'd like to thank DEF CON for giving us this platform.
2651.38s - 2652.94s |  We wanted to speak here for a long time,
2652.94s - 2654.66s |  so this has been really fun.
2654.66s - 2655.96s |  Please, we don't have time for questions,
2655.96s - 2658.50s |  but come find us after if you want to talk more.
2658.50s - 2659.50s |  Thank you, everybody.
2659.50s - 2660.34s |  All right.


